Inpatient glycemic management (IPGM) has become widely accepted as a standard of care (1). Proper glucose measurement is key to safe and effective IPGM (2,3). Bedside glucose monitoring with blood glucose meters is an essential component of IPGM, but has been shown to create confounding analytical and clinical factors (4,5). This occurred when self-monitoring blood glucose meters (SMBG) designed for diabetic patient self-use migrated into the hospital. Subsequently, numerous studies demonstrated confounding factors affecting clinical outcomes in acute care settings. This multicenter observational study is the first to present an algorithm combining four statistical tools to evaluate the analytical and clinical accuracy of a blood glucose monitoring system (BGMS) in critical care patient settings.
In the 1990s through the first decade of the new millennium, glycemic management programs were developed, implemented, and studied to determine the clinical outcome of glycemic control through IV intensive insulin therapy (IIT) in critically ill patients (6–9). The initial outcomes of these glycemic management programs were profound; they significantly reduced postsurgical infections, blood transfusion, acute kidney injury, polyneuropathy, ICU length of stay, and in-hospital mortality (6–8). Unfortunately, follow-up studies reported increased risk for hypoglycemia with an associated enhanced mortality in critically ill patients who received IV IIT (10–12). Central to these adverse events was the unreliability and lack of standardization of glucose measurement.
Historically, the quality of glucose measurement for diabetic patients was assessed using measurement validation protocols established by regulatory and standards agencies in cooperation with manufacturers, in which the comparative device was the YSI 2300 Glucostat (Yellow Springs Instruments, Yellow Springs, OH) and not a commutable central laboratory reference analyzer. As outlined recently, these measurement validation protocols continue to be revised in response to clinical performance concerns in various hospital patient populations (13). In 2003, ISO 15197 required that SMBG analytical accuracy meet the expectation that 95% of glucose measurement fall within 20% of the reference method for values greater than 75 mg/dL and less than or equal to 15 mg/dL for glucose measurements less than 75 mg/dL (14). In 2013, the ISO 15197 analytical accuracy target was tightened (15). Subsequently for hospital use, these targets were commonly achieved in laboratory analytical investigations but were found to be unacceptable in clinical use (3,16). In October 2016, the U.S. Food and Drug Administration (FDA) published new guidelines (16) defining a general approach for verification and validation of performance with separate and distinct criteria for SMBG and BGMS. As evidenced by these new guidelines, the criteria for assessing glucose measurement are still evolving and there is open dialogue on how to properly assess clinical accuracy. Clinical accuracy is a qualitative measure to relate clinical treatment decisions based on a glucose result with clinical outcomes (17). Methods and statistical tools including sensitivity and specificity analyses, error grid analyses, and Monte Carlo simulation models have been individually used to assess clinical accuracy for glucose meters in ambulatory and acute care settings (17–19). As previously reported, individually these tools have limitations across the glucose measurement range because they represent different aspects of assessing analytical performance, for example, bias only, bias + imprecision, estimated total analytical error, and probability of an erroneous glucose result contributing to insulin dosing error (17,20). To date, no investigation has combined these statistical tools to assess glucose measurement clinical accuracy in critically ill patients.
The purpose of this international, multicenter, multidisciplinary observational study was to evaluate the combination of specific statistical tools to assess clinical accuracy of a glucose result in critically ill patients in relation to clinical treatment decisions.
This algorithm, using four distinct statistical tools, was used to determine the accuracy of a BGMS glucose result in critically ill patients compared with a laboratory reference method traceable to definitive method, as recommended by the American Diabetes Association, FDA, and Clinical and Laboratory Standards Institute (CLSI) (16,21,22).
MATERIALS AND METHODS
The study used paired prospective testing and retrospective chart review (February 2013 to February 2014) of critically ill patients aged 2 months to 99 years admitted to ICUs at five international clinical sites (Netherlands, Belgium, and United States). The sites (A, B, C, D, and E) comprised medical, surgical, and burn intensive care patients (Table 1), where institution-specific IV intensive insulin procedures for maintaining glycemic control were employed as the standard of care (Table 1). Patient medical conditions were classified according to the World Health Organization’s International Statistical Classification of Diseases and Related Health Problems, 10th Revision (23). Medications were classification based on the “United States Pharmacopeia” (24). The FDA preapproved the pre-Investigational Device Exemption study protocol, which was subsequently approved by each institution’s ethics committee as part of their standard of care.
Peripheral and central arterial and venous whole blood specimens were collected in lithium heparin blood collection tubes from patients routinely tested for glucose as part of each institution’s glycemic control programs. Capillary whole blood specimens were not included at this time due to the FDA’s comments during protocol development referencing their adverse events database (25), and also based on the recommendations in international consensus guidelines (3) that the standard of care requires the use of arterial or venous specimens. Each whole blood specimen was tested directly after collection using the StatStrip Glucose (Nova Biomedical, Waltham, MA) BGMS and was then immediately centrifuged and the plasma tested on the hospital central laboratory method within 15 minutes. For sites A, B, C, and D, the central laboratory method was plasma glucose hexokinase performed on the Modular P800 platform (Roche Diagnostics, Indianapolis, IN). Sites A, B, C, and D were aligned to National Institute of Standards and Technology (NIST) standard reference materials 917c and 965b. At sites A and B, the plasma hexokinase glucose method NIST alignment was also confirmed directly to an internal gas chromatography isotope dilution mass spectrometry (IDMS) glucose method. For site E, laboratory glucose measurement was performed using a glucose oxidase NIST aligned (917c and 965b) method on the UniCel Synchron DxC (Beckman Coulter, Brea, CA). Additional information about calibration and commutability are provided in Supplemental Table 1 (Supplemental Digital Content 1, https://links.lww.com/CCM/C303).
Clinical Accuracy and Risk Modeling Tools
Four clinical accuracy and risk assessment modeling tools were used to evaluate the risk of mismanagement of dysglycemia associated with BGMS measurements.
Clinical and Laboratory Standards Institute POCT12-A3 Guideline Analysis. Clinical accuracy was assessed using BGMS performance criteria as defined in the CLSI’s 2013 “POCT12-A3: Point-of-care blood glucose testing in acute and chronic care facilities; approved guideline—third edition” (22).
Parkes Error Grid Analysis. The Parkes error grid analysis incorporating recent recommendations for clinical accuracy studies (18) was performed to assess clinical risk associated with glucose measurement differences between the BGMS and central laboratory methods.
Monte Carlo Simulation Modeling. Application of Monte Carlo Simulation of Clinical Risk to the BGMS Trial Data. Simulation of the influence of bias and precision of the BGMS results on the risk of insulin dosing error in critically ill patients was reported in 2013 (19). The region of risk on the contour plot associated with this study was depicted graphically by overlaying a scatter plot of patient data (19). Individual data points were plotted at the average coefficient of variation for the BGMS (3.25%), and the bias values were determined by the difference in BGMS and reference glucose values. The region of clinical risk for operation of the BGMS in critically ill patients is shown by the cluster of data points, and the associated clinical risk is shown by the contour lines (Fig. 3). The straight solid and dashed lines of total analytical error from the original publication were also included as an interpretative guide for the cluster of patient data. Quantitative results were shown in the attached tables of the fraction of all patient data (n = 1,815) bound by specific contour lines. As previously reported, the insulin dosing error rates were categorized as one dosage unit of insulin error, two dosage units of insulin error, and three dosage units of insulin error (19). Further details on the method used to overlay published Monte Carlo simulation contour plots of clinical risk in critical care adult patients to the BGMS trial data in critical care adult patients are presented as supplemental data (Supplemental Digital Content 1, https://links.lww.com/CCM/C303).
Stratified Clinical Sensitivity and Specificity Analysis. Stratified clinical sensitivity and specificity analysis was conducted in order to determine that the BGMS measurement was sufficient for intervention and therapeutic purposes at the medical decision limits of glucose values. The laboratory reference glucose and BGMS glucose results were stratified into glucose categories for insulin dosing and the frequency distributions determined. In each laboratory reference glucose category, sensitivity was assessed by determining the fraction of corresponding BGMS measurements within ± 1 reference method category. The false negative percentage is 100% × (1 – sensitivity). In each BGMS category, the specificity was assessed by the percentage of laboratory reference glucose measurements within ± 1 reference method category. The false positive percentage is 100% × (1 – specificity). Further details on the method used to calculate the sensitivity and specificity in each stratum are presented as supplemental data (Supplemental Digital Content 1, https://links.lww.com/CCM/C303).
Data were analyzed using Analyze-it (version 3.50) for Excel 2007 (Analyse-it Software, Leeds, United Kingdom) and STATA/MP (version 11; StataCorp LP, College Station, TX) and included least squares linear regression. The BGMS mean absolute bias and percent (%) bias were calculated. The percent bias of the BGMS result compared with the hexokinase reference method result was calculated and assessed according to POCT12-A3 (22). Passing and Bablok regression analysis was used for hypoglycemic patient trend analysis.
Complexity of the Patient Population
The retrospective analysis of 1,815 paired glucose measurements from critically ill patients included n = 1,698 patients (paired glucose measurements from 1,692 individual patients and 123 glucose measurements from six burn patients), which included 19 different and complex medical condition categories representing 257 different and specific clinical conditions (23). On average, each patient received 14 medications from 33 different parent drug classes with 144 drug subcategories and 8,016 compounds administered in complex treatment regimens (24). The patient population investigated ranged in age from 2 months to 99 years and represented a wide spectrum of severity of illness (Table 2), receiving multiple therapeutic and polypharmacy medications. The patient population had abnormal ranges of confounding physiological and biochemical substances known to affect the accuracy of the BGMS measurement (4). Detailed information about the breakdown of patient age and glucose ranges, medication classes, and range of confounding physiological and biochemical parameters are provided in Supplemental Tables 2–4 (Supplemental Digital Content 1, https://links.lww.com/CCM/C303).
BGMS Analytical Performance Analysis
Analytical performance was evaluated through comparison of paired patient specimen analyses for the BGMS versus the certified gas chromatography IDMS aligned plasma hexokinase method used in sites A and B (n = 1,245), where the coefficient of correlation for the BGMS versus comparative method was 0.995 with a slope of 1.05 and an intercept of –3.9 mg/dL. The mean percent bias difference between the BGMS and comparative method was –1.35%. These data demonstrated that the BGMS is analytically equivalent to the gas chromatography IDMS aligned reference hexokinase.
Clinical Accuracy Analysis. The data from all five sites demonstrated that 99.3% (1,802 results) of the BGMS measurements were within zone A of the Parkes error grid (Fig. 1). The remaining 0.7% fell into zone B and breakdown analysis showed that these were not clinically significant and would not result in any untoward clinical intervention. Analysis of the collated data from all study sites showed that the performance of the BGMS met the clinical accuracy performance criteria outlined in POCT12-A3 with 95.4% (606/635) of patient sample results within ± 12 mg/dL for glucose values less than 100 mg/dL and 96.5% (1,139/1,180) of patient sample results within ± 12.5% for glucose values greater than 100 mg/dL.
Stratified Clinical Sensitivity and Specificity Analysis. Stratified clinical sensitivity and specificity analysis showed that BGMS measurements were highly sensitive (mean = 95.2%, sd = ± 0.02) and highly specific (mean = 95.8%, sd = ± 0.03) over the glycemic range tested (10 mg/dL intervals between 50 and 150 mg/dL) (Fig. 2).
Monte Carlo Simulation Modeling.Clinical glucose data were overlaid on plots to determine the anticipated probability of insulin dosing error for greater than or equal to 1, greater than or equal to 2, and greater than or equal to 3 insulin dosing error categories. Most BGMS data falls within the boundary of 15% total error and within 0.05% probability of three or more categories of insulin dosing error (Fig. 3). Tabulated analysis was performed with all BGMS data, and the analysis for the simulated sliding insulin scale showed that 1.8% (32/1,815) of critically ill patients had greater than 0.5% chance of three or more insulin dosing errors during treatment and 2.3% (41/1,815) had greater than 20% chance of two or more insulin dosing errors during treatment (Fig. 3).
Hypoglycemic Patient Trend Analysis. A Passing and Bablok bias trend analysis was performed on the clinical dataset from three of the study sites to identify any potential safety issues with the use of the BGMS in the critically ill patient population subset with hypoglycemia. An analysis was not performed on the clinical data from sites C and E due to the small number of hypoglycemic patients: one at site E and none at site C. The BGMS demonstrated 99.1% (223/225) concordance to the central laboratory reference methods in characterizing hypoglycemic patients with glucose less than 70 mg/dL (< 3.9 mmol/L).
The variability observed in glycemic control studies has been associated with nonstandardized glucose methods that are not validated in critically ill patients (2,3). These limitations have resulted in inconsistent outcomes and diminished the utility of BGMS use in high-risk populations (26,27). In response to the published concerns about the suitability of BGMS use in IV IIT, the study team elected to develop and apply a combination of specific statistical tools to thoroughly evaluate the clinical accuracy and the estimated total analytical error of the study BGMS in order to evaluate the device’s suitability for use in critically ill patient care settings. There were no exclusion criteria for patients participating in the study, where the patient population was not limited to diabetic patients, but included all the patients admitted in to the critical care units in each study center. As such, the study included patients with a significant array of medical conditions with abnormal pathophysiologic factors and a vast range of medications known to interfere with the accuracy of many routinely used glucose meters and other glucose measurement methods (4,5).
There were no clinically significant interferences observed, and the BGMS demonstrated substantial equivalence to gas chromatography IDMS traceable plasma hexokinase aligned laboratory reference methods. The use of the different clinical accuracy assessments algorithm demonstrated that the BGMS is accurate and reliable for use in critically ill patients. The significant volume of data generated by this study permitted thorough investigation of device performance across the analytical measuring range and, particularly, the hyper- and hypoglycemic ranges.
In patients identified as hypoglycemic (< 70 mg/dL) using the reference method (225 patients), two (2/225, 0.9%) were identified as normoglycemic using the BGMS. Further review of these two patients did not identify any medical condition, drug combination, or clinical trend, and therefore, the result cannot be explained based upon available information.
The BGMS demonstrated substantial equivalence to the central laboratory reference methods in characterizing hypoglycemic patients with glucose less than 70 mg/dL (< 3.9 mmol/L). Previously published accuracy guidelines and studies lacked sufficient data to properly evaluate clinical accuracy (16). Use of simulation modeling was a beneficial contribution to assess the probability of an inaccurate glucose measurement resulting in an insulin dosing error. The introduction of the stratified clinical sensitivity and specificity analysis as a new tool provided further insight into the quantifiable probability of error (i.e., uncertainty) at the critical decision limits for managing insulin administration. Collectively, the four mathematical and statistical modeling tools enabled the study team to complete an effective and comprehensive clinical accuracy assessment. It is important to note that the study protocol and data analyses have not been applied to other whole blood, point-of-care devices and glucose methods, but, more importantly, the study design and this clinical accuracy algorithm can serve as a model for future studies of in vitro diagnostic methods in critically ill patient care settings, including point-of-care devices and other methods. In the United States, regulation of BGMS has undergone considerable debate following the FDA’s publication of guidelines for manufacturers that separates SMBG from prescription and professional use BGMS, which manufacturers are now required to test in critically ill patient care settings (16,28,29). Subsequently, the Centers for Medicare and Medicaid Services announced that BGMS not cleared for use with critically ill patients would be considered “off-label” when used in these patient populations (30,31), requiring users to comply with the Clinical Laboratory Improvement Amendments of 1988 (32).
Although this study addresses venous and arterial whole blood, an additional study is required using the same clinical accuracy algorithm to determine if capillary whole blood specimens can safely be used in critically ill patient care settings with the study BGMS and other whole blood methods.
The question remains: are all whole blood glucose methods clinically accurate and acceptable for use in critically ill patient care settings? The algorithm used in this study is a rigorous approach combining several statistical tools to effectively validate the clinical accuracy of using bedside glucose monitoring systems in critically ill patients. This algorithm and clinical accuracy assessment approach can be applied to other routinely used whole blood glucose measurement methods, including emerging continuous glucose monitoring systems.
In summary, this clinical accuracy assessment algorithm is an effective tool for comprehensively assessing the validity of using whole blood glucose measurement in critically ill patient care settings.
This study approach and subsequent analyses may serve as an example for future evaluations of other whole blood bedside test systems in critically ill patients.
1. Jacobi J, Bircher N, Krinsley J, et al. Guidelines for the use of an insulin infusion for the management of hyperglycemia in critically ill patients. Crit Care Med. 2012; 40:32513276
2. van den Berghe G, Schetz M, Vlasselaers D, et al. Clinical review: Intensive insulin therapy in critically ill patients: NICE-SUGAR or Leuven blood glucose target? J Clin Endocrinol Metab. 2009; 94:31633170
3. Finfer S, Wernerman J, Preiser JC, et al. Clinical review: Consensus recommendations on measurement of blood glucose and reporting glycemic control in critically ill adults. Crit Care. 2013; 17:229
4. Dungan K, Chapman J, Braithwaite SS, et al. Glucose measurement: Confounding issues in setting targets for inpatient management. Diabetes Care. 2007; 30:403409
5. Kanji S, Buffie J, Hutton B, et al. Reliability of point-of-care testing for glucose measurement in critically ill adults. Crit Care Med. 2005; 33:27782785
6. Furnary AP, Zerr KJ, Grunkemeier GL, et al. Continuous intravenous insulin infusion reduces the incidence of deep sternal wound infection in diabetic patients after cardiac surgical procedures. Ann Thorac Surg. 1999; 67:352360
7. van den Berghe G, Wouters P, Weekers F, et al. Intensive insulin therapy in critically ill patients. N Engl J Med. 2001; 345:13591367
8. Furnary AP, Gao G, Grunkemeier GL, et al. Continuous insulin infusion reduces mortality in patients with diabetes undergoing coronary artery bypass grafting. J Thorac Cardiovasc Surg. 2003; 125:10071021
9. Krinsley JS. Effect of an intensive glucose management protocol on the mortality of critically ill adult patients. Mayo Clin Proc. 2004; 79:9921000
10. Brunkhorst FM, Engel C, Bloos F, et al.: German Competence Network Sepsis (SepNet): Intensive insulin therapy and pentastarch resuscitation in severe sepsis. N Engl J Med. 2008; 358:125139
11. The NICE-SUGAR Study Investigators. Intensive versus conventional glucose control in critically ill patients. N Engl J Med. 2009; 360:12831297
12. The NICE-SUGAR Study Investigators. Hypoglycemia and risk of death in critically ill patients. N Engl J Med. 2012; 367:11081118
13. Rice MJ, Coursin DB. Glucose meters: Here today, gone tomorrow? Crit Care Med. 2016; 44:e97e100
14. International Organization for Standardization: ISO 15197-2003, In Vitro Diagnostic Test Systems—Requirements for Blood-Glucose Monitoring Systems for Self-Testing in Managing Diabetes Mellitus. 2003. Available at: http://www.iso.org/iso/catalogue
. Accessed November 13, 2015
15. International Organization for Standardization: ISO 15197-2013, In Vitro Diagnostic Test Systems—Requirements for Blood-Glucose Monitoring Systems for Self-Testing in Managing Diabetes Mellitus. 2013. Available at: http://www.iso.org/iso/catalogue
. Accessed November 13, 2015
16. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health: Blood Glucose Monitoring Test Systems for Prescription Point-of-Care Use Guidance. 2016. Silver Spring, MD, FDA. Available at: http://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm380325.pdf
. Accessed October 28, 2016
17. Klonoff DC. Point-of-care blood glucose meter accuracy in the hospital setting. Diabetes Spectr. 2014; 27:174179
18. Pfützner A, Klonoff DC, Pardo S, et al. Technical aspects of the Parkes error grid. J Diabetes Sci Technol. 2013; 7:12751281
19. Karon BS, Boyd JC, Klee GG. Empiric validation of simulation models for estimating glucose meter performance criteria for moderate levels of glycemic control. Diabetes Technol Ther. 2013; 15:9961003
20. Simmons DA. How should blood glucose meter system analytical performance be assessed? J Diabetes Sci Technol. 2015; 10:178184
21. Sacks DB, Arnold M, Bakris GL, et al.: National Academy of Clinical Biochemistry: Position statement executive summary: Guidelines and recommendations for laboratory analysis in the diagnosis and management of diabetes mellitus. Diabetes Care. 2011; 34:14191423
22. Clinical and Laboratory Standards Institute: POCT12-A3: Point-of-Care Blood Glucose Testing in Acute and Chronic Care Facilities; Approved Guideline—Third Edition, 2013. Wayne, PA, CLSI. Available at: http://shop.clsi.org/point-of-care-documents/POCT12.html
. Accessed November 13, 2015
23. World Health Organization: International Statistical Classification of Diseases and Related Health Problems 10th Revision, 2015. Geneva, Switzerland, WHO. Available at: http://www.who.int/classifications/icd/en
. Accessed June 2, 2014
24. United States Pharmacopeia: United States Pharmacopeia Reference Standards, 2015. Rockville, MD, USP. Available at: http://www.usp.org/reference-standards/reference-standards-catalog
. Accessed June 2, 2014
25. Department of Health and Human Services, Food and Drug Administration: Manufacturer and User Facility Device Experience Database—(MAUDE), 1991–2015. Silver Spring, MD, FDA. Available at: http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/PostmarketRequirements/ReportingAdverseEvents/ucm127891.htm
. Accessed September 18, 2014
26. Mesotten D, van den Berghe G. Glycemic targets and approaches to management of the patient with critical illness. Curr Diab Rep. 2012; 12:101107
27. Inoue S, Egi M, Kotani J, et al. Accuracy of blood-glucose measurements using glucose meters and arterial blood gas analyzers in critically ill adult patients: Systematic review. Crit Care. 2013; 17:R48
28. FDA Clears Glucose Monitoring System for Use in Hospital Critical Care Units: Press Release of the Department of Health and Human Services, Food and Drug Administration. 2014. Available at: http://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm416144.htm
. Accessed September 24, 2014
29. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health: Self-Monitoring Blood Glucose Test Systems for Over-the-Counter Use Guidance. 2016. Silver Spring, MD, FDA. Available at: http://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm380327.pdf
. Accessed October 28, 2016
30. Department of Health and Human Services, Centers for Medicare and Medicaid Services, Center for Clinical Standards and Quality/Survey & Certification Group: Reissuance of S&C as Draft Only–for Comment, Off-Label/Modified Use of Waived Blood Glucose Monitoring Systems (BGMS). 2015. Baltimore, MD, CMS. Available at: https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/SurveyCertificationGenInfo/Downloads/Survey-and-Cert-Letter-15-11.pdf
. Accessed March 13, 2015
31. New York State Department of Health: Re: Off-Label Use of Glucose Meters. Albany, NY, NYSDOH. Available at: http://www.wadsworth.org/labcert/clep/files/Glucose_meters_off_label_use_1_13_14.pdf
. Accessed January 13, 2014
32. Centers for Medicare and Medicaid Services. Standards and certification: Laboratory Requirements; see regulations at 42 CFR part §493. 1988. Baltimore, MD, CMS, Available at: http://www.ecfr.gov/cgi-bin/text-idx?SID=1248e3189da5e5f936e55315402bc38b&node=pt42.5.493&rgn=div5
. Accessed June 2, 2014