Secondary Logo

Journal Logo

Technology, Computing, and Simulation: Original Clinical Research Report

The Accuracy of 6 Inexpensive Pulse Oximeters Not Cleared by the Food and Drug Administration: The Possible Global Public Health Implications

Lipnick, Michael S. MD*; Feiner, John R. MD*; Au, Paul BS*; Bernstein, Michael BS; Bickler, Philip E. MD, PhD*

Author Information
doi: 10.1213/ANE.0000000000001300
  • Free

Pulse oximeters have become invaluable diagnostic tools because of ease of use, portability, and applicability in a wide range of clinical settings. The increase in utilization and availability of this technology has occurred disproportionately in higher income countries in large part because of significant cost and supply chain barriers in low- and middle-income countries (LMICs). Recent studies estimate that >77,000 operating rooms around the world have no access to pulse oximetry.1 Pulse oximeters can cost >$3000 US dollar (USD) per unit. Until recently, few units were available for <$1000 USD and virtually none for <$50. As a result of increasing popularity and technological advancements, the manufacture of many low-cost pulse oximeters has emerged, although the accuracy of many of these devices is uncertain.

Pulse oximetry computes arterial hemoglobin oxygen saturation from the ratio of the pulsatile to the total transmitted red light divided by the same ratio for infrared light transilluminating the finger, ear, or other tissue. In theory, the derived saturation should be independent of skin pigmentation and many other variables, including hemoglobin concentration, nail polish, dirt, and jaundice. In practice, pulse oximeter measurements are influenced by many variables, and algorithm testing is required to refine the accuracy of these devices.2–11 Such testing often begins with in vitro analysis using a device such as the Fluke SPOT Light Pulse Oximeter Tester (Fluke Biomedical, Everett, WA).12 This device generates electronic signals that are interpreted by the pulse oximeter being tested. Users can manipulate heart rate, Spo2, artifact, and perfusion to varying degrees. Ideally, before clinical use, instrument readings are validated by testing human subjects using arterial blood saturation values (Sao2) measured with a gold standard multiwavelength oximeter. Testing of pulse oximeters in human subjects during hypoxemia is expensive and done at only a few centers in the world. Such testing is not required for marketing these products to consumers for nonmedical use and therefore is often not done by manufacturers of low-cost pulse oximeters.

All pulse oximeters marketed for medical use in the United States are required by the Food and Drug Administration (FDA) to have been tested and certified to be accurate to root mean square error (ARMS) of <3% at Sao2 between 70% and 100%.13 The great majority of calibration and confirmation tests are done on healthy volunteer subjects.

The purpose of this study was to evaluate the accuracy of several low-cost pulse oximeters as a sample of the large number of such devices currently available for purchase on the Internet, in pharmacies, and in retail stores for <$50 USD. We hypothesize that inexpensive pulse oximeters currently available to consumers are inaccurate and would not meet International Organization for Standardization (ISO 80601-2-61:2011) standards for use in clinical practice.13

METHODS

This study was approved by the University of California at San Francisco (UCSF) Committee on Human Research and the Shenzhen University School of Medicine IRB. Written informed consent was obtained from all subjects. Two groups of 11 healthy, nonsmoking subjects were studied; the first 11 subjects were studied at UCSF, and the second group of 11 subjects was studied at Shenzhen, China, using the same protocol. None of the subjects had lung disease, obesity, or cardiovascular problems. They ranged in age between 18 and 40 years and had a range of skin pigments as described in Table 1.

Table 1.
Table 1.:
Demographics

Sample size was decided a priori based on FDA guidelines for accuracy testing. These guidelines require a minimum of 200 data points over an Sao2 range of 70% to 100%, well-balanced in the 70% to 80%, 80% to 90%, and 90% to 100% ranges. In addition, the FDA requires men and women of varying skin tones be included.14

Subjects were studied using identical protocols implemented by these 2 study laboratories that routinely test pulse oximeters for FDA 501(k) certification.14,15 One of the authors was present at the study site in Shenzhen for the experiments. Briefly, study subjects were semisupine (30° head up) with a nose clip, breathing air-nitrogen-carbon dioxide mixtures via a mouthpiece from a partial rebreathing circuit with a voluntarily increased minute ventilation and 10 to 20 L/min fresh gas inflow. An indwelling 22-g radial artery catheter was placed to sample arterial blood for the measurement of Sao2. In the first group of 11 study subjects at the UCSF study site, 3 pulse oximeters were placed on each subject: Starhealth SH-A3 (Star Health Medical, Huoying Hualongyuanzhongli, Changping District, Beijing, China), Jumper FPD-500A (Shenzhen Jumper Medical Equipment, Shenzhen, Guangdong Province, China), and Atlantean SB100 II (Atlantean Corporation, Chubay City, Taiwan). In the second group of 11 study subjects (Shenzhen, China), 3 pulse oximeters were placed on each subject: the Contec CMS50DL, Beijing Choice C20, and Beijing Choice MD300C23. Because none of these devices had any digital or analog output of saturation values, readings were recorded by hand throughout the protocol.

A series of 10 to 12 stable target Sao2 plateaus between 70% and 100% (approximately 70%, 73%, 76%, 80%, 83%, 86%, 89%, 92%, 95%, 98%, and 100%) were sought by an operator who adjusted the inspired air-nitrogen-carbon dioxide mixture breath by breath in response to the computer display of the estimated Sao2 derived from end-tidal gas analysis using LabVIEW 2013 (National Instruments, Austin, TX).2 Input parameters for the computer O2 dissociation curve included hemoglobin P50, base excess, and alveolar-arterial (A − a) oxygen difference. Inputs were adjusted as needed to match the predicted with measured Sao2. At each level, arterial blood was sampled after a plateau of 30 to 60 seconds had been achieved, followed by a second sample at the same plateau 30 seconds later. To ensure that each subject had good circulation to the fingers, each hand was wrapped in a warming pad. “Functional” arterial Sao2 (HbO2/[Hb+HbO2]) was determined by multi-wavelength oximetry using a Radiometer (Copenhagen, Denmark) ABL-90 at UCSF and an OSM3 at Shenzhen. Both instruments were calibrated according to manufacturer recommendations.

Statistical Analysis

Bias was computed as Spo2 - Sao2 from each oximeter’s reading minus the corresponding arterial blood sample value. Bias is reported as mean ± SD, where the SD of the bias represents precision. The SD was calculated according to Bland and Altman with adjustments for multiple measurements for each individual according to the “Method Where the True Value Varies.”16 Limits of agreement are 1.96·SD. The 95% confidence intervals were calculated using bootstrapping (random resampling with replacement) with 50,000 repetitions. ARMS was calculated as the square root of the mean difference between Spo2 − Sao2, squared. The 95% confidence intervals for ARMS were calculated by bootstrapping, as above. An Arms <3% is the accuracy standard used by the FDA. Bias is plotted versus Sao2 as the gold standard. The relationship of bias to Sao2 was analyzed by linear regression accounting for repeated measures. Bias was also analyzed for decadal subgroups of Sao2 (70%–80%, 80%–90%, and 90%–100%) using repeated-measures analysis of variance, with the Tukey-Kramer honestly significant difference for multiple comparison testing. The Levene test was used to compare variances between the different decadal ranges. A P value <0.05 was considered statistically significant. Statistical analysis was performed with JMP 11.0 (SAS Institute, Cary, NC) and Stata 14 (StataCorp, College Station, TX).

RESULTS

Readings from 6 pulse oximeters corresponding to 536 blood samples in 22 healthy subjects were obtained. Characteristics of all study subjects are presented in Table 1.

The ARMS over the range of Sao2 between 70% and 100% was <3%, the FDA clearance threshold, for 2 of the 6 oximeters tested, the Contec CMS50DL and the Beijing Choice C20 (Table 2). The remaining 4 oximeters demonstrated >3% ARMS. This was largely because of high mean bias (Table 2).

Table 2.
Table 2.:
Summary of Data from 6 Low-Cost Pulse Oximeters
Figure 1.
Figure 1.:
Bias (pulse oximeter oxygen saturation [Spo 2] − arterial blood oxygen saturation [Sao 2]) is plotted against Sao 2 measured by either an ABL90 (panels A–C) or OSM3 (panels D–F) hemoximeter (Radiometer). Mean bias is shown by a solid horizontal line. Dashed horizontal lines are the upper and lower limits of agreement. Regression lines and equations are shown on the graphs with 95% confidence intervals for slope. Individual subject data are shown by different colored markers and trendlines. Panels A–C represents one group of 11 subjects, whereas panels D–F represent a different group of 11 subjects (A, Starhealth SH-A3; B, Jumper FPD-500A; C, Atlantean SB100 II; D, Contec CMS50DL; E, Beijing Choice C20; F, Beijing Choice MD300C23).
Figure 2.
Figure 2.:
Bias (pulse oximeter oxygen saturation [Spo 2] − arterial blood oxygen saturation [Sao 2]) is plotted against Sao 2 measured by either an ABL90 (oximeters 1–3) or OSM3 (oximeters 4–6) hemoximeter (Radiometer). Each oximeter is indicated by separate markers.

Figure 1 illustrates bias for individual data points obtained for individual subjects from each oximeter tested. Figure 2 displays bias for each oximeter tested. All oximeters tested demonstrated worsening performance with hypoxia, with mean bias increasing at lower oxygen saturations. This resulted in wide limits of agreement. Data are summarized in Table 2.

DISCUSSION

Pulse oximetry theory predicts that the ratio of pulsatile to total transmitted red light divided by the same ratio for infrared light should depend only on arterial saturation. In practice, there are many factors that must be accounted for to ensure accurate calculation of oxygen saturation by pulse oximetry. Major known variability is caused by anemia, light scattering, venous and tissue pulsation by mechanical force from nearby arteries, pulsatile variations in tissue thickness in the light path other than in the arteries, nail polish, and skin pigment, among others.2 Therefore, the design of accurate pulse oximeters often requires empirically determined correction factors obtained by in vivo comparison of oximeter readings with arterial oxyhemoglobin saturation of volunteer subjects during hypoxemia. Many commonly used low-cost pulse oximeters do not undergo this in vivo testing, and thus, little is known about the accuracy of such devices.

In this study, we tested the accuracy of 6 low-cost pulse oximeters currently available for purchase from popular consumer retailers. The majority (4/6) of the oximeters tested did not meet US FDA standards for accuracy. Unexpectedly, 2 of the 6 oximeters did meet accuracy standards defined by the FDA and ISO, an Arms <3%. Oximeters 4 and 5 sell for approximately $25 USD and performed with similar accuracy for measuring hypoxia in healthy volunteers when compared with popular, clinically approved devices that cost >$3000 USD. Although more expensive units often contain additional features (such as hemoglobin concentration, methemoglobin, and carboxyhemoglobin measurements or enhanced detection during motion or low perfusion states, and faster processing speed), our findings demonstrate that simple, accurate pulse oximeters can likely be developed at significantly lower cost than many units currently available on the market.

It is important to note that although some of the inexpensive units tested here demonstrated accurate saturation readings during hypoxia, none meet current World Health Organization or World Federation of Societies of Anaesthesiologists standards for use in clinical practice.

All instruments tested showed increasingly larger bias (both positive and negative) in oximeter readings at low Sao2 in subjects. It may be that this is a property common to a wide variety of inexpensive pulse oximeters, given that the basic technology is similar. More expensive devices that are FDA-cleared, such as those manufactured by Nellcor, Nonin, Masimo, and others, show a smaller degree of increasing bias during lower Sao2 conditions.2

The magnitude of the oximeter error in all 6 oximeters tested here was relatively small at saturations >90% and probably of no clinical significance. However, there were large and variable errors in performance of pulse oximeters at lower saturations. Clinical use of these devices is therefore of obvious concern. In one subject for oximeters 3 and 6, the device appeared to have locked onto a saturation value and therefore did not detect increasing hypoxemia.

It is not clear to what extent low-cost (<$50 USD) pulse oximeters are used for medical diagnoses worldwide. It has been our experience through extensive time spent in practice environments in LMICs that the use of nontested/non-FDA or Conformité Européene-regulated oximeters is extensive. Furthermore, it has been our experience that the device manufacturers tested here represent a large proportion, if not the majority, of low-cost oximeters currently in use.

To improve anesthesia safety worldwide, conservative estimates indicate that 100,000 additional pulse oximeters are needed, and when accounting for additional care settings such as recovery rooms, >1 million units may be required.17

Our finding that 2 of the low-cost oximeters studied here performed with accuracy to meet FDA clearance standards is particularly relevant for clinical environments in LMICs where access may be limited because of cost. It is in LMICs that most of the world’s surgical disease burden exists with the fewest per capita health care resources (such as health care providers and pulse oximeters) to address the problem.1,18 Furthermore, the purchase of pulse oximeters that cost several thousand—or even several hundred—dollars may not be feasible in these settings, despite research suggesting cost-effectiveness on par with other public health interventions.19

At present, a few initiatives are underway to increase access to low-cost, accurate pulse oximeters in low-income countries. The LifeBox Project aims to equip the world’s operating theaters with pulse oximeters specifically designed and priced for this purpose. We have previously shown that the LifeBox does meet US FDA accuracy standards.20 LifeBox pulse oximeters are donated or purchased by users. The current cost for a LifeBox oximeter is $250, which includes an adult probe, a pediatric probe, a rechargeable battery, delivery by courier anywhere in the world, training materials, and a 2-year warranty (1-year warranty for the probes). Another example of a project aimed at increasing global access to pulse oximeters is the LionsGate Medical Kenek Pulse Oximeter (Lionsgate Technologies, Vancouver, British Columbia, Canada).21 This device costs approximately $50 to $80, although it also requires the purchase of a compatible mobile phone or tablet (iOS) device.

Although the units tested in this study were sold for <$50 USD, this price does not account for additional costs, such as delivery of the unit and replacement batteries, which can add considerably to the overall lifetime cost of these devices.

Nonetheless, based on the data we present here and careful evaluation of the components and features of numerous currently available FDA-cleared pulse oximeters (including screen size, alarms, rechargeable battery, and durable cases), a lower price point for many pulse oximeter units can likely be achieved to target LMIC markets without compromising accuracy or utility.

The present study has several limitations. Study subjects were recruited on a volunteer basis, with limited attempts to recruit subjects of different skin tones in equal proportions. We did not evaluate the impact of skin pigment on saturation although pigment is known to cause inaccurate readings in pulse oximeters.22,23 For individuals with darkly pigmented skin, bias of up to 8% has been reported at lower saturations, in FDA-cleared pulse oximeters tested approximately 10 years ago.22 Although subjects of all skin color are represented in this study, the majority were Asians and Caucasians, and thus applicability to non-Caucasian populations may be limited.

An additional limitation is the use of 2 groups of subjects in 2 different laboratories. Although identical protocols were used, and one of the authors of this article was present during testing at both sites, variability in procedures or differences in study subjects cannot fully be excluded. We studied only 6 oximeters, and thus our results may not be applicable to all non–FDA-cleared oximeters available.

One of the most significant limitations of this study is the ideal conditions under which it was conducted. All volunteers were healthy, with good perfusion values (some with hands wrapped with warming pads) and remained essentially motionless throughout the protocols. Poor perfusion and patient movement are known to significantly affect pulse oximeter accuracy. In clinical scenarios, such as critical illness, factors such as motion artifact and vasoconstriction are likely to play a significant role and further degrade the accuracy of low-cost pulse oximeters, perhaps to a greater degree than more expensive units.11,24,25 Because these relevant clinical factors, among others, were not evaluated, this study should not serve as an endorsement for the clinical use of any of the pulse oximeters tested.

CONCLUSIONS

Inexpensive pulse oximeters have become increasingly available with little regulatory oversight. Most of the commonly sold units tested here demonstrated highly inaccurate saturation readings during hypoxia. A small proportion of the low-cost pulse oximeters tested here performed similarly to more expensive, FDA-cleared units when measuring hypoxia in healthy subjects, although none met World Federation of Societies of Anaesthesiologists standards for clinical use. Caution must be exercised when considering the use of noncertified pulse oximeters to diagnose or treat hypoxia. These findings support the notion that further development of accurate, low-cost oximeters is feasible and should continue to be pursued to improve access to safe clinical care, especially in low-income countries.

DISCLOSURES

Name: Michael S. Lipnick, MD.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Name: John R. Feiner, MD.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Name: Paul Au, BS.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Name: Michael Bernstein, BS.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Name: Philip E. Bickler, MD, PhD.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

This manuscript was handled by: Maxime Cannesson, MD, PhD.

ACKNOWLEDGMENTS

The authors thank Dr. Iain H Wilson, Anesthesiologist and LifeBox Trustee.

REFERENCES

1. Funk LM, Weiser TG, Berry WR, Lipsitz SR, Merry AF, Enright AC, Wilson IH, Dziekan G, Gawande AAGlobal operating theatre distribution and pulse oximetry supply: an estimation from reported data.Lancet2010376105561
2. Adler JN, Hughes LA, Vivilecchia R, Camargo CA Jr.Effect of skin pigmentation on pulse oximetry accuracy in the emergency department.Acad Emerg Med1998596570
3. Bothma PA, Joynt GM, Lipman J, Hon H, Mathala B, Scribante J, Kromberg JAccuracy of pulse oximetry in pigmented patients.S Afr Med J1996865946
4. Ries AL, Prewitt LM, Johnson JJSkin color and ear oximetry.Chest19899628790
5. Emery JRSkin pigmentation as an influence on the accuracy of pulse oximetry.J Perinatol1987732930
6. Ralston AC, Webb RK, Runciman WBPotential errors in pulse oximetry. III: Effects of interferences, dyes, dyshaemoglobins and other pigments.Anaesthesia1991462915
7. Coté CJ, Goldstein EA, Fuchsman WH, Hoaglin DCThe effect of nail polish on pulse oximetry.Anesth Analg1988676836
8. Battito MFThe effect of fingerprinting ink on pulse oximetry.Anesth Analg198969265
9. Scheller MS, Unger RJ, Kelner MJEffects of intravenously administered dyes on pulse oximetry readings.Anesthesiology1986655502
10. Lee KH, Hui KP, Tan WC, Lim TKFactors influencing pulse oximetry as compared to functional arterial saturation in multi-ethnic Singapore.Singapore Med J1993343857
11. Zeballos RJ, Weisman IMReliability of noninvasive oximetry in black subjects during exercise and hypoxia.Am Rev Respir Dis199114412404
12. Fluke Biomedical: The SPOT Light SpO2 Pulse Oximeter 15-second tester.Available at: http://www.flukebiomedical.com/Biomedical/usen/Events/Promos/prosim-needs-DA-A?gclid=CK71pIDe68kCFQqPfgodfqEEEQ#ProSimSpotlight. Accessed December 28, 2015
13. ISO 80601-2-61:2011. Medical electrical equipment—Part 2-61: Particular requirements for basic safety and essential performance of pulse oximeter equipment.Available at: http://www.iso.org/iso/catalogue_detail.htm?csnumber=51847. Accessed October 20, 2015
14. Pulse Oximeters - Premarket Notification Submissions [510(k)s]: Guidance for Industry and Food and Drug Administration Staff.Available at: http://www.fda.gov/RegulatoryInformation/Guidances/ucm341718.htm. Accessed October 21, 2015
15. US Food and Drug Administration. 510(k) Clearances.Available at: http://www.fda.gov/MedicalDevices/ProductsandMedicalProcedures/DeviceApprovalsandClearances/510kClearances/default.htm. Accessed October 20, 2015
16. Bland JM, Altman DGAgreement between methods of measurement with multiple observations per individual.J Biopharm Stat20071757182
17. Global Pulse Oximetry Project. 2008.Available at: http://www.who.int/patientsafety/events/08/1st_pulse_oximetry_meeting_background_doc.pdf. Accessed October 31, 2015
18. Weiser TG, Regenbogen SE, Thompson KD, Haynes AB, Lipsitz SR, Berry WR, Gawande AAAn estimation of the global volume of surgery: a modelling strategy based on available data.Lancet200837213944
19. Burn SL, Chilton PJ, Gawande AA, Lilford RJPeri-operative pulse oximetry in low-income countries: a cost-effectiveness analysis.Bull World Health Organ20149285867
20. Dubowitz G, Breyer K, Lipnick M, Sall JW, Feiner J, Ikeda K, MacLeod DB, Bickler PEAccuracy of the Lifebox pulse oximeter during hypoxia in healthy volunteers.Anaesthesia20136812203
21. Kenek O2 Pulse Oximeter.Available at: http://lgtmedical.com/kenek/O2.html. Accessed October 20, 2015
22. Bickler PE, Feiner JR, Severinghaus JWEffects of skin pigmentation on pulse oximeter accuracy at low saturation.Anesthesiology20051027159
23. Feiner JR, Severinghaus JW, Bickler PEDark skin decreases the accuracy of pulse oximeters at low oxygen saturation: the effects of oximeter probe type and gender.Anesth Analg2007105S1823
24. Jubran A, Tobin MJReliability of pulse oximetry in titrating supplemental oxygen therapy in ventilator-dependent patients.Chest19909714205
25. Noninvasive blood gas monitoring: a review for use in the adult critical care unit. Technology Subcommittee of the Working Group on Critical Care, Ontario Ministry of Health.CMAJ199214670312
Copyright © 2016 International Anesthesia Research Society