Share this article on:

The Accuracy of 6 Inexpensive Pulse Oximeters Not Cleared by the Food and Drug Administration: The Possible Global Public Health Implications

Lipnick, Michael S. MD; Feiner, John R. MD; Au, Paul BS; Bernstein, Michael BS; Bickler, Philip E. MD, PhD

doi: 10.1213/ANE.0000000000001300
Technology, Computing, and Simulation: Original Clinical Research Report

BACKGROUND: Universal access to pulse oximetry worldwide is often limited by cost and has substantial public health consequences. Low-cost pulse oximeters have become increasingly available with limited regulatory agency oversight. The accuracy of these devices often has not been validated, raising questions about performance.

METHODS: The accuracy of 6 low-cost finger pulse oximeters during stable arterial oxygen saturations (SaO2) between 70% and 100% was evaluated in 22 healthy subjects. Oximeters tested were the Contec CMS50DL, Beijing Choice C20, Beijing Choice MD300C23, Starhealth SH-A3, Jumper FPD-500A, and Atlantean SB100 II. Inspired oxygen, nitrogen, and carbon dioxide partial pressures were monitored and adjusted via a partial rebreathing circuit to achieve 10 to 12 stable target SaO2 plateaus between 70% and 100% and PaCO2 values of 35 to 45 mm Hg. Comparisons of pulse oximeter readings (SpO2) with arterial SaO2 (by Radiometer ABL90 and OSM3) were used to calculate bias (SpO2 – SaO2) mean, precision (SD of the bias), and root mean square error (ARMS).

RESULTS: Pulse oximeter readings corresponding to 536 blood samples were analyzed. Four of the 6 oximeters tested showed large errors (up to −6.30% mean bias, precision 4.30%, 7.53 ARMS) in estimating saturation when SaO2 was reduced <80%, and half of the oximeters demonstrated large errors when estimating saturations between 80% and 90%. Two of the pulse oximeters tested (Contec CMS50DL and Beijing Choice C20) demonstrated ARMS of <3% at SaO2 between 70% and 100%, thereby meeting International Organization for Standardization (ISO) criteria for accuracy.

CONCLUSIONS: Many low-cost pulse oximeters sold to consumers demonstrate highly inaccurate readings. Unexpectedly, the accuracy of some low-cost pulse oximeters tested here performed similarly to more expensive, ISO-cleared units when measuring hypoxia in healthy subjects. None of those tested here met World Federation of Societies of Anaesthesiologists standards, and the ideal testing conditions do not necessarily translate these findings to the clinical setting. Nonetheless, further development of accurate, low-cost oximeters for use in clinical practice is feasible and, if pursued, could improve access to safe care, especially in low-income countries.

Published ahead of print April 13, 2016

From the *Department of Anesthesia and Perioperative Care, University of California at San Francisco, San Francisco, California; and Physio Monitor LLC., San Ramon, California.

Accepted for publication January 28, 2016.

Published ahead of print April 13, 2016

Funding: UCSF Hypoxia Research Laboratory.

The authors declare no conflicts of interest.

Reprints will not be available from the authors.

Address correspondence to Michael S. Lipnick, MD, Department of Anesthesia and Perioperative Care, University of California Medical Center, Sciences 256, Box 0542, 513 Parnassus Ave, San Francisco, CA 94143. Address e-mail to Lipnickm@anesthesia.ucsf.edu.

Pulse oximeters have become invaluable diagnostic tools because of ease of use, portability, and applicability in a wide range of clinical settings. The increase in utilization and availability of this technology has occurred disproportionately in higher income countries in large part because of significant cost and supply chain barriers in low- and middle-income countries (LMICs). Recent studies estimate that >77,000 operating rooms around the world have no access to pulse oximetry.1 Pulse oximeters can cost >$3000 US dollar (USD) per unit. Until recently, few units were available for <$1000 USD and virtually none for <$50. As a result of increasing popularity and technological advancements, the manufacture of many low-cost pulse oximeters has emerged, although the accuracy of many of these devices is uncertain.

Pulse oximetry computes arterial hemoglobin oxygen saturation from the ratio of the pulsatile to the total transmitted red light divided by the same ratio for infrared light transilluminating the finger, ear, or other tissue. In theory, the derived saturation should be independent of skin pigmentation and many other variables, including hemoglobin concentration, nail polish, dirt, and jaundice. In practice, pulse oximeter measurements are influenced by many variables, and algorithm testing is required to refine the accuracy of these devices.2–11 Such testing often begins with in vitro analysis using a device such as the Fluke SPOT Light Pulse Oximeter Tester (Fluke Biomedical, Everett, WA).12 This device generates electronic signals that are interpreted by the pulse oximeter being tested. Users can manipulate heart rate, SpO2, artifact, and perfusion to varying degrees. Ideally, before clinical use, instrument readings are validated by testing human subjects using arterial blood saturation values (SaO2) measured with a gold standard multiwavelength oximeter. Testing of pulse oximeters in human subjects during hypoxemia is expensive and done at only a few centers in the world. Such testing is not required for marketing these products to consumers for nonmedical use and therefore is often not done by manufacturers of low-cost pulse oximeters.

All pulse oximeters marketed for medical use in the United States are required by the Food and Drug Administration (FDA) to have been tested and certified to be accurate to root mean square error (ARMS) of <3% at SaO2 between 70% and 100%.13 The great majority of calibration and confirmation tests are done on healthy volunteer subjects.

The purpose of this study was to evaluate the accuracy of several low-cost pulse oximeters as a sample of the large number of such devices currently available for purchase on the Internet, in pharmacies, and in retail stores for <$50 USD. We hypothesize that inexpensive pulse oximeters currently available to consumers are inaccurate and would not meet International Organization for Standardization (ISO 80601-2-61:2011) standards for use in clinical practice.13

Back to Top | Article Outline

METHODS

This study was approved by the University of California at San Francisco (UCSF) Committee on Human Research and the Shenzhen University School of Medicine IRB. Written informed consent was obtained from all subjects. Two groups of 11 healthy, nonsmoking subjects were studied; the first 11 subjects were studied at UCSF, and the second group of 11 subjects was studied at Shenzhen, China, using the same protocol. None of the subjects had lung disease, obesity, or cardiovascular problems. They ranged in age between 18 and 40 years and had a range of skin pigments as described in Table 1.

Table 1

Table 1

Sample size was decided a priori based on FDA guidelines for accuracy testing. These guidelines require a minimum of 200 data points over an SaO2 range of 70% to 100%, well-balanced in the 70% to 80%, 80% to 90%, and 90% to 100% ranges. In addition, the FDA requires men and women of varying skin tones be included.14

Subjects were studied using identical protocols implemented by these 2 study laboratories that routinely test pulse oximeters for FDA 501(k) certification.14,15 One of the authors was present at the study site in Shenzhen for the experiments. Briefly, study subjects were semisupine (30° head up) with a nose clip, breathing air-nitrogen-carbon dioxide mixtures via a mouthpiece from a partial rebreathing circuit with a voluntarily increased minute ventilation and 10 to 20 L/min fresh gas inflow. An indwelling 22-g radial artery catheter was placed to sample arterial blood for the measurement of SaO2. In the first group of 11 study subjects at the UCSF study site, 3 pulse oximeters were placed on each subject: Starhealth SH-A3 (Star Health Medical, Huoying Hualongyuanzhongli, Changping District, Beijing, China), Jumper FPD-500A (Shenzhen Jumper Medical Equipment, Shenzhen, Guangdong Province, China), and Atlantean SB100 II (Atlantean Corporation, Chubay City, Taiwan). In the second group of 11 study subjects (Shenzhen, China), 3 pulse oximeters were placed on each subject: the Contec CMS50DL, Beijing Choice C20, and Beijing Choice MD300C23. Because none of these devices had any digital or analog output of saturation values, readings were recorded by hand throughout the protocol.

A series of 10 to 12 stable target SaO2 plateaus between 70% and 100% (approximately 70%, 73%, 76%, 80%, 83%, 86%, 89%, 92%, 95%, 98%, and 100%) were sought by an operator who adjusted the inspired air-nitrogen-carbon dioxide mixture breath by breath in response to the computer display of the estimated SaO2 derived from end-tidal gas analysis using LabVIEW 2013 (National Instruments, Austin, TX).2 Input parameters for the computer O2 dissociation curve included hemoglobin P50, base excess, and alveolar-arterial (A − a) oxygen difference. Inputs were adjusted as needed to match the predicted with measured SaO2. At each level, arterial blood was sampled after a plateau of 30 to 60 seconds had been achieved, followed by a second sample at the same plateau 30 seconds later. To ensure that each subject had good circulation to the fingers, each hand was wrapped in a warming pad. “Functional” arterial SaO2 (HbO2/[Hb+HbO2]) was determined by multi-wavelength oximetry using a Radiometer (Copenhagen, Denmark) ABL-90 at UCSF and an OSM3 at Shenzhen. Both instruments were calibrated according to manufacturer recommendations.

Back to Top | Article Outline

Statistical Analysis

Bias was computed as SpO2 - SaO2 from each oximeter’s reading minus the corresponding arterial blood sample value. Bias is reported as mean ± SD, where the SD of the bias represents precision. The SD was calculated according to Bland and Altman with adjustments for multiple measurements for each individual according to the “Method Where the True Value Varies.”16 Limits of agreement are 1.96·SD. The 95% confidence intervals were calculated using bootstrapping (random resampling with replacement) with 50,000 repetitions. ARMS was calculated as the square root of the mean difference between SpO2 − SaO2, squared. The 95% confidence intervals for ARMS were calculated by bootstrapping, as above. An Arms <3% is the accuracy standard used by the FDA. Bias is plotted versus SaO2 as the gold standard. The relationship of bias to SaO2 was analyzed by linear regression accounting for repeated measures. Bias was also analyzed for decadal subgroups of SaO2 (70%–80%, 80%–90%, and 90%–100%) using repeated-measures analysis of variance, with the Tukey-Kramer honestly significant difference for multiple comparison testing. The Levene test was used to compare variances between the different decadal ranges. A P value <0.05 was considered statistically significant. Statistical analysis was performed with JMP 11.0 (SAS Institute, Cary, NC) and Stata 14 (StataCorp, College Station, TX).

Back to Top | Article Outline

RESULTS

Readings from 6 pulse oximeters corresponding to 536 blood samples in 22 healthy subjects were obtained. Characteristics of all study subjects are presented in Table 1.

The ARMS over the range of SaO2 between 70% and 100% was <3%, the FDA clearance threshold, for 2 of the 6 oximeters tested, the Contec CMS50DL and the Beijing Choice C20 (Table 2). The remaining 4 oximeters demonstrated >3% ARMS. This was largely because of high mean bias (Table 2).

Table 2

Table 2

Figure 1

Figure 1

Figure 2

Figure 2

Figure 1 illustrates bias for individual data points obtained for individual subjects from each oximeter tested. Figure 2 displays bias for each oximeter tested. All oximeters tested demonstrated worsening performance with hypoxia, with mean bias increasing at lower oxygen saturations. This resulted in wide limits of agreement. Data are summarized in Table 2.

Back to Top | Article Outline

DISCUSSION

Pulse oximetry theory predicts that the ratio of pulsatile to total transmitted red light divided by the same ratio for infrared light should depend only on arterial saturation. In practice, there are many factors that must be accounted for to ensure accurate calculation of oxygen saturation by pulse oximetry. Major known variability is caused by anemia, light scattering, venous and tissue pulsation by mechanical force from nearby arteries, pulsatile variations in tissue thickness in the light path other than in the arteries, nail polish, and skin pigment, among others.2 Therefore, the design of accurate pulse oximeters often requires empirically determined correction factors obtained by in vivo comparison of oximeter readings with arterial oxyhemoglobin saturation of volunteer subjects during hypoxemia. Many commonly used low-cost pulse oximeters do not undergo this in vivo testing, and thus, little is known about the accuracy of such devices.

In this study, we tested the accuracy of 6 low-cost pulse oximeters currently available for purchase from popular consumer retailers. The majority (4/6) of the oximeters tested did not meet US FDA standards for accuracy. Unexpectedly, 2 of the 6 oximeters did meet accuracy standards defined by the FDA and ISO, an Arms <3%. Oximeters 4 and 5 sell for approximately $25 USD and performed with similar accuracy for measuring hypoxia in healthy volunteers when compared with popular, clinically approved devices that cost >$3000 USD. Although more expensive units often contain additional features (such as hemoglobin concentration, methemoglobin, and carboxyhemoglobin measurements or enhanced detection during motion or low perfusion states, and faster processing speed), our findings demonstrate that simple, accurate pulse oximeters can likely be developed at significantly lower cost than many units currently available on the market.

It is important to note that although some of the inexpensive units tested here demonstrated accurate saturation readings during hypoxia, none meet current World Health Organization or World Federation of Societies of Anaesthesiologists standards for use in clinical practice.

All instruments tested showed increasingly larger bias (both positive and negative) in oximeter readings at low SaO2 in subjects. It may be that this is a property common to a wide variety of inexpensive pulse oximeters, given that the basic technology is similar. More expensive devices that are FDA-cleared, such as those manufactured by Nellcor, Nonin, Masimo, and others, show a smaller degree of increasing bias during lower SaO2 conditions.2

The magnitude of the oximeter error in all 6 oximeters tested here was relatively small at saturations >90% and probably of no clinical significance. However, there were large and variable errors in performance of pulse oximeters at lower saturations. Clinical use of these devices is therefore of obvious concern. In one subject for oximeters 3 and 6, the device appeared to have locked onto a saturation value and therefore did not detect increasing hypoxemia.

It is not clear to what extent low-cost (<$50 USD) pulse oximeters are used for medical diagnoses worldwide. It has been our experience through extensive time spent in practice environments in LMICs that the use of nontested/non-FDA or Conformité Européene-regulated oximeters is extensive. Furthermore, it has been our experience that the device manufacturers tested here represent a large proportion, if not the majority, of low-cost oximeters currently in use.

To improve anesthesia safety worldwide, conservative estimates indicate that 100,000 additional pulse oximeters are needed, and when accounting for additional care settings such as recovery rooms, >1 million units may be required.17

Our finding that 2 of the low-cost oximeters studied here performed with accuracy to meet FDA clearance standards is particularly relevant for clinical environments in LMICs where access may be limited because of cost. It is in LMICs that most of the world’s surgical disease burden exists with the fewest per capita health care resources (such as health care providers and pulse oximeters) to address the problem.1,18 Furthermore, the purchase of pulse oximeters that cost several thousand—or even several hundred—dollars may not be feasible in these settings, despite research suggesting cost-effectiveness on par with other public health interventions.19

At present, a few initiatives are underway to increase access to low-cost, accurate pulse oximeters in low-income countries. The LifeBox Project aims to equip the world’s operating theaters with pulse oximeters specifically designed and priced for this purpose. We have previously shown that the LifeBox does meet US FDA accuracy standards.20 LifeBox pulse oximeters are donated or purchased by users. The current cost for a LifeBox oximeter is $250, which includes an adult probe, a pediatric probe, a rechargeable battery, delivery by courier anywhere in the world, training materials, and a 2-year warranty (1-year warranty for the probes). Another example of a project aimed at increasing global access to pulse oximeters is the LionsGate Medical Kenek Pulse Oximeter (Lionsgate Technologies, Vancouver, British Columbia, Canada).21 This device costs approximately $50 to $80, although it also requires the purchase of a compatible mobile phone or tablet (iOS) device.

Although the units tested in this study were sold for <$50 USD, this price does not account for additional costs, such as delivery of the unit and replacement batteries, which can add considerably to the overall lifetime cost of these devices.

Nonetheless, based on the data we present here and careful evaluation of the components and features of numerous currently available FDA-cleared pulse oximeters (including screen size, alarms, rechargeable battery, and durable cases), a lower price point for many pulse oximeter units can likely be achieved to target LMIC markets without compromising accuracy or utility.

The present study has several limitations. Study subjects were recruited on a volunteer basis, with limited attempts to recruit subjects of different skin tones in equal proportions. We did not evaluate the impact of skin pigment on saturation although pigment is known to cause inaccurate readings in pulse oximeters.22,23 For individuals with darkly pigmented skin, bias of up to 8% has been reported at lower saturations, in FDA-cleared pulse oximeters tested approximately 10 years ago.22 Although subjects of all skin color are represented in this study, the majority were Asians and Caucasians, and thus applicability to non-Caucasian populations may be limited.

An additional limitation is the use of 2 groups of subjects in 2 different laboratories. Although identical protocols were used, and one of the authors of this article was present during testing at both sites, variability in procedures or differences in study subjects cannot fully be excluded. We studied only 6 oximeters, and thus our results may not be applicable to all non–FDA-cleared oximeters available.

One of the most significant limitations of this study is the ideal conditions under which it was conducted. All volunteers were healthy, with good perfusion values (some with hands wrapped with warming pads) and remained essentially motionless throughout the protocols. Poor perfusion and patient movement are known to significantly affect pulse oximeter accuracy. In clinical scenarios, such as critical illness, factors such as motion artifact and vasoconstriction are likely to play a significant role and further degrade the accuracy of low-cost pulse oximeters, perhaps to a greater degree than more expensive units.11,24,25 Because these relevant clinical factors, among others, were not evaluated, this study should not serve as an endorsement for the clinical use of any of the pulse oximeters tested.

Back to Top | Article Outline

CONCLUSIONS

Inexpensive pulse oximeters have become increasingly available with little regulatory oversight. Most of the commonly sold units tested here demonstrated highly inaccurate saturation readings during hypoxia. A small proportion of the low-cost pulse oximeters tested here performed similarly to more expensive, FDA-cleared units when measuring hypoxia in healthy subjects, although none met World Federation of Societies of Anaesthesiologists standards for clinical use. Caution must be exercised when considering the use of noncertified pulse oximeters to diagnose or treat hypoxia. These findings support the notion that further development of accurate, low-cost oximeters is feasible and should continue to be pursued to improve access to safe clinical care, especially in low-income countries.

Back to Top | Article Outline

DISCLOSURES

Name: Michael S. Lipnick, MD.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Name: John R. Feiner, MD.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Name: Paul Au, BS.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Name: Michael Bernstein, BS.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Name: Philip E. Bickler, MD, PhD.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

This manuscript was handled by: Maxime Cannesson, MD, PhD.

Back to Top | Article Outline

ACKNOWLEDGMENTS

The authors thank Dr. Iain H Wilson, Anesthesiologist and LifeBox Trustee.

Back to Top | Article Outline

REFERENCES

1. Funk LM, Weiser TG, Berry WR, Lipsitz SR, Merry AF, Enright AC, Wilson IH, Dziekan G, Gawande AA. Global operating theatre distribution and pulse oximetry supply: an estimation from reported data. Lancet 2010;376:1055–61.
2. Adler JN, Hughes LA, Vivilecchia R, Camargo CA Jr. Effect of skin pigmentation on pulse oximetry accuracy in the emergency department. Acad Emerg Med 1998;5:965–70.
3. Bothma PA, Joynt GM, Lipman J, Hon H, Mathala B, Scribante J, Kromberg J. Accuracy of pulse oximetry in pigmented patients. S Afr Med J 1996;86:594–6.
4. Ries AL, Prewitt LM, Johnson JJ. Skin color and ear oximetry. Chest 1989;96:287–90.
5. Emery JR. Skin pigmentation as an influence on the accuracy of pulse oximetry. J Perinatol 1987;7:329–30.
6. Ralston AC, Webb RK, Runciman WB. Potential errors in pulse oximetry. III: Effects of interferences, dyes, dyshaemoglobins and other pigments. Anaesthesia 1991;46:291–5.
7. Coté CJ, Goldstein EA, Fuchsman WH, Hoaglin DC. The effect of nail polish on pulse oximetry. Anesth Analg 1988;67:683–6.
8. Battito MF. The effect of fingerprinting ink on pulse oximetry. Anesth Analg 1989;69:265.
9. Scheller MS, Unger RJ, Kelner MJ. Effects of intravenously administered dyes on pulse oximetry readings. Anesthesiology 1986;65:550–2.
10. Lee KH, Hui KP, Tan WC, Lim TK. Factors influencing pulse oximetry as compared to functional arterial saturation in multi-ethnic Singapore. Singapore Med J 1993;34:385–7.
11. Zeballos RJ, Weisman IM. Reliability of noninvasive oximetry in black subjects during exercise and hypoxia. Am Rev Respir Dis 1991;144:1240–4.
12. Fluke Biomedical: The SPOT Light SpO2 Pulse Oximeter 15-second tester. Available at: http://www.flukebiomedical.com/Biomedical/usen/Events/Promos/prosim-needs-DA-A?gclid=CK71pIDe68kCFQqPfgodfqEEEQ#ProSimSpotlight. Accessed December 28, 2015.
13. ISO 80601-2-61:2011. Medical electrical equipment—Part 2-61: Particular requirements for basic safety and essential performance of pulse oximeter equipment. Available at: http://www.iso.org/iso/catalogue_detail.htm?csnumber=51847. Accessed October 20, 2015.
14. Pulse Oximeters - Premarket Notification Submissions [510(k)s]: Guidance for Industry and Food and Drug Administration Staff. Available at: http://www.fda.gov/RegulatoryInformation/Guidances/ucm341718.htm. Accessed October 21, 2015.
15. US Food and Drug Administration. 510(k) Clearances. Available at: http://www.fda.gov/MedicalDevices/ProductsandMedicalProcedures/DeviceApprovalsandClearances/510kClearances/default.htm. Accessed October 20, 2015.
16. Bland JM, Altman DG. Agreement between methods of measurement with multiple observations per individual. J Biopharm Stat 2007;17:571–82.
17. Global Pulse Oximetry Project. 2008. Available at: http://www.who.int/patientsafety/events/08/1st_pulse_oximetry_meeting_background_doc.pdf. Accessed October 31, 2015.
18. Weiser TG, Regenbogen SE, Thompson KD, Haynes AB, Lipsitz SR, Berry WR, Gawande AA. An estimation of the global volume of surgery: a modelling strategy based on available data. Lancet 2008;372:139–44.
19. Burn SL, Chilton PJ, Gawande AA, Lilford RJ. Peri-operative pulse oximetry in low-income countries: a cost-effectiveness analysis. Bull World Health Organ 2014;92:858–67.
20. Dubowitz G, Breyer K, Lipnick M, Sall JW, Feiner J, Ikeda K, MacLeod DB, Bickler PE. Accuracy of the Lifebox pulse oximeter during hypoxia in healthy volunteers. Anaesthesia 2013;68:1220–3.
21. Kenek O2 Pulse Oximeter. Available at: http://lgtmedical.com/kenek/O2.html. Accessed October 20, 2015.
22. Bickler PE, Feiner JR, Severinghaus JW. Effects of skin pigmentation on pulse oximeter accuracy at low saturation. Anesthesiology 2005;102:715–9.
23. Feiner JR, Severinghaus JW, Bickler PE. Dark skin decreases the accuracy of pulse oximeters at low oxygen saturation: the effects of oximeter probe type and gender. Anesth Analg 2007;105:S18–23.
24. Jubran A, Tobin MJ. Reliability of pulse oximetry in titrating supplemental oxygen therapy in ventilator-dependent patients. Chest 1990;97:1420–5.
25. Noninvasive blood gas monitoring: a review for use in the adult critical care unit. Technology Subcommittee of the Working Group on Critical Care, Ontario Ministry of Health. CMAJ 1992;146:703–12.
© 2016 International Anesthesia Research Society