Technology, Computing, and Simulation: Original Clinical Research Report

The Accuracy of 6 Inexpensive Pulse Oximeters Not Cleared by the Food and Drug Administration: The Possible Global Public Health Implications

Lipnick, Michael S. MD^*; Feiner, John R. MD^*; Au, Paul BS^*; Bernstein, Michael BS^†; Bickler, Philip E. MD, PhD^*

From the ^*Department of Anesthesia and Perioperative Care, University of California at San Francisco, San Francisco, California; and ^†Physio Monitor LLC., San Ramon, California.

Accepted for publication January 28, 2016.

Published ahead of print April 13, 2016

Funding: UCSF Hypoxia Research Laboratory.

The authors declare no conflicts of interest.

Reprints will not be available from the authors.

Address correspondence to Michael S. Lipnick, MD, Department of Anesthesia and Perioperative Care, University of California Medical Center, Sciences 256, Box 0542, 513 Parnassus Ave, San Francisco, CA 94143. Address e-mail to [email protected].

Anesthesia & Analgesia 123(2):p 338-345, August 2016. | DOI: 10.1213/ANE.0000000000001300

Free

BACKGROUND:

Universal access to pulse oximetry worldwide is often limited by cost and has substantial public health consequences. Low-cost pulse oximeters have become increasingly available with limited regulatory agency oversight. The accuracy of these devices often has not been validated, raising questions about performance.

METHODS:

The accuracy of 6 low-cost finger pulse oximeters during stable arterial oxygen saturations (Sao₂) between 70% and 100% was evaluated in 22 healthy subjects. Oximeters tested were the Contec CMS50DL, Beijing Choice C20, Beijing Choice MD300C23, Starhealth SH-A3, Jumper FPD-500A, and Atlantean SB100 II. Inspired oxygen, nitrogen, and carbon dioxide partial pressures were monitored and adjusted via a partial rebreathing circuit to achieve 10 to 12 stable target Sao₂ plateaus between 70% and 100% and Paco₂ values of 35 to 45 mm Hg. Comparisons of pulse oximeter readings (Spo₂) with arterial Sao₂ (by Radiometer ABL90 and OSM3) were used to calculate bias (Spo₂ – Sao₂) mean, precision (SD of the bias), and root mean square error (A_RMS).

RESULTS:

Pulse oximeter readings corresponding to 536 blood samples were analyzed. Four of the 6 oximeters tested showed large errors (up to −6.30% mean bias, precision 4.30%, 7.53 A_RMS) in estimating saturation when Sao₂ was reduced <80%, and half of the oximeters demonstrated large errors when estimating saturations between 80% and 90%. Two of the pulse oximeters tested (Contec CMS50DL and Beijing Choice C20) demonstrated A_RMS of <3% at Sao₂ between 70% and 100%, thereby meeting International Organization for Standardization (ISO) criteria for accuracy.

CONCLUSIONS:

Many low-cost pulse oximeters sold to consumers demonstrate highly inaccurate readings. Unexpectedly, the accuracy of some low-cost pulse oximeters tested here performed similarly to more expensive, ISO-cleared units when measuring hypoxia in healthy subjects. None of those tested here met World Federation of Societies of Anaesthesiologists standards, and the ideal testing conditions do not necessarily translate these findings to the clinical setting. Nonetheless, further development of accurate, low-cost oximeters for use in clinical practice is feasible and, if pursued, could improve access to safe care, especially in low-income countries.

Pulse oximeters have become invaluable diagnostic tools because of ease of use, portability, and applicability in a wide range of clinical settings. The increase in utilization and availability of this technology has occurred disproportionately in higher income countries in large part because of significant cost and supply chain barriers in low- and middle-income countries (LMICs). Recent studies estimate that >77,000 operating rooms around the world have no access to pulse oximetry.^¹ Pulse oximeters can cost >$3000 US dollar (USD) per unit. Until recently, few units were available for <$1000 USD and virtually none for <$50. As a result of increasing popularity and technological advancements, the manufacture of many low-cost pulse oximeters has emerged, although the accuracy of many of these devices is uncertain.

Pulse oximetry computes arterial hemoglobin oxygen saturation from the ratio of the pulsatile to the total transmitted red light divided by the same ratio for infrared light transilluminating the finger, ear, or other tissue. In theory, the derived saturation should be independent of skin pigmentation and many other variables, including hemoglobin concentration, nail polish, dirt, and jaundice. In practice, pulse oximeter measurements are influenced by many variables, and algorithm testing is required to refine the accuracy of these devices.^{^2–11} Such testing often begins with in vitro analysis using a device such as the Fluke SPOT Light Pulse Oximeter Tester (Fluke Biomedical, Everett, WA).^¹² This device generates electronic signals that are interpreted by the pulse oximeter being tested. Users can manipulate heart rate, Spo₂, artifact, and perfusion to varying degrees. Ideally, before clinical use, instrument readings are validated by testing human subjects using arterial blood saturation values (Sao₂) measured with a gold standard multiwavelength oximeter. Testing of pulse oximeters in human subjects during hypoxemia is expensive and done at only a few centers in the world. Such testing is not required for marketing these products to consumers for nonmedical use and therefore is often not done by manufacturers of low-cost pulse oximeters.

All pulse oximeters marketed for medical use in the United States are required by the Food and Drug Administration (FDA) to have been tested and certified to be accurate to root mean square error (A_RMS) of <3% at Sao₂ between 70% and 100%.^¹³ The great majority of calibration and confirmation tests are done on healthy volunteer subjects.

The purpose of this study was to evaluate the accuracy of several low-cost pulse oximeters as a sample of the large number of such devices currently available for purchase on the Internet, in pharmacies, and in retail stores for <$50 USD. We hypothesize that inexpensive pulse oximeters currently available to consumers are inaccurate and would not meet International Organization for Standardization (ISO 80601-2-61:2011) standards for use in clinical practice.^¹³

METHODS

This study was approved by the University of California at San Francisco (UCSF) Committee on Human Research and the Shenzhen University School of Medicine IRB. Written informed consent was obtained from all subjects. Two groups of 11 healthy, nonsmoking subjects were studied; the first 11 subjects were studied at UCSF, and the second group of 11 subjects was studied at Shenzhen, China, using the same protocol. None of the subjects had lung disease, obesity, or cardiovascular problems. They ranged in age between 18 and 40 years and had a range of skin pigments as described in Table 1.

Sample size was decided a priori based on FDA guidelines for accuracy testing. These guidelines require a minimum of 200 data points over an Sao₂ range of 70% to 100%, well-balanced in the 70% to 80%, 80% to 90%, and 90% to 100% ranges. In addition, the FDA requires men and women of varying skin tones be included.^¹⁴

Subjects were studied using identical protocols implemented by these 2 study laboratories that routinely test pulse oximeters for FDA 501(k) certification.^¹⁴^,^¹⁵ One of the authors was present at the study site in Shenzhen for the experiments. Briefly, study subjects were semisupine (30° head up) with a nose clip, breathing air-nitrogen-carbon dioxide mixtures via a mouthpiece from a partial rebreathing circuit with a voluntarily increased minute ventilation and 10 to 20 L/min fresh gas inflow. An indwelling 22-g radial artery catheter was placed to sample arterial blood for the measurement of Sao₂. In the first group of 11 study subjects at the UCSF study site, 3 pulse oximeters were placed on each subject: Starhealth SH-A3 (Star Health Medical, Huoying Hualongyuanzhongli, Changping District, Beijing, China), Jumper FPD-500A (Shenzhen Jumper Medical Equipment, Shenzhen, Guangdong Province, China), and Atlantean SB100 II (Atlantean Corporation, Chubay City, Taiwan). In the second group of 11 study subjects (Shenzhen, China), 3 pulse oximeters were placed on each subject: the Contec CMS50DL, Beijing Choice C20, and Beijing Choice MD300C23. Because none of these devices had any digital or analog output of saturation values, readings were recorded by hand throughout the protocol.

A series of 10 to 12 stable target Sao₂ plateaus between 70% and 100% (approximately 70%, 73%, 76%, 80%, 83%, 86%, 89%, 92%, 95%, 98%, and 100%) were sought by an operator who adjusted the inspired air-nitrogen-carbon dioxide mixture breath by breath in response to the computer display of the estimated Sao₂ derived from end-tidal gas analysis using LabVIEW 2013 (National Instruments, Austin, TX).^² Input parameters for the computer O₂ dissociation curve included hemoglobin P₅₀, base excess, and alveolar-arterial (A − a) oxygen difference. Inputs were adjusted as needed to match the predicted with measured Sao₂. At each level, arterial blood was sampled after a plateau of 30 to 60 seconds had been achieved, followed by a second sample at the same plateau 30 seconds later. To ensure that each subject had good circulation to the fingers, each hand was wrapped in a warming pad. “Functional” arterial Sao₂ (HbO₂/[Hb+HbO₂]) was determined by multi-wavelength oximetry using a Radiometer (Copenhagen, Denmark) ABL-90 at UCSF and an OSM3 at Shenzhen. Both instruments were calibrated according to manufacturer recommendations.

Statistical Analysis

Bias was computed as Spo₂ - Sao₂ from each oximeter’s reading minus the corresponding arterial blood sample value. Bias is reported as mean ± SD, where the SD of the bias represents precision. The SD was calculated according to Bland and Altman with adjustments for multiple measurements for each individual according to the “Method Where the True Value Varies.”^¹⁶ Limits of agreement are 1.96·SD. The 95% confidence intervals were calculated using bootstrapping (random resampling with replacement) with 50,000 repetitions. A_RMS was calculated as the square root of the mean difference between Spo₂ − Sao₂, squared. The 95% confidence intervals for A_RMS were calculated by bootstrapping, as above. An A_rms <3% is the accuracy standard used by the FDA. Bias is plotted versus Sao₂ as the gold standard. The relationship of bias to Sao₂ was analyzed by linear regression accounting for repeated measures. Bias was also analyzed for decadal subgroups of Sao₂ (70%–80%, 80%–90%, and 90%–100%) using repeated-measures analysis of variance, with the Tukey-Kramer honestly significant difference for multiple comparison testing. The Levene test was used to compare variances between the different decadal ranges. A P value <0.05 was considered statistically significant. Statistical analysis was performed with JMP 11.0 (SAS Institute, Cary, NC) and Stata 14 (StataCorp, College Station, TX).

RESULTS

Readings from 6 pulse oximeters corresponding to 536 blood samples in 22 healthy subjects were obtained. Characteristics of all study subjects are presented in Table 1.

The A_RMS over the range of Sao₂ between 70% and 100% was <3%, the FDA clearance threshold, for 2 of the 6 oximeters tested, the Contec CMS50DL and the Beijing Choice C20 (Table 2). The remaining 4 oximeters demonstrated >3% A_RMS. This was largely because of high mean bias (Table 2).

Table 2.:
Summary of Data from 6 Low-Cost Pulse Oximeters

Figure 1.:
Bias (pulse oximeter oxygen saturation [Spo ₂] − arterial blood oxygen saturation [Sao ₂]) is plotted against Sao ₂ measured by either an ABL90 (panels A–C) or OSM3 (panels D–F) hemoximeter (Radiometer). Mean bias is shown by a solid horizontal line. Dashed horizontal lines are the upper and lower limits of agreement. Regression lines and equations are shown on the graphs with 95% confidence intervals for slope. Individual subject data are shown by different colored markers and trendlines. Panels A–C represents one group of 11 subjects, whereas panels D–F represent a different group of 11 subjects (A, Starhealth SH-A3; B, Jumper FPD-500A; C, Atlantean SB100 II; D, Contec CMS50DL; E, Beijing Choice C20; F, Beijing Choice MD300C23).

Figure 2.:
Bias (pulse oximeter oxygen saturation [Spo ₂] − arterial blood oxygen saturation [Sao ₂]) is plotted against Sao ₂ measured by either an ABL90 (oximeters 1–3) or OSM3 (oximeters 4–6) hemoximeter (Radiometer). Each oximeter is indicated by separate markers.

Figure 1 illustrates bias for individual data points obtained for individual subjects from each oximeter tested. Figure 2 displays bias for each oximeter tested. All oximeters tested demonstrated worsening performance with hypoxia, with mean bias increasing at lower oxygen saturations. This resulted in wide limits of agreement. Data are summarized in Table 2.

DISCUSSION

Pulse oximetry theory predicts that the ratio of pulsatile to total transmitted red light divided by the same ratio for infrared light should depend only on arterial saturation. In practice, there are many factors that must be accounted for to ensure accurate calculation of oxygen saturation by pulse oximetry. Major known variability is caused by anemia, light scattering, venous and tissue pulsation by mechanical force from nearby arteries, pulsatile variations in tissue thickness in the light path other than in the arteries, nail polish, and skin pigment, among others.^² Therefore, the design of accurate pulse oximeters often requires empirically determined correction factors obtained by in vivo comparison of oximeter readings with arterial oxyhemoglobin saturation of volunteer subjects during hypoxemia. Many commonly used low-cost pulse oximeters do not undergo this in vivo testing, and thus, little is known about the accuracy of such devices.

In this study, we tested the accuracy of 6 low-cost pulse oximeters currently available for purchase from popular consumer retailers. The majority (4/6) of the oximeters tested did not meet US FDA standards for accuracy. Unexpectedly, 2 of the 6 oximeters did meet accuracy standards defined by the FDA and ISO, an A_rms <3%. Oximeters 4 and 5 sell for approximately $25 USD and performed with similar accuracy for measuring hypoxia in healthy volunteers when compared with popular, clinically approved devices that cost >$3000 USD. Although more expensive units often contain additional features (such as hemoglobin concentration, methemoglobin, and carboxyhemoglobin measurements or enhanced detection during motion or low perfusion states, and faster processing speed), our findings demonstrate that simple, accurate pulse oximeters can likely be developed at significantly lower cost than many units currently available on the market.

It is important to note that although some of the inexpensive units tested here demonstrated accurate saturation readings during hypoxia, none meet current World Health Organization or World Federation of Societies of Anaesthesiologists standards for use in clinical practice.

All instruments tested showed increasingly larger bias (both positive and negative) in oximeter readings at low Sao₂ in subjects. It may be that this is a property common to a wide variety of inexpensive pulse oximeters, given that the basic technology is similar. More expensive devices that are FDA-cleared, such as those manufactured by Nellcor, Nonin, Masimo, and others, show a smaller degree of increasing bias during lower Sao₂ conditions.^²

The magnitude of the oximeter error in all 6 oximeters tested here was relatively small at saturations >90% and probably of no clinical significance. However, there were large and variable errors in performance of pulse oximeters at lower saturations. Clinical use of these devices is therefore of obvious concern. In one subject for oximeters 3 and 6, the device appeared to have locked onto a saturation value and therefore did not detect increasing hypoxemia.

It is not clear to what extent low-cost (<$50 USD) pulse oximeters are used for medical diagnoses worldwide. It has been our experience through extensive time spent in practice environments in LMICs that the use of nontested/non-FDA or Conformité Européene-regulated oximeters is extensive. Furthermore, it has been our experience that the device manufacturers tested here represent a large proportion, if not the majority, of low-cost oximeters currently in use.

To improve anesthesia safety worldwide, conservative estimates indicate that 100,000 additional pulse oximeters are needed, and when accounting for additional care settings such as recovery rooms, >1 million units may be required.^¹⁷

Our finding that 2 of the low-cost oximeters studied here performed with accuracy to meet FDA clearance standards is particularly relevant for clinical environments in LMICs where access may be limited because of cost. It is in LMICs that most of the world’s surgical disease burden exists with the fewest per capita health care resources (such as health care providers and pulse oximeters) to address the problem.^¹^,^¹⁸ Furthermore, the purchase of pulse oximeters that cost several thousand—or even several hundred—dollars may not be feasible in these settings, despite research suggesting cost-effectiveness on par with other public health interventions.^¹⁹

At present, a few initiatives are underway to increase access to low-cost, accurate pulse oximeters in low-income countries. The LifeBox Project aims to equip the world’s operating theaters with pulse oximeters specifically designed and priced for this purpose. We have previously shown that the LifeBox does meet US FDA accuracy standards.^²⁰ LifeBox pulse oximeters are donated or purchased by users. The current cost for a LifeBox oximeter is $250, which includes an adult probe, a pediatric probe, a rechargeable battery, delivery by courier anywhere in the world, training materials, and a 2-year warranty (1-year warranty for the probes). Another example of a project aimed at increasing global access to pulse oximeters is the LionsGate Medical Kenek Pulse Oximeter (Lionsgate Technologies, Vancouver, British Columbia, Canada).^²¹ This device costs approximately $50 to $80, although it also requires the purchase of a compatible mobile phone or tablet (iOS) device.

Although the units tested in this study were sold for <$50 USD, this price does not account for additional costs, such as delivery of the unit and replacement batteries, which can add considerably to the overall lifetime cost of these devices.

Nonetheless, based on the data we present here and careful evaluation of the components and features of numerous currently available FDA-cleared pulse oximeters (including screen size, alarms, rechargeable battery, and durable cases), a lower price point for many pulse oximeter units can likely be achieved to target LMIC markets without compromising accuracy or utility.

The present study has several limitations. Study subjects were recruited on a volunteer basis, with limited attempts to recruit subjects of different skin tones in equal proportions. We did not evaluate the impact of skin pigment on saturation although pigment is known to cause inaccurate readings in pulse oximeters.^²²^,^²³ For individuals with darkly pigmented skin, bias of up to 8% has been reported at lower saturations, in FDA-cleared pulse oximeters tested approximately 10 years ago.^²² Although subjects of all skin color are represented in this study, the majority were Asians and Caucasians, and thus applicability to non-Caucasian populations may be limited.

An additional limitation is the use of 2 groups of subjects in 2 different laboratories. Although identical protocols were used, and one of the authors of this article was present during testing at both sites, variability in procedures or differences in study subjects cannot fully be excluded. We studied only 6 oximeters, and thus our results may not be applicable to all non–FDA-cleared oximeters available.

One of the most significant limitations of this study is the ideal conditions under which it was conducted. All volunteers were healthy, with good perfusion values (some with hands wrapped with warming pads) and remained essentially motionless throughout the protocols. Poor perfusion and patient movement are known to significantly affect pulse oximeter accuracy. In clinical scenarios, such as critical illness, factors such as motion artifact and vasoconstriction are likely to play a significant role and further degrade the accuracy of low-cost pulse oximeters, perhaps to a greater degree than more expensive units.^¹¹^,^²⁴^,^²⁵ Because these relevant clinical factors, among others, were not evaluated, this study should not serve as an endorsement for the clinical use of any of the pulse oximeters tested.

CONCLUSIONS

Inexpensive pulse oximeters have become increasingly available with little regulatory oversight. Most of the commonly sold units tested here demonstrated highly inaccurate saturation readings during hypoxia. A small proportion of the low-cost pulse oximeters tested here performed similarly to more expensive, FDA-cleared units when measuring hypoxia in healthy subjects, although none met World Federation of Societies of Anaesthesiologists standards for clinical use. Caution must be exercised when considering the use of noncertified pulse oximeters to diagnose or treat hypoxia. These findings support the notion that further development of accurate, low-cost oximeters is feasible and should continue to be pursued to improve access to safe clinical care, especially in low-income countries.