In 2016, Roca
et al. proposed an index combining respiratory rate and oxygenation (ROX) to predict the success of oxygen therapy by high-flow nasal cannula in pneumonia patients with acute hypoxemic respiratory failure [ ]. The ROX index is calculated by dividing the patient’s oxygen saturation, by the inspired oxygen concentration (FiO 1 2), and then by the respiratory rate. Only a handful of studies have evaluated the ability of the index to predict adverse outcomes in acute diseases, such as coronavirus disease 2019 (COVID-19) and sepsis [ ]. However, one small preliminary study from a Canadian teaching hospital and a low-resource hospital in Africa looking at three markedly different unselected patient populations found the ROX index to be a powerful discriminator of early mortality [ 2–4 ]. 5
The National Early Warning Score (NEWS) predicts deterioration from sepsis better than systemic inflammatory response syndrome or quick Sequential Organ Failure assessment [
] and is the most widely validated risk score for death within 24 h [ 6 7 , ]. However, the ROX index has been reported to predict the deterioration of COVID-19 patients earlier than NEWS when measured by composite outcomes [ 8 ]. The aim of this study was to compare the ability of the ROX index with NEWS to predict mortality within 24 h of arrival to the hospital of a large population of patients attending several Dutch EDs. 4 Methods
Study design and setting
This retrospective observational multicentre study analyzed data in the Netherlands Emergency Department Evaluation Database (NEED), a registry containing clinical data from all ED visits from the participating hospitals used to benchmark the quality of their care (
). Only the vital signs and mental status measured at the beginning of ED presentation, before ED treatment, are recorded in the NEED, and only one set of vital signs is recorded per patient. Further details on the NEED have been previously published [ www.stichting-need.nl ]. 9 Selection of participants
All consecutive ED patients ≥18 years attending four participating hospitals EDs were included. Inclusion periods varied from 1 January to 31 December 2019, 1 January 2019 to 12 January 2020, and for two hospitals from 1 January 2017 to 31 March 2022. Patients were excluded from the NEED if none or only one vital sign (SBP, heart rate, peripheral oxygen saturation, respiratory rate, or temperature) were registered. In EDs in the Netherlands and elsewhere, vital signs are not registered in all patients, and respiratory rate and temperature are the most frequently omitted. For example, vital signs are often not registered in patients with an ankle distortion or a single fracture, because they are at low risk of adverse events and are often discharged. Respiratory rate is often only registered if patients are considered critically ill by the nurse. In the excluded patients, vital signs were considered missing not at random which prevented the possibility of imputation.
Data collection, default assumptions, and outcomes
Details of how data were collected have been previously described [
]. Briefly, demographic characteristics, type of arrival to the ED, triage category, vital signs, and laboratory tests, as well as disposition and mortality were collected. The primary outcome was death within 24 h of ED arrival; for the primary analysis, it was assumed that no patients discharged from the ED died within 24 h. 9 Statistical analyses
Continuous patient characteristics were described as mean (SD) or median (interquartile range). Categorical data were presented as numbers with proportions [
N, (%)]. Prior to analyses, missing data were substituted by multiple imputations by the chained equation procedure, after which imputation was feasible based on missing value analyses. To enhance the multiple imputation procedure, we also used urea, leukocytes, and intensive care unit admission as predictors in the imputation procedure. When ≥4 vital signs were missing, observations were excluded. These were considered non-randomly missing and despite a comprehensive set of variables in the multiple imputation procedure, including outcomes, we did not consider imputation to be feasible.
We obtained 20 estimates of the missing vital signs for each patient with five iterations each. Regression coefficients and intercepts across imputed sets were averaged to incorporate the variance introduced by the imputation procedure [
]. For each imputation set, we calculated the NEWS and the ROX index. The ROX index, rounded to the nearest integer, was calculated by dividing the patient’s oxygen saturation, by the FiO 10–12 2, and then by the respiratory rate (e.g. 95%/0.21/16 = 28).
Predictive performance was compared between ROX and NEWS using the area under the receiving operating characteristic curves (AUC) with 95% confidence intervals (CIs) and calibration plots. The DeLong test was used to compare AUCs. To evaluate clinical usefulness, Decision Curve Analyses (DCA) were produced, summarizing standardized Net Benefits at a range of thresholds [
]. DCA plots the net benefit of each ROX or NEWS value against the risk encountered at that value. The net benefit for each value of ROX or NEWS tested = (true positives/all) – (false-positives/all × exchange rate), at that value. The exchange rate is determined by the risk at the chosen value; if 10%, the exchange rate = 0.1/(1–0.1), implying that the harm of not being identified is nine times larger than unnecessary identification as high risk. The net benefit was standardized to the prevalence of 24 h-mortality at each value. DCA identifies threshold values of net benefit at different mortality risks that justify the measurement of NEWS or ROX in all patients [ 13 ]. 14
Internal validation was performed by split sample analysis based on a pre-COVID period (till 1 February 2020) and COVID period (all patients after 1 February 2020). Additionally, because we assumed that discharged patients did not experience the outcome, a sensitivity analysis was performed in which a randomly selected 0.04% of the discharged patients were considered deceased within 24-h. This percentage was based on reports in the literature [
]. All analyses were performed in R statistical software [packages dplyr (v1.0.7;2021), rms (v6.2;2021), mice (v45;2011)]. 15 Ethical approval
The study was approved by the medical ethics committee of the Máxima Medical Centre (nr. 21.007). The study conforms to the principles outlined in the Declaration of Helsinki. The study conforms with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis statement [
]. 16 Results
During the study period out of 315 950 patients presented to the four participating EDs 70 732 (22.4%) were excluded because ≥4 vital sign values were missing. These excluded patients were younger, and significantly less likely to be admitted to the hospital or to die than the final population of 245 218 patients included in the study (
Fig. 1). Most included patients (69.1%) had a complete set of the four classic vital signs and oxygen saturation, and only 7.1% had more than one missing value. Fig. 1:
Included and excluded patients according to missing values, hospital and ICU admission, and 24 h and in-hospital mortality. ICU, intensive care unit.
The 1791 patients (0.7%) who died within 24 h were older, more likely to be men, and less likely to be alert (
Table 1). High NEWS and low ROX index values were associated with 24-h mortality ( Fig. 2a and b). The AUC for NEWS was significantly higher than for the ROX index (0.92; 95% CI, 0.91–0.92 versus 0.87; 95% CI, 0.86–0.88; P < 0.01).
Table 1 -
Age, sex, mental status, and vital signs of patients who died within 24 h, those who survived for 24 h and the total study population
75 788 (30.9%)
123 966 (50.6%)
122 208 (50.5%)
61.3 SD 19.4
74.1 SD 13.8
61.2 SD 19.4
SBP (mm Hg)
143.0 SD 21.7
133.0 SD 40.5
143.0 SD 26.5
Heart rate (beats per minute)
86.0 SD 21.2
95.5 SD 27.2
85.9 SD 21.1
Respiratory rate (breaths per minute)
17.9 SD 5.1
23.3 SD 9.4
17.8 SD 5.0
48 255 (19.7%)
Oxygen saturation (%)
96.7 SD 3.3
93.1 SD 7.8
96.7 SD 3.2
36.9 SD 0.9
36.4 SD 1.5
36.9 SD 0.9
30 360 (12.4%)
Fraction of inspired oxygen (FiO
2) 0.24 SD 0.10
0.38 SD 0.23
0.24 SD 0.10
121 561 (49.6%)
74 926 (30.6%)
74 202 (30.7.2%)
166 930 (68.1%)
Means, standard deviations (SD), medians and IQR are shown.
aPatients who needed to be aroused by voice or pain or were unresponsive were NOT alert – all other patients were considered ‘Alert’. Fig. 2:
(a) Observed 24-h mortality and patient numbers according to ROX index determined at emergency department presentation. (b) Observed 24-h mortality and patient numbers according to NEWS values determined at emergency department presentation. NEWS, National Early Warning Score; ROX, respiratory rate and oxygenation.
Approximately half of the study patients (133 019; 54.2%) presented before 2020 and the start of the COVID-19 pandemic. There was no difference between the AUC before and after the covid period for NEWS (0.92; 95% CI, 0.92–0.93 versus 0.91, 95% CI, 0.91–0.93;
P = 0.12), and a small difference in AUC for the ROX index (0.86; 95% CI, 0.85–0.88 versus 0.88; 95% CI, 0.87–0.90; P = 0.02).
Calibration of the ROX index was accurate up to an observed mortality of 5%, after which mortality was underestimated. In contrast, up to 24-h mortality of 3% mortality NEWS slightly under-estimates mortality, and above this level mortality was over-estimated (
Fig. 3). Calibration for ROX improved at the start of the COVID-19 period compared to before, whereas NEWS remained unchanged (see Supplemental Data, Supplemental Digital Content 1, ). The DCA graph shows that the ROX index has a higher net benefit than NEWS for predicted 24-h mortality below 3% ( https://links.lww.com/EJEM/A354 Fig. 4). Fig. 3:
Calibration between predicted and observed 24-h mortality for ROX and NEWS. After discharge from the emergency department, 24-h mortality was assumed to be 0%. NEWS, National Early Warning Score.
Decision Curve Analysis showing the standardized Net Benefit of ROX and NEWS at different predicted risks of 24-h mortality. After discharge from the emergency department, 24-h mortality was assumed to be 0%. NEWS, National Early Warning Score; ROX, respiratory rate and oxygenation.
Sensitivity analysis showed that if the mortality within 24 h of discharged patients was 0.04% the AUC for ROX and NEWS were slightly reduced to 0.85 (95% CI, 0.84–0.86) and 0.89 (95% CI, 0.89–0.90), respectively. However, the predicted and observed 24-h mortality of ROX values now were the same up to a mortality rate of 6%, and the calibration of NEWS was unchanged (Supplemental Data, Supplemental Digital Content 1,
); the ROX index continued to have a higher net benefit than NEWS for predicted 24-h mortality below 3%. https://links.lww.com/EJEM/A354 Discussion
For 24 h mortality predicted to be below 3% (i.e. at ROX index ≥ 14), ROX has a higher net benefit than NEWS, while above a predicted 24 h mortality of 3% the net benefit of NEWS is higher. No difference was observed in the discriminative performance of either NEWS or the ROX index before and after the start of the COVID-19 pandemic. However, the calibration of the ROX index improved during the COVID-19 period.
Strengths and limitations
Although it has the inherent limitation of retrospection, this is a large multicentre study of over a quarter of a million patients. Although a preliminary complete case analysis without imputation produced similar results to our final analysis, the judicious use of imputation reduced selection and other information bias [
]. For example, the FiO 7 2 may not have been routinely recorded for many patients on room air, and excluding these patients would bias results toward patients who were more likely to be severely ill. As our primary outcome of interest was 24-h mortality, we did not exclude patients who were frequent ED attendees from the study. Unfortunately, ROX depends on respiratory rate measurements, which are often measured inaccurately [ ] and we were not able to determine how the respiratory rate was measured in participating centers or verify the accuracy of these measurements. There was no follow-up of patients after ED discharge and our default assumption was that no discharged patients died within 24 h. However, sensitivity analysis showed that the results would not have been significantly changed even in the unlikely event of 0.04% of patients dying within 24 h of discharge [ 17–19 15 , 20 , ]. 21 Interpretation
NEWS was specifically designed to identify patients at risk of imminent death and the risk of anyone dying within 24 h is extremely low [
]. Although our findings can only be applied with certainty to the population studied, there are very few reports in the literature of 24 h mortality above 1% in a general hospital and ED patients. In over 5 million observations reported in a systematic review of the literature, the overall mortality within 24 h was 0.6% and the AUC for 24-h mortality of NEWS was 0.897 [ 8 ]; only three studies [ 8 ] reported a 24-h mortality >1.0%, which ranged from 2.9% [ 22–24 ] to 10.2% [ 23 ]. Therefore, it is probable that our findings can be applied to most ED patients as very few are likely to have a 24-h mortality rate >3%. However, for severely ill and older patients who will inevitably require the measurement of a full set of vital signs NEWS may be more appropriate. 22 Clinical implications
The higher observed discrimination of NEWS for 24-h mortality suggests that compared to the ROX index it is the superior test. However, the AUC is not useful for low-incidence outcomes such as 24-h mortality [
]. Additionally, discrimination does not indicate how close predicted mortality is to observed mortality. In this study, unlike NEWS, the 24-h mortality predicted by ROX was identical to the observed mortality up to a predicted mortality of 5.0%. Despite the lower AUC of ROX, DCA shows that the net benefit of ROX exceeds that of NEWS in patient populations with predicted 24-h mortality of less than 3%. 25
Respiratory rate may often not be measured or can be estimated, or even fabricated, especially if the patient is not thought to be ill [
]. This may hamper the implementation and adoption of ROX. However, devices are now available to measure respiratory rate continuously [ 17 ], and they can also be quickly and reliably obtained at the bedside using the free 26 RRate smartphone application [ ]. NEWS also requires measurement of respiratory rate and a full set of vital signs, which can take up more than 5 min to complete [ 27 28 , ]. Therefore, ROX may be very useful in clinical practice as the initial method of risk assessment in ED patients. The index can be measured more rapidly and repeatedly at the bedside than NEWS and, therefore, could aid monitoring and decision-making in the acute setting for patients needing time-sensitive treatments, such as sepsis or trauma. However, if the predicted mortality is >3.0% (i.e. ROX falls below <14) additional vital signs should be measured to calculate NEWS, and the patient assessed further. 29
It remains to be seen if it is practical to adopt the ROX index as a routine method of ED risk assessment. Wider clinical use will require a cultural change, which recognizes the importance of the accurate measurement and recording of respiratory rate. Furthermore, early warning scores that are better calibrated over a wider risk range than either ROX or NEWS are needed to improve the risk assessment of all patients regardless of their clinical setting [
]. 30 Conclusion
In this patient population, the prediction of 24-h mortality by the ROX index is more accurate than NEWS for most patients likely to be encountered in the ED. ROX may be used as a first screening tool in the ED.
The authors of this unfunded study would like to acknowledge the assistance and cooperation of Leiden University Medical Centre and the board of the Netherlands Emergency department Evaluation Database.
B.G.J.C., J.K., and S.K.N. devised and designed the study, contributed to the analyses, and edited the manuscript. B.G.J.C. and Bd.G. collected the data. W.A.M.H.T. and H.L. are founders of the NEED, collected data, and helped edited the article. J.K. edited the article, takes full responsibility for the study, and acts as a guarantor. All authors have read and approved the article.
Conflicts of interest
J.K. is a major shareholder, director, and chief medical officer of Tapa Healthcare DAC. The other authors have no potential conflicts of interest. For the remaining authors, there are no conflicts of interest.
1. Roca O, Messika J, Caralt B, García-de-Acilu M, Sztrymf B, Ricard J-D, et al. Predicting success of high-flow nasal cannula in pneumonia patients with hypoxemic respiratory failure: the utility of the ROX index. J Crit Care 2016; 35:200–205.
2. Prakash J, Bhattacharya PK, Yadav AK, Kumar A, Tudu LC, Prasad K. ROX index as a good predictor of high flow nasal cannula failure in COVID-19 patients with acute hypoxemic respiratory failure: a systematic review and meta-analysis. J Crit Care 2021; 66:102–108.
3. Lee CU, Jo YH, Lee JH, Kim J, Park SM, Hwang JE, et al. The index of oxygenation to respiratory rate as a prognostic factor for mortality in Sepsis. Am J Emergency Med 2021; 45:426–432.
4. Prower E, Grant D, Bisquera A, Breen CP, Camporota L, Gavrilovski M, et al. The ROX index has greater predictive validity than NEWS2 for deterioration in Covid-19. EClin Med 2021; 35:1–9.
5. Kellett J, Sikakulya FK, Nickel CH. The prediction of early mortality by the ROX index of oxygenation and respiratory rate in diverse Canadian and Ugandan cohorts of unselected patient: a post-hoc retrospective analysis of 80,558 patient observations. Acute Med 2022; 21:68–73.
6. Zonneveld LEEC, van Wijk RJ, Olgers T J, Bouma HR, ter Maaten JC. Prognostic value of serial score measurements of the national early warning score, the quick sequential organ failure assessment and the systemic inflammatory response syndrome to predict clinical outcome in early sepsis. Eur J Emergency Med. 2022; 29;348–356.
7. Gerry S, Bonnici T, Birks J, Kirtley S, Virdee PS, Watkinson PJ, et al. Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology. BMJ Open 2017; 7:e019268.
8. Holland M, Kellett J. A systematic review of the discrimination and absolute mortality predicted by the national early warning scores according to different cut-off values and prediction windows. Eur J Int Med 2022; 98:15–26.
9. Candel BG, Duijzer R, Gaakeer MI, ter Avest E, Sir Ö, Lameijer H, et al. The association between vital signs and clinical outcomes in emergency department patients of different age categories. Emergency Med J. Published Online First: 11 January 2022. doi: 10.1136/emermed-2020-210628
10. Harrell FE. Regression modelling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. 2nd ed. Springer International Publishing 2015.
11. Azur MJ, Stuart EA, Frangakis C, Leaf PJ. Multiple imputation by chained equations: what is it and how does it work? Int J Methods Psychiatr Res 2011; 20:40–49.
12. Vergouwe Y, Royston P, Moons KG, Altman DG. Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol 2010; 63:205–214.
13. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen N, Obuchowski N, et al. Assessing the performance of prediction models: a framework for some traditional and novel measures. Epidemiology 2010; 21:128–138.
14. Vickers AJ, van Calster B, Steyerberg EW. A simple, step-by-step guide to interpreting decision curve analysis. Diagn Progn Res 2019; 3:18.
15. Baker M, Clancy M. Can mortality rates for patients who die within the emergency department, within 30 days of discharge from the emergency department, or within 30 days of admission from the emergency department be easily measured? Emerg Med J 2006; 23:601–603.
16. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. Ann Intern Med 2015; 162:55–63.
17. Loughlin PC, Sebat F, Kellett J. Respiratory rate: the forgotten vital sign—make it count!. Joint Commission J Qual Patient Saf 2018; 44:494–499.
18. Brabrand M, Hallas P, Folkestad L, Lautrup-Larsen CH, Brodersen JB. Measurement of respiratory rate by multiple raters in a clinical setting is unreliable: a cross-sectional simulation study. Crit Care 2018; 44:404–406.
19. Nielsen LG, Folkestad L, Brodersen JB, Brabrand M. Inter-observer agreement in measuring respiratory rate. PLoS One 2015; 10:e0129493.
20. Obermeyer Z, Cohn B, Wilson M, Jena AB, Cutler DM. Early death after discharge from emergency departments: analysis of national US insurance claims data. BMJ 2017; 356:j239.
21. Blom MC, Ashfaq A, Sant’Anna A, Anderson PD, Lingman M. Training machine learning models to predict 30-day mortality in patients discharged from the emergency department: a retrospective, population-based registry study. BMJ Open 2019; 9:e028015.
22. Haegdorens F, Monsieurs KG, De Meester K, Van Bogaert P. The optimal threshold for prompt clinical review: an external validation study of the national early warning score. J Clin Nurs 2020; 29:4594–4603.
23. Lee SB, Kim DH, Kim T, Kang C, Lee SH, Jeong JH, et al. Emergency department triage early warning score (TREWS) predicts in-hospital mortality in the emergency department. Am J Emerg Med 2020; 38:203–210.
24. Liu FY, Qin J, Wang RX, Fan X, Wang J, Sun C, et al. A prospective validation of national early warning score in emergency intensive care unit patients at Beijing. Hong Kong J Emergency Med 2015; 22:137–144.
25. Romero-Brufau S, Huddleston JM, Escobar GJ, Liebow M. ‘Why the C-statistic is not informative to evaluate early warning scores and what metrics to use’. Crit Care 2015; 19:285.
26. McCartan TA, Worrall AP, Ó Conluain R, Alaya F, Mulvey C, MacHale E, et al. The effectiveness of continuous respiratory rate monitoring in predicting hypoxic and pyrexic events: a retrospective cohort study. Physiol Meas 2021; 42:6.
27. Nakitende I, Namujwiga T, Dunsmuir D, Ansermino JM, Wasingya-Kasereka L, Kellett J. Respiratory rates observed over 15 seconds compared with rates measured using the RRate app. Practice-based evidence from an observational study of acutely ill adult medical patients during their hospital admission. Acute Med 2020; 19:15–20.
28. Bellomo R, Ackerman M, Bailey M, Beale R, Clancy G, Danesh V, et al. Vital Signs to Identify, Target, and Assess Level of Care Study (VITAL Care Study) Investigators. And the vital signs to identify, target, and assess level of care study (VITAL Care Study) investigators. Crit Care Med 2012; 40:2349–2361.
29. Dall’Ora C, Griffiths P, Hope J, Briggs J, Jeremy J, Gerry S, et al. How long do nursing staff take to measure and record patients’ vital signs observations in hospital? A time-and-motion study. Int J Nurs Stud 2021; 118:103921.
30. Nissen SK, Candel BGJ, Nickel CH, de Jonge E, Ryg J, Bogh SB, et al. The impact of age on predictive performance of national early warning score at arrival to emergency departments: development and external validation. Ann Emerg Med 2022; 79:354–363.