Secondary Logo

Journal Logo

Original Clinical Report

Stratifying Deterioration Risk by Acuity at Admission Offers Triage Insights for Coronavirus Disease 2019 Patients

Beals, Joseph IV PhD1; Barnes, Jaime J. DO2; Durand, Daniel J. MD3; Rimar, Joan M. DNSc4; Donohue, Thomas J. MD4; Hoq, S. Mahfuz MD5; Belk, Kathy W. BA1; Amin, Alpesh N. MD6; Rothman, Michael J. PhD1

Author Information
Critical Care Explorations: April 2021 - Volume 3 - Issue 4 - p e0400
doi: 10.1097/CCE.0000000000000400


Hospitalized severe acute respiratory syndrome coronavirus 2 disease (COVID-19) patients are susceptible to a dysregulated immune system response giving rise to significant deterioration and elevated rates of ICU utilization, mechanical ventilation, and in-hospital mortality (1,2). In regions with high COVID-19 case rates among the general population, hospitals have been overwhelmed by patient volumes (3–5). These surge conditions have resulted in fatigued, overextended clinical staff having to make rapid assessment of patient risk, a difficult task during the best of times, to allocate limited hospital resources, for example, mechanical ventilators and ICU beds. The challenges faced by caregivers under pandemic conditions cannot be overstated (6). Use of preexisting, well-validated acuity models to support triaging of COVID-19 patients has the potential to ease the burden on clinical staff by supporting them in making difficult care decisions in a more effective and objective fashion.

Among COVID-19 patients, reports indicate that age and comorbidity correlate with mortality risk (7,8). We postulate that both are in essence indirect and inadequate estimates of patient acuity. Age itself is not intrinsically a risk factor but rather a proxy for age-related diseases and infirmities. Comorbidity is a more explicit measure of diseases and infirmities; however, the presence or absence of comorbidities does not directly translate to acuity without a characterization of extent of impairment. Our work evaluates the relationship between impairment and subsequent in-hospital deterioration by explicitly measuring physiologic acuity at admission using the Rothman Index (RI), an extensively validated, commercially available patient condition score (PeraHealth, Charlotte, NC) which is in widespread use by hospitals and health systems across the United States. The RI is a finely graduated acuity score with a large dynamic range, from a maximum of 100 to a minimum of –91, enabling clear distinctions in risk to be made across the acuity spectrum, including in cases of mild impairment. This stands in contrast to other common acuity measures, for example, the National Early Warning Score (NEWS), the Modified Early Warning Score (MEWS), or the Confusion-Urea-Respiratory Rate-Blood Pressure-Age score (CURB-65), which have score ranges of 21, 15, and 6 points, respectively (9–11). Unlike models that are explicitly developed and trained around COVID-19, the RI is suitable for evaluating differences in acuity implications between coronavirus disease (COVID) and non-COVID populations. To calibrate the reader, decreasing RI scores correspond to increasing acuity: 100 implies no physiologic impairment; 65 is typical acuity for patients discharged to skilled nursing facilities (12); 40 is approximately the acuity of patients considered for transfer to an ICU; and 0 corresponds to a 50% likelihood of in-hospital mortality.

RI model features include vital signs, blood oxygen saturation, heart rhythm, eleven different body system nursing assessments, the Braden score, and seven laboratory values. The required data are routinely documented on all hospitalized patients and recorded in the electronic medical record (EMR), necessitating no additional patient assessment or clinical documentation. Software containing the RI algorithm interfaces with the EMR allowing real-time updates using emergency department and inpatient unit data. The model accommodates asynchronous data inputs and is calculated when sufficient inputs (no more than two missing vital signs or nursing assessments) have been received (12). The model development methodology has been published (12–16) and clinically validated as an indicator of patient acuity and early deterioration (17–20).

The RI is unique in its inclusion of the full range of body system nursing assessment data which categorize physical and behavioral evaluations for dozens of factors (e.g., patient stops eating, becomes confused, has trouble walking, develops edema, etc.). Deficits charted by nurses frequently serve as indicators of deterioration earlier than vital sign derangement (13) and allow the RI to perform significantly better as a predictor of adverse outcomes than acuity scores predominantly based on vital signs, such as MEWS (14). Recent work examining the impact of aspirin and anticoagulation on COVID-19 patients identified the utility of the RI in controlling for severity of illness among COVID-19 study populations (21).

Here, we evaluate the ability of the RI model at the time of hospital admission to stratify patients into high or low risk of subsequent severe deterioration with an emphasis on opportunities to support risk stratification in the COVID-19 population.



We analyzed data from LifeBridge Health System’s Sinai Hospital of Baltimore, in Baltimore, MD, and from three Yale New Haven Health System (YNHHS) hospitals in New Haven, Bridgeport, and Greenwich CT. These include urban and suburban community hospitals and an academic medical center.

Non–COVID-19 data were restricted to a timeframe predating changes to admission and care practices provoked by COVID-19, to reflect a representative sample of non-COVID patients. Data from Sinai Hospital of Baltimore included 2019 non-COVID-19 patients admitted and discharged between January 1, and March 15, 2020, and 355 COVID-19 patients admitted and discharged between March 21, and September 14, 2020. Data from YNHHS included 12,639 non–COVID-19 patients admitted and discharged between January 1, and March 15, 2020 and 3,144 COVID-19 patients admitted and discharged between April 1, and June 15, 2020. At all hospitals, COVID-19 patients were diagnosed based on laboratory tests performed either before or during the hospital admission and identified either by indicators in the patient chart or International Classification of Diseases, 10th Edition (ICD-10) diagnosis code U07.1.

The study population was limited to patients 18 years old or older admitted to a medical service (identified by Medicare Severity Diagnosis Related Groups) and who had RI scores during their stay. More than 97% of adult medical patients at each hospital had RI scores. This study was approved with a waiver of informed consent (number 072003) by the Bridgeport Hospital Institutional Review Board and received exempt status from the LifeBridge Health Institutional Review Board.

Population Characteristics

Mean and median age and length of stay (LOS), sex, rates of mortality, rates of discharge to hospice, ICU utilization, and mechanical ventilation, as well as Charlson Comorbidity Index (CCI) for all patients and the subset who expired, are reported for COVID-19 and non–COVID-19 patients. The proportion of COVID-19 and non–COVID-19 patients presenting with each of the individual CCI comorbidities as well as hypertension and obesity is also reported. All comorbidities were identified based on ICD-10 diagnosis codes from the current visit (22). Differences between COVID-19 and non–COVID-19 populations were evaluated for significance with t tests for equal or unequal variance.

RI at Admission as a Predictor of Risk

To evaluate differences in deterioration risk between COVID-19 and non–COVID-19 patients, we compared RI at the time of admission to an inpatient unit with in-hospital mortality rates for COVID-19 and non–COVID-19 populations. We validated the use of the RI with COVID-19 patients by using a single-variable logistic regression model to compare the first RI score in predicting two independently evaluated endpoints: whether the patient expired in the hospital or spent any time in the ICU. Discriminatory performance was assessed using the receiver operating characteristic area under the curve (AUC) together with evaluation of model calibration.

Determining Operating Thresholds for Risk Stratification

Precision (positive predictive value) and recall (sensitivity) values at different RI threshold values were evaluated, and two threshold values were selected for categorizing patients as either low or high risk for subsequent deterioration.

Opportunities to Augment Level of Care Decisions

In order to evaluate whether RI-based risk stratification could provide new insight to providers making care decisions, we analyzed level of care assignment at admission. For patients meeting the RI high risk criteria at admission, we evaluated whether patients were admitted directly to ICU, transferred to ICU following admission (and if so, how long after admission), or never admitted to an ICU, to determine potential opportunities for improved or more timely location assignment on the basis of admission risk.


Population Characteristics

Population characteristics comparing the general medical non–COVID-19 patient population with the COVID-19 population at each hospital are given in Table 1. Statistically significant differences between COVID-19 and non–COVID-19 groups are indicated.

TABLE 1. - Characteristics of Coronavirus Disease 2019 and Noncoronavirus Disease 2019 Populations
Hospital Sinai Hospital of Baltimore Yale New Haven Bridgeport Greenwich
Patient count, n 2,019 355 8,383 1,746 2,822 907 1,434 491
Age (yr), mean, median 57.6, 61 63.8, 65c 57.7, 60 65.4, 66c 58.2, 60 64.0, 65c 58.4, 58 62.9, 62c
Length of stay (d), mean, median 6.8, 5.0 10.3, 8.0c 5.8, 3.9 12.0, 8.6c 5.3, 3.9 10.4, 7.6c 4.2, 3.2 8.6, 6.0c
Male, % 38.7 46.2b 43.1 48c 39.0 49.7c 32.2 57.6c
Expired, % 2.5 16.9c 2.3 15.6c 3.0 16c 2.8 11c
Hospice discharges, % 2.4 5.1b 2.4 3.0 2.1 3.6b 2.8 3.9
ICU utilization during stay, % 12.0 23.7c 11.4 30.5c 17.9 38.8c 10.2 26.7c
Mechanical ventilation, % 5.9 16.6c 3.9 12.7c 3.6 16.9c 2.0 14.7c
CCI all patients, mean, median 2.5, 2 2.5, 2 2.6, 2 2.6, 2 2.4, 2 2.3, 2 1.6, 1 1.5, 1
CCI expired patients, mean, median 5.3, 5 3.7, 3a 5.2, 5 3.8, 3c 4.8, 4 3.5, 3c 4.1, 4 2.7, 2b
Comorbidity % of Patients With Comorbidity
 AIDS/HIV 2.1 2.3 0.4 0.6 0.9 0.4c 0.2 0.4
 Cancer 6.9 3.7a 12.2 4c 8.3 3.2c 6.6 2.4c
 Cerebrovascular disease 13.8 8.7b 7.6 6.8 6.0 4.7 4.8 3.1
 Chronic pulmonary disease 27.2 23.7 30.5 27.2b 31.5 26b 19.6 17.1
 Congestive heart failure 22.9 20.3 23.1 24.1 25.2 21.8a 17.0 10.4c
 Dementia 6.8 19.2c 6.7 21.3c 11.1 22.5c 9.4 12.4
 Diabetes with complications 15.2 19.7a 14.8 20.8c 15.6 19.4b 7.3 9.8
 Diabetes without complications 22.1 33.5c 19.0 33.9c 19.5 29.7c 11.2 24.4c
 Hypertension 35.1 44.5c 29.6 39.6c 28.9 36.4c 24.8 36.3c
 Metastatic carcinoma 3.0 1.1a 5.7 1.5c 3.5 1c 2.7 0.8a
 Mild liver disease 6.4 4.5 7.3 5.8a 6.3 3.2c 4.5 1.8b
 Moderate or severe liver disease 1.9 2.0 2.7 1.5b 2.0 0.9a 1.4 0.6
 Myocardial infarction 12.1 10.1 9.4 9.8 9.8 7.9 6.7 6.9
 Obesity 16.4 24.8c 14.8 29.8c 17.6 25.5c 10.2 16.3c
 Paraplegia and hemiplegia 4.6 2.5 2.3 2.2 2.1 1.8 1.9 1.4
 Peptic ulcer disease 1.0 2.0 1.2 0.8 1.5 0.6a 1.1 1.2
 Peripheral vascular disease 10.3 6.5a 8.2 5.7c 7.9 6.3 4.0 3.7
 Renal disease 19.1 21.1 19.2 24.7c 19.1 23.8a 13.6 13.0
 Rheumatic disease 2.9 3.4 3.9 3.7 3.3 2.5 3.7 2.2
CCI = Charlson Comorbidity Index, COVID = coronavirus disease.
ap < 0.05.
bp < 0.01.
cp < 0.001.

In general, the COVID-19 population is older, more likely to be male, and has longer lengths of stay. They are much more likely to require admission to the ICU, use of a ventilator, and expire in the hospital. However, mean CCI of patients who expire is lower (less acute) for COVID-19 than for the non–COVID-19 population. Preponderant comorbidities include chronic obstructive pulmonary disease, congestive heart failure, diabetes, hypertension, and renal disease, in agreement with other reports (7,8,23), whereas in terms of relative occurrence of comorbidities, dementia, diabetes, obesity, and hypertension are proportionally overrepresented in the COVID-19 population at all four sites.

RI at Admission as a Predictor of Risk

Evaluation of in-hospital mortality as a function of acuity as measured by the RI at admission reveals that COVID-19 patients have a significantly higher mortality risk than non–COVID-19 patients for a given acuity, as shown in Figure 1. Data volumes at Yale New Haven Hospital supported further separating out respiratory medical patients (diagnoses of pneumonia, influenza, acute upper respiratory infection, and acute respiratory failure), and although their mortality risk is higher than that of nonrespiratory, non–COVID-19 patients, it remains far lower than for COVID-19 patients.

Figure 1.
Figure 1.:
Rothman Index (RI) at admission versus mortality rate for coronavirus disease 2019 (COVID-19) (solid red) and non–COVID-19 (dashed blue) populations (error bars are ± 2 sem) for three hospitals. For the Yale New Haven graph, the COVID-19 population is solid red, non–COVID-19 respiratory is dashed blue, and non–COVID-19 nonrespiratory is dotted green.

The median time between assignment to an inpatient location and generation of the first RI score for patients at all four hospitals ranged from 1 to 1.3 hours. Data are bucketed by RI decile, and the mean of each decile plotted; scores from 20 to 40 and below 20 are grouped to ensure reasonable sample sizes (sample count details are shown in Supplemental Fig. S-1,

To validate the performance of the initial RI score as a predictor of subsequent deterioration for both COVID-19 and non–COVID-19 patients, the AUC for discrimination of patients with and without each of two deterioration-related events, in-hospital mortality and need for ICU level care during hospitalization, is calculated (Table 2).

TABLE 2. - Area Under the Curve (95% CI) for Rothman Index Discriminatory Performance for Inpatient Mortality, ICU Utilization
Outcome Sinai Hospital of Baltimore Yale New Haven Bridgeport Greenwich
Mortality 0.91 (0.87–0.95) 0.82 (0.76–0.82) 0.91 (0.89–0.93) 0.81 (0.78–0.83) 0.92 (0.89–0.95) 0.82 (0.78–0.85) 0.90 (0.87–0.94) 0.84 (0.78–0.89)
ICU utilization 0.86 (0.83–0.89) 0.80 (0.75–0.85) 0.83 (0.82–0.85) 0.69 (0.66–0.72) 0.74 (0.72–0.76) 0.67 (0.63–0.70) 0.75 (0.72–0.79) 0.62 (0.57–0.68)
COVID-19 = coronavirus disease 2019.

For both COVID-19 and non–COVID-19 patients, performance at all hospitals using the initial RI at admission ranges from good to excellent for discriminating in-hospital mortality (AUC of 0.81–0.84 for COVID-19 and 0.90–0.92 for non–COVID-19) and fair to good for predicting need for ICU level of care (AUC of 0.62–0.80 for COVID-19 and 0.74–0.86 for non–COVID-19). The RI model is found to be well calibrated using the Hosmer-Lemeshow goodness of fit test, with p values and calibration curves for COVID-19 and non–COVID-19 populations shown in Supplemental Figure S-2 ( This performance is particularly significant in light of the fact that in many instances, the target outcome may have occurred a considerable time (e.g., a number of days) following patient admission.

Operating Thresholds for Risk Stratification

To determine high- and low-risk RI-thresholds for the COVID-19 and non–COVID-19 populations, precision and recall values of different RI threshold values are evaluated as shown in Figure 2. To orient the reader, the right-hand most points indicate a threshold of initial RI less than 100, which encompasses all admitted patients, hence providing 100% sensitivity to mortality. Each point moving from right-to-left represents a 10-point reduction in RI threshold value; thresholds capturing 2% or less of the population are omitted. Points with black markers reflect thresholds of RI less than 50 (COVID-19) and RI less than 40 (non–COVID-19). Patients with lower RI scores, and therefore higher acuity at admission, have a higher probability of expiring in the hospital leading to increasing precision at the expense of decreasing recall.

Figure 2.
Figure 2.:
Precision (i.e., PPV) and recall (i.e., sensitivity) for initial Rothman Index thresholds and inpatient mortality for (A) coronavirus disease 2019 (COVID-19) and (B) non–COVID-19 patients. PPV = positive predictive value, Hosp. = hospital.

Operating point thresholds for performance characterization are selected to be initial RI less than 50 to identify high-risk patients and initial RI greater than 70 to identify low-risk patients within the COVID-19 population. For the non–COVID-19 patients, in light of the lower associated risk for a given acuity, a lesser high-risk threshold of initial RI less than 40 is chosen, and the same low-risk threshold of RI greater than 70 is applied. Operating points are a compromise between sensitivity and positive predictive value. Different operating points could be chosen according to a hospital’s patient placement requirements and available resources.

Table 3 details the characteristics of COVID-19 and non–COVID-19 populations flagged on the basis of being below or above the high- and low-risk thresholds, respectively. Fewer than a third of COVID-19 patients meet the high-risk RI criteria; they have a 34–45% mortality rate. Approximately, half the COVID-19 population meet the low risk RI criteria; they have a mortality rate of only 2–5%. Large differences in LOS and rates of hospice discharge, ICU utilization, and mechanical ventilation between the high- and low-risk groups are also evident. Similarly, sharp contrasts in adverse event rates are also seen between the high- and low-risk groups in the non–COVID-19 populations.

TABLE 3. - Characteristics of Patients in High- and Low-Risk Categories
Risk Group Characteristics Sinai Hospital of Baltimore Yale New Haven Bridgeport Greenwich
High Risk Patients in population n (%) 99 (4.9) 92 (25.9) 729 (8.7) 590 (33.8) 270 (9.6) 290 (32) 99 (6.9) 92 (18.7)
Mortality rate, % 24.5 44.6 17.2 34.6 23.5 33.8 20.2 34.8
Hospice rate, % 3.1 3.3 9.5 5.4 8.5 8.3 14.1 9.8
ICU utilization rate, % 76.5 55.4 53.4 47.1 47.1 51.4 34.3 35.9
Mechanical ventilation rate, % 55.1 40.2 25.7 23.9 21.7 24.1 11.1 23.9
Direct admit to ICU, % 65.3 34.8 43.8 37.6 31.3 36.2 32.3 20.7
 Direct admit to ICU—expired, % 31.2 68.8 26.0 40.2 27.1 34.3 12.5 26.3
Transferred to ICU, % 11.2 20.7 9.6 9.7 15.8 15.2 2.0 15.2
 Transferred to ICU—expired, % 18.2 57.9 18.6 47.4 16.3 31.8 50.0 35.7
No time in ICU, % 23.5 44.6 46.6 52.9 52.9 48.6 65.7 64.1
 No time in ICU—expired, % 8.7 19.5 8.6 28.2 23.6 34.0 23.1 37.3
Days from admit to ICU transfer, mean, median 1.6, 1.1 3.7, 2.2 1.5, 0.7 5.1, 3.2 2.5, 1.0 3.6, 1.2 0.9, 0.9 4.1, 3.4
LOS (d), mean, median 8.8, 6.5 12.7, 10.1 8.9, 6.3 13.8, 10.2 7.7, 6.1 11.6, 9.3 6.7, 5.0 10.0, 6.8
LOS—expired, mean, median 4.0, 2.0 9.4, 6.9 6.6, 4.4 9.8, 6.5 5.1, 3.2 8.2, 4.6 4.7, 3.4 5.8, 5.2
LOS—hospice, mean, median 3.1, 2.8 22.7, 20.0 7.1, 5.0 9.0, 7.2 7.5, 6.7 6.5, 5.1 2.8, 2.3 9.0, 9.1
Low Risk Patients in population n (%) 1,490 (73.8) 181 (51) 4,954 (59.1) 665 (38.1) 1,662 (58.9) 387 (42.7) 978 (68.2) 284 (57.8)
Mortality rate, % 0.4 5.0 0.2 3.3 0.2 2.3 0.3 2.9
Hospice rate, % 4.6 1.1 0.5 0.6 0.4 0.0 0.7 1.4
ICU utilization rate, % 4.0 8.8 3.2 18.2 9.6 23.0 6.0 20.8
Mechanical ventilation rate, % 0.9 6.6 0.5 4.8 0.4 7.0 0.8 9.9
Direct admit to ICU, % 1.8 2.2 1.7 3.9 7.0 9.3 3.2 6.3
 Direct admit to ICU—expired, % 3.7 50.0 0.0 15.4 0.0 8.3 3.2 5.6
Transferred to ICU, % 2.1 6.6 1.5 14.3 2.5 13.7 2.9 14.4
 Transferred to ICU—expired, % 6.3 50.0 9.3 15.8 7.1 11.3 7.1 12.2
No time in ICU, % 96.0 91.1 96.8 81.8 90.4 77.0 94.0 79.2
 No time in ICU—expired, % 0.2 0.6 0.0 0.6 0.1 0.0 0.0 0.9
Days from admit to ICU transfer, mean, median 3.2, 1.6 3.8, 3.4 2.9, 1.7 3.1, 2.2 1.5, 0.9 3.1, 2.2 1.7, 0.7 5.7, 3.5
LOS (d), mean, median 6.1, 4.4 8.1, 6.3 4.8, 3.1 9.1, 6.5 4.3, 3.3 8.0, 5.6 3.4, 2.9 7.4, 5.0
LOS—expired, mean, median 8.7, 6.3 11.2, 9.2 12.6, 9.7 17.4, 15.1 12.2, 13.1 20.3, 17.2 11.6, 10.1 15.9, 14.6
LOS—hospice, mean, median 9.0, 8.9 26.0, 26.0 10.5, 8.0 12.6, 4.4 5.8, 3.9 0.0, 0.0 2.5, 2.7 11.2, 10.7
COVID = coronavirus disease, LOS = length of stay.

Opportunities to Augment Level of Care Decisions

To determine if COVID-19 patients deemed high risk based on admission RI were already recognized and treated as such by providers, we use patient location (i.e., ICU vs non-ICU) as a proxy for provider concern. For three of the four hospitals shown in Table 3 with nonnegligible numbers of patients transferred to ICU during hospitalization (i.e., not directly admitted to ICU), the median time from admission to transfer for COVID-19 patients is two to three times longer than for non–COVID-19 medical patients (1.2–2.3 d and 0.7–1.1 d, respectively). Among patients transferred to ICU, the mortality rate of COVID-19 patients is two to three times higher than for non–COVID-19 patients (32–58% and 16–19%, respectively).


A direct measure of acuity is an effective predictor of physiologic susceptibility to decline as shown in the AUC analysis in Table 2. Importantly, we also see that for a given level of acuity at admission, as measured by the RI (Fig. 1), mortality rates for COVID-19 patients are higher than for the non–COVID-19 medical population; this difference holds even when comparing COVID-19 with a subpopulation of non–COVID-19 medical patients with respiratory-related diagnoses. The higher COVID-19 risk can be understood in light of reports that some COVID-19 patients suffer a significant dysregulated immune system response, often several days into their hospital admission (2,24). Furthermore, as seen in Table 1, we find that the comorbidity burden at admission for those patients who expired in the hospital, as measured by the CCI, was less for COVID-19 patients than for non–COVID-19 patients. These findings imply that a clinician’s sense of how to treat a COVID-19 patient may be led astray by how he or she would treat a non–COVID-19 patient.

Although physicians incorporate many factors into their determination of patient risk, there is a subset of patients for whom risk is not clear-cut. In such instances, reference to an objective acuity-based threshold may help in guiding care decisions. Even in instances when the physician may be uncertain and the acuity-based risk is neither low nor high, but somewhere in the middle (i.e., moderate), an objective score may still be of use, according to circumstances. For example, when beds and resources are scarce (as in a pandemic surge scenario), patients with moderate acuity risk may be treated as lesser risk, and when bed capacity and resources are readily available, a score connoting moderate risk may guide toward closer monitoring or more conservative care decisions out of an abundance of caution.

Given the potential opportunity for acuity model–based risk stratification to provide insight to clinicians, it is important to ask whether the patients retrospectively identified as high risk by the RI were already receiving appropriate and timely care. We note that a large minority of expired COVID-19 patients were never admitted to ICU, although this may have been due to the goals of care for those patients. On the other hand, for patients who were transferred to the ICU, we see that high-risk COVID-19 patients were transferred substantially later in their hospitalization and had a much higher mortality rate than the non–COVID-19 high-risk population. It is possible that if the physician was alerted to the high risk inferred from the initial RI for these COVID-19 patients, there may have been an opportunity to affect outcomes by altering care earlier, whether by more frequent vitals sign measurement, placement in higher level of care, or other means of increased vigilance.

Perhaps equally important, the low-risk group accounted for nearly half of all hospitalized COVID-19 patients and had a substantially lower rate of in-hospital mortality, discharge to hospice, ICU utilization, and mechanical ventilation than the high-risk group. The vast majority (95–97%) of low-risk patients did not expire and had approximately half the LOS of the nonexpiring high-risk population.

Identifying low-risk patients may facilitate decisions to transfer or discharge these patients into lower or nonacute care settings and free hospital and ICU bed capacity for use with higher risk patients. Especially during surge situations, an objective measure may help ensure that resources, supplies, and bed capacity are allocated efficiently. This applies to high-risk patients who may benefit from closer monitoring or more intensive therapies and to low-risk patients, some of whom may not need high levels of care or indeed hospitalization at all.

Robustness, Dynamics, and Practical Considerations

The majority of scores in the public domain which have been evaluated for use with COVID-19 patients, for example, quick Sequential Organ Failure Assessment, NEWS, CURB-65, and their variants (25–28), are based on a limited number of clinical inputs, simple algorithms, and lack a large finely graduated dynamic range. On the other hand, although sophisticated models developed specifically for COVID-19 have proliferated, concerns about their generalizability, robustness, and utility persist, with a recent systematic review unable to recommend a single one for use in practice (29).

Our approach is in essence a middle ground: applying a well-documented, feature-rich acuity score, which nevertheless uses readily available data and has a track record of straightforward implementation at numerous U.S. hospitals. Applying a complex model such as the RI in a simple manner by using cut points avoids problems such as training set artifacts, overfitting, or failure to account for a dynamic care and patient environment, that can arise when model training on a static data set. As our precision recall curves illustrate, alternative cut points could have been chosen. However, the method’s generalizability is shown by similar risk stratification—for identifying both high and low risk patients—using the same cut points at four geographically distinct hospitals with different characteristics: large/small, urban/suburban, academic/community. Even in the face of declining mortality rates, the need for appropriate treatment has not diminished, and the need to know early-on which patients would or would not benefit from more aggressive therapies or warrant closer monitoring remains unaltered.


This work was a retrospective analysis of COVID-19 patients, and as such presupposes knowledge of whether a patient was accurately diagnosed with COVID-19. This work does not account for possible coding error nor uncertainty arising from the possibility of false-positive or false-negative diagnostic laboratory tests. Additionally, care of COVID-19 patients evolved rapidly during the time period spanned by our study data, and the use of treatments such as dexamethasone, remdesivir, convalescent plasma, monoclonal antibodies, proning, and anticoagulation therapies changed both during and subsequent to the period of our study data (30–36) as did perspectives on the benefits of early, aggressive intubation (37,38). During COVID-19 surge conditions, particularly at Yale New Haven Hospital, it may be that some patients were not moved to the ICU due to a more widespread acceptance of palliation, and lack of ICU use may therefore reflect changes in care processes rather than an underrecognition of severity of disease process. Without chart review and more complete contextual data, it is not possible to know what patient goals of care may have been and the extent to which they influenced decisions related to treatment or to receiving care in the ICU. Similarly, we note that identifying high- and low-risk patients is not synonymous with identifying patients who are most or least likely to benefit from treatment which requires a contextual evaluation of individual patient factors and goals. Only via a prospective study can the impact of this acuity score–based risk triage approach be demonstrated.


A direct measure of acuity at the time of admission is shown to be an effective means for identifying patients at risk of significant deterioration, stratifying the risk of critical deterioration in COVID-19 patients. We found that the relationship of comorbidity burden and acuity to mortality is different for COVID-19 and non–COVID-19 patients. COVID-19 patients experience a significantly higher risk of mortality compared with non–COVID-19 medical (respiratory or nonrespiratory) patients at the same level of acuity. This insight may affect treatment decisions, particularly for those COVID-19 patients presenting with modest levels of acuity, where it is not clear what the appropriate level of care should be. Selected acuity thresholds are shown to be a generalizable and robust means of identifying high- and low-risk COVID-19 patients. Use of such an acuity measure at the time of admission may assist front-line clinicians in aligning level of care decisions with hospital and ICU capacity constraints, ensuring that patients are placed appropriately and limited resources are allocated efficiently.


We extend our thanks to Christine Sullivan and Lynn Lewis of Sinai Hospital of Baltimore for their help to make this work both feasible and useful and to Dr. Robert Fogerty of Yale New Haven Hospital for his thoughtful insights and suggestions. We also want to thank the Yale New Haven Health System, LifeBridge Health, the University of California Irvine Medical Center, and PeraHealth, Inc., for support of the researchers involved in the project.


1. Richardson S, Hirsch JS, Narasimhan M, et al. Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID-19 in the New York City Area. JAMA. 2020; 323:2052–2059
2. Catanzaro M, Fagiani F, Racchi M, et al. Immune response in COVID-19: Addressing a pharmacological challenge by targeting pathways triggered by SARS-CoV-2. Signal Transduct Target Ther. 2020; 5:84
3. Remuzzi A, Remuzzi G. COVID-19 and Italy: What next? Lancet. 2020; 395:1225–1228
4. Buckstein M, Skubish S, Smith K, et al. Experiencing the surge: Report from a large New York radiation oncology department during the COVID-19 pandemic. Adv Radiat Oncol. 2020; 5:610–616
5. Keeley C, Jimenez J, Jackson H, et al. Staffing up for the surge: Expanding the New York City Public Hospital workforce during the COVID-19 pandemic. Health Aff (Millwood). 2020; 39:1426–1430
6. Truog RD, Mitchell C, Daley GQ. The toughest triage - allocating ventilators in a pandemic. N Engl J Med. 2020; 382:1973–1975
7. Jain V, Yuan J-M. Predictive symptoms and comorbidities for severe COVID-19 and intensive care unit admission: A systematic review and meta-analysis. Int J Public Health. 2020; 65:533–546
8. Imam Z, Odish F, Gill I, et al. Older age and comorbidity are independent mortality predictors in a large cohort of 1305 COVID-19 patients in Michigan, United States. J Intern Med. 2020; 288:469–476
9. Smith GB, Prytherch DR, Meredith P, et al. The ability of the national early warning score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013; 84:465–470
10. Subbe CP, Kruger M, Rutherford P, et al. Validation of a modified early warning score in medical admissions. QJM. 2001; 94:521–526
11. Lim WS, van der Eerden MM, Laing R, et al. Defining community acquired pneumonia severity on presentation to hospital: An international derivation and validation study. Thorax. 2003; 58:377–382
12. Rothman MJ, Rothman SI, Beals J 4th. Development and validation of a continuous measure of patient condition using the electronic medical record. J Biomed Inform. 2013; 46:837–848
13. Rothman MJ, Solinger AB, Rothman SI, et al. Clinical implications and validity of nursing assessments: A longitudinal measure of patient condition from analysis of the electronic medical record. BMJ Open. 2012; 2:e000646
14. Finlay GD, Rothman MJ, Smith RA. Measuring the modified early warning score and the Rothman index: Advantages of utilizing the electronic medical record in an early warning system. J Hosp Med. 2014; 9:116–119
15. Rothman MJ, Tepas JJ 3rd, Nowalk AJ, et al. Development and validation of a continuously age-adjusted measure of patient condition for hospitalized children using the electronic medical record. J Biomed Inform. 2017; 66:180–193
16. Rothman SI, Rothman MJ, Solinger AB. Placing clinical variables on a common linear scale of empirically based risk as a step towards construction of a general patient acuity score from the electronic health record: A modelling study. BMJ Open. 2013; 3:e002367
17. Alarhayem AQ, Muir MT, Jenkins DJ, et al. Application of electronic medical record-derived analytics in critical care: Rothman Index predicts mortality and readmissions in surgical intensive care unit patients. J Trauma Acute Care Surg. 2019; 86:635–641
18. da Silva YS, Hamilton MF, Horvat C, et al. Evaluation of electronic medical record vital sign data versus a commercially available acuity score in predicting need for critical intervention at a tertiary children’s hospital. Pediatr Crit Care Med. 2015; 16:644–651
19. Tepas JJ 3rd, Rimar JM, Hsiao AL, et al. Automated analysis of electronic medical record data reflects the pathophysiology of operative complications. Surgery. 2013; 154:918–924
20. Gotur D, Zimmerman J. Rothman index as a predictor of post-discharge adverse events in a medical intensive care unit. Chest. 2016; 149:A149–A149
21. Goshua G, Liu Y, Meizlish ML, et al. Admission Rothman index, aspirin, and intermediate dose anticoagulation effects on outcomes in COVID-19: A multi-site propensity matched analysis. Blood. 2020; 136:23–24
22. Quan H, Sundararajan V, Halfon P, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005; 43:1130–1139
23. Singh A, Shaikh A, Singh R, et al. COVID-19: From bench to bed side. Diabetes Metab Syndr. 2020; 14:277–281
24. Johnson S, Gottlieb D. Breaking news: What’s working for COVID-19 patients in the epicenter. Emerg Med News. 2020; 42:1
25. Su Y, Tu GW, Ju MJ, et al. Comparison of CRB-65 and quick sepsis-related organ failure assessment for predicting the need for intensive respiratory or vasopressor support in patients with COVID-19. J Infect. 2020; 81:647–679
26. Volff M, Tonon D, Bourenne J, et al. No added value of the modified NEWS score to predict clinical deterioration in COVID-19 patients. Anaesth Crit Care Pain Med. 2020; 39:577–578
27. Liu FY, Sun XL, Zhang Y, et al. Evaluation of the risk prediction tools for patients with coronavirus disease 2019 in Wuhan, China: A single-centered, retrospective, observational study. Crit Care Med. 2020; 48:e1004–e1011
28. Guo J, Zhou B, Zhu M, et al. CURB-65 may serve as a useful prognostic marker in COVID-19 patients within Wuhan, China: A retrospective cohort study. Epidemiol Infect. 2020; 148:e241
29. Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19 infection: Systematic review and critical appraisal. BMJ. 2020; 369:m1328
30. Stauffer WM, Alpern JD, Walker PF. COVID-19 and dexamethasone: A potential strategy to avoid steroid-related strongyloides hyperinfection. JAMA. 2020; 324:623–624
31. Horby P, Lim WS, Emberson JR, et al.; RECOVERY Collaborative Group. Dexamethasone in hospitalized patients with Covid-19 - preliminary report. N Engl J Med. 2021; 384:693–704
32. Beigel JH, Tomashek KM, Dodd LE, et al.; ACTT-1 Study Group Members. Remdesivir for the treatment of Covid-19 - final report. N Engl J Med. 2020; 383:1813–1826
33. Malani AN, Sherbeck JP, Malani PN. Convalescent plasma and COVID-19. JAMA. 2020; 324:524
34. Gharbharan A, Jordans CCE, GeurtsvanKessel C, et al. Convalescent plasma for COVID-19. A randomized clinical trial. medRxiv. 20202020.07.01.20139857
35. Paul V, Patel S, Royse M, et al. Proning in non-intubated (PINI) in Times of COVID-19: Case series and a review. J Intensive Care Med. 2020; 35:818–824
36. Becker RC. COVID-19 update: Covid-19-associated coagulopathy. J Thromb Thrombolysis. 2020; 50:54–67
37. Tobin MJ, Laghi F, Jubran A. Caution about early intubation and mechanical ventilation in COVID-19. Ann Intensive Care. 2020; 10:78
38. Gattinoni L, Marini JJ, Busana M, et al. Spontaneous breathing, transpulmonary pressure and mathematical trickery. Ann Intensive Care. 2020; 10:88

clinical decision rules; coronavirus disease 2019; decision support systems; hospital mortality; patient acuity; risk assessment

Supplemental Digital Content

Copyright © 2021 2021 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine.