KEY POINTS
Question: What is the accuracy of medical caregivers and MEWS in predicting the development of critical illness?
Findings: In a multicenter, observational study, it was shown that specificity was significantly better when predicted by caregivers, whereas sensitivity was significantly lower for EMS and ED nurses and not significantly different for physicians compared with the MEWS.
Meanings: This suggests that although MEWS is able to correctly predict critical illness, its use leads to overestimation due to a substantial number of false positives. The prediction by medical professionals was proven to be superior.
Recognizing high-risk patients in the emergency department (ED) requiring intensive care decreases morbidity and mortality (1). Previous research revealed that patients who suffered a serious adverse event (SAE), such as ICU admission, cardiac arrests, or death, show abnormal vital signs up to 24 hours before that event (2). Early Warning Scores (EWS) are models based on physiologic variables (e.g., respiratory rate, peripheral oxygen saturation, blood pressure, heart rate, and neurologic status) and are used to improve patient outcomes. Implementing EWS on clinical wards improves early recognition of clinical deterioration and reduces mortality (3).
Although, theoretically, the EWS is simple to use, mistakes in both recording vital parameters and calculating the final score are frequently made (4). In addition, an EWS has low specificity, resulting in a large number of false positives, reflected by false alarms triggered in up to 70–90%, potentially causing alarm fatigue among both nurses and physicians (5–8). Furthermore, following the introduction of EWS in 1997, it has never been compared with standard care, which is clinical gestalt. Clinical gestalt is the ability to judge the severity of illness of a patient based on clinical assessment and subtle patient cues. Clinical assessment consists of a combination of different variables, including vital parameters, patient history, comorbidities, the general impression of a patient, and the professional’s gut feeling. In The Netherlands, the most used EWS is the Modified Early Warning Score (MEWS); therefore, this model was used in this study (Appendix 2, https://links.lww.com/CCM/H306).
Although EWS is increasingly being used in the acute care chain to recognize disease severity, its superiority compared with clinical gestalt remains unproven. Therefore, the aim of this study was to compare the accuracy of predicting the development of critical illness by medical caregivers or MEWS. We hypothesize that the accuracy of predicting critical illness for patients presented at the ED by healthcare providers is superior compared with the EWS.
MATERIALS AND METHODS
Study Design
A prospective observational study was performed in a level-1 trauma center with two different sites and EDs with a combined capacity of about 50,000 patients annually. Data were collected between the March 11, 2021, and October 28, 2021. It is a common daily clinical practice that an ED physician is present during all handovers of Emergency Medical Service (EMS) encounters who were activated prehospital through the national emergency number.
The Medical Ethical Committee board of the Amsterdam Universiteit Medical Center location Academic Medical Center and Vrije Universiteit Medical Center waived the need for official ethical approval because of the strictly observational nature of this study; waiver: W-19_480 19.554, study name: the RECOGnition trial, study acceptance December 18, 2019. Procedures were followed in accordance with the ethical standards of the responsible committee and with the Helsinki Declaration of 1975.
Endpoints and Definitions
The primary aim of this study was to compare the accuracies of healthcare providers and MEWS in predicting the development of critical illness for patients presented at the ED. Critical illness was defined as mortality, direct ICU admission, and the occurrence of SAEs (including indirect ICU admission, sepsis, and myocardial infarction) all within 3 days after ED presentation. Indirect ICU admission was defined as admission to the ICU after initial admission to a general ward.
Secondary endpoints were 28-day all-cause mortality and length of hospital stay. Length of hospital stay was defined as the number of 24-hour periods spent in the hospital. In addition, healthcare providers were asked for their estimation of the severity of illness expressed on a 4-point scale, ranging from 0 to 3 (0 indicating not ill and 3 indicating severely ill). Finally, performance was assessed by sensitivity and specificity when clinical gestalt and MEWS greater than or equal to 3 were combined in predicting critical illness.
Patient Selection
All adult patients (18 yr and over) who were presented at the ED by EMS were assessed for eligibility. Patients transferred from other hospitals were excluded as they had been evaluated by a doctor in a different hospital already, which could influence medical professionals’ predictions. Patients who had been in out-of-hospital cardiac arrest were excluded as well.
Data Collection
Data were collected by a researcher present during EMS presentation between 10 am and 6 pm on workdays, as during this period, most ambulances arrive at the EDs of both centers. Collected baseline variables included age, sex, and a full set of initial vital parameters needed to calculate the MEWS (9).
Directly after EMS handover, all involved healthcare providers present at the time of presentation including EMS and ED nurses, and physicians were asked for their judgment regarding the severity of the illness of the patient. Hereby, their first impression of the patient’s status (based solely on the transfer by EMS nurses, the prehospital vital parameters, and the clinical impression) was acquired. Furthermore, an assessment of the severity of the disease on a 4-point scale (0 indicating not ill to 3 indicating severely ill) and a yes/no questionnaire regarding the prediction of developing critical illness were recorded. In addition, the judgment of healthcare providers regarding 28-day survivability was obtained.
After 28 days, the patient, their first contact, or their general practitioner was contacted to ascertain 28-day survival. In the case of admission to the hospital, further data were collected from the electronic patient files. For patients transferred to other hospitals, the outcome was assessed by calling the ward supervisor on day 3 of their hospital admission. All included data were processed using a standardized data worksheet. Collected data were anonymously processed using an online data collection system (Electronic Data Capture, Amsterdam, The Netherlands).
Modified Early Warning Score
The MEWS was calculated from vital parameters measured during the EMS encounter and at arrival in the ED. The patients’ temperature and neurologic score after ED arrival were used for analysis unless otherwise indicated by the ED nurse. Apart from temperature and neurologic score, no other vital parameters were imputed, as they were considered to be more prone to deviate over short time frames.
Reducing Bias
Observer bias was limited by employing a standard inclusion form, where all results were directly noted, and all questions were asked according to protocol. Furthermore, all medical professionals were interviewed separately.
Exclusion and attrition bias may have occurred on the 28th day of the inclusion period for some patients, as most patients with either a language barrier or lower cognitive status were not interviewed on the 28th day. This bias was reduced by either calling the first contact of the patient or calling their general practitioner to acquire the outcome data needed.
Statistical Analysis
Sample Size Calculation
Our primary hypothesis was that the specificity of medical caregivers would be significantly higher (20%), whereas sensitivity would be noninferior compared with MEWS greater than or equal to 3. Prior research in our centers reported an occurrence of critical illness in around 15% of all patients brought to the ED by EMS (10). Based on these findings, a loss to follow-up of 10%, alpha of 0.05, and a power of 80%, 193 medical judgments were needed to answer our primary outcome. See Appendix 3 (https://links.lww.com/CCM/H306) for the used sample size calculations.
Statistical Analysis
For normally distributed continuous data, the means and sd were calculated. For nonnormally distributed continuous data, medians and interquartile ranges (IQRs) were calculated. Categorical data were presented as absolute values with percentages. Normality of distribution of variables was assessed using visual assessment of histograms. Numerical values with a normal distribution were evaluated using the Student t test, whereas nonnormal distributions were evaluated using the Mann-Whitney U test. The chi-square test was used for evaluating categorical data.
The primary outcome was assessed by calculating sensitivity and specificity for both clinical gestalt and MEWS greater than or equal to 3 and tested for significant differences. The CI for all values was calculated using Wilson statistic. To compare sensitivities, specificities, negative predictive value (NPV), and positive predictive value (PPV) among different models, a McNemar test was used.
As a secondary outcome, the estimation of the severity of illness on a scale from 0 (not ill) to 3 (severely ill) and the MEWS were both assessed as a method of predicting 3-day critical illness and 28-day mortality using the area under the receiver operating characteristic curve (AUROC). In general, the AUROC is characterized using standard terms, where AUROC 0.6–0.7 is considered a poor testing method, 0.7–0.8 is considered fair, 0.8–0.9 is good, and an AUROC greater than 0.9 is considered excellent. The AUROCs for clinician predictions were then compared with the prehospital MEWS using the method described by Hanley and McNeil (11). Finally, the influence of years of experience on the accuracy of the prediction was assessed using linear regression analysis.
For all performed tests, a p value of less than 0,05 was considered significant. Analysis was performed using IBM SPSS Statistics for Windows, Version 27.0 (IBM, Armonk, NY).
RESULTS
During the study period, a total of 800 cases were included in the study. A total of 10 patients were lost to a 3-day follow-up, and a total of 29 patients were lost to a 28-day follow-up. Critical illness within 3 days occurred in 113 patients (14.1%). Patients who developed critical illness were relatively more often primarily presented at the resuscitation or trauma bay compared with the not critically ill (p < 0.001). Prehospital notifications were more often given out in the critically ill group (p = 0.001). A total of 397 patients (49.6%) were admitted to the hospital, and the length of hospital stay was longer for patients who were critically ill compared with the noncritically ill (p = 0.007) (Appendix 1, https://links.lww.com/CCM/H306).
The most frequent diagnosis found in our study population was trauma-related (22.3%), and the second most frequent was stroke (10%). The final diagnosis was missing in 10 patients, nine of whom were transferred to another hospital without a definite diagnosis and one patient passed away in the ED before a diagnosis was made.
Before imputing, a complete MEWS was available for 35.4% of all patients. After imputation, the MEWS was calculated in 94.8% of cases.
In Table 1, the number of patients in each subgroup of critical illness within 72 hours is displayed. In case a patient met multiple criteria, the most severe outcome was primarily counted.
TABLE 1. -
Outcome Measures
| Outcome Measures |
|
Includinga
|
| Critical illness <72 hr, n (%) |
113 (14.1) |
|
| Mortality |
15 |
6 Direct ICU admission |
| 1 Indirect ICU admission |
| 1 RIT consultation (without ICU admission) |
| 1 Sepsis |
| Direct ICU admission (after RIT consultation) |
41 |
4 Sepsis |
| Serious adverse event |
| Indirect ICU admission (after RIT consultation) |
12 |
|
| RIT consultation (without ICU admission) |
4 |
| Sepsis |
23 |
| Acute myocardial infarction |
13 |
| Admission to ER |
5 |
| 28-d mortality, n (%) |
58 (7.2) |
|
RIT = rapid intervention team, ER = emergency department.
aAs some patients qualified as critically ill for multiple reasons, patients who suffered from a less severe cause of critical illness as well were noted here.
The Primary Outcome, Prediction Development of Critical Illness Based on a Yes/No Question
Within 3 days, 113 out of the 790 patients became critically ill (14.1%). MEWS and gestalt of EMS nurses based on the yes/no questionnaire were collected for all 790 patients, whereas the clinical gestalt of ED nurses and physicians was collected 732 and 203 times, respectively.
The clinical gestalt of EMS and ED nurses demonstrated a significantly lower sensitivity compared with MEWS; 41.6% (n = 328) and 41.8% (n = 306) versus 64.8% (n = 512), respectively, in predicting the development of critical illness. However, the clinical gestalt of physicians and MEWS had a similar sensitivity, 60.9% (n = 124) versus 64.8%. Regarding specificity, the gestalt of EMS nurses, ED nurses, and physicians was significantly higher compared with MEWS, 93.2–97.3% versus 70.4%. For gestalt of all healthcare providers, PPV was much higher compared with MEWS, whereas the NPV was similar, as shown in Table 2.
TABLE 2. -
Sensitivity and Specificity for Prediction of 3-d
Critical Illness and 28-d Mortality
| Prediction of Critical Illness Within 3 d |
| Outcome |
Emergency Medical Service Nurse (n = 790) |
p for Diff. With MEWS |
Emergency Department Nurse (n = 732) |
p for Diff. With MEWS |
Physician (n = 203) |
p for Diff. With MEWS |
MEWS ≥3 (n = 790) |
| Sensitivity (95% CI) |
41.6 (38.2–45.1) |
0.001 |
41.8 (38.3–45.4) |
0.003 |
60.9 (54.0–67.4) |
0.581 |
64.8 (61.0–68.4) |
| Specificity (95% CI) |
93.2 (91.2–94.8) |
< 0.001 |
97.3 (95.9–98.3) |
< 0.001 |
96.8 (93.4–98.5) |
< 0.001 |
70.4 (66.8–73.8) |
| Negative predictive value (95% CI) |
90.5 (88.3–92.4) |
0.11 |
91.5 (89.3–93.3) |
0.70 |
89.4 (84.4–92.9) |
0.86 |
92.6 (90.3–94.4) |
| Positive predictive value (95% CI) |
50.5 (47.0–54.0) |
< 0.001 |
70.7 (67.3–73.9) |
< 0.001 |
84.8 (84.9–93.3) |
< 0.001 |
25.8 (22.6–29.3) |
MEWS = Modified Early Warning Score.
Secondary Outcome: Prediction of Severity of Illness Based on a 4-Point Scale
Performance of MEWS and the 4-point scale in recognizing the most severely ill patients was tested. The highest score on the 4-point scale and MEWS greater than or equal to 3 was the used threshold. Performance expressed as the AUROC of ED nurses and physicians in predicting critical illness was significantly better compared with MEWS with an overall performance of 0.809 and 0.848 versus 0.731, p = 0.032 and p = 0.010 (Table 3). However, the performance of EMS nurses was comparable with MEWS with AUROC 0.751 versus 0.731, respectively (p = 0.318).
TABLE 3. -
Area Under the Receiving Operating Characteristics for the Prediction of
Critical Illness Within 72 hr
| Medical Care Provider |
Area Under the Receiving Operating Characteristic for Predicting 3-d Critical Illness (95% CI) |
p for Difference vs MEWS |
| Emergency medical service nurse |
0.751 (0.701–0.802) |
0.318 |
| Emergency department nurse |
0.809 (0.761–0.858) |
0.032 |
| Physician |
0.848 (0.790–0.903) |
0.010 |
| MEWS |
0.731 (0.673–0.789) |
- |
MEWS = Modified Early Warning Score.
A combination of different medical professionals and prehospital MEWS scores was assessed as well. This resulted in an AUROC ranging from 0.691 to 0.704 and did, therefore, not improve the accuracy.
Table 4 shows the different cutoff values and their respectable sensitivities and specificities for predicting critical illness within 3 days. For the prediction by caregivers, the aforementioned scale was used to indicate the cutoff points, ranging from 0 indicating “not ill” to 3 indicating “seriously ill.” For the prediction by the prehospital MEWS score, the cutoff points are based on the total score.
TABLE 4. -
Sensitivity and Specificity per Cutoff Point for Predicting
Critical Illness
| Cutoff Points |
| Medical Care Provider |
Outcome |
0 |
1 |
2 |
3 |
| Emergency medical service nurse (n = 789) |
Sensitivity (95% CI) |
100 (99.5–100) |
93.8 (91.9–95.3) |
83.2 (80.4–85.7) |
57.5 (54.0–60.9) |
| Specificity (95% CI) |
0 (0–0.5) |
23.8 (21.0–26.9) |
47.5 (44.0–51.0) |
85.9 (83.3–88.2) |
| Emergency department nurse (n = 733) |
Sensitivity (95% CI) |
100 (99.5–100) |
96.9 (95.4–97.9) |
84.7 (81.9–87.1) |
62.2 (58.6–65.6) |
| Specificity (95% CI) |
0 (0–0.5) |
26.6 (23.5–29.9) |
53.1 (49.5–56.7) |
91.0 (88.7–92.9) |
| Physician (n = 207) |
Sensitivity (95% CI) |
100 (98.2–100) |
100 (98.2–100) |
93.5 (89.3–96.1) |
60.9 (54.1–67.3) |
| Specificity (95% CI) |
0 (0–1.8) |
36.0 (29.8–42.7) |
58.4 (51.6–64.9) |
89.4 (84.5–92.9) |
| Modified Early Warning Score (n = 642) |
Sensitivity (95% CI) |
100 (99.4–100) |
95.5 (93.6–96.9) |
76.1 (72.7–79.2) |
64.8 (61.0–68.4) |
| Specificity (95% CI) |
0 (0–0.6) |
21.3 (18.3–24.6) |
54.5 (50.6–58.3) |
70.4 (66.8–73.8) |
After assessing the 4-point scale, as shown in Table 4, the sensitivity, specificity, NPV, and PPV were calculated for medical professionals assessing the severity of illness as “seriously ill,” or 3 on the 4-point scale. Medical caregivers’ performance expressed as sensitivity was comparable with the performance of the prehospital MEWS, whereas the specificity was significantly higher.
Secondary Outcome: Prediction of 28-Day Mortality
Judgment of medical caregivers in predicting 28-day mortality expressed as specificity was significantly higher compared with MEWS, 92.9%–96.5% versus 69.2%. Sensitivity was, however, significantly better for MEWS greater than or equal to 3 compared with EMS and ED nurses, whereas the performance of physicians and MEWS greater than or equal to 3 was comparable (Table 5).
TABLE 5. -
Performance in Predicting Mortality Within 28 Days
| Outcome |
Emergency Medical Service Nurse (n = 765) |
p for Differences With MEWS |
Emergency Department Nurse (n = 713) |
p for Differences With MEWS |
Physician (n = 192) |
p for Differences With MEWS |
MEWS Prehospital ≥3 |
| Sensitivity (95% CI) |
31.2 (28.0–34.6) |
< 0.001 |
39.3 (35.8–42.9) |
0.004 |
55.0 (47.9–61.9) |
0.727 |
69.6 (65.9–73.1) |
| Specificity (95% CI) |
92.9 (90.9–94.5) |
< 0.001 |
94.5 (92.6–96.0) |
< 0.001 |
96.5 (92.9–98.3) |
< 0.001 |
69.2 (65.5–72.7) |
| Negative predictive value (95% CI) |
94.3 (92.4–95.7) |
- |
94.8 (93.0–96.2) |
- |
94.9 (90.8–97.2) |
- |
96.6 (94.9–97.8) |
| Positive predictive value (95% CI) |
26.5 (23.5–29.7) |
- |
37.9 (34.3–41.5) |
- |
64.7 (57.7–71.1) |
- |
15.2 (12.6–18.2) |
MEWS = Modified Early Warning Score.
Secondary Outcome: Influence of Years of Experience
EMS nurses had a median experience of 10 years (IQR, 4–20), in which longer experience correlated significantly with the correct prediction of 3-day critical illness (p < 0.001). The ED nurses had a median experience of 5 years (IQR, 3–12), and the physicians had a median experience of 5 years (IQR, 3–11) both with no significant correlation with the correct prediction of 3-day critical illness (p = 0.499 and p = 0.237, respectively).
DISCUSSION
Based on the findings of this study, the clinical gestalt of healthcare providers in the acute care chain in Amsterdam has a significantly higher specificity and PPV in predicting the development of critical illness within 3 days compared with MEWS. The clinical gestalt of physicians has comparable sensitivity to MEWS, whereas prediction by EMS and ED nurses showed a significantly lower sensitivity. Based on our findings, approximately 30% of all ED patients would be considered critically ill without becoming so according to MEWS.
Previous studies aimed to explore the performance of clinical gestalt in predicting patient outcomes. However, this judgment was based on a surrogate of the direct question regarding their clinical gestalt, that is, the presence or absence of a prehospital alert call of EMS to the ED (12). The present study is, thereby, the first that studied the performance of healthcare providers in predicting which patients become critically ill.
As a secondary outcome, the performance of healthcare providers and MEWS in predicting which patients would become critically ill was expressed by calculating the AUROC. Both nurses and physicians have an AUROC higher than 0.75 for predicting 3-day critical illness and was significantly better compared with MEWS. The performance of MEWS expressed in AUROC was 0.731, which is moderate but in concordance with previous studies (13).
The performance of healthcare providers in predicting critical illness based on the yes/no question was adequate and significantly better compared with the MEWS. Interestingly, based on the 4-point severity scale, the prediction of the most severely ill was even better. This is noteworthy as healthcare providers can thus adequately recognize the most severely ill patients. Stating concerns toward the rest of the team or using clinical gestalt as a component to triage patients at the ED may lead to earlier life-saving treatment or interventions, potentially reducing morbidity and mortality.
The predictive accuracy of 28-day mortality was the highest for physicians. Although sensitivity was similar, specificity was significantly better for physicians compared with EMS or ED nurses. All healthcare providers had a significantly higher specificity compared with MEWS. Prior studies reported comparable performance of healthcare providers with AUROC ranging from 0.706 to 0.75 (14,15). An explanation for the difference in the performance of the healthcare providers may be selection bias. Physicians were only attending the handover for those patients that were presented by ambulance after an emergency call and not for the patients presented at the ED on the advice of a general practitioner. This is also the explanation for the difference in the number of interviews between physicians and nurses.
The percentage of critical illness in our study (14.1%) was higher than that reported in previous studies (4.5%) (16,17). This higher occurrence might be explained by the selection of the most severe and ill patients in our level one trauma center. The occurrence of critical illness as stated in this study is in concordance with a study done by the same study group in 2017 (10).
The intent of this observational prospective study was to not influence but simply to observe the management of the patients included in this study. However, in a study by Nickel et al (18), asking physicians whether they would be surprised if their patient died within the next 6 to 12 months might have triggered referral to palliative care. Although the observational nature of this study intended to not influence the healthcare providers’ work, asking them about the prognosis may have influenced their choice of therapy, nonetheless.
Although the chosen time frame from 10 am to 6 pm on working days aimed at including as many patients as possible, it might have introduced selection bias, as previous research has shown that presentations during the night or the weekend might result in worse outcomes (19). However, our main aim was to investigate clinical performance in predicting critical illness in comparison with the MEWS. The type of patients who were admitted to the ED does not influence this estimation.
Patients who had a restricted treatment policy (e.g., do not resuscitate and no ICU admittance) beforehand were not excluded. As healthcare providers may not always have been aware of the restricted treatment policy of the patient, this may have influenced the accuracy of their predictions regarding the development of critical illness. For instance, if a medical professional would have estimated a patient requiring ICU admittance but their “no ICU policy” made this impossible, this would lead to lower sensitivity and specificity, thus distorting the results. This could only have been prevented by excluding patients with a restricted treatment policy in advance, which was impossible in the current setting due to missing information. Retrospectively, establishing which patients had a restricted treatment policy before presentation at the ED could not be determined as this was not registered in all patient files. Another limitation of the findings is that before imputing, a complete MEWS was available for 35.4% of all patients. After imputation, MEWS was available for 94.8% of the cases. However, this may have introduced bias. Despite these limitations, this study was performed in multiple centers, and a large group of patients was assessed with a low dropout rate.
CONCLUSIONS
In this study, we found that medical professionals could predict patients not getting critically ill within 3 days significantly better than the MEWS. Although MEWS may predict those patients that become critically ill, its use leads to overestimation due to a substantial number of false positives. Therefore, healthcare providers’ judgments in the acute care chain remain important, and early escalation of care should be considered in patients they suspect to become critically ill.
REFERENCES
1. Delgado MK, Liu V, Pines JM, et al.: Risk factors for unplanned transfer to intensive care within 24 hours of admission from the
emergency department in an integrated healthcare system. J Hosp Med 2013; 8:13–19
2. Nannan Panday RS, Minderhoud TC, Alam N, et al.: Prognostic value of early warning scores in the
emergency department (ED) and acute medical unit (AMU): A narrative review. Eur J Intern Med 2017; 45:20–31
3. Alam N, Hobbelink EL, van Tienhoven AJ, et al.: The impact of the use of the
Early Warning Score (EWS) on patient outcomes: A systematic review. Resuscitation 2014; 85:587–594
4. Ludikhuize J, Smorenburg SM, de Rooij SE, et al.: Identification of deteriorating patients on general wards; measurement of vital parameters and potential effectiveness of the Modified
Early Warning Score. J Crit Care 2012; 27:424.e7424.e427–424.e74424.e13
5. Subbe CP, Kruger M, Rutherford P, et al.: Validation of a modified
Early Warning Score in medical admissions. QJM 2001; 94:521–526
6. Cvach M: Monitor alarm fatigue: An integrative review. Biomed Instrum Technol 2012; 46:268–277
7. Rothman MJ, Rothman SI, Beals JT: Development and validation of a continuous measure of patient condition using the electronic medical record. J Biomed Inform 2013; 46:837–848
8. Tsien CL, Fackler JC: Poor prognosis for existing monitors in the intensive care unit. Crit Care Med 1997; 25:614–619
9. Teasdale G, Jennett B: Assessment of coma and impaired consciousness. A practical scale. Lancet 1974; 2:81–84
10. Veldhuis LI, Hollmann MW, Kooij FO, et al.: A pre-hospital risk score predicts
critical illness in non-trauma patients transported by ambulance to a Dutch tertiary referral hospital. Scand J Trauma Resusc Emerg Med 2021; 29:32
11. Hanley JA, McNeil BJ: A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983; 148:839–843
12. Fullerton JN, Price CL, Silvey NE, et al.: Is the Modified
Early Warning Score (MEWS) superior to clinician judgement in detecting
critical illness in the pre-hospital environment? Resuscitation 2012; 83:557–562
13. Guan G, Lee CMY, Begg S, et al.: The use of early warning system scores in prehospital and
emergency department settings to predict clinical
deterioration: A systematic review and meta-analysis. PLoS One 2022; 17:e0265559
14. de Groot B, Lameijer J, de Deckere ER, et al.: The prognostic performance of the predisposition, infection, response and organ failure (PIRO) classification in high-risk and low-risk
emergency department sepsis populations: Comparison with clinical judgement and sepsis category. Emerg Med J 2014; 31:292–300
15. Quinten VM, van Meurs M, Wolffensperger AE, et al.: Sepsis patients in the
emergency department: Stratification using the Clinical Impression Score, Predisposition, Infection, Response and Organ dysfunction score or quick Sequential Organ Failure Assessment score? Eur J Emerg Med 2018; 25:328–334
16. Seymour CW, Kahn JM, Cooke CR, et al.: Prediction of
critical illness during out-of-hospital emergency care. JAMA 2010; 304:747–754
17. Kievlan DR, Martin-Gill C, Kahn JM, et al.: External validation of a prehospital risk score for
critical illness. Crit Care 2016; 20:255
18. Nickel CH, Kellett J, Nieves Ortega R, et al.: A simple prognostic score predicts one-year mortality of alert and calm
emergency department patients: A prospective two-center observational study. Int J Clin Pract 2020; 74:e13481
19. van Galen LS, Struik PW, Driesen BE, et al.: Delayed recognition of
deterioration of patients in general wards is mostly caused by human related monitoring failures: A root cause analysis of unplanned ICU admissions. PLoS One 2016; 11:e0161393
