Secondary Logo

Journal Logo

Observational Study

SURvival PRediction In SEverely Ill Patients Study—The Prediction of Survival in Critically Ill Patients by ICU Physicians

Ros, Marijke M. MD1; van der Zaag-Loonen, Hester J. MD, PhD2; Hofhuis, José G.M. PhD1,2; Spronk, Peter E. MD1,2

Author Information
doi: 10.1097/CCE.0000000000000317

Abstract

With an aging population, increasing comorbidities, and technological developments, the demand for ICU services is growing. This growth also raises the need for the discussion on palliative care in patients who are admitted based on the principle “primum non nocere” (first, do no harm) (1,2). Therefore, it is important to use accurate prognostic models, providing appropriate personalized care for each patient.

In the ICU setting, it is common practice to use tools like the Acute Physiology Age and Chronic Health Evaluation (APACHE)–II (3) to estimate the ICU mortality and the Predicted Risk Existing Disease and Intensive Care Therapy (PREDICT) for long-term prognosis (5 yr) of critically ill patients (4). These models are limited by its inherent group approach, which makes it impossible to translate group-based numbers to individual patients. In addition, these prediction models take data into account that can only be acquired after an observational period of at least 8 hours. The estimation of intensive care physicians regarding prediction of hospital mortality in critically ill patients might be more accurate (5). However, it is unknown if physicians can reliably predict long-term survival of critically ill patients. They might overpredict the risk of mortality, in line with two-objective prognostic models (6), which could translate to unjustified fatalism when considering the treatment options in critically ill patients.

The surprise question (SQ), “Would I be surprised if this patient died in the next 12 months?”, is a tool to identify patients at high risk of death in the next year (7). It is used in Medical Oncology to recognize the phase of palliative care (8). A recent meta-analysis highlighted a wide degree of accuracy of the SQ as a prognostic tool (9) and as a poor, to modestly, predictive tool for death in noncancer illness (8).

The value of the SQ, answered by intensive care physicians in the setting of seriously ill patients, is currently unknown. Our research question was “Is the SQ in the acute setting (within 2 hours of admission) a useful tool to predict the risk of mortality in critically ill patients, during ICU stay, hospital stay and after one year?”. Because of answering the SQ in the first 2 hours of admission, we tried to measure the clinical insight of the physician (not influenced by clinical course). We hypothesized that the SQ in the acute setting has additive value for predicting the risk of mortality in critically ill patients.

MATERIALS AND METHODS

Study Design and Participants

The design of this study was a single-center, prospective, observational cohort study. The study included all patients of 18 years and older, who were admitted to the ICU of Gelre hospitals, location Apeldoorn, between April 2013 and April 2018. Gelre hospitals is a university-affiliated teaching hospital with 850 beds. The ICU is a 12-bed mixed medical-surgical ICU. Only the first admission of a patient during this period was evaluated, all ICU readmissions of the same patient during these years were excluded. After informing the Gelre Hospital Medical Ethics Review Board about the research, a waiver for this study was obtained (reference number: 2019_11). During the study period, the medical ICU staff consisted of six intensivists, who had 5, 8, 10, 10, 11, 18 years of experience, respectively (median 10 yr). No changes in the staff occurred during the study period.

Patients were treated according to (inter)–national standards. In addition, the physician filled in the Dutch National Intensive Care Evaluation (NICE) registry (10) for each ICU admission. This was a standard procedure at all ICU admissions. The NICE registry collects demographic information, acute physiologic variables, patient outcomes, and clinical diagnoses of all admissions within 24 hours. In this NICE registry, the physician was also asked to answer the following SQs with either “yes” or “no”: “I expect that the patient is going to survive the ICU admission” (SQ 1), “I expect that the patient is going to survive the hospital stay” (SQ 2), and “I expect that the patient is going to survive one year after ICU admission” (SQ 3). These SQs were all answered within 2 hours of ICU admission. Variables, of which the ICU physician had knowledge at the time of answering the SQs, for example, age and sex of the patient, were marked.

We composed a database of anonymized data regarding demographic information, APACHE III scores, date of ICU admission, date of death (if applicable), and the answers of physicians about survival expectation of the patients.

Outcomes

The primary endpoints of this study were ICU survival, hospital survival, and 1-year survival. Secondary outcomes included positive predictive values (PPVs) and negative predictive values (NPVs) of responses to the SQs by ICU physicians.

Statistical Analysis

Data were analyzed with IBM SPSS Statistics Version 25. The differences in baseline between the included and excluded patients were tested by independent sample t tests, chi-square tests, and Mann-Whitney U tests. PPVs and NPVs of the different SQs were calculated for the three outcomes. The Kappa statistic was calculated to determine agreement between observed and predicted values (11). With the use of multivariable logistic regression analysis, the independent relation between the SQ and actual mortality within 1 year was studied. We performed a stepwise approach. First, all baseline variables were entered into a multivariable model. Through backward selection, the variables which had an association with the outcome, expressed by a p value of less than 0.3, were retained in the model. Next, the SQ was added to the model. The classification table was used to assess the effect size of the addition.

RESULTS

Patient Characteristics

A total of 3,140 patients were included. Fifty patients were excluded from analysis because of missing data regarding the SQs, 568 patients were excluded because of readmission (Fig. 1). Baseline characteristics between the included and excluded patients differed for some variables (Supplement 1 Tables 1 and 2,https://links.lww.com/CCX/A483). In the missing data group, there were more cerebrovascular accidents and cardiovascular insufficiency at the start. Less patients in the missing group survived 1 year (64%) in comparison with the included group (77%; p = 0.04). The readmission group differed especially in chronic obstructive pulmonary disease (COPD) and confirmed infection. Besides that, the readmission group is more often respiratory insufficient (11% vs 4%; p = 0.00) and required mechanical ventilation after 24 hours of ICU admission more often (50 vs 46%; p = 0.04).

F1
Figure 1.:
Flowchart inclusions.

Baseline characteristics of the study population were listed according to the SQ estimation of de ICU physician for 1-year survival (Table 1). The mean age of the included patients was 63.5 ± 16.6 years, and 1,793 patients (57%) were male. Mortality after 1 year was 23%, and 270 of 3,140 patients (9%) did not survived the ICU admission. Patients who had been expected by the physician to survive 1 year were younger (61.3 yr; 95% CI, 60.6–62.0) than patients who were expected to die within 1 year (72.5 yr; 95% CI, 71.5–73.5; p ≤ 0.001). In addition, those patients had less comorbidities, like COPD and diabetes, in comparison with the patients who were expected to die in 1 year (Table 1). Almost all variables were different between the groups “expected alive” and “expected to die”. The mean APACHE III score of patients expected alive after 1 year was 59.6 (95% CI, 58.7–60.5) and 82.0 (95% CI, 80–84; p = 0.04) in patients who were expected to have died within 1 year.

TABLE 1. - Baseline Characteristics
Expected Alive Expected Dead
Variables SQ 3: 1 Year Survival (n = 2,531—80.6%) SQ 3: 1 Year Survival (n = 609—19.4%) Total Group, n = 3,140
Age, yr, mean (sd)a 61.3 (16.8) 72.5 (12.6) 63.5 (16.6)
Gender, male, % (CI)a 57.8 (55.9–59.7) 45.8 (41.8–49.8) 57.1 (55.4–58.8)
Diabetes, % (CI)a 15.4 (14.0–16.8) 20.5 (17.3–23.7) 16.4 (15.1–17.7)
Chronic obstructive pulmonary disease, % (CI)a 12.7 (11.4–14.0) 20.2 (7.0–23.7) 14.1 (12.9–15.3)
Acute renal insufficiency, % (CI) 5.6 (4.7–6.5) 10.7 (8.2–13.2) 6.6 (5.7–7.5)
Chronic renal insufficiency, % (CI)a 2.6 (2.0–3.2) 5.6 (3.8–7.4) 3.2 (2.6–3.8)
Chronic dialysis, % (CI)a 0.7 (0.4–1.0) 1.0 (0.2–1.8) 0.7 (0.4–1.0)
Confirmed infection, % (CI)a 27.2 (25.5–28.9) 33.2 (29.5–36.9) 28.3 (26.7–29.9)
Gastrointestinal bleeding, % (CI)a 2.6 (2.0–3.2) 4.4 (2.8–6.0) 3.0 (2.4–3.6)
Cerebrovascular accident, % (CI)a 1.2 (0.8–1.6) 2.3 (1.1–3.5) 1.4 (1.0–1.8)
Intracranial mass, % (CI)a 0.2 (0.0–0.4) 2.0 (0.9–3.1) 0.6 (0.3–0.9)
Immunocompromized, % (CI)a 9.6 (8.5–10.7) 12.0 (9.4–14.6) 10.0 (9.0–11.0)
Neoplasm, % (CI)a 3.7 (3.0–4.4) 14.6 (11.8–17.4) 5.8 (5.0–6.6)
Cirrhosis, % (CI)a 0.6 (0.3–0.9) 2.6 (1.3–3.9) 1.0 (0.7–1.3)
Hematologic malignancy, % (CI)a 1.2 (0.8–1.6) 3.8 (2.3–5.3) 1.7 (1.2–2.2)
Cardiovascular insufficiency, % (CI)a 0.8 (0.5–1.1) 5.9 (4.0–7.8) 1.8 (1.3–2.3)
Cardiopulmonary resuscitation < 24 hr before admission, % (CI)a 1.5 (1.0–2.0) 6.7 (4.7–8.7) 2.5 (2.0–3.0)
Respiratory insufficiency, % (CI)a 3.1 (2.4–3.8) 9.2 (6.9–11.5) 4.3 (3.6–5.0)
Mechanical ventilation at 0 hr, % (CI)a 29.3 (27.5–31.1) 41.9 (38.0–45.8) 31.7 (30.1–33.3)
Mechanical ventilation after 24 hr, % (CI) 42.1 (40.2–44.0) 61.9 (58.0–65.8) 45.9 (44.2–47.6)
Admission typea, % (CI)
 Medical 65.2 (63.3–67.1) 79.5 (76.3–82.7) 68.0 (66.4–69.6)
 Acute surgery 26.3 (24.6–28.0) 17.1 (14.1–20.1) 24.5 (23.0–26.0)
 Elective surgery 8.5 (7.4–9.6) 3.1 (1.7–4.5) 7.5 (6.6–8.4)
Acute ICU admission, % (CI)a 92.9 (91.9–93.9) 98.0 (96.9–99.1) 93.9 (93.1–94.7)
ICU stay duration, d, median (IQR) 1 (1–4) 2 (1–5) 2 (1–4)
Hospital stay, d, median (IQR) 11 (4–20) 9 (5–17) 10 (5–18)
ICU survival, % (CI) 91.4 (90.4–92.4)
Hospital survival, n (%) 87.5 (86.3–88.7)
1-yr survival, n (%) 86.2 (84.9–87.5) 40.4 (36.5–44.3) 77.3 (75.8–78.8)
Acute Physiology Age and Chronic Health Evaluation III, mean (sd) 59.6 (23.1) 82.0 (25.2) 64.0 (25.2)
IQR = interquartile range, SQ = surprise question.
aVariables known by the physician at the time of answering the SQ (components of the National Intensive Care Evaluation register).

Expectation of ICU Survival (SQ 1)

For 2,987 out of 3,140 patients (95% CI, 94–96), the physician expected survival of ICU admission. The NPV was 94% (Table 2). For 153 patients (5%), the physician expected mortality during ICU admission. The PPV of this SQ was 64% (95% CI, 62–66) (Table 2). The accuracy of this SQ was 93% (95% CI, 92–94) with a kappa statistic of 0.43 (95% CI, 0.37–0.49).

TABLE 2. - Predicted Outcome Versus Actual Outcome for Surviving ICU Stay, Hospital Stay, and 1 Year, Respectively
Predicted Outcomes Actual Outcomes
Not Surviving Surviving
ICU Stay (n = 270) Hospital Stay (n = 392) 1 yr (n = 713) ICU Stay (n = 2,870) Hospital Stay (n = 2,748) 1 yr (n = 2,427)
Not surviving ICU stay (n = 153) 98 55
Hospital stay (n = 252) 148 104
1 yr (n = 609) 363 246
Surviving ICU stay (n = 2,987) 172 2,815
Hospital stay (n = 2,888) 244 2,644
1 yr (n = 2,531) 350 2,181
ICU stay: positive predictive value, 64.1%; negative predictive value, 94.2%; mean accuracy, 93% (95% CI, 92–94%); Kappa = 0.43 95% CI 0.37–0.49 (moderate agreement). Hospital stay: positive predictive value, 58.7%; negative predictive value, 91.6%; mean accuracy, 89% (95% CI, 88–90%); Kappa = 0.40, 95% CI, 0.35–0.45 (fair agreement). 1 yr: positive predictive value, 59.6%; negative predictive value, 86.2%; mean accuracy, 81% (95% CI, 80–82%); Kappa = 0.43, 95% CI, 0.39–0.47 (moderate agreement).

Expectation of Hospital Survival (SQ 2)

The intensivist was also asked what he/she expected regarding hospital survival, which could be different from ICU survival. For 2,888 patients (92%; 95% CI, 91–93), the physician expected survival of hospital admission. The negative predicted value was 92% (95% CI, 91–93) (Table 2).

For 252 patients (8%), the physician expected mortality during the hospital admission. The positive predicted value of this SQ was 59% (95% CI, 53–65). The accuracy of this SQ was 89% (95% CI, 88–90%) with a kappa statistic of 0.40 (95% CI, 0.35–0.45).

Expectation of 1 Year Survival (SQ 3)

In a group of 2,531 patients (81%; 95% CI, 79–82%), the physician expected 1-year survival after ICU admission. For 609 patients (19%; 95% CI, 16–22%), the physician expected 1-year mortality. The negative predicted value was 86% (95% CI, 85–87),and the positive predicted value was 60% (95% CI, 56–64). The physician’s prediction of survival was more accurate for ICU and hospital admission in comparison with 1-year survival (Table 2). The mean accuracy for the expectation of 1-year survival in this study was 81% (95% CI, 80–82) with a kappa statistic of 0.43 (95% CI, 0.39–0.47).

Logistic Regression

All available patient characteristics, known to the intensivist at time of ICU admission, were added to a multivariable model. Model 1, the basic model, showed the association between these patient characteristics and 1-year survival (Table 3). Excluded variables through backward selection were gender, COPD, gastrointestinal bleeding, and acute ICU admission. These variables that did not significantly contribute to the model were excluded using a backward exclusion approach.

TABLE 3. - Logistic Regression
Model 1 Model 2
Variables OR 95% CI p OR 95% CI p
Surprise question 1.06 1.05–1.07 0.00 0.19 0.16–0.24 0.00
Age 1.04 1.03–1.05 0.00
Diabetes 0.72 0.57–0.90 0.00 0.70 0.55–0.90 0.01
Chronic renal insufficiency 1.33 0.82–2.17 0.25 1.20 0.72–2.03 0.48
Chronic dialysis 4.26 1.59–11.39 0.00 4.66 1.66–13.09 0.00
Confirmed infection 0.77 0.63–0.94 0.01 0.72 0.58–0.89 0.00
Cerebrovascular accident 1.87 0.96–3.66 0.07 1.62 0.79–3.34 0.19
Intracranial mass 11.73 3.59–38.32 0.00 6.95 2.03–23.86 0.00
Immunocompromised 0.59 0.45–0.79 0.00 0.52 0.39–0.70 0.00
Neoplasm 3.58 2.55–5.03 0.00 2.15 1.49–3.09 0.00
Cirrhosis 6.22 2.93–13.18 0.00 3.48 1.59–7.63 0.00
Hematologic malignancy 4.01 2.18–7.37 0.00 3.02 1.59–7.63 0.00
Cardiovascular insufficiency 2.25 1.26–4.00 0.01 1.24 0.67–2.30 0.50
Cardiopulmonary resuscitation 3.62 2.15–6.10 0.00 2.67 1.53–4.64 0.00
Respiratory insufficiency 1.81 1.20–2.72 0.00 2.21 0.78–1.88 0.41
Mechanical ventilation at start 1.81 1.47–2.23 0.00 1.63 1.31–2.01 0.00
Admission type 2.60 1.69–4.02 0.00 1.82 1.16–2.85 0.01
1.32 0.84–2.08 0.28 1.15 0.72–1.84 0.55
Nagelkerke pseudo r 2, % 25.4 33.7
OR = odds ratio.
Association of patient characteristics and 1 yr. Model 1: basic model (excluded variables through backward selection were “gender, chronic obstructive pulmonary disease, gastrointestinal bleeding, and acute ICU admission”). Model 2: Model 1 plus surprise question 3.

Model 2 shows model 1 plus SQ3. Model 2 was correctly predicting the outcome for 81.5% of the cases, compared with 79.6% in model 1 (p < 0.001). This ratio was also visible in the receiver operating characteristic curves; model 1 showed an area under the curve (AUC) of 0.78, and model 2 showed an AUC of 0.82 (Fig. 2). These results showed an added value for predicting 1-year survival of the SQ besides the patient characteristics used in APACHE III scores.

F2
Figure 2.:
Blue: Receiver operating characteristic (ROC) curve basic model (model 1). Area under the curve 0.778. Contains variables: age, diabetes, chronic renal insufficiency, chronic dialysis, confirmed infection, cerebrovascular accident, intracranial mass, immunocompromised, neoplasm, cirrhosis, hematologic malignancy, cardiovascular insufficiency, cardiopulmonary resuscitation, respiratory insufficiency, mechanical ventilation at start, admission type. Red: ROC curve basic model with addition of the surprise question (SQ) (model 2). Area under the curve 0.822. Contains all variables of the basic model and SQ3.

DISCUSSION

The SUrvival PRediction In SEverly ill patients (SURPRISE) study shows that critical care clinicians are reasonably good at predicting 1-year survival in ICU patients: the accuracy of the SQ for ICU admission, hospital admission, and 1-year survival in 3,140 patients was 93%, 89%, and 81% respectively. The PPV of the SQ for 1-year survival is 60%, and the NPV is 86%. The kappa statistics show fair to moderate agreement between the observed and predicted values.

The clinical value of the SQ in ICU patients however is challenging. Previous studies looked at the additive role of the SQ in ICU patients for predicting mortality. Hadique et al (12) used the SQ in a prediction model, besides Charlson comorbidity index and APACHE III score. They reported a strong discrimination and calibration to predict 6-month mortality in medical ICU patients (sensitivity 74%, specificity 82%, mean accuracy 75% [range, 69–82% (p = 0.336)]). However, the SQ was answered by physicians who had knowledge of the physical patient status. Other studies have demonstrated that physicians’ predictions relate reasonably with the predictions derived from de APACHE-II and PREDICT models (6). However, in our study, the SQ was asked in an earlier phase (within 2 hr) of the ICU admission, which inherently does not incorporate the patient’s clinical trajectory during the first 1–2 days of ICU stay. That is why the clinical insight plays a more important role in our study. However, it should be taken into account that the SQ is an useful tool, triage in critically ill patients with limited life expectancy is nuanced and usually takes more than 2 hours of clinical assessment.

All available studies who analyzed the SQ in ICU patients differ in study population, time to follow-up, and the moment of answering the SQ. In the study of Litton et al (6), the SQ was analyzed for 2-year mortality in 2,497 ICU patients. The included patients were young (mean, 53.8 yr) and received mechanical ventilation frequently (86%) (6). The physician’s estimations were correct in 80.4% (95% CI 78.9–82.1), with a PPV of 57.4% and a NPV of 88.8%. The SQ was answered for each patient within the first 24 hours of ICU admission. In contrast, the study of Hadique et al (12) estimated 6-month mortality in 1,049 patients. The included patients had a mean age of 61 years and an average APACHE III score of 72.4. This study population is comparable with patients of our study. They reported a mean accuracy of the physicians using the SQ of 76% (range, 67–84%) (12). However, the SQ was answered for each patient within the first 12–24 hours of ICU admission and in that way differs from our study where SQ had to be scored within the first 2 hours of admission. The moment of answering the SQ is determinative because knowing the occurrences of the first hours after the acute moment, and knowing the APACHE score for example, may influence the estimation of the physician. Indeed, the SQ may be of clinical value if the (right) estimation was made in the acute moment.

The observed 1-year mortality in our SURPRISE study was 22.7% with a physician-predicted mortality of 19.4%. The study of Litton et al (6) shows an observed 2-year mortality of 23.4% and a physician-predicted mortality of 26.5%. They reported extremes of age, extent of comorbidities, and severity of APACHE-II score as independent risk factors for an inaccurate physician prediction. However, the studied duration of mortality differs, in the SURPRISE study, the physicians tended to underpredict 1-year mortality instead of overprediction. This is probably due to the minimal available information at the time of prediction (within 2 hr, not knowing APACHE scores). Indeed, there was only a similar observed rate of mortality despite 1-year versus 2-year observation. This is probably due to differences in study populations (patients in the study of Litton et al [6] were younger) or because the mortality rate is higher in the first year after ICU admission compared with the second year.

In other, nonacute departments, it also seems difficult to estimate the survival time of patients. A meta-analysis by White et al (9) shows a wide degree of accuracy, from poor to reasonable, reported across studies using the SQ with a time frame of 6–12 months in nonacute departments. This analysis contains 22 complete data studies (five were in oncology) with 25.718 estimates of physicians and nurses. It shows a pooled accuracy of the SQs of approximately 75%. The included studies using a shorter time frame for the SQ had a slightly better accuracy. The study of Haydar et al (13) demonstrated a NPV of 98% for the modified SQ, “Would you be surprised if this patient died in the next 30 days?”, to predict in-hospital mortality in older patients receiving care in the emergency department (ED). In our SURPRISE study, the accuracy of the SQ also decreases over time. It seems easier to predict shorter time periods, like ICU and hospital survival, than predicting 1-year survival.

Overall, studies about the SQ show controversial results for clinical practice about the prognostic value. However, identification of patients with limited life expectancy can increase the implementation of an early palliative approach and therefore improve quality of life. The study of Zeng et al (14) demonstrated that educating emergency medicine (EM) physicians about the SQ can increase the number of palliative care consults in the ED. These consults also help to decrease costs by avoiding unwanted healthcare interventions. Ouchi et al (15) shows an additive value of the SQ especially for older patients who visited the ED. They report a low PPV (32%) and high NPV (90%) with a C-statistic of 0.67. The SURPRISE study shows higher PPV (60%) and comparable NPV (86%) of the SQ, probably because there was more information available for the intensivists in comparison with EM physicians. In conclusion, identifying the need for palliative care interventions by using the SQ is helpful in the decision-making process with patients and family members.

Our study has a number of strengths. First, we prospectively included a large cohort of ICU patients over time. Second, there were many different variables available because of the Dutch national standardized NICE registry (10). Third, ICU staffing and hospital staffing/organizations in the ICU did not change during the study nor the admission and discharge policy. The participating intensivists were a heterogeneous experienced group (median 10 yr). Several limitations of our study should also be acknowledged. First, because of missing SQ data, 50 patients were excluded from the final analysis, and some patient characteristics do show significant differences (Supplement 1 Table 1, https://links.lww.com/CCX/A483). Second, we did not use a validation cohort to test our results for our prognostic model. Finally, this is a single-center study from the Netherlands. The data may be different in other settings, in particular regarding differences in ICU admission policies, ICU capacity, and local ethical regulations and protocols.

CONCLUSIONS

The frequently overlooked simple and cheap SQ is probably an useful tool, besides other available patient characteristics, to evaluate the expected prognosis of acutely admitted critically ill patients. As such, the SQ may be helpful in the decision-making process with the patient and family members whether or not to escalate or discontinue certain therapeutic interventions.

REFERENCES

1. Piers RD, Azoulay E, Ricou B, et al.; APPROPRICUS Study Group of the Ethics Section of the ESICM. Perceptions of appropriateness of care among European and Israeli intensive care unit nurses and physicians. JAMA. 2011; 306:2694–2703
2. Gulini JEHMB, Nascimento ERPD, Moritz RD, et al. Predictors of death in an intensive care unit: Contribution to the palliative approach. Rev Esc Enferm USP. 2018; 52:e03342
3. Knaus WA, Zimmerman JE, Wagner DP, et al. APACHE-acute physiology and chronic health evaluation: A physiologically based classification system. Crit Care Med. 1981; 9:591–597
4. Ho KM, Knuiman M, Finn J, et al. Estimating long-term survival of critically ill patients: The PREDICT model. PLoS One. 2008; 3:e3226
5. Scholz N, Bäsler K, Saur P, et al. Outcome prediction in critical care: Physicians’ prognoses vs. scoring systems. Eur J Anaesthesiol. 2004; 21:606–611
6. Litton E, Ho KM, Webb SA. Comparison of physician prediction with 2 prognostic scoring systems in predicting 2-year mortality after intensive care admission: A linked-data cohort study. J Crit Care. 2012; 27:423.e9–423.15
7. Moss AH, Ganjoo J, Sharma S, et al. Utility of the ‘surprise’ question to identify dialysis patients with high mortality. Clin J Am Soc Nephrol. 2008; 3:1379–1384
8. Downar J, Goldman R, Pinto R, et al. The “surprise question” for predicting death in seriously ill patients: A systematic review and meta-analysis. CMAJ. 2017; 189:E484–E493
9. White N, Kupeli N, Vickerstaff V, et al. How accurate is the ‘surprise question’ at identifying patients at the end of life? A systematic review and meta-analysis. BMC Med. 2017; 15:139
10. Van de Klundert N, Holman R, Dongermans DA, et al. Data resource profile: The Dutch national intensive care evaluation (NICE) registry of admissions to adult intensive care units. Int J Epidemiol. 2015; 44:1850–1850h
11. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33:159–174
12. Hadique S, Culp S, Sangani RG, et al. Derivation and validation of a prognostic model to predict 6-month mortality in an intensive care unit population. Ann Am Thorac Soc. 2017; 14:1556–1561
13. Haydar SA, Strout TD, Bond AG, et al. Prognostic value of a modified surprise question designed for use in the emergency department setting. Clin Exp Emerg Med. 2019; 6:70–76
14. Zeng H, Eugene P, Supino M. Would you be surprised if this patient died in the next 12 months? Using the surprise question to increase palliative care consults from the emergency department. J Palliat Care. 2020; 35:221–225
15. Ouchi K, Jambaulikar G, George NR, et al. The “surprise question” asked of emergency physicians may predict 12-month mortality among older emergency department patients. J Palliat Med. 2018; 21:236–240
Keywords:

critical care; intensive care units; (hospital) mortality; palliative medicine; patient-centered care; survival analysis

Supplemental Digital Content

Copyright © 2021 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine.