Patients acutely admitted to the ICU have a high mortality, and survivors may suffer from long-term morbidity and reduced quality of life (12). These critically ill patients frequently present with clinical signs of circulatory shock such as low blood pressure, oliguria, and skin mottling. These signs often guide treatment, assuming that they indicate vital organ hypoperfusion and are associated with increased mortality (3–7). Indeed, guidelines on the management of shock recommend treating patients based on clinical examination, supplemented with critical care ultrasonography (CCUS) (8).
Data on the prognostic value of clinical examination findings are inconsistent. Previous studies have identified different predictors of mortality such as low blood pressure (6910), oliguria (35), prolonged capillary refill time (CRT) (1112), and skin mottling (413). They often evaluated one or two clinical signs in isolation, instead of assessing a combination of signs and symptoms, which would more accurately reflect daily clinical practice. Furthermore, most studies had relatively small sample sizes or included a selected subgroup such as patients with sepsis, cardiogenic shock, and severe trauma (eTable 1, Supplemental Digital Content 2, http://links.lww.com/CCM/E746).
The prognostic value of clinical examination remains to be established in a large, consecutive cohort of critically ill patients. Compared with well-established prognostic scores, which are complex to calculate and unsuited for individual patient prognostication (1415), a simple bedside clinical examination might better inform caregivers in their decision making. Accordingly, our aim was to evaluate which clinical examination findings were independently associated with 90-day mortality in acutely admitted ICU patients. In addition, we hypothesized that combined clinical examination findings would have similar prognostic value compared with existing prognostic scores.
MATERIALS AND METHODS
Design, Setting, and Patients
The prospective, observational, single-center Simple Intensive Care Studies-I (SICS-I) was conducted following a prewritten protocol and statistical analysis plan (SAP; see Supplement 1, Supplemental Digital Content 1, http://links.lww.com/CCM/E745, or clinicaltrials.gov: NCT02912624). All consecutive patients admitted to the ICU of the University Medical Center Groningen were eligible for inclusion. Adult patients who had an unplanned ICU admission and were expected to stay for at least 24 hours were included. Patients were excluded if their ICU admission was planned preoperatively, if acquiring research data interfered with clinical care due to continuous resuscitation efforts (e.g., mechanical circulatory support), or if informed consent was not provided. In unresponsive patients, informed consent was first obtained from the legal representatives and at a later time if the patient recovered consciousness. If the patient died before consent was obtained, the study data were used and legal representatives were informed on the study. The local institutional review board approved the study (M15.168207).
All included patients underwent clinical examination followed by CCUS within the first 24 hours of their ICU admission (eTable 2, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). Researchers conducted the clinical and CCUS examinations, and their findings were not revealed to caregivers.
All clinical examinations were standardized, and cutoff values for abnormal clinical signs were predefined in the protocol (clinicaltrials.gov: NCT02912624). A total of 19 clinical signs per patient were recorded (eTable 1, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). Respiratory rate, heart rate and rhythm, arterial blood pressures, and central venous pressures were recorded from the bedside monitor. Patients were auscultated for the presence of cardiac murmurs and crepitations. Clinical signs reflecting organ perfusion were obtained from the three organs readily accessible to clinical examination: cerebral (mental status), renal (urine output), and skin perfusion (CRT, central-to-peripheral temperature difference [ΔTc-p] and skin mottling). Mental status was assessed according to the categories “Alert,” “responsive to Voice,” “responsive to Pain,” and “Unresponsive” and was scored irrespective of sedation use. Urine output was scored 1 and 6 hours prior to the clinical examination, adjusted for body weight, and considered decreased if less than 0.5 mL/kg/hr. CRT was the time for skin color to fully return after applying firm pressure at the sternum, index finger, and knee for 15 seconds and considered prolonged if greater than 4.5 seconds (16). ΔTc-p was the difference between central temperature measured by a bladder thermistor catheter and peripheral temperature measured by a skin probe on the big toe and dorsum of the foot and considered abnormal if greater than 7°C (1718). The degree of skin mottling was rated at the knee according to a score from 0 to 5, where 0–1 was regarded as mild, 2–3 was regarded as moderate, and 4–5 was regarded as severe mottling (19).
The primary outcome (dependent) variable was 90-day all-cause mortality obtained through the municipal record database. Sensitivity analyses were conducted using all-cause mortality at 7- and 30-day follow-ups.
Sample Size and Missing Data
The sample size was based on the estimation that half of the number of acute ICU admissions per year (N = 1,500) would fulfill the inclusion criteria. The potentially detectable difference was calculated using skin mottling as an example for the case inclusion exceeded 1,000 patients: a significant mortality difference of 9% for skin mottling with 84% power and a maximal type 1 error risk of 0.015 could be detected (20). Missing values were considered missing at random because these depended on other observed patient characteristics (such as age and mechanical ventilation) and a significant Little’s test (21). Multiple imputations (20 times) for missing data were conducted, and parameter estimates and standard errors were combined using Rubin’s formula (2223).
The aims of our primary analyses were twofold: first, a multivariable logistic regression analysis was conducted to identify the clinical examination findings that independently predict mortality at 90-day follow-up and, second, the discriminative performance of this model was compared with that of the Simplified Acute Physiology Score-II (SAPS-II), Acute Physiology and Chronic Health Evaluation-IV (APACHE-IV), and Sequential Organ Failure Assessment (SOFA). Analyses were conducted with Stata Version 15.1 (StataCorp, College Station, TX, USA) on the imputed dataset following our published SAP (Supplement 1, Supplemental Digital Content 1, http://links.lww.com/CCM/E745).
Model Development and Validation.
Unadjusted and age- and sex-adjusted regression analyses were conducted on 19 clinical signs. A p-value of less than 0.25 threshold was used for inclusion in the multivariable models, which was constructed using forward stepwise regression by adding blocks of variables. The multivariable model was adjusted for age (covariate) and norepinephrine infusion rate (mediator) under the pathophysiological mechanism that norepinephrine alters most clinical signs. The final model was internally validated with bootstrap sampling. For bootstrap sampling, 1,075 patients were repeatedly drawn with replacement from the imputed dataset (for a more in-depth explanation, see eFig. 1, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). In total, 100 bootstrap samples were drawn, and the final model was reconstructed in each sample. Each variable from the final model was considered internally validated if it was significant in at least 80 of the 100 bootstrapped models (2024). Calibration of the multivariable models was checked with calibration plots and Hosmer–Lemeshow tests. Discrimination of the final model was evaluated with receiver operating characteristic (ROC) curves (25). Dominance analysis was used to determine the relative importance of independent variables in each multivariable model (26). Our multivariable model was compared with the SAPS-II (reference model), APACHE-IV, and SOFA scores by 1) analyzing differences between the area under the ROC curves (AUC) using the method proposed by DeLong et al (27) and 2) constructing reclassification tables and calculating the net reclassification improvement (28).
Sensitivity and Subgroup Analyses.
In sensitivity analyses, we assessed whether the statistically significant predictors of 90-day mortality were also predictive of 7- and 30-day mortalities. Time dependency was also investigated by conducting a multivariable Cox regression analysis on 90-day mortality.
Two planned subgroup analyses were conducted, in which only the clinical examination findings that were statistically significant in the primary analysis were evaluated. First, patients were stratified by vasopressor use. Second, patients were stratified by underlying pathology that could influence the clinical measurements, that is, acute liver failure or post-orthotopic liver transplantation (OLT), heart failure, septic shock, cardiac arrest, and CNS pathology.
The SICS-I was designed to address multiple hypotheses on six different outcomes, and therefore, the mortality outcome was adjusted for multiple hypothesis testing (29). Supplement 1 (Supplemental Digital Content 1, http://links.lww.com/CCM/E745) contains the details or our SAP, but in short, a p-value of 0.015 indicated statistical significance and p-values between 0.015 and 0.05 indicated suggestive significance with an increased family-wise error rate (2030). For our secondary (subgroup) analyses, a p-value of less than 0.05 indicated statistical significance due to the hypothesis-generating purpose. Accordingly, primary analyses are presented with 98.5% CIs and secondary (subgroup) analyses with 95% CIs.
Amendments to the SAP.
For our primary analysis, multivariable logistic regression analyses were used instead of Cox regression because the outcome (90-day mortality) was fixed, time to event was considered less relevant, and our statistical methods would be more in line with that in the literature. Findings of the Cox regression analyses are reported in Supplement 1 (Supplemental Digital Content 1, http://links.lww.com/CCM/E745) and Supplement 2 (Supplemental Digital Content 2, http://links.lww.com/CCM/E746).
We intended to conduct a multivariable regression analysis of clinical examination findings adjusted for the SAPS-II. Since the SAPS-II also contains various clinical examination findings, we realized that such a model would have little clinical relevance and instead used this score as the reference model. We compared the performance of our clinical examination model to the SAPS-II, APACHE-IV, and SOFA.
Patient Characteristic and Outcome
A total of 1,442 patients were assessed for eligibility between March 27, 2015, and July 22, 2017. The inclusion criteria were fulfilled in 1,212 patients, of whom 137 patients were not included for various reasons (Fig. 1). In the final analysis, 1,075 patients (89%) were included. The median time from ICU admission to inclusion was 15 hours (IQR = 8–20 hr). The proportion of missing values per variable is presented in eTable 2 (Supplemental Digital Content 2, http://links.lww.com/CCM/E746) and per case (eFig. 2, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). One third of the patients was admitted after acute or complicated surgery, and the most common admission diagnoses were of cardiovascular or respiratory origin (Table 1).
After 90 days, 298 patients (28%) had died and eight patients (1%) were lost from follow-up due to emigration to or residence in another country. Patients who died within 90 days were significantly older, were more often mechanically ventilated, had higher positive end-expiratory pressures and lower diastolic blood pressures and mean arterial pressures (p < 0.015; Table 2). Most clinical signs reflecting organ perfusion differed: patients who died had significantly lower urine outputs, colder extremities, longer CRTs, and more severe skin mottling during the clinical examination.
Clinical Examination and 90-Day Mortality
Both unadjusted and age- and sex-adjusted analyses showed that most clinical examination findings were associated with 90-day mortality (eTable 3, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). Multivariable logistic regression adjusted for age and norepinephrine infusion rate showed that five clinical examination findings, that is, higher respiratory rate, higher systolic blood pressure, lower central temperature, altered consciousness, and decreased urine output, were independently associated with 90-day mortality (Fig. 2). The variables atrial fibrillation, diastolic blood pressure, and severe skin mottling were of suggestive statistical significance due to a p-value of greater than 0.015 and a statistical significance in less than 80 of the 100 bootstrap replications (Fig. 2).
The multivariable logistic regression analysis was repeated with systolic blood pressure in quartiles because this variable had a U-shaped relationship with mortality (eFig. 3, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). In this model, only the highest quartile (i.e., a systolic blood pressure > 133 mm Hg) had a suggestive statistically significant association with mortality (OR = 1.65; 98.5% CI, 0.90–3.05; p = 0.046; eTable 4, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). In the complete case analysis, all variables except diastolic blood pressure remained statistically significant (eTable 5, Supplemental Digital Content 2, http://links.lww.com/CCM/E746).
Performance of Clinical Examination When Compared with Prognostic Scores
When comparing AUC’s to three established ICU prognostic risk scores, the clinical examination model was comparable to the SAPS-II and APACHE-IV and significantly better than the SOFA score (Fig. 3). The clinical examination model distinguished 817 patients (76%) correctly into survivor or nonsurvivor. The number of patients correctly classified was 810 (75%) for the SAPS-II, 818 (76%) for the APACHE-IV, and 800 (74%) for the SOFA. The net reclassification improvement of the clinical examination model was 3.8% compared with the SAPS-II (p = 0.09), 5.0% compared with the APACHE-IV (p = 0.025), and 12% compared with the SOFA (p < 0.001).
Sensitivity Analysis: Clinical Examination and Short-Term Mortality
The relation of clinical examination findings over time was studied using logistic regression analyses on 7- and 30-day mortalities and a Cox regression. Severe skin mottling had stronger associations with 7-day mortality (OR = 3.06; 95% CI, 1.34–6.98; p = 0.008) compared with 90-day mortality (OR = 2.45; 95% CI, 1.12–5.34; p = 0.025). Systolic and diastolic blood pressures were not statistically significantly associated with 7-day mortality (eTable 7, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). Atrial fibrillation and diastolic blood pressures were not statistically significantly associated with 30-day mortality (eTable 7, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). Results of the multivariable Cox regression were comparable to the logistic regression analysis used in the main analysis (eTable 8, Supplemental Digital Content 2, http://links.lww.com/CCM/E746).
In two predefined subgroup analyses, the patient population was stratified by vasopressor use and by underlying pathology. In these analyses, only the eight clinical examination findings that were statistically significant in the primary analysis were tested (eFig. 5, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). In patients without vasopressors, only a higher respiratory rate and an altered consciousness had statistically significant associations with mortality (p < 0.001; eTable 9, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). In patients receiving vasopressors, a higher respiratory rate, atrial fibrillation, lower central temperature, nonresponsiveness, decreased urine output, and severe skin mottling were independently associated with 90-day mortality (eTable 9, Supplemental Digital Content 2, http://links.lww.com/CCM/E746). In patients admitted with septic shock, only age and skin mottling over the knee were significantly associated with mortality (OR = 3.22; 95% CI, 1.31–7.94; p = 0.011). In the subgroups of patients admitted with acute liver failure or post-OLT, heart failure, with a CNS pathology, or after cardiac arrest, there were too few events (i.e., < 40) to assess any meaningful independent associations.
Clinical examination in 1,075 adult patients acutely admitted to the ICU had reasonable prognostic accuracy. Five of the 19 tested clinical examination findings, that is, increased respiratory rate, increased systolic blood pressure, lower core temperature, altered consciousness, and decreased urine output, were independently associated with 90-day mortality. The predictive and discriminative value of a simple clinical examination approached that of the SAPS-II and APACHE-IV and outperformed the SOFA score.
In line with previous studies, we found that clinical signs reflecting cerebral, renal, and skin hypoperfusion were independently associated with mortality in the critically ill (5611). In our data, severe skin mottling had a suggestively significant association with an OR of 2.48 (95% CI, 1.13–5.44), whereas others who assessed the persistence of skin mottling over time found a stronger association with an OR of 16 (95% CI, 11–1,568) (19) and an OR of 3.29 (95% CI, 2.08–5.19) (4). The independent association of decreased urine output with 90-day mortality confirms findings from the FINNAKI studies (531). Similar to the modified early warning score, an altered consciousness regardless of sedation significantly predicted mortality in the critically ill (32). All abovementioned variables may reflect the severity of critical illness on the first day of ICU admission and as such identify patients at higher risk for circulatory failure and mortality. An alternative explanation might be that patients with these clinical signs are treated differently.
The reasonable performance of our prediction models on 90-day mortality is in line with previous models derived from similar cohorts (33). All prognostic scores performed worse than expected from previous literature (1415). The inclusion criteria of the SICS-I may explain this discrepancy: we studied 90-day mortality in patients acutely admitted to the ICU, whereas most prognostic scores perform best in evaluating in-hospital mortality or specific populations such as patients with trauma or suspected infection (3435). The use of an unselected population may produce unbiased risk estimates and increases external validity (24). The main disadvantage of this approach is that average associations may be neutral or balanced out by different characteristics in different subgroups. The secondary analyses in clinically different subgroups were conducted to explore such associations: for example, a high systolic blood pressure was no longer independently associated with mortality in patients requiring vasopressors or in patients with septic shock (eTable 9, Supplemental Digital Content 2, http://links.lww.com/CCM/E746).
Implications and Generalizability
The SICS-I provides evidence that a thorough clinical examination conducted on the first day of ICU admission may be used to obtain a rough estimation of 90-day mortality. By establishing the prognostic value of 19 clinical examination findings, we set the first step for a parsimonious clinical examination, that is, the fewest number of clinical signs that yield the most prognostic value (36). These simple and easily obtainable clinical variables may better inform physicians in their clinical decision making. The examinations were conducted within 24 hours of ICU admission, usually in the morning, and after primary resuscitation efforts. There was no prespecified moment in time for the examination, which may decrease generalizability of the results. Nonetheless, this research practice does reflect daily clinical care where most patients are routinely assessed in the morning, regardless of the time that has passed since ICU admission.
The dynamic care process of the critically ill patient may limit an accurate prediction of 90-day mortality with a single clinical examination or a prognostic score, which reflects a baseline mortality risk based on medical history and findings from the first 24 hours of ICU admission. The clinical status and treatment of a critically ill patient change frequently, and repeated clinical examinations might predict the individual patient prognosis more accurately. Previous studies have already shown that prolonged mechanical ventilation with high pressures or persistently low blood pressures, skin mottling, a decrease in urine output and increasing central-to-peripheral temperature gradients have strong associations with mortality (45937–39). Our clinical examination was limited to a single time point, which could explain why not all these common prognostic variables were also statistically significant in the SICS-I. Future research should study the variation of clinical examination and associated interventions over time to assess its prognostic value (40).
Strengths and Limitations
The SICS-I was an unselected, single-center cohort of consecutive ICU patients, and its findings require external validation in an independent cohort. To address this limitation, we assessed the robustness of our findings by adjusting for multiple outcomes, conducting multiple imputations and sensitivity analyses, and internally validating each predictive variable by bootstrap sampling. We evaluated a heterogeneous ICU population, and certain prognostic associations may be more pronounced in patient subgroups, which is why we studied clinically relevant subgroups in our secondary analyses. Our findings do not apply to pediatric or electively admitted patients.
The clinical examination findings collected in our study were not shared with caregivers. However, some of these findings (i.e., blood pressure and heart rate) were also assessed by caregivers and may have informed subsequent treatment decisions or were influenced by their interventions. The predictive value of a clinical variable measured at baseline therefore included the value of this variable combined with the subsequent intervention(s) to correct such a value. Since treatment strategies between physicians and countries differ, this fact may explain why different studies identify different predictors of mortality, in addition to population differences and other confounders. The prognostic value of clinical variables will be more transparent in a randomized setting where interventions are given based on different clinical treatment targets (4142).
A simple clinical examination, which can be performed in any critically ill patient in any setting, has reasonable discriminative value for assessing 90-day mortality in a single-center cohort of acutely admitted ICU patients. In this study, a single, protocolized clinical examination had similar prognostic abilities compared with the SAPS-II and APACHE-IV and outperformed the SOFA score.
We would like to thank Chris H.L. Thio from the Department of Epidemiology of the University Medical Center Groningen for helping with the multiple imputations of our missing values. The SICS Study Group members include the following: project leaders: Geert Koster, MD; Frederik Keus, MD, PhD; and Iwan C.C. van der Horst, MD, PhD; research coordinator: Willem Dieperink, PhD; researchers who conducted patient inclusions: Roos Bleijendaal, MD; Yasmin F. Cawale, MD; Ramon P. Clement, MD; Devon Dijkhuizen, BSc; Ruben J. Eck, MD; Bart Hiemstra, MD; Anja Haker, BSc; Casper D.H. Hilbink, MD; Thomas Kaufmann, MD; Martiene Klasen; MD, Manon Klaver, MD; Laura J. Schokking, BSc; Victor W. Sikkens, MD; Madelon Vos, MD; Justin Woerlee, MD; and Renske Wiersema, BSc.
1. Sakr Y, Vincent JL, Ruokonen E, et al.; Sepsis Occurrence in Acutely Ill Patients Investigators: Sepsis and organ system failure are major determinants of post-intensive care unit mortality
. J Crit Care 2008; 23:475–483
2. Vincent JL, Marshall JC, Namendys-Silva SA, et al.; ICON investigators: Assessment of the worldwide burden of critical illness
: The intensive care over nations (ICON) audit. Lancet Respir Med 2014; 2:380–386
3. Macedo E, Malhotra R, Bouchard J, et al. Oliguria is an early predictor of higher mortality
in critically ill patients. Kidney Int 2011; 80:760–767
4. Coudroy R, Jamet A, Frat JP, et al. Incidence and impact of skin mottling over the knee and its duration on outcome in critically ill patients. Intensive Care Med 2015; 41:452–459
5. Vaara ST, Parviainen I, Pettilä V, et al.; FINNAKI Study Group: Association of oliguria with the development of acute kidney injury in the critically ill. Kidney Int 2016; 89:200–208
6. Houwink AP, Rijkenberg S, Bosman RJ, et al. The association between lactate, mean arterial pressure, central venous oxygen saturation and peripheral temperature and mortality
in severe sepsis: A retrospective cohort analysis. Crit Care 2016; 20:56
7. Hiemstra B, Eck RJ, Keus F, et al. Clinical examination
for diagnosing circulatory shock. Curr Opin Crit Care 2017; 23:293–301
8. Cecconi M, De Backer D, Antonelli M, et al. Consensus on circulatory shock and hemodynamic monitoring. Task force of the European Society of Intensive Care Medicine. Intensive Care Med 2014; 40:1795–1815
9. Dünser MW, Takala J, Ulmer H, et al. Arterial blood pressure during early sepsis and outcome. Intensive Care Med 2009; 35:1225–1233
10. Varpula M, Tallgren M, Saukkonen K, et al. Hemodynamic variables related to outcome in septic shock. Intensive Care Med 2005; 31:1066–1071
11. Champion HR, Sacco WJ, Hannan DS, et al. Assessment of injury severity: The triage index. Crit Care Med 1980; 8:201–208
12. Ait-Oufella H, Bige N, Boelle PY, et al. Capillary refill time exploration during septic shock. Intensive Care Med 2014; 40:958–964
13. de Moura EB, Amorim FF, da Cruz Santana AN, et al. Skin mottling score as a predictor of 28-day mortality
in patients with septic shock. Intensive Care Med 2016; 42:479–480
14. Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993; 270:2957–2963
15. Zimmerman JE, Kramer AA, McNair DS, et al. Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality
assessment for today’s critically ill patients. Crit Care Med 2006; 34:1297–1310
16. Schriger DL, Baraff L. Defining normal capillary refill: Variation with age, sex, and temperature. Ann Emerg Med 1988; 17:932–935
17. Curley FJ, Smyrnios NA. Irwin RS, Cerra FB, Rippe JM. Routine monitoring of critically ill patients. In: Intensive Care Medicine. 2003, pp Philadelphia, Lippincott Williams & Wilkins, 250–270
18. Lima A, Jansen TC, van Bommel J, et al. The prognostic value of the subjective assessment of peripheral perfusion in critically ill patients. Crit Care Med 2009; 37:934–938
19. Ait-Oufella H, Lemoinne S, Boelle PY, et al. Mottling score predicts survival in septic shock. Intensive Care Med 2011; 37:801–807
20. Hiemstra B, Nolte IM, Tio CH, et al. Detailed statistical analysis plan of the Simple Intensive Care Studies-I (SICS-I). ClinicalTrials.gov
. 2000 Feb 29 - Bethesda, MD, National Library of Medicine (US). Identifier NCT00287391. Available at: https://clinicaltrials.gov/ProvidedDocs/24/NCT02912624/SAP_000.pdf
. Accessed July 10, 2019
21. Little RJA. A test of missing completely at random for multivariate data with missing values. J Am Stat Assoc 1988; 83:1198–1202
22. White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med 2011; 30:377–399
23. Jakobsen JC, Gluud C, Wetterslev J, et al. When and how should multiple imputation be used for handling missing data in randomised clinical trials—A practical guide with flowcharts. BMC Med Res Methodol 2017; 17:162
24. Labarère J, Renaud B, Bertrand R, et al. How to derive and validate clinical prediction models for use in intensive care medicine. Intensive Care Med 2014; 40:513–527
25. Afessa B, Gajic O, Keegan MT. Severity of illness and organ failure assessment in adult intensive care units. Crit Care Clin 2007; 23:639–658
26. Azen R, Traxel N. Using dominance analysis to determine predictor importance in logistic regression. J Educ Behav Stat2009; 34:319–347
27. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 1988; 44:837–845
28. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27:157–172; discussion 207
29. Hiemstra B, Eck RJ, Koster G, et al.; SICS Study Group: Clinical examination
, critical care ultrasonography and outcomes in the critically ill: Cohort profile of the Simple Intensive Care Studies-I. BMJ Open 2017; 7:e017170
30. Jakobsen JC, Wetterslev J, Winkel P, et al. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Med Res Methodol 2014; 14:120
31. Varis E, Pettilä V, Poukkanen M, et al.; FINNAKI Study Group: Evolution of blood lactate and 90-day mortality
in septic shock. A post hoc analysis of the FINNAKI study. Shock 2017; 47:574–581
32. Subbe CP, Kruger M, Rutherford P, et al. Validation of a modified early warning score in medical admissions. QJM 2001; 94:521–526
33. Granholm A, Perner A, Krag M, et al. Development and internal validation of the Simplified Mortality
Score for the Intensive Care Unit
(SMS-ICU). Acta Anaesthesiol Scand 2018; 62:336–346
34. Boyd CR, Tolson MA, Copes WS. Evaluating trauma care: The TRISS method. Trauma score and the injury severity score. J Trauma 1987; 27:370–378
35. Redfern OC, Smith GB, Prytherch DR, et al. A Comparison of the quick sequential (sepsis-related) organ failure assessment score and the national early warning score in non-ICU patients with/without infection. Crit Care Med 2018; 46:1923–1933
36. Sackett DL. The rational clinical examination
. A primer on the precision and accuracy of the clinical examination
. JAMA 1992; 267:2638–2644
37. Damuth E, Mitchell JA, Bartock JL, et al. Long-term survival of critically ill patients treated with prolonged mechanical ventilation: A systematic review and meta-analysis. Lancet Respir Med 2015; 3:544–553
38. Su L, Zhang R, Zhang Q, et al. The effect of mechanical ventilation on peripheral perfusion index and its association with the prognosis of critically ill patients. Crit Care Med 2019; 47:685–690
39. van Genderen ME, Bartels SA, Lima A, et al. Peripheral perfusion index as an early predictor for central hypovolemia in awake healthy volunteers. Anesth Analg 2013; 116:351–356
40. Wiersema R, Castela Forte JN, Kaufmann T, et al. Observational study
protocol for repeated clinical examination
and critical care ultrasonography within the simple intensive care studies. J Vis Exp 2019; 143:e58802
41. Pettilä V, Merz T, Wilkman E, et al. Targeted tissue perfusion versus macrocirculation-guided standard care in patients with septic shock (TARTARE-2S): Study protocol and statistical analysis plan for a randomized controlled trial. Trials 2016; 17:384
42. Hjortrup PB, Haase N, Bundgaard H, et al.; CLASSIC Trial Group; Scandinavian Critical Care Trials Group: Restricting volumes of resuscitation fluid in adults with septic shock after initial management: The CLASSIC randomised, parallel-group, multicentre feasibility trial. Intensive Care Med 2016; 42:1695–1705