Journal Logo

Feature Articles

Performance of a Clinical Decision Support Tool to Identify PICU Patients at High Risk for Clinical Deterioration*

Dewan, Maya MD, MPH1,2,3; Muthu, Naveen MD4; Shelov, Eric MD, MBI4; Bonafide, Christopher P. MD, MSCE4; Brady, Patrick MD, MSc1,3,5; Davis, Daniela MD, MSCE6; Kirkendall, Eric S. MD, MBI5,7; Niles, Dana MS6; Sutton, Robert M. MD, MSCE6; Traynor, Danielle RN6; Tegtmeyer, Ken MD1,2; Nadkarni, Vinay MD, MS6; Wolfe, Heather MD, MSHP6

Author Information
Pediatric Critical Care Medicine: February 2020 - Volume 21 - Issue 2 - p 129-135
doi: 10.1097/PCC.0000000000002106


In children’s hospitals, most cardiopulmonary resuscitation (CPR) events occur in PICUs (1). Many children who require CPR die during or shortly after the event, and those who survive are often left with new disabilities (2–6). Identifying the children most likely to require CPR is important because evidence-based interventions currently exist to improve their outcomes. Examples include “just in time” CPR training, team-based simulation, and bundles for tracheal intubation (7–14).

Our conceptual framework for identifying and improving outcomes for the PICU patients most likely to require CPR is based on the principles of risk mitigation through situational awareness (15). Situational awareness is the ability to monitor and recognize cues that increase the awareness of what is happening around you, integrate information to develop a comprehensive picture of the current state, and extrapolate forward to determine if the knowledge obtained will adversely influence the situation immediately or in the near future (16,17). The improvement of situational awareness within the PICU and development of mitigation plans to prevent CPR events is predicated on the accurate early identification of high-risk patients.

The baseline method used to identify PICU patients most likely to require CPR at the study hospitals was based on clinician intuition alone; patients that were of concern to attending physicians were identified followed by discussion of these patients bid at situational awareness huddles. This approach had a poor positive predictive value (PPV) and a problematic number needed to screen (NNS) of 37–45 patients for each clinical deterioration event, defined as a cardiac arrest or a code bell activation with response of the unit code team. NNS is analogous to number needed to treat. This number signifies the number of patients labeled as high risk (potentially leading providers to institute mitigation plans or receive just in time training) for each patient who experienced a clinical deterioration event. As a first step to improve the identification of high-risk PICU patients, we developed a paper checklist of factors with excellent sensitivity and specificity and a NNS of six patients for each clinical deterioration event (18). Despite the accuracy of the paper checklist, we found that a major barrier to implementation was the time needed to manually screen the criteria for each patient.

Due to this implementation barrier, we shifted our focus to adapting the paper checklist into an automated electronic health record (EHR)-based clinical decision support tool, the PICU Warning Tool. Our aim was to examine the retrospective test characteristics of the PICU Warning Tool with a pragmatic approach to the tradeoffs between predictive accuracy and the effort required to implement.


Study Design

We performed a retrospective observational cohort study to evaluate the performance of a replicated version of the PICU Warning Tool to predict clinical deterioration events in PICU patients within a 24-hour window following a positive high-risk screen. The Institutional Review Boards of Cincinnati Children’s Hospital Medical Center (CCHMC) and Children’s Hospital of Philadelphia (CHOP) approved this study.


The study PICUs are tertiary care pediatric medical/surgical ICUs with 55 and 35 beds respectively and combined yearly admissions of over 6,000 patients/yr with a baseline clinical deterioration rate of 2–3% of all PICU patients. The cardiac surgical ICU and neonatal ICU were excluded.

Selection of the Cohort

The study included all patients admitted to the PICU at CHOP and CCHMC from July 1, 2014, to June 30, 2015, the year prior to the initiation of any focused situational awareness work at either institution. No patients were excluded.

The primary exposure of interest was determination of high-risk status during PICU admission via the PICU Warning Tool. The primary outcome of interest was clinical deterioration event within 24 hours of a positive screen. The date and time of the deterioration event was used as the index time point. Only the initial deterioration event was included for each patient as the goal of the PICU Warning Tool was to establish initial situational awareness for high-risk patients. This also accounted for multiple sampling.

Identification of High-Risk Patients

The primary exposure of interest was high-risk patient status as determined by the automated PICU Warning Tool. The original paper-based checklist, on which the PICU Warning Tool is based, included 15 equally weighted single parameter high-risk patient factors that predicted clinical deterioration events (18). We began the process with a review of the paper checklist to determine if the items were computable and translatable into an automated clinical decision support tool. Some criteria were excluded because the data were not reliably recorded as discrete elements, or were not documented at all, such as “provider intuition.” Although documentation fields could be added to the EHR to capture these missing elements, we chose to build a system without requiring any additional clinical documentation in order to meet the clinical decision support best practice of integration into existing workflows (19). Due to these limitations, we excluded five of the original criteria and modified others (Table 1). We added a criterion for severe cardiac dysfunction in an attempt to capture an additional high-risk population that was lost due to the excluded criteria; we have further described this process and its adherence to clinical decision support best practices in a prior manuscript (20).

Review of Original Paper Checklist Criteria and Translation to PICU Warning Tool

We replicated the predictions of the real-time PICU Warning Tool by retrospectively querying the institutional data warehouse to identify all patients that would have flagged as high risk by the PICU Warning Tool, as well as the specific criteria that caused the patient to be identified as high risk. The EHR was screened hourly for patients meeting high-risk criteria per the established definitions. Patients who met any single criteria were considered high-risk patients.

Identification of Patients Who Experienced Clinical Deterioration

The outcome of interest was clinical deterioration event within 24 hours of a positive high-risk screen by the PICU Warning Tool. Clinical deterioration was defined as a cardiac arrest or a code bell activation with response of the unit code team. These events were identified through multiple means including the use of a quality improvement database in place at the time of the study that tracked CPR and respiratory emergency events (CHOP), an electronic log of code button presses with manual chart review (CCHMC), and a review of all code sheets documented during that time period (CHOP, CCHMC).

Data Analysis

We evaluated the sensitivity, specificity, PPV, and negative predictive value of the overall performance of the PICU Warning Tool to retrospectively identify patients who would experience clinical deterioration within 24 hours of a positive screen. A second sensitivity analysis evaluated the test characteristics of the PICU Warning Tool to predict a clinical deterioration event at any point during a patient’s PICU admission following a positive high-risk flag. Secondary analysis included standard diagnostic test characteristics of individual components of the EHR-based PICU Warning Tool including likelihood ratios (LRs), sensitivity, specificity, PPV, and negative predictive value of each component within the combined data set from both institutions and evaluated separately. Categorical variables between institutions were compared using the chi-square test or Fisher exact test.


A total of 6,233 patients were evaluated between the two centers. There was a total of 233 clinical deterioration events experienced by 154 individual patients (2.5%). At CCHMC, 3.7% of patients (83/2,241 total patients) experienced a clinical deterioration event and 21.6% (485/2,241 total patients) met high-risk criteria during their PICU stay. At CHOP, 1.8% of patients (71/3,992 total patients) experienced clinical deterioration and 9.6% (385/3,992 total patients) met high-risk criteria during their PICU stay (Fig. 1). Both the number of clinical deterioration events and the percent of patients meeting high-risk criteria were significantly different at the two institutions (p < 0.0001 for both).

Figure 1.
Figure 1.:
Cohort identification. CCHMC = Cincinnati Children’s Hospital Medical Center, CHOP = Children’s Hospital of Philadelphia.

The percent of patients meeting individual high-risk criteria was also different between centers for three of the criteria—pulmonary hypertension, high mean airway pressure, and profound acidosis (Table 2). The highest frequency criteria was vasoactive shock at both centers followed by profound acidosis and high mean airway pressure.

Percent of Patients Meeting Individual High-Risk Criteria by Center

The overall PPV of the EHR-based tool was 7.1% (95% CI, 5.9–8.6%) with a NNS of 14 patients (95% CI, 12–17) for each index clinical deterioration event. The tool performed with a PPV of 8.0% and a NNS of 13 at CCHMC as compared with CHOP with a PPV of 6.0% and a NNS of 17. All of these test characteristics were inferior to the original paper-based tool with a PPV of 17.3% and an NNS of 6 (Table 3).

Test Characteristics of the Electronic PICU Warning Tool at Each Center and Overall As Compared With the Original Paper-Based Tool

The predictive value of the individual criteria varied. The most predictive of the individual criteria was elevated lactic acidosis with a positive LR of 6 (95% CI, 2.6–13.9), high mean airway pressure had a positive LR of 4.8 (95% CI, 3.3–7.0), and profound acidosis had a positive LR of 3.6 (95% CI, 2.4–5.5) (Fig. 2). The least predictive were extracorporeal membrane oxygenation and hyperkalemia which did not accurately predict any clinical deterioration events followed by intracranial hypertension with a positive LR of 1 (95% CI, 0.2–2.9). Pulmonary hypertension only predicted clinical deterioration at CHOP and renal replacement therapy only predicted clinical deterioration events at CCHMC. Complete test characteristics of the individual criteria are presented in Supplemental Table 1 (Supplemental Digital Content 1,

Figure 2.
Figure 2.:
Positive likelihood ratios with confidence limits for individual criteria and overall for prediction. ECMO = extracorporeal membrane oxygenation.

To predict clinical deterioration at any point following a positive screen during the patient’s PICU admission, the PICU Warning Tool had improved test characteristics with a sensitivity of 60.4% (95% CI, 52.2–68.2%), specificity of 87.2% (95% CI, 86.3–88.0%), PPV 10.7% (9.4–12.1%), negative predictive value of 98.9% (95% CI, 98.6–99.1%), and a NNS of 10 (95% CI, 9–11).


Using data from two institutions, we evaluated the performance of an automated clinical decision support tool, the PICU Warning Tool, derived from a previously published single-center paper checklist. The key findings of this study were that, with adaptation from paper to automated to ease screening, the sensitivity decreased from 100% to 40% to predict clinical deterioration within 24 hours of a positive high-risk screen, and the NNS increased from 6 to 14. As a practical example, using the PICU Warning Tool rather than the paper checklist, a 30-bed PICU with yearly admissions of 2,000 patients would have on average five high-risk patients per week requiring mitigation planning and just in time training. However, this would only train the teams of 40% of patients who would experience clinical deterioration within 24 hours and 60% of patients who would experience clinical deterioration during their PICU admission. Using the paper checklist, this same PICU would have on average one high-risk patient per week requiring just in time training and would train 100% of the teams caring for patients who experience clinical deterioration. Although this example may overestimate the performance of the paper checklist in a real-life setting, as it was only studied over a 3-month period at a single center, it does raise important concerns about the often unstudied conversion from a paper-based clinical decision support tool to an automated one.

To summarize, in adapting the tool from paper to electronic, we eliminated the workload burden of manual screening but markedly reduced the tool’s sensitivity and more than doubled the NNS. Depending on the recommended action associated with screening positive, this could increase providers’ workload burden of developing mitigation plans or delivering just in time training, limiting its feasibility and acceptability. More standard situational awareness improvements around structured communication tools (21), standardized handoffs (22), and huddles (16,23) are likely feasible for most centers with a NNS of 14 patients for each clinical deterioration event. However, more intensive training centered around individual at-risk patients (5,8,24) and team-based simulations (12–14) may not be feasible at this NNS. It is important to note that although we have focused on individual level patient identification and mitigation, prior work demonstrates that with improved situational awareness and a proactive approach to identify high-risk patients, improvements can be seen in prevention of high-risk events at a systems level (23,25–27).

This difference in test characteristics identified here demonstrates the importance of evaluating the translation from paper to automated clinical decision support tools. Optimizing the human-automation interaction may lead to improved outcomes in this context. Prior literature demonstrates that automation does not replace the human activity but changes it in planned and unplanned ways (28). To the extent to which the clinical decision support automation provides the right information at the right time, it can decrease the team workload and improve the team situational awareness (29). Although the performance of the automated tool may be inferior to that of the paper tool, our use of clinical decision support best practices (19,30,31) and our implementation plan for effective automation design (32), may lead to improved patient outcomes despite the higher NNS.

We speculate that the decrease in sensitivity and increase in NNS is likely related to the removal of components that are known to be predictive including high-risk intubations (33) and prior clinical deterioration events (34). In addition, provider intuition (i.e., identification of high-risk patients by physician judgment) has been shown to be more predictive than early warning scores alone when completed by attendings (35). As the original evaluation of the paper checklist did not include evaluation of the individual criteria, it was not possible for us to know a priori the impact of removing these elements. It is reasonable to study the addition of these criteria to the future prospective implementation of this tool to see if we are able to improve the sensitivity of the PICU Warning Tool using a combined automated screening component and a manual entry component from those criteria identified during huddle.

This work has several limitations. First, the retrospective evaluation of this automated clinical decision support tool was completed at only two centers, and the frequency of deterioration events at these centers was low. External validation in a larger number of pediatric centers is important prior to widespread adoption. Second, these data were collected retrospectively. Although designed to mimic the real-time queries of the prospective tool, there may have been missed indications due to documentation errors such as incorrect documentation of a patient’s mean airway pressure or diagnosis in the problem list. Last, we only evaluate the first clinical deterioration event of each patient. It is possible the predictability of the tool changes with subsequent events.

We will next implement and study the PICU Warning Tool as a prospective high-risk patient identifier on its impact on clinical deterioration rate. To improve the performance of the tool, we plan to add the missing criteria through manual identification to investigate if the sensitivity of the PICU Warning Tool increases using combined automated screening and manual entry of criteria identified bid during safety huddle. As we decrease the prevalence of clinical deterioration events, the PPV will fall and the NNS of the tool will rise. It will therefore be imperative that we track additional outcomes including situational awareness among team members, use of mitigation plans, and quality of resuscitation response in addition to predictive ability.


An automated clinical decision support translation of a paper checklist for PICU patients at risk for clinical deterioration demonstrates reduced accuracy in prediction with a NNS of 14 rather than 6 despite improved feasibility. We speculate that improved feasibility of identification of high-risk patients using automated EHR warning tools may allow for risk mitigation and just in time preparation and that test characteristics may improve with incorporation of more provider and procedural concerns.


1. Berg RA, Sutton RM, Holubkov R, et al.; Eunice Kennedy Shriver National Institute of Child Health and Human Development Collaborative Pediatric Critical Care Research Network and for the American Heart Association’s Get With the Guidelines-Resuscitation (formerly the National Registry of Cardiopulmonary Resuscitation) Investigators: Ratio of PICU versus ward cardiopulmonary resuscitation events is increasing. Crit Care Med 2013; 41:2292–2297
2. Sutton RM, French B, Niles DE, et al. 2010 American Heart Association recommended compression depths during pediatric in-hospital resuscitations are associated with survival. Resuscitation 2014; 85:1179–1184
3. Cheng A, Lin Y, Nadkarni V, et al. The effect of step stool use and provider height on CPR quality during pediatric cardiac arrest: A simulation-based multicentre study. CJEM 2018; 20:80–88
4. Abella BS, Edelson DP, Kim S, et al. CPR quality improvement during in-hospital cardiac arrest using a real-time audiovisual feedback system. Resuscitation 2007; 73:54–61
5. Niles DE, Nishisaki A, Sutton RM, et al. Improved retention of chest compression psychomotor skills with brief “Rolling Refresher” training. Simul Healthc 2017; 12:213–219
6. Lin Y, Wan B, Belanger C, et al. Reducing the impact of intensive care unit mattress compressibility during CPR: A simulation-based study. Adv Simul (Lond) 2017; 2:22
7. Sutton RM, Niles D, Meaney PA, et al. Low-dose, high-frequency CPR training improves skill retention of in-hospital pediatric providers. Pediatrics 2011; 128:e145–e151
8. Niles D, Sutton RM, Donoghue A, et al. “Rolling Refreshers”: A novel approach to maintain CPR psychomotor skill competence. Resuscitation 2009; 80:909–912
9. Sutton RM, Niles D, Meaney PA, et al. “Booster” training: Evaluation of instructor-led bedside cardiopulmonary resuscitation skill training and automated corrective feedback to improve cardiopulmonary resuscitation compliance of Pediatric Basic Life Support providers during simulated cardiac arrest. Pediatr Crit Care Med 2011; 12:e116–e121
10. Wolfe H, Zebuhr C, Topjian AA, et al. Interdisciplinary ICU cardiac arrest debriefing improves survival outcomes*. Crit Care Med 2014; 42:1688–1695
11. Parker MM, Nuthall G, Brown C 3rd, et al.; Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network: Relationship between adverse tracheal intubation associated events and PICU outcomes. Pediatr Crit Care Med 2017; 18:310–318
12. Knight LJ, Gabhart JM, Earnest KS, et al. Improving code team performance and survival outcomes: Implementation of pediatric resuscitation team training. Crit Care Med 2014; 42:243–251
13. Stewart C, Shoemaker J, Keller-Smith R, et al. Code team training: Demonstrating adherence to AHA guidelines during pediatric code blue activations. Pediatr Emerg Care 2017 Oct 16. [Epub ahead of print]
14. Gilfoyle E, Koot DA, Annear JC, et al. Improved clinical performance and teamwork of pediatric interprofessional resuscitation teams with a simulation-based educational intervention. Pediatr Crit Care Med 2017; 18:e62–e69
15. Brady PW, Wheeler DS, Muething SE, et al. Situation awareness: A new model for predicting and preventing patient deterioration. Hosp Pediatr 2014; 4:143–146
16. Goldenhar LM, Brady PW, Sutcliffe KM, et al. Huddling for high reliability and situation awareness. BMJ Qual Saf 2013; 22:899–906
17. Endsley MR. Measurement of situation awareness in dynamic systems. Hum Factors J Hum Factors Ergon Soc 1995; 37:65–84
18. Niles DE, Dewan M, Zebuhr C, et al. A pragmatic checklist to identify pediatric ICU patients at risk for cardiac arrest or code bell activation. Resuscitation 2016; 99:33–37
19. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: Making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10:523–530
20. Shelov E, Muthu N, Wolfe H, et al. Design and implementation of a pediatric ICU acuity scoring tool as clinical decision support. Appl Clin Inf 2018; 9:576–587
21. Haig KM, Sutton S, Whittington J. SBAR: A shared mental model for improving communication between clinicians. Jt Comm J Qual Patient Saf 2006; 32:167–175
22. Starmer AJ, Sectish TC, Simon DW, et al. Rates of medical errors and preventable adverse events among hospitalized children following implementation of a resident handoff bundle. JAMA 2013; 310:2262–2270
23. Brady PW, Muething S, Kotagal U, et al. Improving situation awareness to reduce unrecognized clinical deterioration and serious safety events. Pediatrics 2013; 131:e298–e308
24. Cheng A, Brown LL, Duff JP, et al. Improving cardiopulmonary resuscitation with a CPR feedback device and refresher simulations (CPR CARES Study): A randomized clinical trial. JAMA Pediatr 2015; 169:137–144
25. Hueckel RM, Turi JL, Cheifetz IM, et al. Beyond rapid response teams: Instituting a “Rover Team” improves the management of at-risk patients, facilitates proactive interventions, and improves outcomes. In: Advances in Patient Safety: New Directions and Alternative Approaches (Vol 3). 2008Rockville, Agency for Healthcare Research and Quality.
26. Sandquist M, Tegtmeyer K. No more pediatric code blues on the floor: Evolution of pediatric rapid response teams and situational awareness plans. Transl Pediatr 2018; 7:291–298
27. McClain Smith M, Chumpia M, Wargo L, et al. Watcher initiative associated with decrease in failure to rescue events in pediatric population. Hosp Pediatr 2017; 7:710–715
28. Nickerson RS. Reviews of human factors and ergonomics, volume I. Ergon Des Q Hum Factors Appl 2011; 14:31
29. Wright MC, Kaber DB. Effects of automation of information-processing functions on teamwork. Hum Factors 2005; 47:50–66
30. Osheroff J, Teich J, Levick D, et al. Improving Outcomes With Clinical Decision Support: An Implementer’s Guide. 2012Second Edition. Chicago, IL, Healthcare Information and Management Systems Society (HIMSS).
31. Wright A, Phansalkar S, Bloomrosen M, et al. Best practices in clinical decision support: The case of preventive care reminders. Appl Clin Inform 2010; 1:331–345
32. Klein G, Woods DD, Bradshaw JM, et al. Ten challenges for making automation a “team player” in joint human-agent activity. IEEE Intell Syst 2004; 19:91–95
33. Graciano AL, Tamburro R, Thompson AE, et al. Incidence and associated factors of difficult tracheal intubations in pediatric ICUs: A report from National Emergency Airway Registry for Children: NEAR4KIDS. Intensive Care Med 2014; 40:1659–1669
34. Gupta P, Pasquali SK, Jacobs JP, et al. Outcomes following single and recurrent in-hospital cardiac arrests in children with heart disease: A report from American Heart Association’s Get With the Guidelines Registry-Resuscitation. Pediatr Crit Care Med 2016; 17:531–539
35. Sweney JS, Poss WB, Grissom CK, et al. Comparison of severity of illness scores to physician clinical judgment for potential use in pediatric critical care triage. Disaster Med Public Health Prep 2012; 6:126–130

cardiopulmonary resuscitation; clinical decision support systems; clinical deterioration; pediatric intensive care units

Supplemental Digital Content

Copyright © 2019 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies