Journal Logo

Online Clinical Investigations

Checklist for Early Recognition and Treatment of Acute Illness and Injury: An Exploratory Multicenter International Quality-Improvement Study in the ICUs With Variable Resources

Vukoja, Marija MD, PhD1,2; Dong, Yue MD3; Adhikari, Neill K. J. MDCM, MSc4–6; Schultz, Marcus J. MD, PhD7–10; Arabi, Yaseen M. MD11; Martin-Loeches, Ignacio MD, PhD12–14; Hache, Manuel MD15; Gavrilovic, Srdjan MD1,2; Kashyap, Rahul MB BS, MBA3; Gajic, Ognjen MD, MSc16; for the Checklist for Early Recognition and Treatment of Acute Illness and Injury (CERTAIN) Investigators of the SCCM Discovery Network Checklist for Early Recognition and Treatment of Acute Illness and Injury (CERTAIN) Investigators MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MD MB, BCh MD MD MD MS MD

Author Information
doi: 10.1097/CCM.0000000000004937


The delivery of high-quality evidence-based medicine is expected to minimize complications and improve outcomes in critically ill patients. However, quality-improvement studies in ICU have yielded conflicting results (1–3). Most were performed in high-income country (HIC) ICUs, and no large-scale global quality-improvement efforts have been undertaken in low- and middle-income countries (LMICs) (4) despite their appeal for resource-limited settings where the burden of critical illness is high, and multiple barriers preclude evidence-based care (5). In LMICs, however, a systematic approach to error prevention with the use of checklist has been shown to improve patients’ safety in surgical theaters (6).

To facilitate timely and accurate best practice delivery in critically ill patients in ICUs with variable resources, an international collaboration developed the Checklist for Early Recognition and Treatment of Acute Illness and Injury (CERTAIN), a clinical decision support tool that provides evidence-based checklists for structured ICU admission and rounding. Based on user-centered design, we developed CERTAIN that is fast, easy to use and applicable in diverse clinical settings (7,8). We hypothesized that care assisted by CERTAIN would improve the processes of care and outcomes of critically ill patients in ICUs with variable resources.


Design and Setting

This was a pragmatic, prospective, before-after quality-improvement study conducted between November 1, 2013, and December 31, 2017. In the preimplementation phase (phase 1, 3 mo or minimum 50 patients per ICU, whatever was reached first), we collected baseline data including ICU characteristics, demographics, baseline severity, comorbidities, limitations on life support interventions, daily care processes, and clinical outcomes of patients admitted to the participating ICUs. Following structured CERTAIN implementation (phase 2), the same variables were collected in the postimplementation phase (phase 3, 6 mo or minimum 150 patients per ICU, whatever was reached first). A detailed description of the methods is published (8). The study was approved by the research ethics boards of all participating hospitals.

The study included ICUs in 10 LMICs and five HICs, across five continents (eTable 1, Supplemental Digital Content 1,; country income status was defined by the World Bank (9). Those ICUs was approached to participate based on previous collaboration with the investigators and through contacts established through members of the European Society of Intensive Care Medicine global working group and American Thoracic Society international health committee. The participating ICUs had variable size (median 12 beds, range, 5–52) and resources; a detailed description is given in eTable 1 (Supplemental Digital Content 1,

Intervention Details and Phases of Implementation

CERTAIN is a web-based decision support tool displaying relevant clinical information with a systematic approach. It incorporates evidence-based checklists, algorithms, and educational modules on performing critical procedures and has been reported previously (8). CERTAIN consists of two modules: an admission module and a rounding module (eFigs. 1–3, Supplemental Digital Content 1, A mobile version and paper version were also provided in case of problems with internet connection. Items of admission and rounding checklists are presented in Figures 1 and 2. The PDF format of the admission checklist, rounding checklist, and individual syndrome cards are also available for download on the CERTAIN website: The tool was tested in a simulated environment before the implementation phase (10,11).

Figure 1.
Figure 1.:
Checklist for Early Recognition and Treatment of Acute Illness and Injury admission module checklist abbreviated (full version is in the supplement, Supplemental Digital Content 2, BP = blood pressure, BUN = blood urea nitrogen, CHF = congestive heart failure, Cr = creatinine, Billi = bilirubin, DNI = do not intubate, DNR = do not resuscitate, ECG = electrocardiogram, Glu = glucose, Hb = hemoglobin, HR = heart rate, INR = international normalized ratio, Lam = lactate, Plt = platelets, RR = respiratory rate, SARS = severe acute respiratory syndrome, SC = subcutaneous, SpO 2 = blood oxygen saturation, TB = tuberculosis, Temp = temperature, UO = urine output, US = ultrasound, V tach = ventricular tachycardia.
Figure 2.
Figure 2.:
Checklist for Early Recognition and Treatment of Acute Illness and Injury rounding module checklist abbreviated (full version is in the supplement, Supplemental Digital Content 3, HEM = hematology, Spont. = spontaneous, vent. = ventilation.

CERTAIN implementation was done through an education program that engaged with local teams including at least three ICU members. After 2–4 weeks of access to online curriculum (slide presentations, published papers), the team conducted a follow-up online remote training via simulation coaching and debriefing (eFig. 4, Supplemental Digital Content 1, (8,11). Following a train-the-trainer session, the local trainer was certified to train their local staff. Once local physicians and nurses completed the training, the participating center proceeded to clinical implementation. The core of the intervention was a structured approach to admission and daily rounding by using the checklist to prompt clinicians to follow best care practices. At the discretion of individual ICUs, a dedicated ICU team member used the CERTAIN checklist to prompt other team members to review checklist items within the tool during admission and rounding.

To enter the postimplementation phase, all centers were required to meet greater than 80% adherence to both the rounding and admission checklists for 4 consecutive weeks during the implementation phase. Adherence was tracked weekly through self-reported online form. CERTAIN was implemented sequentially in each ICU, so that by the end of the study, all ICUs had received the intervention (eFigs. 5 and 6, Supplemental Digital Content 1,

Patients and Data Collection

In both pre- and postimplementation phases, data were collected prospectively on consecutive patients (≥ 18 yr) admitted to the study ICUs. Readmissions, patients who were admitted for simple monitoring, patients who had a planned ICU admission for routine postoperative surveillance lasting less than 24 hours after uncomplicated surgery (this and previous criterion as defined by the site investigators), and patients who were transferred from an ICU outside the participating hospital were excluded.

Data were collected using the secure Research Electronic Data Capture system (12). All data collectors were trained by the coordinating team and via online tutorials. The coordinating center audited data collection and met investigators regularly via monthly videoconference.

Study Outcomes and Definitions

The study’s coprimary outcomes include the following: 1) nonadherence to daily processes of care and 2) ICU and hospital length of stay (LOS). Secondary outcomes include 1) ICU mortality, 2) hospital mortality, 3) 28-day mortality, 4) blood product utilization, 5) infections, and 6) survival time. We observed the nonadherence to processes on days 0 (the day of ICU admission), 1, 2, 3, 7, 14, and 21, as long as the patient remained in the ICU. Nonadherence was recorded for the following 10 daily care processes: 1) deep vein thrombosis (DVT) prophylaxis, 2) peptic ulcer prophylaxis, 3) oral care, 4) head of bed (HOB) elevation to at least 30° above horizontal, 5) spontaneous breathing trial (SBT), 6) family conference discussion (for patients on mechanical ventilation), 7) assessment of central venous catheter (CVC) removal (for patients with CVC), 8) assessment for urinary catheter removal (for patients with urinary catheter), 9) assessment to continue or discontinue current antimicrobials (for patients receiving antimicrobial therapy), and 10) assessment to continue or discontinue current sedation (for patients receiving sedation). Nonadherence to these daily processes of care was quantified as an incidence rate, defined as the ratio of the number of observations of not receiving basic care procedures among eligible patients (events) to the number of total observations of in which the specific intervention was indicated (exposure) expressed per 1,000 days of specific intervention (eTable 3, Supplemental Digital Content 1,

The published study protocol defined the primary outcome as adherence to care practices; we changed this to nonadherence because it was a “rare” event, and we used Poisson regression for analysis. Modifications from the original protocol were done before data analysis and are presented in eTable 7 (Supplemental Digital Content 1,

Blood product utilization was measured by documenting RBC, fresh frozen plasma (FFP), and platelet transfusions. For infections, we recorded incidence rates of ventilator-associated pneumonia (VAP), catheter-related bloodstream infections (CRBSIs), and urinary catheter infections. Although we provided standard operating definitions for the infection outcomes (13), the final diagnosis was based on medical records; we did not request microbiologic data to validate this diagnosis. For ICU and hospital LOS, the interval between the ICU admission date and the ICU or hospital discharge date was calculated in days. Patient status at ICU, hospital discharge, and at 28 days was assessed by reviewing the medical records and by postdischarge telephone contact (for discharged patients), respectively.

We did not assess adherence to some of the prespecified outcomes (e.g., shock resuscitation and sepsis treatment) due to limited resources for real-time data collection (eTable 5, Supplemental Digital Content 1, The incidence rates of VAP, CRBSI, and urinary catheter infections, and RBC, FFP, and platelet transfusions were added after the study protocol publication but before study completion.

Statistical Analysis

Detailed statistical methods are presented in the Supplementary Digital Content (Supplemental Digital Content 1, Categorical data are reported as counts and proportions and continuous data as mean (sd) or median (interquartile range [IQR]).

The study encompasses six sets of clinical outcomes that were measured before and after CERTAIN implementation. These outcomes were studied in all centers. Assessments of CERTAIN’s effect were performed by developing linear, logistic, and Poisson models as the data required. These models were used to estimate ratios that expressed the effect of CERTAIN. The model’s variables included CERTAIN implementation, center (to account for the clustering of patients within center), and patient characteristics (to account for imbalances between the pre/post implementation phases). For example, in the model for mortality, the regression coefficient (sd) for the CERTAIN implementation variable was used to estimate the adjusted odds ratio (aOR) and its CI. The p values are based on the likelihood ratio test using nested and full models. All tests are two sided, and a p value of less than 0.05 was considered statistically significant. All statistical analysis was conducted using the R statistical software, Version 3.4.(R Foundation for Statistical Computing, Vienna, Austria. URL

Revised Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) were used as a framework for conducting the study (14,15).


Study Flow and Patient Characteristics

A total 55 centers were approached, of which 46 accepted to participate and 34 centers completed the study. A total of 4,256 patients completed the study: 1,447 in preimplementation and 2,809 in postimplementation phases (Fig. 3). Twelve centers did not complete the study: eight centers dropped out during preimplementation phase, three during implementation phase, and one center in final postimplementation phase. Reasons for drop-out included lack of resources and change in leadership leading to a shift in priorities. Baseline demographics and severity of illness of patients in pre- and postimplementation phases were similar. More patients in the postimplementation phase were admitted from home, whereas patients in preimplementation phase had more comorbidities and more limitations of life support interventions at ICU admission (Table 1). The median duration of ICU study participation was 96 weeks (IQR, 54–139 wk) (eTable 4, Supplemental Digital Content 1,

TABLE 1. - Baseline Characteristics of Patients in Pre- and Postimplementation Groups
Pre Implementation Post Implementation p
N = 1,447 N = 2,809
Age, median (IQR) 62 (46–75) 62 (47–74) 0.92
Gender, female, n (%) 588 (40.6) 1169 (41.6) 0.56
Hospital admission source, n (%) < 0.001
 Home 680 (47.0) 1605 (57.2)
 Nursing home 35 (2.4) 39 (1.4)
 ED 444 (30.7) 791 (28.2)
 Outside hospital ED 228 (15.8) 280 (10.0)
 Other 59 (4.1) 91 (3.2)
Life support limitation, n (%) 150 (10.4) 137 (4.9) < 0.001
Sequential Organ Failure Assessment score, median (IQR) 6 (4–10) 6 (4–9) 0.07
Mechanical ventilation, n (%) 822 (57.0) 1680 (60.4) 0.04
Use of vasoactive medications, n (%) 583 (40.7) 1111 (40.1) 0.75
Antimicrobial medication, n (%) 1198 (83.6) 2256 (81.8) 0.17
Comorbidities, n (%)
 Congestive heart failure 270 (18.7) 376 (13.4) < 0.001
 Cardiac arrhythmias 201(13.9) 296 (10.5) 0.001
 Valvular disease 93 (6.4) 110 (3.9) < 0.001
 Pulmonary circulation disorders 135 (0.9) 193 (0.7) 0.05
 Hypertension 643 (44.4) 1204 (42.9) 0.34
 Neurologic disorders 215 (14.9) 274 (9.7) < 0.001
 Diabetes 384 (26.5) 713 (25.3) 0.66
 Hypothyroidism 48 (3.3) 111 (3.9) 0.34
 Renal failure 229 (15.8) 359 (12.8) 0.007
 Liver disease 111 (7.7) 142 (5.1) < 0.001
 Peptic ulcer disease excluding bleeding 40 (2.7) 48 (1.7) 0.03
 AIDS 9 (0.6) 12 (0.4) 0.53
 Lymphoma 15 (1.0) 49 (1.7) 0.10
 Metastatic cancer 64 (4.4) 140 (5.0) 0.46
 Solid tumor without metastasis 80 (5.5) 196 (7.0) 0.08
 Rheumatoid arthritis/collagen vascular disease 29 (2.0) 55 (2.0) 1.00
 Coagulopathy 54 (3.7) 81 (2.9) 0.16
 Obesity 124 (8.6) 123 (4.4) < 0.001
 Weight loss 62 (4.3) 79 (2.8) 0.01
 Fluid and electrolyte disorders 169 (11.7) 238 (8.5) < 0.001
 Blood loss anemia 50(3.5) 102 (3.1) 0.84
 Deficiency anemia 63 (4.3) 61 (2.2) < 0.001
 Alcohol abuse 65 (4.5) 90 (3.2) 0.04
 Drug abuse 23 (1.6) 46 (1.6) 1.00
 Psychosis 24 (1.7) 40 (1.4) 0.64
 Depression 59 (4.1) 75 (2.7) 0.02
 Other 390 (26.9) 703 (25.0) 0.19
 None 88 (6.1) 362 (12.9) < 0.001
ED = emergency department, IQR = interquartile range.
p based on χ2 when comparing categorical variables or Kruskal-Wallis rank-sum test when comparing quantitative variables.

Figure 3.
Figure 3.:
Study flow. IRB = institutional review board.

Primary Outcomes

Following implementation of CERTAIN, there was a significant decrease in nonadherence to eight of 10 targeted daily care processes compared with preimplementation period (Table 2 and Fig. 4). There were 2,840 mechanically ventilated patients resulting in 10,069 observation days (3453 in preimplementation and 6,616 in postimplementation phases).

Figure 4.
Figure 4.:
Checklist for Early Recognition and Treatment of Acute Illness and Injury (CERTAIN): change in nonadherence to daily care processes before and after CERTAIN implementation. “Mech vent” denotes that those care practices were only measured in mechanically ventilated patients. CVC = central venous catheter, DVT = deep vein thrombosis, HOB = head of bed, SBT = spontaneous breathing trial, UC = urinary catheter.

Compared with the preimplementation period, the incidence of nonadherence to DVT and peptic ulcer prophylaxis was significantly reduced in the postimplementation period (adjusted incidence rate ratio [aIRR], 0.74; 95% CI, 0.68–0.81 and 0.46; 95% CI, 0.38–0.57, respectively). There was a decrease in nonadherence to daily assessment of CVCs (from 647 to 566 per 1,000 CVC days; aIRR, 0.85; 95% CI, 0.81–0.90) and urinary catheters (674 vs 533 per 1,000 urinary catheter days; aIRR, 0.84; 95% CI, 0.80–0.88) (Table 2). There was also an improvement in adherence to daily assessment of medications. The incidence rate for not assessing the need for antimicrobials was reduced from 340 to 219 per 1,000 observation days (aIRR, 0.66; 95% CI, 0.62–0.71). Omissions in daily assessment of sedation need decreased from 359 to 212 per 1,000 observation days (aIRR, 0.62; 95% CI, 0.57–0.67) (Table 2). The intervention was also associated with reduced nonadherence to the best practices of conducting daily SBTs and holding family conferences. There was no change in nonadherence to daily oral care and HOB elevation.

TABLE 2. - The Incidence Rates of Nonadherence to Daily Care Processes Before and After Checklist for Early Recognition and Treatment of Acute Illness and Injury Implementation
Total 4,256 Pre Intervention, N = 1,447 Post Intervention, N = 2,809
Observed Event Incidence Rate (95% CI) Observation Days, n Observed Events, n Patients, n Incidence Rate (95% CI) Observation Days, n Observed Events, n Patients, n Adjusted for Center Effects Incidence Rate Ratio p
Mechanical ventilation
Per 1,000 ventilator days Per 1,000 ventilator days
 No deep vein thrombosis prophylaxis 268 (251–286) 3,453 926 966 200 (189–211) 6,616 1,322 1,874 0.74 (0.68–0.81) < 0.001
 No peptic ulcer prophylaxis 55 (48–64) 3,453 193 966 28 (24–32) 6,616 185 1,874 0.46 (0.38–0.57) < 0.001
 No documented assessment of spontaneous breathing trial 563 (539–589) 3,453 1,946 966 450 (434–467) 6,616 2,980 1,874 0.81 (0.76–0.86) < 0.001
 No documented family conference/ discussion 416 (395–438) 3,453 1,437 966 361 (347–376) 6,616 2,391 1,874 0.86 (0.81–0.92) < 0.001
 No daily oral care 45 (38–52) 3,453 155 966 42 (38–48) 6,616 281 1,874 0.94 (0.77–1.16) 0.60
 No head of bed elevation at 30° 39 (33–46) 3,453 135 966 37 (32–42) 6,616 244 1,874 1.00 (0.81–1.23) 0.98
Per 1,000 CVC days Per 1,000 CVC days 1,896
 No documented assessment for CVC removal 647 (622–673) 3,922 2,538 990 566 (549–583) 7,308 4,136 1,896 0.85 (0.81–0.90) < 0.001
Urinary catheters,
 No documented assessment for urinary catheter removal (per 1,000 urinary catheter days) 674 (653–697) 5,268 3,553 1,322 564 (549–578) 10,228 5,766 2,609 0.84 (0.80–0.88) < 0.001
Antimicrobials and sedation
Per 1,000 d of use Per 1,000 d of use
 No documented assessment to continue or discontinue current antimicrobials 340 (324–356) 5,168 1,758 1,310 219 (209–228) 9,497 2,079 2,498 0.66 (0.62–0.71) < 0.001
 No documented assessment to continue or discontinue current sedation 359 (338–381) 3,049 1,092 941 212 (201–224) 5,436 1,251 1,895 0.62 (0.57–0.67) < 0.001
CVC = central venous catheter.

The mean ICU LOS was 6.5 days (sd 3.0 d) in pre- and 5.8 days (sd 2.8 d) days in postimplementation phases, and the mean hospital LOS was 14.9 days (sd 3.1 d) in preimplementation versus 14.6 days (sd 2.8 d) in postimplementation phases (adjusted ratio of geometric means [aRoGM], 0.86; 95% CI, 0.80–0.92, and 0.92; 95% CI, 0.85–0.97, respectively). Results were similar after adjusting for baseline patient covariates, income level, and center (aRoGM, 0.86; 95% CI, 0.80–0.92 and 0.92; 95% CI, 0.86–0.98, respectively) (Table 3).

Secondary Outcomes

The overall (combined pre- and postimplementation phases) ICU, hospital, and 28-day mortality were 24.8%, 29.8%, and 31.7%. There was a significant ICU mortality reduction from 28.8% (preimplementation phase) to 23.5% (postimplementation phase) (aOR, 0.84; 95% CI, 0.71–0.99). Similarly, the hospital mortality reduction was 34.2% to 28.8% (aOR, 0.81; 95% CI, 0.69–0.95). Finally, the 28-day mortality reduction was 36.9% to 30.9% (aOR, 0.82; 95% CI, 0.70–0.95). The reductions in ICU, hospital, and 28-day mortality remained significant after adjusting for baseline imbalances, income status, and center characteristics (aOR, 0.74; 95% CI, 0.63–0.87; 0.74, 0.64–0.86; 0.70, 0.66–0.89, respectively) (Table 3).

TABLE 3. - The patient outcomes before and after Checklist for Early Recognition and Treatment of Acute Illness and Injury Implementation
Total 4,256 Pre Intervention, N = 1,447 Post Intervention, N = 2,809 Effect Measure (95% CI) p
Length of stay
Geometric Mean (sd), d Geometric Mean (sd), d Adjusted Ratio of Geometric Means
 ICU 6.5 (3.0) 5.8 (2.8) 0.86 (0.80–0.92) < 0.001
 Hospital 14.9 (3.1) 14.6 (2.8) 0.91 (0.85–0.97) 0.003
Mortality (%) Death Patient, n Mortality (%) Death Patient, n Adjusted OR p
 ICU mortality 28.8 (26.5–31.2) 406 1,409 23.5 (21.9–25.1) 648 2,762 0.84 (0.71–0.99) 0.041
 Hospital mortality 34.2 (31.7–36.7) 477 1,396 28.8 (27.1–30.5) 793 2,757 0.81 (0.69–0.95) 0.01
 28 d mortality 36.9 (34.4–39.5) 504 1,367 30.9 (29.2–32.7) 845 2,731 0.82 (0.70–0.95) 0.012
OR = odds ratio.

From 2,886 patients with CVCs (68% of total; 990 (68%) in preimplementation and 1,896 (67%) in postimplementation phases), 97 patients (3.4%) had CRBSI. There was a reduction of CRBSI in postimplementation phase (4.4% vs 2.8%; aOR, 0.58; 95% CI, 0.38–0.90). Urinary catheter infection was present in 58 of 1,322 patients (4.39%) in preimplementation and 134 of 2,609 (4.02%) in postimplementation phases (aOR, 0.87; 95% CI, 0.62–1.23). The risk of VAP remained unchanged before and after CERTAIN implementation (18.7% vs 20.0%; aOR, 1.02; 95% CI, 0.80–1.28). The incidence rate of RBC and platelet transfusion did not differ between the study phases. The incidence rate of FFP transfusion was lower in postimplementation compared with preimplementation phase (aIRR, 0.75; 95% CI, 0.66–0.85) (eTable 8, Supplemental Digital Content 1,

Post Hoc Analyses

In a post hoc subgroup analysis according to World Bank–defined country income status, the improvements were more pronounced in LMICs (reduction in nonadherence in 8/10 processes) compared with HICs (reduction in nonadherence in 6/10 processes) (eTables 5 and 6, Supplemental Digital Content 1, Omissions in adherence to SBT, assessment for CVC and urinary catheter removal, and antimicrobial and sedation use decreased in both HIC and LMICs, whereas omissions in DVT and peptic ulcer prophylaxis and holding family conferences were reduced only in LMICs. Omissions in adherence to HOB elevation decreased only in HICs. The implementation of CERTAIN was associated with reduction in incidence rates of CRBSI and FFP transfusions in LMICs but not HICs (eTables 5 and 6, Supplemental Digital Content 1,

In a related post hoc analysis, significant interaction between income status and CERTAIN implementation effects on ICU, hospital, and day 28 mortality was observed (p < 0.001, p = 0.004, p = 0.002 respectively). After adjusting for baseline imbalances, the ICU mortality (LMIC: aOR, 0.63; 95% CI, 0.53–0.75; HIC: aOR, 1.18; 95% CI, 0.83–1.68), hospital mortality (LMIC: aOR, 0.65; 95% CI, 0.55–0.77, HIC: aOR, 1.03; 95% CI, 0.75–1.43), and 28-day mortality (LMIC: aOR, 0.67; 95% CI, 0.56–0.79, HIC: aOR, 1.19; 95% CI, 0.87–1.63) were reduced only in LMICs. The survival analysis stratified according to income status reported similar differences in CERTAIN effect (eFig. 8, Supplemental Digital Content 1,


In this multinational study of ICUs with variable resources, prompting the ICU clinical team with CERTAIN admission and rounding checklists was associated with lower nonadherence to most daily care processes, reduced LOS, and reduced mortality. To our knowledge, this is the first global effort to introduce a checklist into everyday clinical practice to improve adherence to ICU daily care processes and patient outcomes. We demonstrated that CERTAIN implementation is associated with significant improvements of evidence-based care delivery.

CERTAIN implementation was associated with improvement of eight of 10 practices with no improvement in HOB elevation and daily oral care practice. The baseline adherence to many processes of care was high, with the greatest improvement in processes with lower adherence in preimplementation phase. Previous studies on checklist use in ICUs have yielded conflicting results (1,2,6,16–20). Improvements were seen in HOB elevation, daily interruption of sedative infusions, peptic ulcer disease and deep venous thrombosis prophylaxis, oral care for ventilated patients, electrolyte repletion, initiation of physical therapy, and documentation of restraint orders (21,22). Overall, the results were not consistent across studies (23–25). This variation in effect may be due to differences in checklist selection and outcome definition. The way in which checklists are implemented may also influence the results, as active prompting has been shown to be superior to a checklist alone (20). Most studies, including this one, used a before-after design and thus are prone to secular trends. Importantly, previous studies were mostly performed in HICs that have more resources for quality-improvement interventions and an established organizational safety culture. Notably, daily care processes in our study improved more in LMICs than HICs, which may relate to higher adherence to baseline processes of care in HICs but also fewer centers enrolled from HICs.

In LMIC ICUs, lack of structured training and lack of physicians are major barriers to implementing evidence-based medicine (26). A systematic approach and provision of low-cost basic critical care procedures can minimize death and reduce costly complications in all environments (27,28). Simple interventions such as checklists improved outcomes in surgical and trauma patients globally without significantly increasing costs (6,29). So far, data on efficacy of such quality-improvement processes in ICUs of developing countries are scarce. Recently, a quality-improvement initiative in sepsis was associated with improved compliance to quality indicators and reduced mortality in a middle-income country (30). Advances in information technology have yielded opportunities for online remote training and quality improvement without costly on-site education. Such use of collaborative network infrastructure has already shown to improve adaptation of care practices in HIC community ICUs (3).

We observed a significant decrease in number of CVC infections in the postimplementation phase with no effect on prevention of VAP and urinary catheter infections. Studies on preventing ICU infections with quality-improvement interventions have shown different results (16,19,31). This heterogeneity of effect may be due to nonuniform definitions of ICU infections. Mortality was significantly lower in the post-CERTAIN phase. In a stratified analysis according to income status, the mortality reduction was present only in LMICs and remained significant after adjusting for baseline imbalances between pre- and postimplementation groups. In contrast, a randomized trial to introduce prompting and checklists did not decrease hospital mortality and ICU infections in Brazilian ICUs (2) but did show modest improvements in four of seven processes of care. The main difference is that in our study, the intervention (training and checklist) not only included a rounding checklist (CERTAIN rounds) but also a structured admission and resuscitation checklist (CERTAIN admission). Similar to our findings, a nonrandomized evaluation of the global implementation of a surgical checklist was associated with marked reductions in mortality and postoperative complications in economically diverse group of hospitals (6), but the same effect was not observed in a HIC setting (32).

Our study has several limitations. Before-after studies are prone to confounding by secular improvements and Hawthorne effect. Several aspects of management may have changed over time; for example, transfusion triggers may have declined. Therefore, cause and effect cannot be reliably determined. Second, in this pragmatic study, data were collected by bedside clinicians rather than dedicated study coordinators. We encouraged adherence to the intervention throughout the study period and measured it via self-reporting rather than direct observation. Third, the baseline characteristics of pre- and postintervention groups differed. The preimplementation cohort had more comorbidities and more limitations on life-sustaining measures, both of which could contribute to the mortality differences seen. Nevertheless, the association of CERTAIN with decreased LOS and mortality remained after adjustment for the baseline imbalances. Although the statistical analysis may be able to control for some of this confounding, residual confounding may persist. Fourth, the diagnosis of CVC and urinary catheter infections and VAP were based on medical records rather than adjudication including microbiological data. Fifth, interventions such as catheter insertion bundles, nutrition, or decubitus ulcer prevention were not examined. The evaluated daily care processes were mostly addressed by the CERTAIN rounding checklist, and the impact of the admission checklist and decision support algorithms on outcomes could not be reliably assessed in the absence of validated measures. Sixth, ICU-specific factors including staffing, number of beds, baseline performance, and prior internal quality-improvement interventions may have influenced the results. The time frame in which the ICUs adopted and implemented the intervention differed significantly due to the duration of the study period. Challenges in CERTAIN implementation are discussed elsewhere (33). Only those hospitals motivated to improve quality care were committed time and resources to complete the study (eFig. 7, Supplemental Digital Content 1, However, the observed association persisted after adjustment for center. The patients who were transferred from an ICU outside the participating hospital were excluded from this study; the results may not be applicable on those patients. Last, some of the elements of SQUIRE guidelines including cost and strategic trade-offs, opportunity cost, and sustainability were not analyzed. Given these limitations, additional carefully designed studies are required to validate CERTAIN association with improved outcomes.

Despite these limitations, the results of this study show that systematic checklist implementation is feasible even in resource-limited settings and is associated with improvements in processes of care. To further facilitate CERTAIN use in daily work flow, we enabled translation into local language and developed a paper version of the tool. In addition, we have conducted educational courses (domestics, international, and online programs) and developed a train-the-trainer course to share our experience globally (


The implementation of a structured electronic decision tool with checklists in international ICUs has shown an association with reductions in nonadherence to basic care processes, CRBSI risk, and mortality.


In memoriam of Dr. Bekele Afessa, one of the pioneers of critical care practice improvement whose teachings inspired this work. We would like to thank Dr. Chris Farmer, Dr. Terri Clemmer, Dr. Atul Gawande, and Dr. Ken Hillman for helpful advice during study design and conduct. We thank Michael Malinchoc, MS, for performing statistical analysis. We thank the participating ICUs and their teams for their enthusiastic work and contribution to this project.


1. Ko HC, Turner TJ, Finnigan MA. Systematic review of safety checklists for use by medical care teams in acute hospital settings–limited evidence of effectiveness. BMC Health Serv Res. 2011; 11:211
2. Cavalcanti AB, Bozza FA, Machado FR, et al.; Writing Group for the C-ICUI, the Brazilian Research in Intensive Care. Effect of a quality improvement intervention with daily round checklists, goal setting, and clinician prompting on mortality of critically ill patients: A randomized clinical trial. JAMA. 2016; 315:1480–1490
3. Scales DC, Dainty K, Hales B, et al. A multifaceted intervention for quality improvement in a network of intensive care units: A cluster randomized trial. JAMA. 2011; 305:363–372
4. Vukoja M, Riviello ED, Schultz MJ. Critical care outcomes in resource-limited settings. Curr Opin Crit Care. 2018; 24:421–427
5. Adhikari NK, Fowler RA, Bhagwanjee S, et al. Critical care and the global burden of critical illness in adults. Lancet. 2010; 376:1339–1346
6. Haynes AB, Weiser TG, Berry WR, et al.; Safe Surgery Saves Lives Study Group. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009; 360:491–499
7. Barwise A, Garcia-Arguello L, Dong Y, et al. Checklist for early recognition and treatment of acute illness (CERTAIN): Evolution of a content management system for point-of-care clinical decision support. BMC Med Inform Decis Mak. 2016; 16:127
8. Vukoja M, Kashyap R, Gavrilovic S, et al. Checklist for early recognition and treatment of acute illness: International collaboration to improve critical care practice. World J Crit Care Med. 2015; 4:55–61
9. World Bank Country and Lending Groups. World Bank Country and Lending Groups 2017. Available at: Accessed April 30, 2019
10. Sevilla-Berrios R, O’Horo JC, Schmickl CN, et al. Prompting with electronic checklist improves clinician performance in medical emergencies: A high-fidelity simulation study. Int J Emerg Med. 2018; 11:26
11. Shao M, Kashyap R, Niven A, et al. Feasibility of an international remote simulation training program in critical care delivery: A pilot study. Mayo Clin Proc Innov Qual Outcomes. 2018; 2:229–233
12. Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009; 42:377–381
13. National Healthcare Safety Network. CDC/NHSN Surveillance Definitions for Specific Types of Infections 2013. Available at: Accessed September 13, 2020
14. Revised Standards for Quality Improvement Reporting Excellence. Available at: Accessed October 10, 2020
15. Hempel S, Shekelle PG, Liu JL, et al. Development of the quality improvement minimum quality criteria set (QI-MQCS): A tool for critical appraisal of quality improvement intervention publications. BMJ Qual Saf. 2015; 24:796–804
16. Bion J, Richardson A, Hibbert P, et al.; Matching Michigan Collaboration & Writing Committee. ‘Matching Michigan’: A 2-year stepped interventional programme to minimise central venous catheter-blood stream infections in intensive care units in England. BMJ Qual Saf. 2013; 22:110–123
17. Dubose J, Teixeira PG, Inaba K, et al. Measurable outcomes of quality improvement using a daily quality rounds checklist: One-year analysis in a trauma intensive care unit with sustained ventilator-associated pneumonia reduction. J Trauma. 2010; 69:855–860
18. Marsteller JA, Sexton JB, Hsu YJ, et al. A multicenter, phased, cluster-randomized controlled trial to reduce central line-associated bloodstream infections in intensive care units*. Crit Care Med. 2012; 40:2933–2939
19. Pronovost P, Needham D, Berenholtz S, et al. An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med. 2006; 355:2725–2732
20. Weiss CH, Moazed F, McEvoy CA, et al. Prompting physicians to address a daily checklist and process of care and clinical outcomes: A single-site study. Am J Respir Crit Care Med. 2011; 184:680–686
21. Berenholtz SM, Milanovich S, Faircloth A, et al. Improving care for the ventilated patient. Jt Comm J Qual Saf. 2004; 30:195–204
22. Byrnes MC, Schuerer DJ, Schallom ME, et al. Implementation of a mandatory checklist of protocols and objectives improves compliance with a wide range of evidence-based intensive care unit practices. Crit Care Med. 2009; 37:2775–2781
23. Narasimhan M, Eisen LA, Mahoney CD, et al. Improving nurse-physician communication and satisfaction in the intensive care unit with a daily goals worksheet. Am J Crit Care. 2006; 15:217–222
24. Pronovost P, Berenholtz S, Dorman T, et al. Improving communication in the ICU using daily goals. J Crit Care. 2003; 18:71–75
25. Agarwal S, Frankel L, Tourner S, et al. Improving communication in a pediatric intensive care unit using daily patient goal sheets. J Crit Care. 2008; 23:227–235
26. Haniffa R, Pubudu De Silva A, de Azevedo L, et al. Improving ICU services in resource-limited settings: Perceptions of ICU workers from low-middle-, and high-income countries. J Crit Care. 2018; 44:352–356
27. Schell CO, Gerdin Warnberg M, Hvarfner A, et al. The global need for essential emergency and ritical care. Crit Care. 2018; 22:284
28. Lee JS, Roberts SWP, Götsch K, et al. Caring for critically ill patients in humanitarian settings. Am J Respir Crit Care Med. 2019; 199:572–580
29. Lashoher A, Schneider EB, Juillard C, et al. Implementation of the world health organization trauma care checklist program in 11 centers across multiple economic strata: Effect on care process measures. World J Surg. 2017; 41:954–962
30. Machado FR, Ferreira EM, Schippers P, et al.; SPDM against sepsis project. Implementation of sepsis bundles in public hospitals in Brazil: A prospective study with heterogeneous results. Crit Care. 2017; 21:268
31. Cachecho R, Dobkin E. The application of human engineering interventions reduces ventilator-associated pneumonia in trauma patients. J Trauma Acute Care Surg. 2012; 73:939–943
32. Urbach DR, Govindarajan A, Saskin R, et al. Introduction of surgical safety checklists in Ontario, Canada. N Engl J Med. 2014; 370:1029–1038
33. Coville H, Kashyap R, Suzuki R, et al. Challenges of implementing a stepped wedge knowledge delivery intervention in 20 countries. Crit Care Med. 2016; 44:106


Checklist for Early Recognition and Treatment of Acute Illness and Injury (CERTAIN) Investigators: Aida Mujakovic, MD, Dragana Markotic, MD, Zoran Karlovic, MD, Min Shao, MD, Zhigang Chang, MD, Feihu Zhou, MD, Hongjun Kang, MD, Jianjun Gui, MD, Yimin Li, MD, Sibei Chen, MD, Shouhong Wang, MD, Xin Zhao, MD, Haitao Lan, MD, Lin Dou, MD, Yan Kang, MD, Xuelian Liao, MD, Wei Liu, MD, Alan Sustic, MD, Danijel Knezevic, MD, Jose Yunen, MD, Rajyabardhan Pattnaik, MD, Chandan Dey, MD, Varma Muralidhar, MD, Mradul Kumar Daga, MD, Harpreet Singh, MD, Emily Naylor, Juvelikian Georges, MD, Jose Guillermo Dominguez Cherit, MD, Su Jung Choi, MD, Fatima Ajaz, MD, Carolina L. Tapia, MD, Erric Cinco, MD, Anna Kluzik, MD, Joanna Kaik, MD, Maja Surbatovic, MD, Mihailo Stojic, MD, Uros Petrovic, MD, Svetislava Milic, MD, Sixtus Ruyumbu, MD, Ozlem Cakin, MD, Atilla Ramazanoglu, MD, Vakil Abhay, MD, Syed Anjum Khan, MD, Joseph Poterucha, MD, Rose M. Peterson, Parvez Mir, MD, Hong-I Liao, MD, Jovan Matijasevic, MD, Pedja Kovacevic, MD, Sasa Dragic, MD, Milka Jandric, MD, Danica Momcicevic, MD, Hong Bo, MD, Jun Guo, MD, Bo Wang, MD, Moldovan Sabov, MD, Amelia Barwise, MB, BCh, Reina Suzuki, MD, Hongchuan Coville, MD, Lisbeth Garcia Arguello, MD, Lei Fan MS, Oguz Kilickaya, MD.


checklist; global health; intensive care; quality improvement

Supplemental Digital Content

Copyright © 2021 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine and Wolters Kluwer Health, Inc.