Moderator: Lillian S. Kao, MD, MS
S. Kao, MD, MS, CMQ is Professor, Vice-Chair for Research and Faculty
Development, and Vice-Chair for Quality of Care in the Department of Surgery at
the McGovern Medical School at The University of Texas Health Science Center at
Houston. She is the co-founder and co-director of the Center for Surgical
Trials and Evidence-based Practice (C-STEP). She is also faculty for the
Masters Program in Clinical Research at McGovern Medical School, for which she
co-directs courses in Clinical Trials and Advanced Clinical Study Design. Her
research interests include surgical infections, surgical quality improvement,
and implementation of evidence-based practices.
Haugen et al performed a two-hospital, five-specialty, stepped wedge cluster randomized controlled trial in the Netherlands to evaluate the effect of the World Health Organization (WHO) checklist on surgical outcomes. After adaptation, the checklist consisted of 20 items at 3 critical steps of the procedure: the "sign in" before induction of anesthesia, the "time out" before start of surgery, and the "sign out" before the head surgeon left the operating room. All patients undergoing elective or emergency surgery requiring all 3 steps of the checklist were included. The primary outcomes were both major and minor complications and in-hospital mortality up to 30 days after surgery. The secondary outcome was patients' length of stay. Based on an intention-to-treat analysis, the trial demonstrated a significant reduction in complication rates from 19.9% to 12.4% (p<0.001). Based on a per-protocol analysis, the complication rates were reduced to 11.5% (p<0.001) when all 3 steps of the checklist were completed, resulting in a number needed to treat of 12. The checklist also resulted in a non-statistically significant reduction in mortality from 1.6% to 1.0% (p=0.151) and a significant reduction in mean length of stay from 7.8 to 7.0 days (p=0.022).
Several other multi-center studies of checklists have been performed, the majority of which were pre- and post-intervention studies which can be prone to biases caused by regression to the mean and secular trends. The authors in this study chose to use a stepped wedge design whereby each "cluster", which in this trial was one of five surgical specialties (cardiothoracic, neurosurgery, orthopedic, general, and urologic), received the intervention starting on a different month that was assigned in a random order. A significant advantage of this design was that each cluster served as its own control as well as a control for the other clusters. Furthermore, adjusting for time in the multivariable analysis allowed for minimization of confounding based on month of checklist initiation. A second advantage to the stepped wedge design was that all participants received the intervention, which was preferable since the intervention is perceived to have a high chance of benefit and a low risk of harm. On the other hand, in a randomized trial, only a proportion of participants (i.e., half if there is 1:1 allocation) receive the potential benefits of the intervention. Third, contamination is minimized with a cluster randomized trial where the unit of randomization is not an individual but rather a group of individuals. Contamination refers to adoption of the intervention by those in the control arm and results in underestimation of the treatment effect. Lastly, there may be logistical, practical, and financial reasons to use the stepped wedge design. For example, not all clusters may be ready to implement the intervention at the same time. Disadvantages of the stepped wedge design include the large amount of data and the complexity of the statistical analysis required.
There were a few limitations to this study. First, because participants in a given specialty or cluster are more similar to each other than to those in other specialties, the analysis should have adjusted for intra-cluster correlation which would have increased the calculated sample size. Second, more information about implementation measures would have been helpful. For example, in this trial, complete compliance was not observed in 27% of procedures randomized to receive the checklist. Details regarding whether procedures that did not receive the complete intervention were systematically different than those that did would have been helpful in informing future implementation efforts. Third, blinding was not performed in that the healthcare providers were aware of the intervention. Although a Hawthorne effect could have been present whereby participants improve based on observation alone, the authors argued in a letter to the editor that this was unlikely due to the duration of the study over several months and the lack of 100% compliance with the checklist. Furthermore, the authors argued that the participants were blinded as to the primary and secondary outcomes of the study.
The importance of this study is that it used the most rigorous, feasible study design to demonstrate improved patient outcomes after checklist implementation at two hospitals. A cluster randomized trial without the stepped wedge design would have been difficult given the small number of hospitals and specialties and would have restricted the benefits of the checklist to only half of the participants. However, questions remain regarding the mechanism(s) by which checklists improve patient outcomes and the optimal strategies for effective implementation.
- Multiple studies have been performed evaluating the effect of the surgical safety checklist on patient outcomes across a wide variety of settings. These studies have had conflicting results. In the absence of known harms associated with the checklist, do you think that more studies need to be done that focus on patient outcomes? What studies, if any, are needed to better inform effective dissemination and implementation of surgical safety checklists?
- What do you think was the mechanism by which checklists improved patient outcomes in this and other checklist studies? Increased compliance with evidence-based process measures? This trial did not measure the effect of checklists on compliance with process measures. Improved safety culture? Previous studies have reported a correlation between checklist effectiveness and change in safety attitudes; however, Haugen et al did not identify any improvement in safety culture.
- In this trial, the authors adapted the checklist to their context prior to implementation. Do you think that such adaptation contributed to the effectiveness of the checklist? What implementation factors do you think influence effectiveness the most? For institutions that do not currently employ the checklist, what can be done to maximize its successful implementation?
Haugen AS, Softeland E, Almeland SK et al. Reply to "Letter to the Editor Concerning the Article-Effect of the World Health Organization Checklist on Patient Outcomes: A Stepped Wedge Cluster Randomized Controlled Trial". Ann Surg 2016; 263: 324-5.