The Spillover Effects of Quality Improvement Beyond Target Populations in Mechanical Ventilation : Critical Care Explorations

Secondary Logo

Journal Logo

Quality Improvement Report

The Spillover Effects of Quality Improvement Beyond Target Populations in Mechanical Ventilation

Greenwald, Andrew S. MD1; Hauw-Berlemont, Caroline MD1; Shan, Mingxu MS2; Wang, Shuang PhD2; Yip, Natalie MD3; Kurlansky, Paul MD4; Argenziano, Michael MD4; Bennett, Bridgette RN1; Langone, Danielle PA1; Moitra, Vivek MD1

Author Information
Critical Care Explorations 4(11):p e0802, November 2022. | DOI: 10.1097/CCE.0000000000000802

Abstract

KEY POINTS

  • Question: If a quality improvement project is successful for one subset of patients in an ICU, do the benefits of that program extend to other patients in the ICU?
  • Findings: After the successful implementation of a quality improvement program to improve postoperative mechanical ventilation times in coronary artery bypass grafting patients, patients in the same ICU who underwent certain valve replacement procedures also experienced a significant reduction in postoperative mechanical ventilation times.
  • Meaning: Positive effects of a quality improvement program can “spill over” to patients in an ICU, despite them not being the target of the quality improvement program.

Mechanical ventilation time has been proposed as a quality indicator of care after cardiac surgery (1–3) because prolonged mechanical ventilation after cardiac surgery impacts perioperative mortality (4), morbidity (5–7), length of stay (8,9), and healthcare costs (10). To reduce mechanical ventilation times among patients who undergo coronary artery bypass grafting (CABG) surgery, we developed a mechanical ventilation weaning protocol. In addition to developing and implementing the protocol, we provided monthly audit and feedback reports to physicians and nurses on the length of mechanical ventilation for CABG patients only.

Quality improvement programs usually focus on targeted outcomes and rarely report or evaluate the unintended consequences—either positive or negative—of program interventions (11–13). Unintended consequences are part of what is known as the “spillover effect,” that is, an effect that an intervention can have on a population not eligible for a program (14). Although the spillover effect has been observed in the context of policy and financial incentives in healthcare, little is known about spillover effects in the context of quality improvement processes. To our knowledge, no quality improvement studies of mechanical ventilation have evaluated the unintended consequences of a quality improvement initiative on adult patients excluded from or not included in the initiative (15). We therefore sought to determine the impact of protocol development, protocolization, and monthly audit and feedback on mechanical ventilation times on colocated patients who were excluded from a protocol (in this case, patients undergoing isolated aortic or mitral valve repair or replacement [left-sided valves]).

MATERIALS AND METHODS

Setting

The Columbia University Irving Medical Center Human Research Protection Office Institutional Review Board approved this retrospective study (Protocol Number AAAT5677, Title: STS, Approved February 10, 2021), which was conducted in a 738-bed academic medical center using data from July 2015 to May 2018. Informed consent was waived because this was a retrospective medical record review. The study was completed in accordance with the values of the World Medical Association’s Helsinki Declaration of 1975. The medical center is a quaternary referral center for open heart surgeries, which include valve surgery, bypass grafting, cardiac transplants, temporary and durable mechanical circulatory support device insertions, and complex aortic procedures. Patients who underwent CABG and left-sided valve surgery were cared for in the same geographical locations (a 21-bed ICU or a 10-bed ICU). CABG and left-sided valve surgery patients were admitted to each of these units and managed by the same nursing and physician staff.

Quality Improvement Process

We compared three phases. Phase 0 (P0) from July 2015 to December 2015 was the pre-intervention or “before” phase. Phase 1 (P1) from January to August 2016 represented the period during which the protocol was being developed with audit and feedback and before-protocol implementation (“during” phase). Phase 2 (P2) from September 2016 to May 2018 represented the post-intervention period of protocol implementation and audit and feedback (“after” phase). Both P1 and P2 included audit and feedback.

In January 2016, a multidisciplinary group developed a ventilator weaning protocol for CABG patients. The protocol was developed by ICU nurses, advanced practice providers, and physicians. During the P1 period (protocol development and audit and feedback of ventilation times), the protocol was revised by ICU clinicians via peer-to-peer discussions to adjust weaning parameters such as dosing of vasopressors or weaning steps. The protocol was implemented in August 2016 (Appendix A, https://links.lww.com/CCX/B93). During P1 and P2, the medical director provided monthly feedback (median ventilation hours and reintubation data) to all caregivers for CABG patients only via email. Reintubation was defined as endotracheal intubation at any point after extubation. The emails also included median mechanical ventilation times stratified by percentile compared with the Society for Thoracic Surgeons (STS) database (16). The intention of focusing solely on CABG patients was to provide an achievable scope for the quality improvement program. There were no other interventions in the ICU to reduce mechanical ventilation times in cardiac surgery patients before the implementation of this initiative. Furthermore, there were no other changes in standards of care, protocols, or staffing practices during this time.

Measures

Our primary outcome measure for this study was mechanical ventilation times in excluded left-sided valve patients. We compared the level change of mechanical ventilation times for CABG patients and left-sided valve patients to determine whether the protocol was successful and then compared the rate of change of mechanical ventilation times over time between the included (CABG) and excluded (left-sided valve) populations. We did not analyze patients who had more complex procedures such as ventricular assist device insertions, cardiac transplantation, and aortic surgery to limit the heterogeneity of our control group.

Statistical Analysis

We created a cohort of CABG patients to include only patients who received mechanical ventilation for less than 24 hours. Because cardiac surgery patients may experience prolonged mechanical ventilation for postoperative complications such as postoperative bleeding, reoperations, and stroke, we excluded from the analysis patients with mechanical ventilation times of greater than 24 hours (even if they were treated with the protocol) to limit heterogeneity. We compared the cohort of CABG patients with a cohort of patients undergoing isolated left-sided valve procedures. All the data were extracted from the STS Database.

Using the monthly median mechanical ventilation times across P0, P1, and P2 from July 2015 to May 2018 (35 data points for 35 months), we performed an interrupted time series analysis to compare ventilation time trends in both the CABG and valve patient populations. We analyzed two components: 1) the level of change in intercepts, that is, the before-protocol period (P0), during-protocol period (P1), and after-protocol period (P2) and 2) the difference in slopes compared with pre-protocol in two subsequent time intervals (P1 and P2).

For univariate analyses, patient-level ventilation times of P0 versus P1, P0 versus P2, P1 versus P2, and P0 versus P2 were compared with a two-sample t test to examine associations between ventilation times and the intervention. To investigate whether patient populations before and after the intervention were comparable, we compared demographic data using Pearson chi-square tests for categorical variables and two-sample t tests for continuous variables.

To examine intervention effects on ventilation time within each surgical procedure, we further performed logistic regressions using patient-level data to determine associations between intervention (binary outcome variable for “before” [P0] and “after” [P2]) and ventilation time adjusting for patient characteristics as potential confounding variables within CABG patients and within valve patients, respectively. To examine whether the intervention had different effects on ventilation times for patients undergoing different surgical procedures, we also fitted a logistic regression model with the surgical procedure × ventilation time interaction term adjusting for potential confounders. Analyses were performed using R Version 3.5.1, Vienna, Austria (17). Statistical significance was considered at 0.05 level.

RESULTS

Cohort Characteristics

Between July 2015 and May 2018, a total of 1976 patients underwent surgery. Of these, 58.20% had CABG surgery (Supplement Tables 1–3, https://links.lww.com/CCX/B93). After initiation of the protocol in August 2016, median mechanical ventilation times for isolated CABG patients ventilated for less than 24 hours decreased significantly from 11 (P0) to 6.22 hours (P2) (p < 0.001) (Supplement Table 1, https://links.lww.com/CCX/B93). For the CABG cohort, the STS risk of predicted morbidity or mortality was significantly lower in P2 than in P0 (0.13 vs 0.16; p < 0.001) (Supplement Table 1, https://links.lww.com/CCX/B93). The rate of reintubations was not significantly different in the CABG group between P0 5.96% (95% CI, 2.73–11.3%) versus P2 3.47% (2.24–5.12%; p = 0.178).

There was a similar reduction in postoperative ventilation times from P0 to P2 for patients excluded from the protocol who received isolated valve surgery. The median ventilation time for patients ventilated less than 24 hours decreased from 10.0 to 6.0 hours (p < 0.001) (Supplement Table 1, https://links.lww.com/CCX/B93). For the valve cohort, the STS risk of predicted morbidity or mortality was significantly lower in P2 than in P0 (0.15 vs 0.18; p < 0.001). There was no difference in predicted prolonged mechanical ventilation between P0 and P2 (Supplement Table 1, https://links.lww.com/CCX/B93). The number of reintubations in this population was also not significantly different between P0 1.72% (0.36–5.04%) versus P2 2.79% (1.49–4.77%; p = 0.478). After adjustment for patient characteristics in the logistic model, the association between postoperative ventilation time and intervention was significant (p < 0.0001). The results of the statistical modeling are found in Supplement Table 3 (https://links.lww.com/CCX/B93) and Appendix B (https://links.lww.com/CCX/B93).

Interrupted Time Series Analysis

The level of change for both the CABG and valve patient populations was significantly different between P0 and P2, demonstrating that the intervention not only had an effect in the target population of CABG patients but also had a spillover effect in the valve patients (Fig. 1). In the multiple interrupted time series analysis, there was no significant difference in the level between CABG and valve patients for P0 (p = 0.3416) and P2 (p = 0.8464). There was no difference in the slope of change between the CABG and valve patient populations across time cohorts (P0 vs P1 [p = 0.8809]; P1 vs P2 [p = 0.3834]; P0 vs P2 [p = 0.7672]), which suggests that the rate of change in mechanical ventilation times was similar between included and excluded patients (Appendix C, https://links.lww.com/CCX/B93).

F1
Figure 1.:
Trends in monthly median time of ventilation with trendlines per period. There were three phases to the quality improvement initiative: phase 0 (P0), pre-intervention or “before” phase; phase 1 (P1), creation of the protocol with audit and feedback (“during” phase); and phase 2 (P2), post-intervention period of implementation with audit and feedback (“after” phase). CAB = Coronary Artery Bypass.

DISCUSSION

After initiation of a weaning protocol with audit and feedback, we observed a reduction in mechanical ventilation times in patients who underwent CABG surgery. We found similar changes in other patient populations within the same ICU, which was an unintended consequence, or spillover effect, of the program. Fortunately, we found that rates of reintubation did not significantly increase throughout implementation of the protocol in either the CABG or valve populations. Although reintubation is a coarse measure of failure of extubation, utilization rates of noninvasive mechanical ventilation such as bilevel support or high-flow nasal cannula were not available; however, we feel that use of these other strategies does not constitute a failure of extubation. The reduced slope of decrease during P2 (the period of audit and feedback and protocol implementation) compared with P1 (the period of protocol development) suggests that the phase during which the protocol was created showed the greatest decrease in ventilation times. We estimate that this was because the greatest changes to clinical practice occurred during these months, and P2 reflects the adoption of these practices into the unit standard of care. Our analyses suggest that our practices were sustained in both cohorts for at least 21 months after the protocol was rolled out (Fig. 1). The interrupted time series analysis demonstrates that the rate of change for ventilation times was most significant during the development phase of the protocol, was sustained over time in the post-intervention period, was significantly different in both duration of ventilation and rate of change from before to after the intervention, and was similar in both the on-target and off-target populations.

The implementation of quality improvement processes in a complex healthcare environment can have desirable or undesirable consequences (18,19). A framework for these outcomes has been proposed to facilitate and encourage evaluation of unexpected facets of quality improvement projects (11–13). Yet only a few studies have reported the unexpected positive consequences, or “pleasant surprises,” of quality improvement programs, and those reports are limited as they involve anecdotal evidence (20,21). A study of surgical residents reported a “halo effect” after surgical residents who received focused education about venous thromboembolism (VTE) prophylaxis in trauma patients increased their prescribing rates of VTE prophylaxis in nontrauma patients, even though the education program had solely aimed for change in the care of trauma patients (22). We believe our investigation is the first ICU quality improvement study to systematically observe, report, and quantitatively evaluate the positive consequence of project implementation on patients excluded from a quality improvement initiative. We believe that the mechanism of the spillover effect in our program was a change in the practice pattern, or “culture,” of the entirety of our ICU team. Because the nurses, respiratory therapists, physician’s assistants, and physicians saw the early success of the protocol during the P1 period, we posit that this led to a change in our local biases about early extubation after CABG surgery. Our team made the implicit leap that patients undergoing straightforward valve surgery should be treated the same way. Of course, this could have been a negative assumption on behalf of our team; however, as rates of reintubation did not change, we are fortunate that untoward effects did not become apparent.

Identifying and reporting positive consequences can inform the design of quality improvement programs. Our observation of reduced mechanical ventilation times in valve patients was a desirable consequence, or a pleasant surprise, that was not associated with an increase in reintubation rates, a potential trade-off (13). As the original quality improvement project was meant solely to target CABG patients, and the effect on valve patients was found only post hoc in preparing this analysis, we use the term “pleasant surprise.” We believe that this pleasant surprise resulted from a quality improvement initiative that sought to engage all frontline providers to create a protocol and provide timely, nonpunitive audit and feedback and contextualized peer-peer discussions (23). A recognized obstacle to change in organizational culture in healthcare is a lack of staff engagement from a perceived lack of ownership by participants (24), an effect that we sought to mitigate by involving all clinical disciplines of ICU staff. The empowerment of the team may have been responsible for the spillover effect (25,26). Rather than implementing a top-down, physician-led approach, we included nurses, respiratory therapists, mid-level providers, fellows, and residents in the development of the protocol.

We believe that the monthly reporting of mechanical ventilation times was crucial for the success of the initiative because all providers could see the ventilation times improve drastically during the P1 period. It has not escaped our attention that little effect on mechanical ventilation times was seen after the protocol was considered final; however, we believe this is a result of the practical effects being applied during the P1 period and a ceiling for improvement having been reached in that time. Critically, after implementation of the protocol, all members of the care team took ownership of the project, which likely had a central role in its success. This sort of protocol has already proved its efficacy in the past (27,28).

Our study has some limitations. Mechanical ventilation times were very high before implementation of this project; thus, it is possible that achieving reductions in mechanical ventilation times was not challenging. Although the CABG patient population in P2 had a lower risk for increased duration of mechanical ventilation, this observation was not seen in the valve patients who experienced a similar reduction in mechanical ventilation times. The ICU team was not involved in selecting the patients who underwent cardiac surgery during either study period; hence, the decrease in risk for increased mechanical ventilation was coincidental. Further, it is notable that much of the reduction in mechanical ventilation time was noted to occur during P1. We estimate that this happened because of the broad involvement of staff in the development of the protocol before formal implementation. Thus, reduction in mechanical ventilation became a conscious practice-changing effort on behalf of the whole ICU team.

CONCLUSIONS

Our data suggest that protocolization for early extubation and feedback to providers about their practice can affect both the desired population and patients excluded from the protocol: this was identified as an unintended consequence or pleasant surprise. Our mechanical ventilation study reports and evaluates this positive consequence of implementation. Patients who are both the intended beneficiary of change and other patients who are cared for by the same healthcare team are “at risk” for beneficial or detrimental effects from efforts to improve quality. This suggests that practice improvement focused on one specific facet of practice can have pleiotropic effects on culture change.

ACKNOWLEDGMENTS

We would like to thank Jennifer Holmes, ELS, for her invaluable editorial assistance.

REFERENCES

1. Jacobs JP, He X, O’Brien SM, et al.: Variation in ventilation time after coronary artery bypass grafting: An analysis from the Society of Thoracic Surgeons adult cardiac surgery database. Ann Thorac Surg 2013; 96:757–762
2. Holman WL, Peterson ED, Athanasuleas CL, et al.: Alabama coronary artery bypass grafting cooperative project: Baseline data. Alabama CABG Cooperative Project Study Group. Ann Thorac Surg 1999; 68:1592–1598
3. Gaies M, Werho DK, Zhang W, et al.: Duration of postoperative mechanical ventilation as a quality metric for pediatric cardiac surgical programs. Ann Thorac Surg 2018; 105:615–621
4. Camp SL, Stamou SC, Stiegel RM, et al.: Can timing of tracheal extubation predict improved outcomes after cardiac surgery? HSR Proc Intensive Care Cardiovasc Anesth 2009; 1:39–47
5. Marelich GP, Murin S, Battistella F, et al.: Protocol weaning of mechanical ventilation in medical and surgical patients by respiratory care practitioners and nurses: Effect on weaning time and incidence of ventilator-associated pneumonia. Chest 2000; 118:459–467
6. He S, Chen B, Li W, et al.: Ventilator-associated pneumonia after cardiac surgery: A meta-analysis and systematic review. J Thorac Cardiovasc Surg 2014; 148:3148–3155.e1–e5
7. Erdil N, Gedik E, Donmez K, et al.: Predictors of postoperative atrial fibrillation after on-pump coronary artery bypass grafting: Is duration of mechanical ventilation time a risk factor? Ann Thorac Cardiovasc Surg 2014; 20:135–142
8. Hawkes CA, Dhileepan S, Foxcroft D: Early extubation for adult cardiac surgical patients. Cochrane Database Syst Rev 2003; 4:CD003587
9. Probst S, Cech C, Haentschel D, et al.: A specialized post anaesthetic care unit improves fast-track management in cardiac surgery: A prospective randomized trial. Crit Care 2014; 18:468
10. Cheng DC, Karski J, Peniston C, et al.: Early tracheal extubation after coronary artery bypass graft surgery reduces costs and improves resource use. A prospective, randomized, controlled trial. Anesthesiology 1996; 85:1300–1310
11. Manojlovich M, Lee S, Lauseng D: A systematic review of the unintended consequences of clinical interventions to reduce adverse outcomes. J Patient Saf 2016; 12:173–179
12. Toma M, Dreischulte T, Gray NM, et al.: A balanced approach to identifying, prioritizing, and evaluation all potential consequences of quality improvement: Modified Delphi study. BMJ Open 2019; 9:e023890
13. Toma M, Dreischulte T, Gray NM, et al.: Balancing measures or a balanced accounting of improvement impact: A qualitative analysis of individual and focus group interviews with improvement experts in Scotland. BMJ Qual Saf 2018; 27:547–556
14. Angelucci M, Di Maro V: Program evaluation and spillover effects. J Development Effectiveness 2016; 8:22–43
15. Witte MK, Mahle WT, Pasquali SK, et al.; Pediatric Heart Network Investigators: Spillover of early extubation practices from the pediatric heart network collaborative learning study. Pediatr Crit Care Med 2021; 22:204–212
16. Performance Measure Descriptions. Society for Thoracic Surgeons National Database, 2011. Available at: https://www.sts.org/quality-safety/performance-measures/descriptions#ProlongedIntubation. Accessed November 9, 2022
17. R Core Team: R: A Language and Environment for Statistical Computing. Vienna, Austria, R Foundation for Statistical Computing, 2018. Available at: https://www.R-project.org/. Accessed November 9, 2022
18. Chang SY, Sevransky J, Martin GS: Protocols in the management of critical illness. Crit Care 2012; 16:306
19. Dijkstra R, Wensing M, Thomas R, et al.: The relationship between organisational characteristics and the effects of clinical guidelines on medical performance in hospitals, a meta-analysis. BMC Health Serv Res 2006; 6:53
20. Duvoisin G, Fischer C, Maucort-Boulch D, et al.: Reduction in the use of diagnostic tests in infants with risk factors for early-onset neonatal sepsis does not delay antibiotic treatment. Swiss Med Wkly 2014; 144:w13981
21. Paul M, Andreassen S, Tacconelli E, et al.; TREAT Study Group: Improving empirical antibiotic treatment using TREAT, a computerized decision support system: Cluster randomized trial. J Antimicrob Chemother 2006; 58:1238–1245
22. Lau BD, Streiff MB, Hobson DB, et al.: Beneficial “halo effects” of surgical resident performance feedback. J Surg Res 2016; 205:179–185
23. Sinuff T, Muscedere J, Rozmovits L, et al.: A qualitative study of the variable effects of audit and feedback in the ICU. BMJ Qual Saf 2015; 24:393–399
24. Dixon-Woods M, McNicol S, Martin G: Ten challenges in improving quality in healthcare: Lessons from the Health Foundation’s programme evaluations and relevant literature. BMJ Qual Saf 2012; 21:876–884
25. Donovan AL, Aldrich JM, Gross AK, et al.; University of California, San Francisco Critical Care Innovations Group: Interprofessional care and teamwork in the ICU. Crit Care Med 2018; 46:980–990
26. Vander Schaaf EB, Cornett AC, Randolph GD: Improving culture, one quality improvement project at a time. J Public Health Manag Pract 2018; 24:57–62
27. Cove ME, Ying C, Taculod JM, et al.: Multidisciplinary extubation protocol in cardiac surgical patients reduces ventilation time and length of stay in the intensive care unit. Ann Thorac Surg 2016; 102:28–34
28. Cook DJ, Pulido JN, Thompson JE, et al.: Standardized practice design with electronic support mechanisms for surgical process improvement: Reducing mechanical ventilation time. Ann Surg 2014; 260:1011–1015
Keywords:

cardiovascular; extubation; mechanical ventilation; quality and patient safety; ventilation

Supplemental Digital Content

Copyright © 2022 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine.