Clinicians and researchers often tout the newest breakthrough or latest successful intervention. Sharing wins, however, is often done at the expense of sharing obstacles, failures, and subsequent adjustments, which are the cornerstone of quality improvement (QI).1–3 Here, we share 3 key lessons from 2 hospital-based QI initiatives—the Ohio Timely Recognition of Abuse Injuries (TRAIN) Collaborative and the University of Pittsburgh Medical Center (UPMC) Child Abuse Initiative (UPMC-CAI). Both focus on early identification, proper evaluation, and accurate reporting of child maltreatment. These are important clinical issues because many children who die or nearly die from maltreatment had been evaluated previously by a medical professional who did not recognize abuse and/or did not report it to Child Protective Services.
The TRAIN Collaborative consists of 6 children’s hospitals in Ohio. Modeled after the Institute for Healthcare Improvement’s (IHI) Breakthrough Series Collaborative, TRAIN convened an expert panel and conducted an iterative series of learning sessions and rapid cycles of change. The collaborative focused on improving the health care provider’s recognition of, and response to, potentially abusive injuries in infants 6 months of age and younger.
The UPMC-CAI was a collaboration between UPMC Children’s Hospital of Pittsburgh (CHP) and 13 general emergency departments (EDs) in the UPMC hospital system. Key to this initiative was a child abuse clinical decision support system consisting of a universal child abuse screen4 and triggers developed based on natural language processing and orders placed in the electronic health record (EHR), a pop-up alert for providers, a physical abuse order set, and a child abuse reporting form to assist providers in documenting necessary information for Child Protective Services.
Both initiatives showed success. The TRAIN Collaborative reduced recurrent injury by nearly 75%.5 The UPMC-CAI demonstrated a 4-fold increase in identification of potentially abusive injuries in infants and toddlers.6 Although both experienced success, both also identified several setbacks related to (1) staff turnover; (2) unanticipated differences between academic and community hospitals; and (3) failure to invest early enough or robustly enough in data collection.
LESSON 1: ANTICIPATE STAFF TURNOVER
Although planning for the unexpected is essential for any long-term initiative, staff turnover should be anticipated. Over several years, personnel will experience promotions, relocations, illnesses, or family changes. These changes can be sudden and allow little time to respond.
For UPMC-CAI success, each of the 13 EDs had a child abuse response team consisting of a nurse—usually a sexual assault nurse examiner—and the ED director. The oversight team at CHP needed to contact the ED site team quickly when further evaluation/reporting of a child seen at their site was required. Sometimes ED site team members were promoted, moved to another institution, or went on leave and contact information, including emails and phone numbers, became outdated. Given the time sensitivity of follow-up evaluation, these personnel changes often led to the CHP oversight team intervening directly with families or Child Protective Services. This process was not optimal for many reasons, including loss of empowerment for ED sites to intervene on behalf of patients evaluated in their hospital. Although changes in staffing likely did not impact the outcome, they increased the workload for the CHP oversight team and affected the ability of ED site teams to learn from possible missed abuse cases.
A QI collaborative leader should evaluate staffing periodically. Asking staff “if you were promoted tomorrow and no longer able to complete this project, who would you identify as a potential replacement and why?” yields valuable insight and guides a succession plan. Inability to identify such a person is a red flag that should prompt additional consideration. Leaders may be called upon to support a critical role on an interim basis; familiarity with colleagues and access to key data and budget and compliance support may soften the hardship of unexpected personnel changes.
LESSON 2: NO 2 HOSPITALS ARE THE SAME
In his 1998 article, Kilo7 identified the underlying premises of the IHI Breakthrough Series model, many which had a demonstrable impact on the collaboratives. With TRAIN, leaders identified substantial gaps in knowledge and practice in participating hospitals, including basic QI methodology knowledge. There was broad variation in care for children and discrepant use of evidence-based recommendations, including skeletal surveys for suspected abuse, support services for families, and social worker involvement/interventions. Although the collaborative understood the clear importance of describing and disseminating best practices to participating sites, putting this into practice was more complicated. Many community hospitals contract with private physician groups rather than directly employing emergency medicine and radiology clinicians to serve their patient population. Although these providers are crucial to implementing child abuse–related QI work, significant capacity—which neither TRAIN nor UPMC-CAI had—was needed to provide outreach and educate multiple community hospitals and their contracted physician groups. When outreach is possible, identifying contracted clinicians willing to serve as champions can be challenging. Simply having a memorandum of understanding signed by hospital or system leadership does not necessarily translate to a mandate with contracted physician groups to attend learning sessions, change current practice, or contribute data. A needs assessment before the project launch can be immensely helpful in identifying knowledge and resource gaps. Identify a liaison to work with community hospital partners as that person can act as a translator, problem solver, and ombudsman.
LESSON 3: INVEST EARLY AND ROBUSTLY IN DATA COLLECTION
QI initiatives, particularly those including both community and academic sites, may need as much, if not more, data management and analysis support compared with traditional multicentered research projects.
Data collection decisions for a multicenter QI project cannot be top-down. Project goals and partner-specific goals and the associated data to collect for each may be remarkably different. Collaborative leaders should address these differences before data collection begins. TRAIN leadership created an expert panel to discuss data before launching the collaborative but failed to follow the IHI recommendation to implement periodic planning group assessments to address changes. For example, the initial data collection plan did not include reviews of deaths of previously evaluated children. Soon after launching, it became clear that this was a data point of great importance to several participating sites. Retrospectively collecting these new data was time consuming and caused significant stress on limited resources at some participating sites. Another important decision relates to how data are collected. Software and resources (even resources as simple as an internet connection) are taken for granted in the academic setting but may not be readily available or fully understood at participating sites. For example, as part of the UPMC-CAI, nurses at each site received a weekly data report as an Excel file (Microsoft Corp, Redmond, WA). It soon became clear that the nurses were not comfortable in manipulating the data files. In hindsight, training about weekly report interpretation assumed a level of Excel software understanding which many nurses did not have. Understanding site-specific resources and comfort level with data collection/presentation before launch is critical because addressing deficiencies or making changes to data forms or processes over time is expensive and time consuming.
Failures, big or small, are an integral component of the QI process. Although no provider engaging in this work strives to fail, it is a disservice to colleagues if failures are not recognized and promoted as an opportunity to learn and improve. Between 2 large multicenter pediatric QI collaboratives focused on child abuse, leaders identified obstacles that added unnecessary time and cost to otherwise successful work. Anticipating staff unavailability and ensuring backup communication plans, developing site-specific approaches that address the unique resource deficiencies of individual hospitals, and implementing a data collection strategy that accommodates all sites including regularly scheduled ongoing review can improve the efficiency of a large QI collaborative. Multisite collaboratives benefit from strong local leadership. Identifying local leaders who are fully engaged can help navigate particular nuances and site-specific challenges. As the leader of a multisite collaborative, visiting participating sites to understand their specific processes, strengths, and barriers helps to anticipate problems and develop solutions before the collaborative begins. Although the collaboratives discussed above address child maltreatment, the barriers and strategies presented may interest anyone considering initiating a large-scale QI initiative.
The authors have no financial interest to declare in relation to the content of this article.
1. Donnelly LF. Avoiding failure: tools for successful and sustainable quality-improvement projects. Pediatr Radiol. 2017;47:793797.
2. The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Improvement. IHI Innovation Series white paper. 2003. Boston, MA: Institute for Healthcare Improvement; Available at: www.IHI.org
4. Rumball-Smith J, Fromkin J, Rosenthal B, et al. Implementation of routine electronic health record-based child abuse screening in General Emergency Departments. Child Abuse Negl. 2018;85:5867.
5. Thackeray J. Data Presented at the Casey Family Programs Safety Convening. 2018.Phoenix, AZ.
6. Rosenthal B, Skrbin J, Fromkin J, et al. Integration of physical abuse clinical decision support at two general emergency departments. J Am Med Inform Assoc. 2019.
7. Kilo CM. A framework for collaborative improvement: lessons from the Institute for Healthcare Improvement’s breakthrough series. Qual Manag Health Care. 1998;6:113.