Joly, Brenda M. PhD, MPH; Booth, Maureen MRP; Shaler, George MPH; Conway, Ann PhD
Learning collaboratives, or quality improvement (QI) collaboratives, often are characterized as short-term, multiagency consortia developed to spread new knowledge and innovation to improve the quality and delivery of services.1 The concept of learning collaboratives, while new to public health, has established credibility in the health care sector and the Institute for Healthcare Improvement's (IHI's) model is one of the most common approaches.2 The IHI model creates an intense and time-limited environment in which known experts (ie, faculty) work with practitioners through a series of alternating learning and action cycles to test a “change package” to bring about desired improvement. Results show remarkable improvements within participating organizations, including increased use of preventive services for newborns and children3; better access to medical services4; lower caesarean section rates5; improvement in the provision of care for diabetes and heart disease6; and improved mental health service use among youth.7 Research also confirms that the structure and features of a learning collaborative impact its success and that not all learning collaboratives are created equal.8
A Multisite Mini-Collaborative Approach
Building off the successful application of learning collaboratives in health care, the Multi-State Learning Collaborative: Lead States in Public Health Quality Improvement (MLC) initiative required each of 16 participating states to implement learning collaboratives, referred to as “mini-collaboratives,” in 2 of 10 target areas over the 3-year grant. A review of this initiative and the process and outcome target areas are reported elsewhere.9 The IHI learning collaborative was referenced as a model for states to use in establishing their mini-collaboratives. Once grants were awarded, however, broad discretion was given to states in how they organized, staffed, and implemented their mini-collaboratives.
The working definition of a mini-collaborative provided by program sponsors speaks to the flexibility of the approach: A mini-collaborative includes any combination of local and state public health agencies or public health partners within a state that are focused on improving public health performance related to the selected target area.10 From the start, the MLC initiative envisioned mini-collaboratives as having 2 distinct but complementary purposes. On the one hand, mini-collaboratives were meant to expose state and local health departments (LHDs) to the tools and techniques of quality management. They also provided an opportunity to apply QI skills to improve one or more high priority target areas. States approached these dual aims very differently. Some emphasized QI tools and techniques, independent of the chosen target area. Other states used the target area as the springboard for learning both the evidence base on a content area and the quality management tools necessary to facilitate improvement. Unlike the IHI model, very few states used mini-collaboratives to test a prescribed “change package.” Across the 16 MLC states, mini-collaboratives varied in terms of their focus, length, composition, size, rigor, and emphasis on QI.
Under contract with the Robert Wood Johnson Foundation, the University of Southern Maine served as the external evaluators for the MLC. A description of the overall evaluation framework, design, and methodology to assess this initiative has been presented elsewhere.11 The purpose of this article is to examine, through a case study approach, the use of mini-collaboratives to build QI capacity and performance in public health.
The guiding framework for our case study was based, in large part, on a review of the literature on traditional collaboratives (eg, those that applied the IHI model). While our review confirmed that the MLC mini-collaborative approach differed from most health care collaboratives that focus on implementing a defined model or standard of care, we believed that key features of successful health care collaboratives might be transferable and we set about to test these assumptions. A review of effective collaboratives and how conditions can be optimized for sustaining change once a collaborative ends identified the following 6 elements.
1. Sponsorship and faculty. The credibility and independence of the sponsor, its alignment with the goals of the collaborative, and its ability to recruit expert faculty (eg, those with subject-matter knowledge and a practical understanding of what motivates changes in an area) contribute to the caliber of entities who participate and its ultimate success.2,12
2. Topic. A topic should (1) address an evidence-based practice known to bring about desirable change, but has not been widely applied, and (2) focus on a priority where change is warranted and outcomes are available for tracking change.2,12 When teams work on different areas of improvement, there is less opportunity to apply best practices, change concepts, or research evidence.8
3. Participants. Teams should be composed of those affected by the proposed change, have the authority to conduct small scale tests of change, and can help in making change become standard practice throughout the organization.2,12 Senior leaders must be committed to supporting the change, and assuming responsibility for its sustainability.2 Participants must be willing to learn change theory, skills for breaking down problems, and techniques for analyzing and managing issues that may get in the way of change.8
4. Advance work. The research suggests that good collaboratives begin long before the first face-to-face meeting. Sponsors must have designed realistic and clear aims; measurable strategies for monitoring its progress; and concrete evidence-based changes that will be the focus of learning.2
5. Measurable targets. The use of specific measures for monitoring progress is a common feature across successful collaboratives. Teams that do not set targets early in the process and do not routinely report on progress are less successful in learning quality methods and achieving improvements.8
6. Learning sessions and action cycles. A key component of a collaborative is the in-person learning session interspersed with action cycles. Learning sessions focus on sharing the evidence base needed for change, common barriers to implementation, and strategies for optimal performance improvement.2,13 Successful learning sessions typically focus on mutual learning and teamwork. They also allow time for teams to apply ideas in their home settings with the help of faculty.8 Action cycles generally focus on testing and implementing changes in an organization, collecting data, and measuring impact. Effective collaboratives have clear expectations about team assignments during action cycles and established mechanisms for teams to stay in contact.2
Given the flexible design of the MLC mini-collaborative process described earlier, our evaluation questions and the literature review, we developed a case study theory and related protocols to guide our efforts. We hypothesized that several factors would be associated with a successful mini-collaborative including (1) having sufficient time to plan and implement the mini-collaborative, (2) holding productive learning sessions, (3) recruiting a strong faculty, (4) providing sufficient training and technical assistance, (5) engaging members to actively participate, (6) promoting senior leadership buy-in and commitment for QI, (7) disseminating evidence and best practice information related to the desired change, (8) using a reputable improvement model, (9) monitoring activities and outcomes through the use of a work plan, (10) providing resources to participating organizations, (11) communicating clear expectations to all members, and (12) developing a cohesive team or learning environment.
On the basis of these constructs, this study (1) tests our theory on the key drivers of a successful mini-collaborative, (2) describes the use of QI approaches and techniques, including facilitating factors and impediments, and (3) identifies key attributes affecting the spread and sustainability of QI efforts within an LHD. The results are intended as a guide for public health professionals interested in implementing QI learning collaboratives, adopting QI tools, or broadening the reach of their existing QI efforts.
We used a multisite case study methodology to evaluate MLC mini-collaboratives through the perspective of a subset of their LHD participants. This type of research methodology is often used in QI research14,15 and is frequently applied to empirical inquiry where real-life contexts are often inseparable from the main issues of interest.16,17 The approach enabled our evaluation team to collect “on-the-ground” information about mini-collaboratives as well as the adoption and implementation of QI within LHDs. This study was reviewed and approved by the University of Southern Maine's institutional review board and consent was obtained from participants.
Our primary source of data collection was based on a series of semistructured interviews (B.M.J., G.S., M.B., A.C., MLC Case Studies: State Level Interview Protocol, unpublished data, 2009–2011; B.M.J., G.S., M.B., A.C., MLC Case Studies: Local Level Interview Protocol, unpublished data, 2009–2011; B.M.J., G.S., M.B., A.C., MLC Case Studies: State Level Follow-up Interview Protocol, unpublished data, 2011; B.M.J., G.S., M.B., A.C., MLC Case Studies: Local Level Follow-up Interview Protocol, unpublished data, 2011; B.M.J., G.S., M.B., A.C., Faculty and Subject Matter Telephone Interview Protocol, unpublished data, 2011), the majority of which were conducted during a site visit. We also included observations of mini-collaborative meetings (when possible) and the collection and review of documents related to mini-collaborative efforts. Using multiple sources of evidence is a notable feature of a hallmark case study17 and was a core element of our evaluation design.
Study participants and data collection
State and local participants
All 16 MLC states participated in the case studies and provided information about their mini-collaboratives. The sponsors (eg, state-level agencies) and other stakeholders responsible for organizing and implementing the mini-collaborative were invited to participate in our case studies. In addition, QI teams from 2 participating LHDs were included. Local health departments were selected in consultation with state sponsors of the mini-collaborative. Although we did not implement a strict a priori selection criteria, whenever possible, LHDs with different characteristics and levels of QI experience were selected to broadly represent a range of experiences. The Table shows descriptive information about participating LHDs.
TABLE Characteristi...Image Tools
Because we could conduct only a limited number of case studies each year yet wanted to follow a subset of states over a 2-year period, we incorporated 3 cycles of data collection:
1. Cycle 1. In 2009, site visits and semistructured interviews with state sponsors and LHD QI teams were conducted in 12 of the 16 MLC states (Indiana, Iowa, Florida, Michigan, Minnesota, Missouri, Montana, New Jersey, Oklahoma, South Carolina, Washington, and Wisconsin) resulting in 36 group interviews. The selection of initial states was largely based on the target area and timing of their mini-collaboratives. We tried to select sites where the mini-collaborative was still in an early stage of implementation. The state- and local-level interview protocols focused on how and why the target area was selected, the composition and cohesiveness of the collaborative and LHD QI teams, expectations and goals, the selected improvement model, roles within the mini-collaborative, barriers and context, use of QI methods, and technical assistance and training. Each interview lasted approximately 60 to 90 minutes and was digitally recorded and transcribed verbatim for analysis.
2. Cycle 2. In 2010, site visits and face-to-face semistructured interviews with state sponsors and LHD QI teams were conducted in the 4 remaining MLC states (Kansas, Illinois, New Hampshire, North Carolina). Similar protocols and processes were used during the second round of site visits. Although New Hampshire's mini-collaborative did not include LHDs, we elected to include data captured during our interviews with QI teams of participating local agencies because we felt their perspective was valuable. Efforts to meet with members of the mini-collaborative in North Carolina were unsuccessful; therefore, this state was subsequently dropped from the analysis.
3. Cycle 3. In 2010, we conducted follow-up telephone interviews with the state MLC lead as well as the lead person from each of the LHDs in the original 12 case study sites to learn about progress, new insights, and lessons learned. Several of these mini-collaboratives had ended or were near completion; therefore, the sites were well positioned to reflect on their efforts.
We conducted telephone interviews with 6 faculty members from different states. An interview protocol (B.M.J., G.S., M.B., A.C., Faculty and Subject Matter Telephone Interview Protocol, unpublished data, 2011) was used to guide the discussion. Selection of faculty was based on their level of involvement and knowledge of QI efforts in public health. Faculty interviews lasted approximately 1 hour each and focused on using mini-collaboratives to build QI capacity and improve target areas.
In general, we used interpretative phenomenological analysis as a framework for analyzing our qualitative data. This approach originated in health psychology18,19 and has been used successfully in public health.20 Our analyses applied standard techniques to code the data based on themes and connections between themes.21 We reviewed the literature, cross-referenced case study data with other survey data (B.M.J., M.B., G.S., The QI Maturity Tool, unpublished data, version 1.0, February 2009; version 2.0, February 2010; version 3.0, February 2011), and held periodic meetings with professional colleagues to discuss emerging patterns. This process helped to compare, validate, and extend our findings.
Key drivers of a successful mini-collaborative
Our findings underscore lessons from the literature on features that make for an effective collaborative and provide early evidence supporting our case study theory on key drivers of a successful mini-collaborative. Nine of the 12 factors we hypothesized were identified as relevant including (1) advanced planning efforts that detailed the mini-collaborative process, expectations, and measures of success; (2) the selection of qualified faculty; (3) timely and skill-based training and technical assistance; (4) the engagement of and commitment from senior leaders from participating LHDs; (5) the application of evidence to achieve desired change; (6) the adoption of a credible improvement model; (7) evaluation of efforts and outcomes; (8) the articulation of clear roles, responsibilities, and goals to all collaborative members; and (9) the availability of resources to implement small scale change. In addition to these factors, our case studies revealed 2 additional areas that were linked to successful QI collaboratives in public health: the selection of target areas and prior experience with QI and learning collaboratives. In terms of target selection, LHDs were motivated to participate in mini-collaboratives when a target area had relevance to their accreditation aspirations or areas of interest. Not surprisingly, outcome target areas (eg, obesity) were more challenging to address than process targets (eg, community health assessments) in the 6- to 18-month span of most mini-collaboratives. To offset this challenge, we learned that outcome target areas were often converted into shorter-term process goals. Interestingly, having an evidence-based intervention to address a specific target area was not a key factor in target area selection except in a few states.
Conducting mini-collaboratives was a new role for MLC states and most lacked expertise in doing so. However, their individual and collective knowledge about how to run a successful mini-collaborative advanced significantly after the first round of mini-collaboratives. By 2010, every state had developed more detailed work plans, AIM statements, and general goals for their mini-collaboratives. Many incorporated a formal pre-/postassessment of QI knowledge and skills so that the curriculum could be tailored to the competencies of LHD participants. States also documented their efforts with greater detail, through the use of program manuals, formal memorandums of understanding with participating LHDs, and charters. We found that these activities gave LHDs greater confidence in the collaborative process and a clearer understanding of their roles as participants.
We found that the rigor of the curriculum and the enthusiasm of LHD participants were more robust among states interviewed in 2010 than in 2009. This confirmed our sense that states needed time to build expertise in running mini-collaboratives. These states also benefited from resources gained from pioneer mini-collaboratives, which were captured during national meetings and documented in an e-library maintained by grant sponsors. Subject-matter experts spoke favorably about the mini-collaborative model and saw it as a powerful tool for teaching new skills and facilitating peer networks for ongoing sharing once collaboratives ended. Although the mini-collaborative approach was viewed favorably, its variability across sites made any definitive conclusions on its ideal structure and length difficult. Subject-matter experts cautioned that the duration of a mini-collaborative should fit the task and that 12 months may be a reasonable timespan for planning and implementing most mini-collaboratives looking to achieve short-term goals.
Application of QI tools and techniques at the local level
In all 16 states, LHDs were required to implement a QI project as a requirement of participation in a mini-collaborative (eg, development of a community health assessment, workforce development). For many LHDs, the QI project was integral to the mini-collaborative; for others it was a stand-alone effort. Unlike traditional learning collaboratives, most LHDs worked on different QI projects. The diverse interests and activities of LHD participants challenged the concept of traditional collaboratives where curricula and learning are focused on common issues. Because of diverse QI projects, it often was not possible to address the unique needs of each LHD participant during shared learning sessions and some LHDs struggled when left alone to work on their QI projects during action cycles. Some states had resources to step in and provide individual technical assistance in these cases; others did not. Despite the obvious advantages of requiring a common QI project across LHDs, several states were reluctant to dictate the content of a QI project with specified interventions. Subject-matter experts saw the value of common QI projects as a way to scientifically examine outcomes.
The case studies revealed that the application of QI tools was an important driver of change and that several QI methods were particularly dominant including Plan-Do-Study-Act, affinity diagrams, fishbone diagrams, and run charts. However, findings suggested that LHDs often had difficulty determining the right tools to apply in their QI projects. States often struggled in finding the right balance between training participants on many QI tools versus training on a narrow set of tools for specific situations. Although our case studies did not focus on whether tools were applied appropriately across all participating sites, our findings revealed the importance of linking tools to QI projects and giving LHD participants a depth rather than breadth of understanding.
Case study results showed that LHD participants see QI projects as extra work that could easily be trumped by real work priorities. In their practice of QI, LHDs learned that change requires both the hard skills related to QI techniques and the soft skills of communication and relationship building. Most mini-collaboratives only focused on the hard skills. Local health department participants spoke of the need to establish better working relationships with state and local agency staff as a prerequisite to soliciting their help and participation on QI projects. Similarly, LHD participants noted that engaging community partners in their QI projects was often a new and challenging effort above and beyond the technical work of a QI project.
Sustainability and spread
Obstacles and facilitating factors
There was limited evidence of sustainability and broad spread of QI within MLC states. For LHDs participating in a mini-collaborative, the event had a major impact and will likely influence their work going forward, though it is unclear whether the impact will be program-specific or felt more broadly across the agency. Many obstacles were identified as thwarting its spread throughout an agency: no easily identified champion to push QI; lack of funding to get properly trained; difficulty in identifying opportunities to practice QI skills; lack of confidence in knowing the right QI tools or techniques to use; and difficulty connecting QI project work to national accreditation. Local health departments were more likely to report optimistic forecasts of sustainability and spread when there was a commitment of the state to provide ongoing QI technical assistance or when, as a result of the mini-collaborative, new connections were made with local partners. Also, LHD participants who were part of an agency QI team seemed to have greater confidence that their work would continue and potentially grow. Participants were more apt to indicate sustainability and spread when local leadership was perceived as playing a large role in fostering QI within an agency.
We asked state sponsors about their views on using mini-collaboratives in other statewide initiatives. More than half would use the model again as a tool for learning and engagement of LHDs. Other states indicated that the target area would determine whether a mini-collaborative model was appropriate. Changes in state leadership and local governance were mentioned as important determinants of future use, as was relevance to LHD priorities. The need for strategies to engage legislators, boards, and county commissioners was addressed by several states that saw these leaders as having influential roles in securing sufficient resources necessary to sustain QI at the state and local levels. Mini-collaboratives were seen as a useful means to economize and connect QI on statewide and local levels through joint priority setting, shared learning, and resource development in select areas.
The case studies were designed to advance our understanding of (1) the key drivers of a successful QI learning collaborative, (2) how local public health agencies implement QI strategies including the features that impede and enhance those efforts, and (3) the sustaining impact these efforts have on building capacity for future improvement. Our case studies were not set up to measure the success of MLC mini-collaboratives through quantifiable improvements in target areas. Data to do so were not available to substantiate and compare such claims across LHDs, given the variability described earlier. Our findings suggest that the impact of mini-collaboratives can be better understood as a catalyst for engaging LHDs in the theory and practice of QI. The absence of quantifiable measures and data collection reflect how these mini-collaboratives were conceived and perceived. For the most part, mini-collaboratives were not organized to create a step-by-step ladder to improvement. Their purpose was as much inspirational as tutorial. The process of learning, building awareness, and creating a peer community was seen as more fundamental than the result. Within this context, the mini-collaboratives have been successful.
We also saw mini-collaboratives change over the course of the MLC initiative in ways that suggest the more disciplined approach used in traditional learning collaboratives. The case studies identified a direction, if not a path, for the successful mini-collaborative of the future.
* A clear, unambiguous, and quantifiable goal for a mini-collaborative is essential to finding the right faculty, developing a coherent curriculum, and recruiting committed participants.
* A strong evidence base in the selected target area is the starting point for a mini-collaborative.
* The desire to provide LHDs with the flexibility to choose their own QI projects must be carefully weighed against the value of focusing on a common issue and intervention.
* The work of the mini-collaborative includes the aggregate learning cycles and the individual agency action cycles. Resources must be adequate to support both.
* Team-based learning is easier to reinforce and sustain in an agency than individual learning once a mini-collaborative is over.
* The mini-collaborative should be part of a broader state strategy for building QI capacity within LHDs that relate to enhancing performance in priority areas.
Our case study approach had several limitations that influenced our findings. First, this qualitative approach was not designed to evaluate the QI projects that were implemented as part of an LHD's participation in a mini-collaborative. The diversity of QI projects within and across MLC states, the multitude of interventions, the lack of standard measurements against which performance can be assessed, and the absence of required data collection made such an evaluation impossible. Instead, we focused on the perceived effectiveness of state-sponsored mini-collaboratives in building QI capacity and sustaining and spreading those gains over time. Second, the case studies provided a snapshot in time and may not capture the full impact of a mini-collaborative. Some site visits occurred in the early stages of a collaborative, and others were scheduled toward the end. Sometimes we looked at second or third generation collaboratives within a state, in others we observed a state's only collaborative over an extended period. Finally, despite our best efforts to include a group of LHDs representative of all LHDs participating in mini-collaboratives, the diversity and variability of mini-collaboratives across states made that goal unrealistic. Similarly, subject-matter experts selected for interviews do not represent faculty across mini-collaboratives. Although the evaluation cannot overcome these limitations, a more recent set of case studies conducted in 2011 has provided additional quantifiable information about the impact of the MLC initiative at the LHD level and we intend to publish these findings in the near future.
Our original case study theory was drawn from existing research and our results yielded findings that frequently aligned with this literature. But we also found that the use of learning collaboratives in public health is unique in several important ways. First, there is no obvious focal point for sponsorship that brings together the requisite combined skills of QI and subject-matter expertise. We learned that some states were ill equipped to assume this role and scrambled to find experienced faculty who could design an integrated curricula. Second, the evidence base for improving public health processes and practices is slim. Unlike traditional learning collaboratives where the task is to apply evidence-based practices, the MLC mini-collaboratives often developed interventions or adapted interventions from other settings. Testing and validating these approaches will be a necessary next step to developing a body of research that supports public health practice transformation.
© 2012 Lippincott Williams & Wilkins, Inc.