Cornett, Amanda MPH; Thomas, Monecia MHA; Davis, Mary V. DrPH, MSPH; Mahanna, Elizabeth MPH; Cordova, Andrea BA; Herring, Claire MHA; Lea, C. Suzanne PhD; Harrison, Lisa Macon MPH; Randolph, Greg D. MD, MPH
The public health system is the first line of defense to keep communities healthy and protected from disasters. However the ability of public health agencies to perform the core functions of public health has been compromised because of limitations in funding and organizational capacity and the lack of a skilled workforce. This performance gap highlights the need to improve the overall public health infrastructure.1,2
Efforts in the last decade have begun to improve the infrastructure. The 10 Essential Services of Public Health defined the core public health services and provided the foundation for the development of accreditation standards. These standards have been used by both state and local public health accreditation systems to evaluate the capacity of agencies to perform the core functions.3,4 Although these efforts will help drive improvement, accreditation alone cannot create improvement. Instead, it must be a part of an overall performance management system that includes ongoing measurement and quality improvement (QI).4,5 In essence, accreditation is the “assessment” component that identifies gaps for improvement and QI is the “action” component that uses tools and strategies to impact accreditation findings by identifying the root cause(s) of an issue, testing and implementing solutions, and continuously measuring the effectiveness of the improvements and the quality of services.6
North Carolina (NC) is one of many states actively working to improve the public health system by integrating accreditation and QI. In 2005, NC was the first to develop and legislatively mandate an accreditation program to assure that local health departments (LHDs) across the state had the capacity to perform the 10 Essential Services and 3 core functions of assessment, assurance, and policy development.3 This accreditation program strives to standardize the performance of LHDs and drive statewide adoption of QI within NC's decentralized public health system, a system in which 85 LHDs operate autonomously under local authorities and receive local and state funding to perform public health responsibilities. In addition, as a participant in the Multi-State Learning Collaborative (MLC)—a Robert Wood Johnson Foundation–supported initiative to inform accreditation efforts and explore the use of QI within public health—LHDs across the state were introduced to QI via distance learning trainings, regional face-to-face workshops, and QI learning collaboratives. These activities helped increase awareness and interest in applying QI and emphasized the need to build the capacity of LHDs to use QI to improve the performance of their agency to ultimately impact health outcomes.
In 2009, the NC Public Health Foundation and the NC Division of Public Health (DPH), with financial support from local foundations, created the NC Center for Public Health Quality (CPHQ). In late 2010, CPHQ was able to greatly expand its impact with the National Public Health Improvement Initiative funded by the Centers for Disease Control and Prevention. The mission of CPHQ is to create an infrastructure to foster and support QI among all local and state public health professionals by providing training and technical assistance, sharing evidence-based practices, providing performance measurement, leading strategic QI initiatives, and engaging public health leadership to support QI. The overall infrastructure of CPHQ was modeled after The North Carolina Hospital Association Center for Hospital Quality and Patient Safety, because of its demonstrated ability to engage NC hospitals in successful QI projects.7
During the first year of operation, CPHQ created a QI training program to build the capacity of LHD staff to apply QI methods and tools in their daily work. This article describes the adaptation and creation of the Public Health Quality Improvement (PH QI 101) training program, the formative evaluation of the effectiveness of the program, and lessons learned from program development.
Intervention development—pilot testing of an existing health care QI training program
In June 2009, teams from 4 LHDs and a curriculum development team (comprising public health professionals from CPHQ, DPH, and the NC Institute for Public Health (NCIPH)) attended the NC Area Health Education Centers (AHEC)/NC Hospital Association's (NCHA's) QI program to pilot test and adapt it for LHD staff. Before the first workshop, each LHD chose a QI project and assembled a team of 3 to 6 staff. CPHQ provided funding to each team for program tuition, travel reimbursements, and supplies for their QI project.
The AHEC/NCHA program is a 5-month program adapted from the Institute for Healthcare Improvement's Breakthrough Series (BTS) Collaborative model. The BTS model has demonstrated successful results in health care8 and most recently in public health9,10 and provides an opportunity for a group of organizations to collaborate over a 9- to 18-month period to learn QI methods to improve performance. The BTS model utilizes the Model for Improvement (MFI),11 a QI framework that focuses on 4 fundamental components: (1) setting an aim, (2) developing measures to track improvements, (3) identifying ideas for change, and (4) rapidly testing and adapting changes with plan-do-study-act (PDSA) cycles.
The AHEC/NCHA program was chosen as the model for this training program for several reasons: The program has successfully attracted teams from hospitals, medical practices, and other health care organizations across NC; it has demonstrated substantial gains in increasing participants' confidence to apply QI; and it has shown measurable improvements in patient care. In addition, the curriculum development team believed that if LHD staff learned to use the same QI framework (the MFI) as other health care and public health system partners; it would facilitate future collaborations and improvement initiatives at the community across the state.
Intervention-–overview of the public health quality improvement program (PH QI 101)
On the basis of the feedback from the 4 LHD pilot teams, the curriculum development team capitalized on the strengths of the AHEC/NCHA program and made adaptations, which we believed would increase its applicability to public health. (Table 1 summarizes strengths and opportunities for improvement identified by the pilot sites and adaptations made to the AHEC/NCHA program.)
Figure provides an overview of the adapted PH QI 101 training program. The objectives of the program are to help participants:
FIGURE . Overview of...Image Tools
* Understand, select, and use QI methods and tools in their daily work activities to improve the efficiency and effectiveness of services provided;
* Coach others within their local agency to use QI methods and tools; and
* Develop a plan to incorporate QI methods and tools within their local agency so that QI becomes “the way to do business,” thus creating a culture of continuous QI.
The PH QI 101 training program provides instruction in the MFI, Lean principles and tools, and change management topics including creating effective teams, motivating others to change, and sustaining and spreading QI. Lean was incorporated into the training program on the basis of pilot test results of 10 LHDs within the Northeastern North Carolina Public Health Partnership. Lean is a continuous process improvement method, adapted from the manufacturing industry, that focuses on identifying and eliminating non–value-added (wasteful) activities as defined from the customer's perspective.12 Lean utilizes the Kaizen improvement event (a 4-day, on-site improvement event within the LHD) to create rapid improvement within a given process.13 The Kaizen improvement event provides the opportunity to observe a process, identify wasteful activities, and immediately test and implement changes to the process using PDSA cycles.
Before participating in the 6-month training program, an LHD chooses a QI project on the basis of gaps identified in accreditation findings, employee/client satisfaction surveys, organizational strategic goals, and/or program monitoring results. Each LHD assembles a team of 3 to 6 staff members, who participate in all components of the training program as described next:
* A 2-month prework phase occurs prior to the first face-to-face workshop and consists of three 1-hour webinars. The webinars provide instruction on selecting a project, assembling a team, developing a project aim statement, and using QI tools (eg, a process/value stream map and observational walks) to understand the chosen process for improvement;
* A 2-day face-to-face workshop provides instruction on developing measures to track improvements, using QI tools to identify changes, and testing changes using PDSA cycles. In addition, during the first workshop, local health directors (from each participating team) attend a half-day face-to-face breakout session to learn how to create an organization that supports and sustains QI. The session provides instruction in Kotter's 8-Steps of Change14 and includes a panel discussion with other local health directors who have implemented and sustained QI throughout their agency;
* A 4-month “action period” provides the opportunity for LHD teams to apply QI tools within their agency. Teams make changes using PDSA cycles and participate in a Kaizen improvement event. (Participation in a Kaizen improvement event is optional, but LHDs are strongly encouraged to participate.) In addition, LHD teams collaborate via monthly webinars and report on project progress by submitting monthly reports;
* A final 2-day face-to-face workshop provides the opportunity for teams to celebrate their successes, share lessons learned, and develop a plan to sustain their project and spread QI throughout their agency;
* Individualized technical assistance and coaching is provided by faculty via e-mail and phone throughout the program; and
* Quarterly “alumni” webinars occur after the final workshop to assess progress and provide additional instruction in QI methods and tools.
Intervention-–implementation of the PH QI 101 training program
The CPHQ launched the first cohort of the PH QI 101 training program in February 2010 with 8 LHDs. Local health departments were recruited through site visits, presentations, and exhibits at local and regional public health meetings, and e-mail newsletters. The first 8 LHDs that registered for the program were selected to participate. Each LHD received funding from CPHQ for the Kaizen improvement event, travel reimbursement, and supplies for their QI project. (Table 2 provides a description of cohort 1 participants.) An additional 23 LHDs were placed on a waiting list for upcoming cohorts of the training program. All 85 NC LHDs will have the opportunity to have a team participate in the program within a 5-year period.
Measurement and evaluation of the PH QI 101 training program
The CPHQ partnered with the NCIPH, to conduct a hybrid internal and external formative evaluation of the program. The evaluation examined how well the training achieved the goal of increasing the capacity of participants to conduct QI. Evaluation activities addressed the 4 levels of a training evaluation as outlined by Kirkpatrick.15 Participant reaction was assessed through measures of participant satisfaction and suggestions for improving the training program (second workshop evaluation form). Learning was assessed through measures of the extent to which participants thought training goals were achieved (workshop evaluation forms) and participant confidence to conduct QI activities (pre/postsurveys). Application of QI skills to work was examined through achievement of project aims by the end of the training program. Organizational change was evaluated through measures of sharing of QI tools with other staff members, which indicates a climate of continuous improvement (pre/postsurveys). Items on the pre/postsurvey had been pilot tested as part of another QI training evaluation project.16 Items on workshop evaluations had been previously used on CPHQ and NCIPH workshop evaluations.
Analysis of evaluation items included descriptive statistics to identify trends in participant satisfaction and major areas for program improvement. For pre/postmeasures of confidence to perform QI skills and sharing of QI tools with other staff, we compared pre/postmean scores and confidence intervals for trends. The study protocol was determined to be exempt by the institutional review board at University of North Carolina at Chapel Hill. All data were protected on password protected servers and individual identifiers removed.
Local health departments in the PH QI 101 training program
Eight LHDs (40 staff members and 6 local health directors) participated in the first cohort of the PH QI 101 training program. The 8 LHDs varied in size of population served (Table 2). Seven agencies had received accreditation prior to the training. Twenty-six percent of participants had previously participated in a QI project.
Program evaluation results
Forty-two participants (staff and health directors) representing the 8 LHDs completed the presurvey and 30 individuals completed the postsurvey. Thirty-six individuals completed workshop 1 and 38 completed workshop 2 evaluation forms. Variation in completion rates reflect the fact that not all staff participated in all training components and, in some health departments, staff were unable to complete the entire training program.
Reaction to the training program
Table 3 shows participant satisfaction with various components of the training program and the number of participants that provided ratings for each component (evaluation administered at the end of workshop 2). As a benchmark of high satisfaction, we examined the percentage of participants who rated each program component as a 5 or 6 on a scale of 1 = poor and 6 = excellent. Participant satisfaction ratings ranged from 60% for the prework phase to 94% who reported that they were very satisfied with the overall training program.
Participants provided specific suggestions to improve the training through multiple feedback mechanisms. These included improving prework webinar content, reducing content repetition, increasing program pace, and standardizing and structuring the role of coaches. Respondents also reported some initial challenges with training program technology (eg, use of an online server and use of webcams), which improved over the course of program delivery.
After workshop 1, 91% of 35 participants agreed or strongly agreed that the workshop content helped them learn to apply QI methods and tools in their daily work activities. Following workshop 2, 84% of participants agreed or strongly agreed that the workshop content helped them demonstrate the ability to use the MFI and Lean to design a new QI project (n = 38).
Confidence to conduct QI projects
Each participant was asked to rate their confidence in conducting a QI project on a scale of 1 = not at all confident to 6 = completely confident on pre/postsurveys. Preprogram mean confidence to conduct a QI project was 4.43 (95% confidence interval 4.04–4.81; n = 42) and postprogram mean confidence was 4.80 (95% confidence interval 4.53–5.07; n = 30). Using a rating of 5 or 6 as confident, a total of 70% (n = 30) of participants were confident to conduct a QI project following the training program as compared with 48% of participants (n = 42) prior to the program.
Application to work
During the training program, LHD teams implemented a wide variety of improvement projects within their agency. All 8 teams chose to participate in a Kaizen improvement event. Of the 8 projects implemented, 7 resulted in measurable improvements in outcome or process goals. Table 4 summarizes the QI projects and improvements. These results were obtained at program completion and will be reexamined 6 months posttraining.
An additional measure of applicability of content to participant work is use of it on the job. Participants reported using content in the following ways. Ninety-three percent of participants indicated using data and display methods and 87% indicated using QI tools (Pareto charts, run charts, PDSA cycles) and 63% indicated using the MFI and Lean.
Participants were asked to report sharing of QI methods and tools with coworkers on a pre/postsurvey. Before participating in the training program, 43% of participants had shared QI methods and tools with coworkers and 100% of participants reported sharing tools following the training. Among those who had shared tools, participants reported on average they had shared them with 9 coworkers prior to the training program. Posttraining, this question was asked differently on the survey. Among those who had shared tools, more than a third had shared them with more than 10 coworkers. Eighty-seven percent of participants indicated sharing a variety of QI tools (ie, Pareto charts, fishbone diagrams, and PDSA cycles) and 80% indicated sharing QI data and display methods with their coworkers.
This article describes the development and implementation of a new QI training program for LHD staff. The program was adapted from an existing program for health care professionals.
Staff and health directors from 8 LHDs successfully completed the program with nearly all reporting that they were satisfied with the program. Participants' perceived ability to conduct a QI program was higher following program completion.
The 8 LHDs that participated in the program varied in the size of population served. Previous evidence shows that larger agencies are more likely to engage in QI activities than smaller agencies.17 Results from this training program suggest that despite smaller size and limited human and financial resources, agencies of all sizes can successfully implement QI within their agency. The smaller LHDs may actually benefit more in some cases because increasing efficiency will allow them to do more work with fewer resources. In addition, participating in the training may provide a form of continuing education and professional development that employees in smaller agencies might not otherwise receive. To ensure that all smaller agencies in NC continue to successfully complete the training and implement QI, we believe it will be important to provide financial assistance to cover training and travel costs and to continue to prioritize smaller agencies to participate in future trainings.
Several adaptations were made to the AHEC/NCHA program to create the PH QI 101 training. On the basis of preliminary participant and faculty feedback, the leadership session (provided during workshop 1) and instruction in Lean principles and tools were successful additions to the PH QI 101 training. The leadership session aimed to facilitate health directors' understanding of the importance of establishing an infrastructure to support QI. To foster QI within an agency, both a top-down—leadership creates the vision and urgency to do QI, and a bottom-up—the workforce is equipped to apply QI in their daily activities approach is needed.18–20
In addition, preliminary feedback suggests that incorporating Lean principles and tools, specifically the Kaizen improvement event, strengthened the training. The Kaizen improvement event provided LHD staff the opportunity to gain hands-on experience using QI tools to test and implement changes that rapidly improved daily work flow. Many participants felt that the event helped increase staff buy-in and excitement for QI. Future evaluations will focus on how both of these additions advance QI in NC LHDs.
Although the program shows promising success, we identified the following areas for improvement on the basis of participant and faculty feedback. Although the leadership session created awareness and helped health directors begin to identify ways they could increase agency-wide support of QI, we believe involving health directors earlier in the training process and on an ongoing basis will be most beneficial. To address this, a half-day face-to-face workshop will be held prior to the prework phase. Health directors and the person responsible for leading the QI project will be required to attend. The session will outline the expectations of the PH QI 101 training program, discuss the role of the health director in supporting the QI team, and introduce Kotter's 8 Steps of Change14 to help leaders identify ways they can sustain QI. In addition, a webinar will be held midpoint of the training program with a final half-day face-to-face session held after workshop 2.
Technical assistance and coaching is important and helps teams successfully complete the training program. Although teams were successful in increasing their confidence and reaching project goals, many participants felt that technical assistance varied between faculty experts. This impression may have resulted from limited guidance given to faculty regarding their role as a coach. To address this issue, we created a roles and responsibilities document to review with each faculty prior to the start of the program. A faculty's role as a coach is to provide feedback on all completed assignments (eg, aim statement, measures, and PDSA cycles), conduct at least 2 phone calls per month with their assigned QI team(s), review monthly progress reports, and attend monthly conference calls with other faculty to review the progress of their assigned QI team(s). In addition, faculty will be assigned a maximum of 2 QI teams to allow adequate time to interact and provide the needed technical assistance.
Although participants were highly satisfied with the workshops, many recommended changing the structure of the workshops such that it is spread over 3 days rather than 2 (eg, ½ day the first day, 1 full day the second day, and a ½ the third day). This will allow participants the opportunity to process the material and provide travel time to and from the workshop.
In addition, we offer the following recommendations on the basis of our learnings. First, identify and collaborate with key stakeholders and partners within local and state agencies to build on existing programs or resources. Second, establish a standard method of QI that is consistent with other industries and organizations within your state. Instruction in both the MFI and Lean principles provides LHDs in North Carolina an opportunity to collaborate with businesses, hospitals, and other health care organizations in their communities on larger system improvement initiatives that focus on population health outcomes since these are common QI methods used in health care and business settings in North Carolina.
It is also important to recruit a cadre of faculty with experience leading and implementing QI initiatives in public health. Faculty expertise should consist of both theoretical and practical knowledge of how to apply QI. Utilizing LHD staff (who have implemented QI) as faculty provides a great opportunity as it creates an environment of peer-to-peer learning and provides continued professional development for staff. Finally, continuous improvement and measurement should be built into the development of a training program. Following each component of the program, rapid feedback was collected from participants to identify changes to immediately improve the content and structure of the program.
To successfully spread and disseminate the training program, it will be important to consider the characteristics of LHDs. It is important to recruit innovators and early adopters who can create early wins and generate buy-in from other LHDs. These organizations tend to be more likely to take risks and be successful in adopting innovative approaches.21 Success stories from participants in this first cohort have generated such demand that there is currently a waitlist for LHDs to participate in the training program despite doubling the training capacity in late 2010. Continued success and dissemination will require identifying additional strategies to engage LHDs who are more averse to change and less interested in adopting QI.21
This study has several limitations. Although the 8 LHDs that participated in the program reflect a diverse range of agencies in North Carolina, they may not be a representative sample of those in other states. In addition, this training program does require a significant investment of human and financial resources. Finally, this is a formative evaluation; findings are preliminary and have been used to inform program improvements. Despite these limitations, we believe many aspects of this training model can be adapted by other states interested in developing a QI training program.
A commonly held belief about systems is that “Every system is perfectly designed to get the results that it gets.” Over the last 10 years, several efforts have been initiated to improve the performance of the public health system, including accreditation and QI. Quality improvement involves a set of knowledge and skills that are new to the public health workforce. Thus, to ensure that we have a system that supports and sustains performance improvement, it will be critical to increase the capacity of the public health workforce to use QI methods and tools to monitor and continuously improve services. In addition, we must engage public health leaders to create a vision and urgency to improve and transform public health. The PH QI 101 training program is one model that others can use to begin creating a system to support and sustain QI in public health organizations.
2. Institute of Medicine. The Future of the Public's Health in the 21st Century. Washington, DC: National Academies Press; 2002.
3. Beitsch LM, Mays GP, Corso L, Chang C, Brewer R. States gathering momentum: promising strategies for accreditation and assessment activities in multistate learning collaborative applicant states. J Public Health Manag Pract. 2007;13(4):364.
5. Mays G. Can Accreditation Work in Public Health? Lessons From Other Service Industries. A Report Prepared for the Robert Wood Johnson Foundation. Little Rock, AR: University of Arkansas for Medical Sciences; 2004.
6. Baker S, Beitsch L, Landrum L, Head R. The role of performance management and quality improvement in a national voluntary public health accreditation system. J Public Health Manag. 2007;13(4):427–429.
8. Institute for Healthcare Improvement. The Breakthrough Series: IHI's Collaborative Model for Achieving Breakthrough Improvement. Boston, MA: IHI Innovation Series White Paper; 2003.
9. Gillen S, McKeever J, Edwards K, Thielen L. Promoting quality improvement and achieving measurable change: the lead states initiative. J Public Health Manag Pract. 2010;16(1):55–60.
10. Lotstein D, Seid M, Ricci K, Leuschner K, Margolis P, Lurie N. Using quality improvement methods to improve public health emergency preparedness: prepare for pandemic influenza. Health Aff. 2008;27(5):w328–w339.
11. Langley GL, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Guide to Enhancing Organizational Performance. 2nd ed. San Francisco, CA: Jossey-Bass; 1996.
12. Womack J, Jones D. Lean Thinking. 2nd ed. New York, NY: Simon & Schuster Inc; 2003.
13. Martin K, Osterling M. The Kaizen Event Planner: Achieving Rapid Improvement in Office, Service, and Technical Environments. New York, NY: Productivity Press; 2007.
14. Kotter JP. Leading change: why transformation efforts fail. Harv Business Rev. January 2007:92–107.
15. Kirkpatrick DL. Evaluating Training Programs. 2nd ed. San Francisco, CA: Berrett-Koehler Publishers Inc; 1998.
16. Davis MV, Vincus A, Eggers M, et al. Effectiveness of public health quality improvement training approaches: application, application, application. J Public Health Manag Pract. E1–E7.
17. Beitsch LM, Leep C, Shah G, Brooks RG, Pestronk RM. Quality improvement in local health departments: results of the NACCHO 2008 survey. J Public Health Manag Pract. 2010;16(1):48.
18. Davis MV. Opportunities to advance quality improvement in public health. J Public Health Manag Pract. 2010;16(1):8.
19. Lesneski CD, Massie SE, Randolph GD. Continuous quality improvement in public health organizations. In: Sollecito WA, Johnson JK eds. McLaughlin and Kaluzny's Continuous Quality Improvement in Health Care. 4th ed. Sudbury, MA: Jones & Bartlett Learning; 2013.
20. Riley WJ, Rasons HM, Duffy GL, Moran JW, Henry B. Realizing transformational change through quality improvement in public health. J Public Health Manag Pract. 2010;16(1):72.
21. Rogers EM. Diffusion of Innovations. 5th ed. New York, NY: Free Press; 2003.
public health; quality improvement; training curriculum; workforce development
© 2012 Lippincott Williams & Wilkins, Inc.