Domain 9 of the Public Health Accreditation Board Standards and Measures, Version 1.0 calls for health departments to evaluate and continuously improve processes, programs, and interventions.1Strengthening the Community of Practice for Public Health Improvement (COPPHI) is a multiyear project led by the National Network of Public Health Institutes and funded by the Robert Wood Johnson Foundation.2 The COPPHI project expands upon previous initiatives to nurture and expand a community of practice that is advancing accreditation, with a significant focus on quality improvement (QI) to build capacity to meet the Public Health Accreditation Board standards and measures set forth in Domain 9.3–6
One of the primary components of COPPHI is the QI Award Program that provides small grants and distance-based QI coaching to state, local, tribal, and territorial health departments. During each of 2 QI award cycles, 30 sites received $5000 and QI coaching to engage in QI projects that address local priorities across all accreditation standards and measures (see the Table for a description of each project). The limited funding allowed participating sites to support staffing, software, travel, training, meeting facilities, and supplies for their QI work.
The QI Coaching Model
The National Network of Public Health Institutes facilitated a team of 9 QI coaches who were selected through a competitive application process. Coaches were assigned to work with up to 4 sites each to provide individually tailored technical assistance to support health department QI projects/QI project teams. Although the coaching was provided primarily from a distance, the model did allow for 1 in-person meeting with the QI team lead from each site during the Open Forum for QI in Public Health, a twice annual conference that focuses on QI and accreditation. Up to 15 additional hours of coaching were provided from a distance. The QI coaches also assisted with documentation of QI project results for submission to the Public Health Quality Improvement Exchange, PHQIX.org.
Data Sources and Methods
Data sources include QI coaching satisfaction surveys completed by each of the sites, site progress reports, QI coaching logs, and QI coach meeting notes gathered during the first cycle of the QI Award Program (30 sites). The QI coaching satisfaction survey was designed to assess the quality and effectiveness of the individual technical assistance provided by the QI coaches. It was administered to all project team members identified by each QI award site via a Web-based platform midway through the QI project and upon completion of their work. The overall response rate for cycle 1 was 46% (n = 47/98).*
The QI coaches were asked to maintain technical assistance logs that documented the amount of time spent providing the coaching, the methods used, and the topic areas covered. Coaches also provided input during regularly scheduled meetings between COPPHI project staff and the QI coaches. Technical assistance logs and notes from these meetings were used to gain further insight into the effectiveness of the QI coaching model and to identify mid-course adjustments.
Progress reports collected from the 30 sites were designed to assess increased capacity related to QI. Progress report data were collected from sites via a Web-based platform at the mid-project point and upon conclusion of their work. The overall response rate for the progress reports was 100%, although the response rate for individual questions varied.
QI coaching hours and methods
On the basis of findings from the QI coaching satisfaction surveys, QI coaching was provided to 85 staff members across the 30 sites. The methods used to provide distance-based QI coaching included phone (96%), e-mail (81%), and Web-based communication, such as webinars or video conferencing (19%).
Review of the technical assistance logs reveal the highest number of hours spent in direct communication between a coach and a single site was 16.8 whereas the fewest number of reported hours was 3.8. Six coaches offered a breakdown of the number of hours (in increments of 0.25) spent per site over the course of the entire QI project. These data show that some coaches averaged 14 hours per site whereas others averaged only 5.6 hours per site. In addition to the variations noted across sites in general, variations among different sites assigned to the same coach were also noted. The detailed log from one coach demonstrated that more time was needed for planning and preparation for the QI coaching sessions (12.8 hours) than for actual sessions themselves (9.7 hours). From review of the logs, it is not clear whether the variance in time needed for planning and preparation was due to differences across sites, coaches, or specific coaching topics.
Site impressions of QI coaching
Nearly 80% of cycle 1 respondents (37/47) indicated that they strongly agreed or agreed that distance-based QI coaching is effective. Open-ended feedback from participating sites suggests that having access to a QI coach allowed for validation of their QI work, helped focus project scope and prioritize activities, and provided clarification and feedback on the use of QI tools and process throughout each step of project implementation. Sites also noted the usefulness of technical expertise, particularly relating to measuring progress and analyzing data. Many responses also highlighted the importance of having a coach who was willing to be flexible and available.
Data from the final cycle 1 reports revealed several accomplishments from participating sites, including increased experience with QI processes and the initiation of QI spread within the health department. All sites (n = 30) developed an aim statement and refined the statement at least once—frequently more often—throughout the course of their work. Almost all sites reported using other QI processes, including collecting and analyzing data (100%; n = 29), mapping a process (93%; n = 27), setting measurable objectives (86%; n = 25), and testing the effects of an intervention (83%; n = 24). The QI Award Program sites also indicated that they were able to develop other important resources to support their QI processes, including: QI team charters, surveys, focus group guides, educational flyers, and PowerPoint trainings.
Twenty-nine of 30 sites noted that their QI work had stimulated some level of additional activity within and beyond the project team. The 2 most frequently reported activities included sharing QI project results internally and/or externally (80%; n = 24) and providing training to staff in QI methods (63%; n = 19). The one site that indicated that QI had not spread further within the health department stated that rather it has stimulated further advancement of its community health assessment efforts.
Timing of in-person meeting
During the first cycle of the QI Award Program, the QI coaches and the participating sites met in person about 4 months after the project work had started. During the second cycle, the QI coaches met with their sites at the beginning of the project. Feedback from sites and QI coaches from both cycles indicated that having at least 1 in-person session at the beginning of the project was invaluable to establishing a solid working relationship.
Suggestions to improve coaching
When asked for their top recommendation for improving the QI coaching process, responses included communicating clearer expectations, planning more structured coaching sessions, and using interactive technology. Findings suggest that it is essential that the roles of the site and the QI coach be made explicitly clear. Sites and QI coaches reported that it was useful to have clarity around the QI coaching process, including (1) what will occur during each coaching session and what preparation is needed, and (2) who will be responsible for which steps in the QI coaching process.
On a related note, the lack of availability of their coach was a frustration noted by some sites; likewise, some coaches noted difficulties getting started with their teams. For the QI coaching model to be effective, it is vital that the site and the QI coach be available to respond to requests. Scheduling and availability did become more difficult when the QI coach and the site are located in distant time zones.
As much of the work of QI is highly visual, such as developing a cause and effect diagram, in-person meetings were also found to be helpful. If meeting in person is not possible, it is incumbent on both the site and the QI coach to ensure the use of technology that is as interactive as possible, including video conferencing and webinar platforms.
Finally, the QI project time frame and the amount of distance-based QI coaching allotted were the same for all sites, regardless of topic being addressed or the team's familiarity with QI. Feedback from coaches suggests that some sites needed less assistance whereas others needed more. In addition, many coaches expressed difficulty keeping the momentum going over the full course of the project because many of the projects could have been completed on a shorter time frame.
Approximately 80% of QI satisfaction survey respondents indicated that they strongly agreed or agreed that distance-based coaching is effective, and sites also reported increased experience with QI processes, initiation of QI spread within the health department, and additional activity within and beyond the project team due to the QI Award Program. While a number of accomplishments were realized, and the satisfaction level with distance-based coaching was generally high, certain challenges were also reported by both sites and QI coaches, which suggests that a one-size-fits-all approach to QI coaching is not adequate. Time should be spent on the front end to accurately scope the hours of QI coaching needed (including preparation time), the expectations of sites and QI coaches, and the technologies that will be used to respond to each site's individual needs.
The preliminary findings related to distance-based QI coaching provided to the first 30 sites participating in the QI Award Program have been used to inform programmatic improvements. As cycle 2 concludes, data from cycle 2 will be aggregated with the cycle 1 data to provide a more robust picture of the effectiveness of distance-based QI coaching. In addition, an analysis will be conducted to identify the accreditation standards and measures that each QI project addressed, which will provide insight into how QI may be used to increase accreditation readiness related to specific standards and measures. As QI becomes more ingrained in health department culture, understanding what makes the QI coaching model effective and communicating those findings with key stakeholders can continue to increase QI capacity among public health practitioners and assist health departments with preparation for Public Health Accreditation Board accreditation.