The national accreditation program, implemented by the Public Health Accreditation Board (PHAB), is designed to advance the quality and performance of public health departments in the United States. To be recognized as an accredited health department, applicants assess themselves against a set of Standards and Measures that are designed to support and advance the health department's quality improvement (QI) and performance management efforts.1 QI in public health is
the use of a deliberate and defined improvement process ... focused on activities that are responsive to community needs and improving population health. It refers to a continuous and ongoing effort to achieve measurable improvements in the efficiency, effectiveness, performance, accountability, outcomes, and other indicators of quality in services or processes which achieve equity and improve the health of the community.2 (p6)
The implementation of QI activities by health departments can lead to positive changes within the health department, such as improvements in efficiency and effectiveness.3 , 4 For example, QI activities have helped build and develop workforce capacity5 , 6; increased efficiencies such as time7 and costs saved8 or reduced steps in a process2; improved effectiveness3 such as customer and staff satisfaction, data quality,9 and public health programs and services2; and enhanced supportive environment and culture for QI within public health agencies.10–13 When QI is integrated across the health department, both within specific projects and across all programs and activities, it can be considered integrated into the health department's culture.2 The PHAB accreditation program is working to strengthen QI and performance management infrastructure, including activities, processes, and culture, of public health departments throughout the country.14
To assess the QI and performance management benefits of PHAB accreditation, NORC at the University of Chicago conducted Web-based surveys of applicant and accredited health departments at 3 different steps of the accreditation process. NORC received funding support to conduct the surveys and other data collection activities from (1) the Robert Wood Johnson Foundation (RWJF)15 , 16 and (2) PHAB, through funding provided by RWJF.17 This article describes selected survey data related to the perceived QI and performance management benefits of the PHAB accreditation program, as reported by applicant and accredited health departments that completed the surveys. Additional quantitative and qualitative data have been published describing internal and external accreditation benefits related to accountability, transparency, communication, and collaboration reported by applicant and accredited health departments, among other evaluation findings.18
We conducted 3 Web-based surveys of health departments that applied for PHAB accreditation. The surveys were designed to gather feedback from applicant health departments as they completed 3 milestones in the accreditation process. These milestones were as follows: (1) registering their intent to apply for accreditation (referred to as the applicant survey); (2) achieving accreditation (referred to as the accredited survey); and (3) 1 year after achieving accreditation (referred to as the follow-up survey). The NORC Institutional Review Board reviewed the study protocols and determined them to be exempt from full review. A description of the survey population, data collection procedures, and methods for analysis for each of the 3 surveys follows.
We used a total population sample to survey the universe of health departments that applied for and achieved PHAB accreditation between November 2013 and May 2017. During this time frame, we surveyed all applicant and accredited health departments at 3 points in time. For the applicant survey, we contacted every health department that had registered its intent to apply for accreditation in PHAB's online system, e-PHAB. For the accredited survey, we contacted every health department that achieved accreditation between November 2013 and May 2017. For the follow-up survey, we contacted every health department that reached its 1-year anniversary of achieving accreditation.
To determine which health departments to contact, PHAB provided a list of every health department that had reached each milestone. PHAB also provided the name and contact information for each health department's director and designated accreditation coordinator. To recruit participation in the survey, we sent an e-mail invitation to each health department's director. The invitation included a link to complete the Web-based survey. We also sent an e-mail to each accreditation coordinator, informing him or her that we had contacted his or her health department director regarding the survey. We requested that the health department director complete the survey but accepted responses from the accreditation coordinator or another staff designee. We sent several reminders to both the health department director and the accreditation coordinator, encouraging them to complete the survey. Each survey remained in the field for approximately 6 weeks, and we sent no more than 3 reminders for each survey.
Between November 2013 and May 2017, we fielded the survey to multiple cohorts of health departments as they reached each of the accreditation milestones. The applicant survey was sent to 15 cohorts of health departments prior to their participation in the accreditation training for applicant health departments; 267 health departments received the applicant survey. The accredited survey was sent to 14 cohorts of health departments that had achieved accreditation; 160 health departments received the accredited survey. The follow-up survey was sent to 13 cohorts of health departments after they reached their 1-year anniversary of achieving accreditation; 118 health departments received the follow-up survey. Across the 3 surveys, we received 479 responses—231 responses to the applicant survey (response rate of 87%), 145 responses to the accredited survey (response rate of 91%), and 103 responses to the follow-up survey (response rate of 87%). See Table 1.
We created 3 survey instruments with a combination of open- and closed-ended response questions that were designed to gather information from respondents about the PHAB accreditation process, the experience of applicants, and the initial benefits of accreditation. The applicant survey included questions about the motivators and anticipated internal benefits of PHAB accreditation, current QI activities, and current QI and performance management infrastructure. The accredited survey included questions about the internal benefits, changes in QI activities, changes in QI and performance management infrastructure, and other perceived benefits resulting from PHAB accreditation. The follow-up survey included questions about the benefits, changes in QI activities, QI, and performance management infrastructure, and other benefits or outcomes experienced in the past year as a result of being accredited.
Each survey instrument maintained a core set of questions between November 2013 and May 2017. In late 2015, each survey instrument was modified slightly to include new questions about perceived QI and performance management benefits of PHAB accreditation. Among the 15 cohorts that received the applicant survey, 7 cohorts received the revised version; among the 14 cohorts that received the accredited survey, 6 received the revised version; and among the 13 cohorts that received the follow-up survey, 6 received the revised version. We received 219 responses to the new questions added to the revised survey—90 responses for the applicant survey, 94 responses for the accredited survey, and 57 responses for the follow-up survey.
When data collection began in November 2013, the PHAB accreditation program had been underway for approximately 2 years. Therefore, some health department cohorts did not receive invitations to complete the applicant or accredited surveys because they had already passed the designated milestones in the accreditation process. In addition, health departments vary in the amount of time it takes to undergo the accreditation process—some may take a shorter or longer time to achieve accreditation after registering in e-PHAB. Thus, not all health departments that complete the applicant survey at one point in time subsequently receive the accredited or follow-up surveys at the same time.
NORC researchers prepared data collected by reviewing and cleaning the raw data, including recoding response options and corresponding variables as appropriate. We created 3 clean data files for all quantitative data gathered from each survey using SAS and Excel. A corresponding survey codebook was created with an index of survey questions, indicating whether the survey question was core or added in 2015. For quantitative data, we conducted univariate analyses to determine frequency distributions and averages. We conducted these analyses for each survey, grouping all responses to the core survey questions and grouping responses to the added questions in 2015.
We conducted longitudinal analyses for a subset of health departments (n = 72) that received and responded to both the applicant and accredited surveys. The longitudinal analyses allowed us to compare changes within the health departments over time, from before undergoing the accreditation process to after achieving accreditation. To date, an insufficient number of health departments have completed other combinations of surveys (ie, accredited and follow-up surveys); therefore, longitudinal data from these respondents are not presented. Survey data continue to be collected with funding from RWJF,17 so there is future opportunity to conduct longitudinal analyses with a greater number of health departments, as the number of health departments that have responded to multiple surveys increases over time.
Characteristics of survey respondents
Across the 3 surveys, we received 479 responses from 324 unique health departments; 155 health departments responded to 2 or more of the surveys. Among the unique responses, 88% were local health departments and 11% were state health departments; there were also 2 tribes, 2 multijurisdictional applicants,* and 1 integrated health system.† Of the 35 unique state health departments, the governance structure for public health was as follows: 23 decentralized, 10 centralized, and 2 mixed. Of the 284 unique local health departments, 88% were in states with decentralized public health governance structures, 4% were in centralized states, and 8% were in mixed governance states. Among the local applicants, 40% had a population of less than 100 000, 39% had a population between 100 000 and 500 000, and 21% had a population above 500 000. Survey responses were received from applicant and accredited health departments across 45 states.
QI processes and activities
Following their accreditation decision, public health departments were sent the accredited survey and were asked about the immediate benefits of PHAB accreditation. Nearly all respondents to the accredited survey reported ongoing QI activities. The majority of recently accredited health departments (97%) reported that because of their participation in the accreditation process, they had implemented or planned to implement new strategies for QI, compared with 67% of health departments that reported doing so before undergoing accreditation. Ninety-six percent of recently accredited health departments also reported that because of their participation in PHAB accreditation, they had implemented new strategies to evaluate their agency's effectiveness and quality, compared with 83% of health departments that reported doing so before undergoing accreditation. In addition, 95% of recently accredited health departments reported that because of their participation in the accreditation process, they have informed or plan to inform decisions based on their QI or performance management processes, compared with 75% of applicant health departments that reported doing so.
QI process and activities: Longitudinal analysis
Longitudinal analysis was conducted for a subset of health departments that received and responded to both the applicant and accredited surveys. Data from these analyses indicate an overall increase in the level of agreement among respondents regarding their health department's use of QI and performance management strategies following accreditation, compared with before undergoing the accreditation process. After becoming accredited, 69% of respondents strongly agreed that they had implemented or planned to implement new QI strategies, more than 3 times the percentage of respondents who strongly agreed with this statement before undergoing PHAB accreditation (18%). A similar change is seen in the respondents who strongly agreed that they used strategies to monitor and evaluate effectiveness and quality. In addition, 55% of respondents to the accredited survey strongly agreed that they used information from QI processes to inform decisions compared with 13% of respondents to the applicant survey who strongly agreed before undergoing accreditation. See Table 2.
QI knowledge and staff training
The applicant, accredited, and follow-up surveys were revised in late 2015 to include new questions regarding QI trainings and perceived changes in QI knowledge among staff. Fewer health departments received the revised versions of these surveys, and the data that follow reflect a smaller number of respondents. Overall, respondents indicated that accreditation has resulted in improved or an increased number of staff members trained in QI and perceived improvements in knowledge of QI practices among staff. One year after achieving PHAB accreditation, nearly two-thirds (62%) of respondents reported that more than three-fourths of their health department staff members had QI and/or performance management training compared with 53% of recently accredited and 44% of applicant health departments that responded to the survey. Similarly, the percentage of respondents who reported that less than a quarter of health department staff members had QI and/or performance management training decreased over time. One year after achieving PHAB accreditation, 4% of respondents reported that less than a quarter of staff members had QI and/or performance management training, compared with 12% of recently accredited and 33% of applicant health departments that responded to the survey. See Table 3.
As more public health department staff members receive training on QI and performance management, their familiarity and knowledge of these practices increase. The revised applicant, accredited, and follow-up surveys asked respondents to report on the perceived level of QI knowledge among staff in their public health department. Overall, there was a shift in the percentage of staff members who were reportedly knowledgeable of and practicing QI among accredited health departments. Before undergoing accreditation, approximately a quarter (24%) of respondents reported that a subset of staff members were familiar with QI, and less than one-fifth (19%) reported that the majority of staff members were knowledgeable and practiced or used QI. Following the accreditation decision, 2% of respondents to the accredited survey reported that a subset of staff members were familiar with QI, and 23% reported that the majority of staff members were knowledgeable and practiced or used QI. One year after the accreditation decision, 2% of respondents to the follow-up survey reported that a subset of staff members were familiar with QI and 38% reported that the majority of staff members were knowledgeable and practiced or used QI. See Table 4.
QI culture and infrastructure
The follow-up survey included several core questions regarding QI culture and infrastructure; all health departments surveyed received these questions. One year after the accreditation decision, 97% strongly agreed or agreed that accreditation had stimulated QI and performance improvement opportunities within their health department; 95% reported that they perceived that their health department's awareness of or focus on QI efforts had improved as a result of accreditation; and 92% strongly agreed or agreed that accreditation had strengthened the culture of QI within their public health department. Respondents also reported on the benefits they had experienced as a result of their health department's strengthened QI infrastructure. Respondents reported that their health department's QI culture had strengthened their performance management system (82%), decreased time/cost or improved process quality (63%), and improved public health outcomes achieved (32%).
The QI environment reported by survey respondents also changed over time, from before undergoing accreditation to 1 year after the accreditation decision. The following data are from questions that were added to the applicant, accredited, and follow-up surveys when they were revised in late 2015 and reflect a smaller number of respondents. Prior to accreditation, 6% of respondents to the revised applicant survey reported that QI was “our culture” and 28% of respondents reported that QI was “conducted informally; sporadic program efforts.” One year after achieving accreditation, 18% of respondents to the revised follow-up survey reported that QI was their culture and 7% of respondents reported that QI was conducted informally or sporadically. These data indicate a shift toward a stronger QI culture among health departments 1 year after they are accredited. See Table 5.
Accredited health departments reported multiple QI and performance management benefits as a result of undergoing PHAB accreditation. These include improved awareness of or focus on QI efforts, increased staff members trained in QI and/or performance management, and perceived increases in staff QI knowledge. The effect of PHAB accreditation on QI activities, processes, and culture has begun to be documented,14 and evaluation findings presented confirm that the accreditation program stimulates QI and performance improvement opportunities within health departments that have completed the accreditation process. Accredited health departments have implemented new QI strategies, are using strategies to evaluate effectiveness and quality, and are using information from QI processes to inform decision making at a greater level after achieving accreditation. This is compared with before undergoing the accreditation program, as reported by respondents. A greater proportion of staff members within accredited health departments have received training in QI and/or performance management than health departments that had not yet completed the accreditation program. In turn, the perceived level of QI knowledge among accredited health department staff was reportedly greater than the perceived QI knowledge among health departments that had not yet undergone accreditation. The cumulative effects of increased QI activities, increased staff QI training, and perceived increase in staff QI knowledge as a result of accreditation have also led to perceived improvements in QI culture. The percentage of health department respondents who reported QI as being a key part of their health department's culture increased 3-fold from before completing PHAB accreditation to 1 year after the accreditation decision.
There are several limitations to be noted. First, the surveys were sent to each public health department's director but some surveys were completed by the accreditation coordinator or another staff designee. Regardless of the respondent, the survey data are reflective of that individual's perspective and do not necessarily reflect the perceptions of public health department staff or clients. This limitation should be considered when interpreting findings. Second, survey respondents may have interpreted some questions differently than others. For example, we did not include a definition of “QI culture” in the surveys, so respondents may have provided different responses based on their own interpretations. Third, the surveys were completed by health departments that had either registered their intent to apply for accreditation or had already completed the accreditation program. Therefore, the respondents are subject to selection bias and the data are not intended to be representative of the broader universe of health departments in the United States. Fourth, we did not conduct inferential statistics for the analysis of survey data, as we used a total population sample to survey the universe of health departments that applied for and achieved accreditation. Finally, because PHAB accreditation is still relatively new, the data are likely not representative of the public health field because those health departments that have already undergone the process may be considered early adopters. In addition, to date, the health departments that have received and responded to the follow-up survey and, to a lesser degree, the accredited survey, may be considered earlier adopters than the health departments that have received and responded to the applicant survey.
Implications for Policy & Practice
- In practice, accreditation appears to be an effective tool for promoting QI and performance management within public health departments.
- Compared with applicants, accredited health departments reported greater QI knowledge among health department staff.
- Evaluation data demonstrate that undergoing and achieving accreditation has stimulated public health departments' QI and performance management activities, including staff QI training, perceived staff QI knowledge, and other improvements in efficiency and effectiveness.
2. Riley WJ, Moran JW, Corso LC, Beitsch LM, Bialek R, Cofsky A. Defining quality improvement
in public health
. J Public Health
Manag Pract. 2010;16(1):5–7.
3. Dilly JA, Bekemeier B, Harris JR. Quality improvement
interventions in public health
systems: a systematic review. Am J Prev Med. 2012;42(5)(suppl 1):S58–S71.
4. Riley W, Parsons H, McCoy K, et al Introducing quality improvement
methods into local public health
departments: structured evaluation
of a statewide pilot project. Health Serv Res. 2009;44(5, pt 2):1863–1879.
5. Cornett A, Thomas M, Davis MV, et al Early evaluation
results from a statewide quality improvement
training program for local public health
departments in North Carolina. J Public Health
Manag Pract. 2012;18(1):43–51.
6. Davis MV, Vincus A, Eggers M, et al Effectiveness of public health quality improvement
training approaches: application, application, application. J Public Health
Manag Pract. 2012;18(1):E1–E7.
7. Harrison LM, Shook ED, Harris G, Lea CS, Cornett A, Randolph GD. Applying the model for improvement in a local health department: quality improvement
as an effective approach in navigating the changing landscape of public health
practice in Buncombe County, North Carolina. J Public Health
Manag Pract. 2012;18(1):19–26.
8. McLees AW, Nawaz S, Thomas C, Young A. Defining and assessing quality improvement
outcomes: a framework for public health
. Am J Public Health
. 2015;105(suppl 2):S167–S173.
9. Mason M, Schmidt R, Gizzi C, Ramsey S. Taking improvement action based on performance results: Washington State's experience. J Public Health
Manag Pract. 2010;16(1):24–31.
10. Davis MV, Mahanna E, Joly B, et al Creating quality improvement
culture in public health
agencies. Am J Public Health
11. Livingood WC, Sabbagh R, Spitzfaden S, et al A quality improvement evaluation
case study: impact on public health
outcomes and agency culture. Am J Prev Med. 2013;44(5):445–452.
12. Joly BM, Booth M, Shaler G, Conway A. Quality improvement
learning collaboratives in public health
: findings from a multisite case study. J Public Health
Manag Pract. 2012;18(1):87–94.
13. Davis MV, Cornett A, Mahanna E, See C, Randolph G. Advancing quality improvement
in public health
departments through a statewide training program. J Public Health
Manag Pract. 2016;22(2):E21–E27.
18. Kronstadt J, Meit M, Siegfried A, Nicolaus T, Bender K, Corso L. Evaluating the impact of national public health accreditation
—United States, 2016. MMWR Morb Mortal Wkly Rep. 2016;65(31):803–806.
* The respondent was the lead health department of the multijurisdictional application.
† The respondent was the state office in the centralized state integrated local public health department system.