Share this article on:

Quality Improvement and Performance Management Benefits of Public Health Accreditation: National Evaluation Findings

Siegfried, Alexa, MPH; Heffernan, Megan, MPH; Kennedy, Mallory, BA; Meit, Michael, MA, MPH

Journal of Public Health Management and Practice: May/June 2018 - Volume 24 - Issue - p S3–S9
doi: 10.1097/PHH.0000000000000692
QUALITY IMPROVEMENT & PERFORMANCE MANAGEMENT: Research Reports: Research Full Report

Objective: To identify the quality improvement (QI) and performance management benefits reported by public health departments as a result of participating in the national, voluntary program for public health accreditation implemented by the Public Health Accreditation Board (PHAB).

Design: We gathered quantitative data via Web-based surveys of all applicant and accredited public health departments when they completed 3 different milestones in the PHAB accreditation process.

Participants: Leadership from 324 unique state, local, and tribal public health departments in the United States.

Results: Public health departments that have achieved PHAB accreditation reported the following QI and performance management benefits: improved awareness and focus on QI efforts; increased QI training among staff; perceived increases in QI knowledge among staff; implemented new QI strategies; implemented strategies to evaluate effectiveness and quality; used information from QI processes to inform decision making; and perceived achievement of a QI culture. The reported implementation of QI strategies and use of information from QI processes to inform decision making was greater among recently accredited health departments than among health departments that had registered their intent to apply but not yet undergone the PHAB accreditation process. Respondents from health departments that had been accredited for 1 year reported higher levels of staff QI training and perceived increases in QI knowledge than those that were recently accredited.

Conclusions: PHAB accreditation has stimulated QI and performance management activities within public health departments. Health departments that pursue PHAB accreditation are likely to report immediate increases in QI and performance management activities as a result of undergoing the PHAB accreditation process, and these benefits are likely to be reported at a higher level, even 1 year after the accreditation decision.

Public Health Research Department, NORC at the University of Chicago, Bethesda, Maryland.

Correspondence: Alexa Siegfried, MPH, Public Health Research Department, NORC at the University of Chicago, 4350 East-West Hwy, Ste 800, Bethesda, MD 20814 (siegfried-alexa@norc.org).

This research was conducted by NORC at the University of Chicago with support from (1) the Robert Wood Johnson Foundation (RWJF) and (2) the Public Health Accreditation Board through funding provided by RWJF.

The authors declare no conflicts of interest.

The national accreditation program, implemented by the Public Health Accreditation Board (PHAB), is designed to advance the quality and performance of public health departments in the United States. To be recognized as an accredited health department, applicants assess themselves against a set of Standards and Measures that are designed to support and advance the health department's quality improvement (QI) and performance management efforts.1 QI in public health is

the use of a deliberate and defined improvement process ... focused on activities that are responsive to community needs and improving population health. It refers to a continuous and ongoing effort to achieve measurable improvements in the efficiency, effectiveness, performance, accountability, outcomes, and other indicators of quality in services or processes which achieve equity and improve the health of the community.2 (p6)

The implementation of QI activities by health departments can lead to positive changes within the health department, such as improvements in efficiency and effectiveness.3 , 4 For example, QI activities have helped build and develop workforce capacity5 , 6; increased efficiencies such as time7 and costs saved8 or reduced steps in a process2; improved effectiveness3 such as customer and staff satisfaction, data quality,9 and public health programs and services2; and enhanced supportive environment and culture for QI within public health agencies.10–13 When QI is integrated across the health department, both within specific projects and across all programs and activities, it can be considered integrated into the health department's culture.2 The PHAB accreditation program is working to strengthen QI and performance management infrastructure, including activities, processes, and culture, of public health departments throughout the country.14

To assess the QI and performance management benefits of PHAB accreditation, NORC at the University of Chicago conducted Web-based surveys of applicant and accredited health departments at 3 different steps of the accreditation process. NORC received funding support to conduct the surveys and other data collection activities from (1) the Robert Wood Johnson Foundation (RWJF)15 , 16 and (2) PHAB, through funding provided by RWJF.17 This article describes selected survey data related to the perceived QI and performance management benefits of the PHAB accreditation program, as reported by applicant and accredited health departments that completed the surveys. Additional quantitative and qualitative data have been published describing internal and external accreditation benefits related to accountability, transparency, communication, and collaboration reported by applicant and accredited health departments, among other evaluation findings.18

Back to Top | Article Outline

Methods

We conducted 3 Web-based surveys of health departments that applied for PHAB accreditation. The surveys were designed to gather feedback from applicant health departments as they completed 3 milestones in the accreditation process. These milestones were as follows: (1) registering their intent to apply for accreditation (referred to as the applicant survey); (2) achieving accreditation (referred to as the accredited survey); and (3) 1 year after achieving accreditation (referred to as the follow-up survey). The NORC Institutional Review Board reviewed the study protocols and determined them to be exempt from full review. A description of the survey population, data collection procedures, and methods for analysis for each of the 3 surveys follows.

Back to Top | Article Outline

Survey population

We used a total population sample to survey the universe of health departments that applied for and achieved PHAB accreditation between November 2013 and May 2017. During this time frame, we surveyed all applicant and accredited health departments at 3 points in time. For the applicant survey, we contacted every health department that had registered its intent to apply for accreditation in PHAB's online system, e-PHAB. For the accredited survey, we contacted every health department that achieved accreditation between November 2013 and May 2017. For the follow-up survey, we contacted every health department that reached its 1-year anniversary of achieving accreditation.

Back to Top | Article Outline

Data collection

To determine which health departments to contact, PHAB provided a list of every health department that had reached each milestone. PHAB also provided the name and contact information for each health department's director and designated accreditation coordinator. To recruit participation in the survey, we sent an e-mail invitation to each health department's director. The invitation included a link to complete the Web-based survey. We also sent an e-mail to each accreditation coordinator, informing him or her that we had contacted his or her health department director regarding the survey. We requested that the health department director complete the survey but accepted responses from the accreditation coordinator or another staff designee. We sent several reminders to both the health department director and the accreditation coordinator, encouraging them to complete the survey. Each survey remained in the field for approximately 6 weeks, and we sent no more than 3 reminders for each survey.

Between November 2013 and May 2017, we fielded the survey to multiple cohorts of health departments as they reached each of the accreditation milestones. The applicant survey was sent to 15 cohorts of health departments prior to their participation in the accreditation training for applicant health departments; 267 health departments received the applicant survey. The accredited survey was sent to 14 cohorts of health departments that had achieved accreditation; 160 health departments received the accredited survey. The follow-up survey was sent to 13 cohorts of health departments after they reached their 1-year anniversary of achieving accreditation; 118 health departments received the follow-up survey. Across the 3 surveys, we received 479 responses—231 responses to the applicant survey (response rate of 87%), 145 responses to the accredited survey (response rate of 91%), and 103 responses to the follow-up survey (response rate of 87%). See Table 1.

TABLE 1

TABLE 1

Back to Top | Article Outline

Survey instruments

We created 3 survey instruments with a combination of open- and closed-ended response questions that were designed to gather information from respondents about the PHAB accreditation process, the experience of applicants, and the initial benefits of accreditation. The applicant survey included questions about the motivators and anticipated internal benefits of PHAB accreditation, current QI activities, and current QI and performance management infrastructure. The accredited survey included questions about the internal benefits, changes in QI activities, changes in QI and performance management infrastructure, and other perceived benefits resulting from PHAB accreditation. The follow-up survey included questions about the benefits, changes in QI activities, QI, and performance management infrastructure, and other benefits or outcomes experienced in the past year as a result of being accredited.

Each survey instrument maintained a core set of questions between November 2013 and May 2017. In late 2015, each survey instrument was modified slightly to include new questions about perceived QI and performance management benefits of PHAB accreditation. Among the 15 cohorts that received the applicant survey, 7 cohorts received the revised version; among the 14 cohorts that received the accredited survey, 6 received the revised version; and among the 13 cohorts that received the follow-up survey, 6 received the revised version. We received 219 responses to the new questions added to the revised survey—90 responses for the applicant survey, 94 responses for the accredited survey, and 57 responses for the follow-up survey.

When data collection began in November 2013, the PHAB accreditation program had been underway for approximately 2 years. Therefore, some health department cohorts did not receive invitations to complete the applicant or accredited surveys because they had already passed the designated milestones in the accreditation process. In addition, health departments vary in the amount of time it takes to undergo the accreditation process—some may take a shorter or longer time to achieve accreditation after registering in e-PHAB. Thus, not all health departments that complete the applicant survey at one point in time subsequently receive the accredited or follow-up surveys at the same time.

Back to Top | Article Outline

Data analysis

NORC researchers prepared data collected by reviewing and cleaning the raw data, including recoding response options and corresponding variables as appropriate. We created 3 clean data files for all quantitative data gathered from each survey using SAS and Excel. A corresponding survey codebook was created with an index of survey questions, indicating whether the survey question was core or added in 2015. For quantitative data, we conducted univariate analyses to determine frequency distributions and averages. We conducted these analyses for each survey, grouping all responses to the core survey questions and grouping responses to the added questions in 2015.

We conducted longitudinal analyses for a subset of health departments (n = 72) that received and responded to both the applicant and accredited surveys. The longitudinal analyses allowed us to compare changes within the health departments over time, from before undergoing the accreditation process to after achieving accreditation. To date, an insufficient number of health departments have completed other combinations of surveys (ie, accredited and follow-up surveys); therefore, longitudinal data from these respondents are not presented. Survey data continue to be collected with funding from RWJF,17 so there is future opportunity to conduct longitudinal analyses with a greater number of health departments, as the number of health departments that have responded to multiple surveys increases over time.

Back to Top | Article Outline

Results

Characteristics of survey respondents

Across the 3 surveys, we received 479 responses from 324 unique health departments; 155 health departments responded to 2 or more of the surveys. Among the unique responses, 88% were local health departments and 11% were state health departments; there were also 2 tribes, 2 multijurisdictional applicants,* and 1 integrated health system. Of the 35 unique state health departments, the governance structure for public health was as follows: 23 decentralized, 10 centralized, and 2 mixed. Of the 284 unique local health departments, 88% were in states with decentralized public health governance structures, 4% were in centralized states, and 8% were in mixed governance states. Among the local applicants, 40% had a population of less than 100 000, 39% had a population between 100 000 and 500 000, and 21% had a population above 500 000. Survey responses were received from applicant and accredited health departments across 45 states.

Back to Top | Article Outline

QI processes and activities

Following their accreditation decision, public health departments were sent the accredited survey and were asked about the immediate benefits of PHAB accreditation. Nearly all respondents to the accredited survey reported ongoing QI activities. The majority of recently accredited health departments (97%) reported that because of their participation in the accreditation process, they had implemented or planned to implement new strategies for QI, compared with 67% of health departments that reported doing so before undergoing accreditation. Ninety-six percent of recently accredited health departments also reported that because of their participation in PHAB accreditation, they had implemented new strategies to evaluate their agency's effectiveness and quality, compared with 83% of health departments that reported doing so before undergoing accreditation. In addition, 95% of recently accredited health departments reported that because of their participation in the accreditation process, they have informed or plan to inform decisions based on their QI or performance management processes, compared with 75% of applicant health departments that reported doing so.

Back to Top | Article Outline

QI process and activities: Longitudinal analysis

Longitudinal analysis was conducted for a subset of health departments that received and responded to both the applicant and accredited surveys. Data from these analyses indicate an overall increase in the level of agreement among respondents regarding their health department's use of QI and performance management strategies following accreditation, compared with before undergoing the accreditation process. After becoming accredited, 69% of respondents strongly agreed that they had implemented or planned to implement new QI strategies, more than 3 times the percentage of respondents who strongly agreed with this statement before undergoing PHAB accreditation (18%). A similar change is seen in the respondents who strongly agreed that they used strategies to monitor and evaluate effectiveness and quality. In addition, 55% of respondents to the accredited survey strongly agreed that they used information from QI processes to inform decisions compared with 13% of respondents to the applicant survey who strongly agreed before undergoing accreditation. See Table 2.

TABLE 2

TABLE 2

Back to Top | Article Outline

QI knowledge and staff training

The applicant, accredited, and follow-up surveys were revised in late 2015 to include new questions regarding QI trainings and perceived changes in QI knowledge among staff. Fewer health departments received the revised versions of these surveys, and the data that follow reflect a smaller number of respondents. Overall, respondents indicated that accreditation has resulted in improved or an increased number of staff members trained in QI and perceived improvements in knowledge of QI practices among staff. One year after achieving PHAB accreditation, nearly two-thirds (62%) of respondents reported that more than three-fourths of their health department staff members had QI and/or performance management training compared with 53% of recently accredited and 44% of applicant health departments that responded to the survey. Similarly, the percentage of respondents who reported that less than a quarter of health department staff members had QI and/or performance management training decreased over time. One year after achieving PHAB accreditation, 4% of respondents reported that less than a quarter of staff members had QI and/or performance management training, compared with 12% of recently accredited and 33% of applicant health departments that responded to the survey. See Table 3.

TABLE 3

TABLE 3

As more public health department staff members receive training on QI and performance management, their familiarity and knowledge of these practices increase. The revised applicant, accredited, and follow-up surveys asked respondents to report on the perceived level of QI knowledge among staff in their public health department. Overall, there was a shift in the percentage of staff members who were reportedly knowledgeable of and practicing QI among accredited health departments. Before undergoing accreditation, approximately a quarter (24%) of respondents reported that a subset of staff members were familiar with QI, and less than one-fifth (19%) reported that the majority of staff members were knowledgeable and practiced or used QI. Following the accreditation decision, 2% of respondents to the accredited survey reported that a subset of staff members were familiar with QI, and 23% reported that the majority of staff members were knowledgeable and practiced or used QI. One year after the accreditation decision, 2% of respondents to the follow-up survey reported that a subset of staff members were familiar with QI and 38% reported that the majority of staff members were knowledgeable and practiced or used QI. See Table 4.

TABLE 4

TABLE 4

Back to Top | Article Outline

QI culture and infrastructure

The follow-up survey included several core questions regarding QI culture and infrastructure; all health departments surveyed received these questions. One year after the accreditation decision, 97% strongly agreed or agreed that accreditation had stimulated QI and performance improvement opportunities within their health department; 95% reported that they perceived that their health department's awareness of or focus on QI efforts had improved as a result of accreditation; and 92% strongly agreed or agreed that accreditation had strengthened the culture of QI within their public health department. Respondents also reported on the benefits they had experienced as a result of their health department's strengthened QI infrastructure. Respondents reported that their health department's QI culture had strengthened their performance management system (82%), decreased time/cost or improved process quality (63%), and improved public health outcomes achieved (32%).

The QI environment reported by survey respondents also changed over time, from before undergoing accreditation to 1 year after the accreditation decision. The following data are from questions that were added to the applicant, accredited, and follow-up surveys when they were revised in late 2015 and reflect a smaller number of respondents. Prior to accreditation, 6% of respondents to the revised applicant survey reported that QI was “our culture” and 28% of respondents reported that QI was “conducted informally; sporadic program efforts.” One year after achieving accreditation, 18% of respondents to the revised follow-up survey reported that QI was their culture and 7% of respondents reported that QI was conducted informally or sporadically. These data indicate a shift toward a stronger QI culture among health departments 1 year after they are accredited. See Table 5.

TABLE 5

TABLE 5

Back to Top | Article Outline

Discussion

Accredited health departments reported multiple QI and performance management benefits as a result of undergoing PHAB accreditation. These include improved awareness of or focus on QI efforts, increased staff members trained in QI and/or performance management, and perceived increases in staff QI knowledge. The effect of PHAB accreditation on QI activities, processes, and culture has begun to be documented,14 and evaluation findings presented confirm that the accreditation program stimulates QI and performance improvement opportunities within health departments that have completed the accreditation process. Accredited health departments have implemented new QI strategies, are using strategies to evaluate effectiveness and quality, and are using information from QI processes to inform decision making at a greater level after achieving accreditation. This is compared with before undergoing the accreditation program, as reported by respondents. A greater proportion of staff members within accredited health departments have received training in QI and/or performance management than health departments that had not yet completed the accreditation program. In turn, the perceived level of QI knowledge among accredited health department staff was reportedly greater than the perceived QI knowledge among health departments that had not yet undergone accreditation. The cumulative effects of increased QI activities, increased staff QI training, and perceived increase in staff QI knowledge as a result of accreditation have also led to perceived improvements in QI culture. The percentage of health department respondents who reported QI as being a key part of their health department's culture increased 3-fold from before completing PHAB accreditation to 1 year after the accreditation decision.

There are several limitations to be noted. First, the surveys were sent to each public health department's director but some surveys were completed by the accreditation coordinator or another staff designee. Regardless of the respondent, the survey data are reflective of that individual's perspective and do not necessarily reflect the perceptions of public health department staff or clients. This limitation should be considered when interpreting findings. Second, survey respondents may have interpreted some questions differently than others. For example, we did not include a definition of “QI culture” in the surveys, so respondents may have provided different responses based on their own interpretations. Third, the surveys were completed by health departments that had either registered their intent to apply for accreditation or had already completed the accreditation program. Therefore, the respondents are subject to selection bias and the data are not intended to be representative of the broader universe of health departments in the United States. Fourth, we did not conduct inferential statistics for the analysis of survey data, as we used a total population sample to survey the universe of health departments that applied for and achieved accreditation. Finally, because PHAB accreditation is still relatively new, the data are likely not representative of the public health field because those health departments that have already undergone the process may be considered early adopters. In addition, to date, the health departments that have received and responded to the follow-up survey and, to a lesser degree, the accredited survey, may be considered earlier adopters than the health departments that have received and responded to the applicant survey.

Back to Top | Article Outline

Implications for Policy & Practice

  • In practice, accreditation appears to be an effective tool for promoting QI and performance management within public health departments.
  • Compared with applicants, accredited health departments reported greater QI knowledge among health department staff.
  • Evaluation data demonstrate that undergoing and achieving accreditation has stimulated public health departments' QI and performance management activities, including staff QI training, perceived staff QI knowledge, and other improvements in efficiency and effectiveness.
Back to Top | Article Outline

References

1. Public Health Accreditation Board. Guide to national public health department initial accreditation. http://www.phaboard.org/wp-content/uploads/Guide-to-Accreditation-final_LR2.pdf. Adopted June 2015. Accessed May 31, 2017.
2. Riley WJ, Moran JW, Corso LC, Beitsch LM, Bialek R, Cofsky A. Defining quality improvement in public health. J Public Health Manag Pract. 2010;16(1):5–7.
3. Dilly JA, Bekemeier B, Harris JR. Quality improvement interventions in public health systems: a systematic review. Am J Prev Med. 2012;42(5)(suppl 1):S58–S71.
4. Riley W, Parsons H, McCoy K, et al Introducing quality improvement methods into local public health departments: structured evaluation of a statewide pilot project. Health Serv Res. 2009;44(5, pt 2):1863–1879.
5. Cornett A, Thomas M, Davis MV, et al Early evaluation results from a statewide quality improvement training program for local public health departments in North Carolina. J Public Health Manag Pract. 2012;18(1):43–51.
6. Davis MV, Vincus A, Eggers M, et al Effectiveness of public health quality improvement training approaches: application, application, application. J Public Health Manag Pract. 2012;18(1):E1–E7.
7. Harrison LM, Shook ED, Harris G, Lea CS, Cornett A, Randolph GD. Applying the model for improvement in a local health department: quality improvement as an effective approach in navigating the changing landscape of public health practice in Buncombe County, North Carolina. J Public Health Manag Pract. 2012;18(1):19–26.
8. McLees AW, Nawaz S, Thomas C, Young A. Defining and assessing quality improvement outcomes: a framework for public health. Am J Public Health. 2015;105(suppl 2):S167–S173.
9. Mason M, Schmidt R, Gizzi C, Ramsey S. Taking improvement action based on performance results: Washington State's experience. J Public Health Manag Pract. 2010;16(1):24–31.
10. Davis MV, Mahanna E, Joly B, et al Creating quality improvement culture in public health agencies. Am J Public Health. 2014;104(1):e98–e104.
11. Livingood WC, Sabbagh R, Spitzfaden S, et al A quality improvement evaluation case study: impact on public health outcomes and agency culture. Am J Prev Med. 2013;44(5):445–452.
12. Joly BM, Booth M, Shaler G, Conway A. Quality improvement learning collaboratives in public health: findings from a multisite case study. J Public Health Manag Pract. 2012;18(1):87–94.
13. Davis MV, Cornett A, Mahanna E, See C, Randolph G. Advancing quality improvement in public health departments through a statewide training program. J Public Health Manag Pract. 2016;22(2):E21–E27.
14. Public Health Accreditation Board. PHAB accreditation impact series. http://www.phaboard.org/wp-content/uploads/PHAB-Accreditation-Impact-Series-1-QI-Research-Addendum-updated-January-2017.pdf. Updated January 2017. Accessed May 31, 2017.
15. Robert Wood Johnson Foundation. Evaluating the short-term outcomes of national accreditation of tribal, state, local, and territorial public health agencies, 2015-2017. http://www.rwjf.org/en/library/grants/2015/02/evaluating-the-short-term-outcomes-of-national-accreditation-of-.html. Accessed May 30, 2017.
16. Robert Wood Johnson Foundation. Evaluating the short-term outcomes of national accreditation of tribal, state, local, and territorial public health agencies, 2017-2020. http://www.rwjf.org/en/library/grants/2016/11/evaluating-the-short-term-outcomes-of-national-accreditation-of-.html. Accessed May 30, 2017.
17. NORC at the University of Chicago. Initial evaluation of the public health accreditation program: final report. http://www.phaboard.org/wp-content/uploads/PHAB-Final-Report_December-2016.pdf. Published December 2016. Accessed May 31, 2017.
18. Kronstadt J, Meit M, Siegfried A, Nicolaus T, Bender K, Corso L. Evaluating the impact of national public health accreditation—United States, 2016. MMWR Morb Mortal Wkly Rep. 2016;65(31):803–806.

* The respondent was the lead health department of the multijurisdictional application.
Cited Here...

† The respondent was the state office in the centralized state integrated local public health department system.
Cited Here...

Keywords:

accreditation; evaluation; public health; quality improvement

Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved.