In October 2005, five states (Illinois, Michigan, Missouri, North Carolina, and Washington) were funded by the Robert Wood Johnson Foundation (RWJF) to participate in a Multistate Learning Collaborative (MLC) to enhance their existing performance and capacity assessment or accreditation programs for public health departments. Additional goals of the MLC were to provide opportunities for state and local public health practitioners to learn from each other, develop a strong network with their peers and relevant experts, inform the broader public health practice community about their processes and lessons learned, and facilitate the exchange and identification of best practices. The findings and lessons learned from the MLC have informed a parallel process called the Exploring Accreditation project, funded by RWJF and the Centers for Disease Control and Prevention (CDC) to explore the feasibility and desirability of developing a voluntary national accreditation model for state and local health departments.
These efforts and other recent performance improvement initiatives reflect the need and desire for greater accountability and consistency in public health practice. Since the landmark 1988 Institute of Medicine report on the future of public health,1 policy makers and public health professionals have sought to clarify the core functions of public health, articulate the essential services, and promulgate standards for measuring the performance of health departments and public health systems. More than one third of all states currently have documented accreditation programs, certification efforts, or performance improvement initiatives,2 reflecting various public health standards and quality improvement models. The five states selected to participate in the MLC represented varied approaches and levels of maturity. The purpose of this article is to report (1) lessons learned and challenges related to performance assessment in the five MLC states; (2) lessons learned from participating in the learning collaborative; and (3) recommendations for states considering performance or capacity assessment, accreditation, or a similar accountability system.
In June 2006, an interview tool was created to develop a better understanding of the experiences of the five MLC states related to the assessment or accreditation of public health departments. Seven open-ended questions were asked about the states' reasons for pursuing accreditation or accountability programs; how they built a foundation for their programs; how they developed their standards and program; how they promoted or demonstrated the impact of accreditation or assessment; what they learned from participating in the MLC; and recommendations to other states. During July and August of 2006, subjects representing a range of roles within the public health systems in the five MLC states were sent the interview tool. They either e-mailed their responses or participated in telephone or face-to-face interviews conducted by the investigators. Investigators contacted participants who e-mailed their responses if there were questions related to their comments.
Fifteen (15) individuals were interviewed, three from each of the five states. Seven respondents represented state health departments, six represented public health institutes, and two represented local health departments. Participant involvement in their state's performance assessment or accreditation program ranged from 1 to 16 years. Table 1 details respondent affiliations and major themes.
Reasons for Pursuing Performance Assessment or Accreditation
The most common reasons for pursuing performance assessment or accreditation included documenting accountability, assuring a consistent level of public health capacity, identifying gaps in health department performance, and complying with state law. Most states viewed performance assessment or accreditation as a tool to document accountability. A Michigan respondent called accreditation a mechanism for demonstrating to stakeholders that community needs were being met and financial resources were being used effectively. It was also viewed as a tool to ensure and demonstrate consistency in public health service delivery so that every individual could expect a consistent level of public health services regardless of where they lived. Most states saw performance assessment or accreditation as a way to evaluate the performance of public health departments and identify gaps within and between departments that could be used for continuous quality improvement. In Washington, a state law mandating some type of assessment program was viewed as a reason for pursuing accreditation or performance assessment. Washington's laws require the development of basic standards for public health to strengthen the public health system and improve the health of individuals in the state.
Building a Foundation for Success
Several elements emerged as crucial to the success of these states' accreditation or assessment programs. Respondents consistently mentioned establishing buy-in through partnerships and stakeholder involvement in the process from the outset. Participants from Michigan described their state's broadly based approach, and included state and local health officials, individuals from the Michigan Public Health Institute, and persons representing the association for local health directors as partners. These individuals' and organizations' participation, in the form of collaboration and support, was deemed critically important at all stages of the process.
In North Carolina, local health directors, local boards of health, and county commissioners provided important support for accreditation. Also, nontraditional partners, such as representatives of the broader healthcare community (hospitals and medical societies) and legislators, served as members of North Carolina's Public Health Task Force, established to develop recommendations to strengthen the state's public health system. Missouri's program also involved nontraditional partners, including the state nurses association, medical societies, and academic partners, such as schools of nursing.
Most states emphasized the importance of defining the accountability system and necessary components. An Illinois respondent explained that communicating in a common language about what accreditation means for local health departments was important because of confusion between the existing Illinois certification program and the accreditation process being explored. In North Carolina, it was important to define the roles, responsibilities, and expectations within and outside the partnership. North Carolina communicated clear descriptions of the roles of the various partners and kept county commissioners informed of the process, whether or not they had day-to-day involvement.
Additional components of performance assessment or accreditation that needed to be defined included the following: what it would cost to fund the program; how much time and energy the stakeholders would have to invest in the process; and the number of staff and training required at the state and local levels. A few respondents stressed dedicated resources and additional funding to be in place before any program could begin. Illinois is a case in point for the importance of providing resources for performance assessment. When their certification program was established in the 1990s, they lacked funding to implement the program and suffered a significant loss of staff through an early retirement incentive program at the state level. Without adequate capacity, the state health department could not support the technical and data systems operations and evaluate the program, and reviews of local health department applications for certification were delayed. Moreover, the health plans developed by local health departments as part of the certification process could not be implemented because of a lack of resources. These shortcomings engendered distrust between the state agency and the local health departments. Illinois is still addressing the relics of this lack of trust in their attempt to build their new accreditation program within the context of their existing certification program. For them to move forward, adequate funding and resources, reestablishment of trust, and local buy-in will be crucial.
Strong leadership and champions at state and local levels were also important drivers of performance assessment or accreditation. A Michigan participant claimed that it was essential not only to have champions for accreditation, but also to sustain their role and identify alternates. Leaders and champions in the five MLC states included state health officers, other state health department staff, local health officers, and legislators.
Piloting the Program
Participants consistently mentioned pilot testing the program before implementation as an essential way to identify needed changes and to make improvements before the launch. The Michigan pilot program was tested in four sites representing one county, one health district, an urban center, and a rural setting, and numerous changes were made before statewide implementation. Missouri piloted a three-tiered accreditation program in seven local health departments to test methods and tools and determine the strengths and weaknesses of their program. The pilot helped identify a workforce priority, the need for staff recruitment and retention. Illinois piloted their accreditation program before the implementation to identify areas for improvement, particularly because of their history of strained relationships and distrust. Washington conducted an entire assessment cycle in more than 70 state and local sites in 2000 to test their standards and the assessment process. The clarifications in the standards and the improvements made to the site visit process resulting from this comprehensive testing helped ensure the success of the subsequent cycles in 2002 (baseline) and 2005, by allowing all sites to gain experience in interpreting performance requirements and preparing for the site visits.
North Carolina piloted the components of their accreditation system to identify functional deficiencies and make needed improvements to their standards, documentation requirements, site visit orientation, and other processes. It allowed planners to identify issues related to dividing the work among site visitors, developing policies for obtaining additional documentation during the site visit, structuring the site visit exit interview, and identifying the type and magnitude of training needed by local health department staff and site visitors. The pilot phase taught them the importance of paying honoraria for site visitors as a recruitment tool, organizing daily conferences during the site visit, and focusing the site visit exit interview on strengths and opportunities for improvement.
To develop standards, Michigan and Washington each created a committee of state and local representatives to create, review, and modify all standards, measures, and tools. The process also served to establish trust, build partnerships, and promote buy-in. A “performance and accountability-based relationship between state and local partners” was deemed important to this process, as were standards that were “understandable, valid, reliable, and clearly communicated.” Respondents from Michigan recommended that standards committees test the standards and measures before assuming that they are too difficult to meet, and, when developing guidance for “how to meet a standard,” take care that the intent or scope of the standards is not altered.
Missouri and Illinois linked their standards to existing tools, including the 10 Essential Public Health Services (EPHS),3 the Operational Definition of a Functional Local Health Department,4 and the core public health functions.1 Missouri used the 10 EPHS to frame their standards and the Operational Definition to streamline the language. Illinois' practice standards for certification closely mirrored those of the 10 EPHS, although they preceded them. For accreditation, Illinois is using the 10 EPHS and the Operational Definition of a Functional Local Health Department to develop standards.
North Carolina struggled with when to use the “nonapplicable” response category, and how this category would count toward achieving a certain standard. For example, if a standard related to hiring a local health director, but the incumbent had been in place for several years, how should the health department be judged on this standard? North Carolina planners decided to establish timeframes for various standards, requiring documentation on a hiring only if it had occurred within the last 48 months, for example.
Promoting and Demonstrating the Impact of the Program
A few states have found capturing data pertaining to the actual cost of assessment or accreditation challenging, although it is recognized that promoting such programs would be easier if it could be shown that costs are proportional to the value added to local health departments. Most states noted the need to evaluate the assessment program itself and provide feedback and opportunities for improving it. North Carolina has conducted an extensive evaluation of its accreditation program from its inception to improve the standards, forms used, how site visits are conducted, and other elements. For Illinois, a continuous formative evaluation is seen as a useful process that will allow them to marshal, reallocate, or bring in new resources for their accreditation program. Michigan evaluates every cycle of their program. When local health departments expressed dissatisfaction with key accreditation process components during its second cycle of reviews, Michigan paused the on-site review process component of their program and administered a survey to local health departments and state agency reviewers. Numerous changes were made mid-cycle to improve and strengthen the program.
Washington not only evaluates its assessment program but also demonstrates the impact of its program by aggregating assessment results and sharing them statewide. Since the test cycle in 2000, Washington has aggregated the performance results for all local health departments, for all state level sites, and for the public health system as a whole. In 2002 and 2005, local health departments received results based on how they performed compared to the benchmarks of their entire peer group and how their performance was related to overall budget. These results enabled Washington to identify systemwide priorities for improvement and to strengthen collaboration, including areas of state performance that support work at the local level and vice versa.
Several respondents mentioned the importance of sharing the successes of the accreditation program with partners, and using accredited local health departments to communicate their stories to nonaccredited health departments and other stakeholders to promote the program. Incentives and financial resources were identified as important in promoting and strengthening accreditation or assessment, and providing opportunities for quality improvement. For example, in Illinois, certification was linked to opportunities for local health protection grants, which generated local health department interest and practice changes.
Participating in the MLC
Participation in the MLC highlighted the range of possible approaches to performance assessment or accreditation. Even though there were varying approaches, participants still identified similarities in state goals and challenges. Many felt that the Collaborative allowed them to share ideas, learn from one another, and reach better solutions collectively. Missouri and Michigan plan to jointly create a database of effective practices in accreditation and develop standards based on these best practices. Learning from the approaches of other states, Illinois was able to develop a universal message to build a common understanding of accreditation in their state.
Recommendations to Other States
Respondents made two main recommendations for states interested in accreditation or assessment programs. The first was to engage broad stakeholder and leadership (ie, legislators and state and local health officials) support and participation in the process from the beginning. The second important recommendation was to understand that planning, implementing, and evaluating an accreditation program for public health departments involve an ongoing process that requires long-term commitment and a focus on the goals, benefits, and anticipated outcomes.
Performance assessment is seen as a tool to document accountability, ensure a consistent level of public health capacity, and evaluate and identify gaps in health department performance. In addition to these functions, many public health professionals believe that performance and capacity assessment or accreditation can raise the visibility of public health in the community, improve an agency's chances of competing successfully for funds through grants and contracts, improve employee morale and job satisfaction, and serve as a platform for quality improvement.
For most of the states participating in the MLC, establishing partnerships that ensure stakeholder involvement in all aspects of the program was an important element for the success of performance assessment or accreditation programs. These findings are similar to the findings of Thielen,5 who concluded from interviews with eight states that leadership at both levels was key to the success of their performance assessment or accreditation programs. We believe that the partnerships and engagement with the broader public health community gave these programs the breadth and depth of insight and expertise necessary to enhance wide acceptance. Strong leadership and champions at the state and local levels also often translate into financial or political support for getting programs such as these accomplished. Broad and high-level engagement is recommended for other states considering such programs. If a national voluntary program were to be considered, broad stakeholder and leadership engagement at the national level would be important as well.
After ensuring strong support, pilot testing was perceived as the most important step planners did to set the stage for successful programs. Besides identifying areas for improvement, pilot programs help ensure success by familiarizing public health departments with the process before it is “formalized,” giving them not only ownership of the program through their input into its development but also practice at fulfilling the requirements and a deeper understanding of what will be required of them. This type of preparation may make the actual program run more smoothly once it is under way.
The goal with standard development activities was to enhance buy-in and maximize the utility of the standards. Most states used a collaborative approach to developing standards with broad acceptance and utility, and most used existing public health tools to establish their standards, judging that familiarity would make them more acceptable to public health departments.
Continuous evaluation of the assessment programs and constructive feedback to public health agencies were seen as supportive of the performance assessment programs. It is believed that if states and agencies can see the benefit to their programs of having this type of assessment, and if they can see the performance assessment programs improving over time, they will be more supportive of these programs. The states in this study would benefit from even more evaluation of the programs' impacts.
Based on participant responses, several goals of the MLC were realized: (1) public health practitioners were able to learn from other states and locals and apply these learnings in their state; (2) participants developed a strong network with relevant peers; and (3) the collaborative facilitated the exchange and identification of best practices. These goals continue to be realized.
Given the momentum and growing interest in public health accreditation and performance assessment efforts, the need to understand, document, and share critical elements of success is increasingly important. Although there were only five states in this study, these states are considered to have well-established performance management and accountability systems. Findings across two or more of these states have relevance to others attempting to establish these programs. They also have relevance to a potential national voluntary program. On the other hand, these states differ greatly in their programs6 and these differences leave room for further research into what factors affect the success of such programs. Particularly fruitful might be a comparison of states that have full-fledged accreditation programs with those that do not, and states providing incentives, or have dedicated funds, with those that do not.
Further research will also be needed as more states establish standards, design programs, and move forward with comprehensive performance assessment of their agencies. Fifteen percent (15%) of the nation's population resides in the 5 MLC states. As their programs mature, they could teach us more about the outcomes, in terms of improved capacity, performance, and health outcomes, made possible through strong assessment programs. Only one of these states (Washington) evaluates at both the local and state health department levels. It might be fruitful to look more closely at the challenges in developing a multitiered assessment system to understand why more states do not focus on multiple levels and how to facilitate more multileveled assessment programs. Finally, the sample in this study was quite small and somewhat limited: we interviewed three individuals from each of the states, and focused most of our attention on state health departments and public health institute personnel, with only two interviewees being from local health departments. This was partly because the state and institute personnel were more involved in the MLC; however, as programs mature, local public health impressions of the process could enrich our understanding further.
Through a collaborative learning process, the MLC states have learned from each other, shared resources, and identified practices for adoption. These states' lessons regarding the context, impetus, structure, and key ingredients are likely transferable to others who may be interested in developing or enhancing public health accreditation or performance improvement programs.