Joly, Brenda M. PhD, MPH; Polyak, Georgeen PhD; Davis, Mary V. DrPH, MSPH; Brewster, Joan MPA; Tremain, Beverly PhD; Raevsky, Cathy BA; Beitsch, Leslie M. MD, JD
The elusive holy grail of public health is the achievement of improved community health status indicators.1 For years, practitioners have been serving on public health's front lines, with the implicit underlying assumptions that their efforts to mobilize high-risk individuals to adopt healthier lifestyles and improve the environment will lead the community to better overall health. Nonetheless, even the most optimistic among us must acknowledge that there is but scant evidence linking the hard work of dedicated public health professionals to changes in community health outcomes.
On the more positive side, there is a growing body of evidence documenting positive intermediate outcomes for specific public health interventions, many of which are documented in The Guide to Community Preventive Services.2 While public health has long been grappling with the overall lack of an evidence base supporting the benefits and outcomes of the overall public health enterprise, the rest of the healthcare sector has continued to advance forward, embracing accreditation as at least part of the answer. Today, we are approaching a crossroad. As the opportunities increase for governmental public health agencies to participate in accreditation processes, will we finally be able to demonstrate that missing linkage? In short, does accreditation lead us to the promised land: can it move us on the journey toward achieving and demonstrating the links to community health outcomes? Or, must we be satisfied with other, less ambitious results, reflecting improved performance of public health agencies?
In this article, we describe accreditation in health and other industries, and its relationship to outcomes. Despite differences across sectors, we examine important lessons that might have meaningful public health translations, as well as influences in and on public health that make translation problematic. Finally, maintaining our gaze firmly on the ultimate prize, we propose a logic model whereby we can assess the impact accreditation has on community public health outcomes.
Lessons From Service Delivery Organizations
In 2004, Mays conducted an extensive review of accreditation programs in healthcare, education, social service, and public service industries, primarily among US service delivery organizations.3 This review examined the purpose and goals of accreditation programs, design features and implementation processes, and their outcomes and impact.
Mays reviewed 11 studies, 9 observational and 2 experimental, for the impact of accreditation.2 As a group, these programs have positive effects on service quality, operations, and service-related outcomes for the organizations that participate in accreditation, providing moderate support that accreditation can have beneficial effects.
The literature on the impact of accreditation programs in service and healthcare settings is limited and the available studies primarily use observational designs.2 Results from these studies indicating differences in outcomes between accredited and unaccredited organizations may be subject to two sources of bias. First, selection bias may occur because organizations that are already of higher quality may self-select to participate in an accreditation program. Second, a program effect may occur. In this case, organizations that participate in an accreditation program may improve their service quality to achieve program standards, rather than having met the standards prior to undergoing participation.2 Determining the impact of accreditation programs may depend on outcomes studied, research designs, and controls for selection bias.2 Furthermore, accreditation outcomes are difficult to define and can vary between stakeholders, users, observers, and accreditation programs.4 Anecdotal evidence of outcomes in other industries' organizations includes decreased liability expenses, increased efficiency, and increased performance.2 Industry-wide benefits included increased uniformity, increased marketing effects, and positive impacts on staff training and service quality related to sharing of model practices.2
Accreditation in Public Health
To date, no studies have examined the impact of public health accreditation programs on the public health system or health outcomes. Other performance improvement activities, such as the National Public Health Performance Standards Program,5 the Illinois certification program,6 and Washington's Performance and Capacity Assessment Program,7 have analyzed the relationship among performance improvement activities and agency performance, system performance, and, in some cases, health outcomes.8
Findings from the Turning Point Performance Management Collaborative survey of state health agency performance management activities indicated that performance management activities have resulted in improved structures and processes (eg, contracting, policies, staff development).9 In several states, these improvements reportedly resulted in positive health and health-related outcomes, such as immunization rates and coronary bypass surgery survival rates.8 As an example, the Florida quality improvement and performance measurement system involving the state and local health departments was associated with improvements in selected health status indicators.10
As with the accreditation literature in service and health fields, the public health system performance measurement and improvement literature are primarily observational. In addition, results depend on outcomes studied and relatively new metrics, such as the public health performance standards, National Association of County and City Health Officials' Operational Definition of a Functional Local Health Department,11 and performance improvement inputs and outcomes. Furthermore, study results may be confounded by the selection bias of the states and local agencies that participate in these processes. Despite these cautions, Mays concluded that accreditation programs have the potential to improve public health service delivery, operations, and outcomes.2 The 2003 Institute of Medicine report, The Future of the Public's Health,12 also recognized the potential positive outcomes of accreditation for state and local health departments.
Particular Characteristics of Public Health
While progress in introducing an accreditation process to public health can be informed by the experiences and modeling of other service industries, there are some specific issues that make any evaluative activity
problematic in public health. Governmental public health is notoriously underfunded.13,14 This factor is critical in the attempt to relate the types and levels of outputs local agencies are able to produce. While one might assume that an accredited agency has the financial resources to provide a minimum level of services that corresponds with established model practices, more work is needed to understand the complex relationship between resources, priorities, and different types of outcomes for both accredited and nonaccredited agencies.
Second, time is a significant factor in the development of chronic diseases. The prevention of some tobacco-related illnesses such as cancer and heart disease may take decades to develop. The outcomes of some prevention programs will not be available within the time frame of an accreditation process, and conversely, attempts to link chronic morbidity and mortality outcomes to the current time may actually be the result of public health activities 30 years in the past. This issue was recognized in a study conducted by Hutchison and Turnock in Illinois.7
Lastly, the lack of an agreed-upon set of “public health outcomes” poses a problem. There are several sources that lend themselves to the development of such a construct. For example, Healthy People 2010 offers 10 Leading Health Indicators and more than 400 objectives.15 The United Health Foundation America's Health Rankings bases its state-by-state comparisons on a set of components that includes personal behaviors, community environment, health policies, and health status outcomes.16 However, overall, community-level system outcomes remain a topic with a very limited published research base.
Linking Accreditation/Performance and Outcomes in Public Health
In spite of these limitations, there is significant interest in better understanding the link between public health performance and outcomes. Recently, the National Public Health Systems Research Agenda was published and the need to explore performance and health outcomes was ranked as one of the top three priorities.17 A recent entire issue of the journal Health Affairs focused on the public health system and emphasized the importance of public health systems becoming accountable for health outcomes.18 Given much of the dialogue at the national level stemming from the Exploring Accreditation19 Project, the Multi-State Learning Collaborative and other efforts, the need to strengthen the evidence regarding performance and outcomes remains critical to guiding policy and public health practice. This important empirical linkage will likely influence the adoption, successful implementation, and credibility of a public health accreditation program.
As noted, the scientific base to measure, detect, and predict the nature and extent of public health outcomes in relationship to accreditation status is in its infancy. Given the overall lack of evidence, the development of a conceptual model provides a structure for further exploring anticipated outcomes and relationships. The logic model depicted below in Figure 1 illustrates the potential link between public health agency accreditation and public health outcomes, including (but not limited to) measures of health status. The underlying assumptions are as follows: (1) public health efforts result in positive changes to health status and (2) accreditation leads to quality improvement that, in turn, leads to the use of best practices thereby impacting community health.
The logic model was drafted by the authors using a participatory approach. The model draws from the work of the Exploring Accreditation Steering Committee, the existing Multi-State Learning Collaborative logic model, and a conceptual framework developed by Handler and colleagues20 used to explore the relationships between public health practice, performance, and outcomes. The proposed model focuses on inputs, strategies, outputs, and outcomes, with emphasis on accredited public health agencies as the input of interest. In our approach, we explicitly acknowledge that accreditation is an important, but not exclusive factor, in producing health outcomes. In addition, the model recognizes the complex interplay of contextual factors that have the ability to influence accreditation behavior, health outcomes, system performance, and ongoing research.
The major strategies included in the model focus on (1) maintaining performance and quality improvement systems; (2) sharing, documenting, and implementing model practices; (3) adhering to public health performance standards; (4) promoting the value of public health and agency accreditation; and (5) participating in ongoing accreditation-related efforts on a routine basis.
The right-hand side of the logic model showcases three levels of outcomes: short-term, intermediate, and long-term. The short-term outcomes focus on (1) enhanced performance, quality, and consistency related to the delivery of public health services; (2) increased accountability, efficiency, effectiveness, visibility, and perceived value of health departments; and (3) an increase in positive program-specific policies and outcomes. The intermediate outcomes include (1) agency-level changes resulting in enhanced public health services and model practices; (2) community-level changes reflected in the environment (eg, increased opportunity for physical activity) and through health behaviors (eg, decreased smoking rates); and (3) systems-level changes resulting in increased financial, political, and community support. The long-term outcomes focus on a network of high-functioning health departments that lead to a stronger public health system that ultimately results in improvements in population health outcomes as measured by morbidity, mortality, disparities, injuries, disabilities, and quality of life (the sought-after holy grail). The myriad factors that impact public health performance, outcomes, agencies, and systems research (eg, funding, leadership, political climate, community characteristics) are also incorporated and categorized as contextual factors.
Using the Logic Model
Ongoing public health systems research and evaluation are essential for identifying the strength and association of the relationships outlined in the logic model, even while acknowledging and controlling for system characteristics, priorities, funding, and other contextual factors and determinants of health that impact community-level outcomes. While there are a multitude of potential public health outcomes and indicators, many of which are related to long-term measures of health status, there is a critical need to build an evidence base that focuses on multiple levels of more proximate outcomes. These include capacity, practices, knowledge, systems, behavior, and policy-related outcomes, all of which likely influence long-term measures of health status.
While the methodological challenges of research on public health accreditation and outcomes are considerable, continuing research efforts are needed to help us address important questions. The proposed logic model depicted in Figure 1 is intended to be used, in part, as a roadmap to help identify and prioritize research and evaluation questions as well as to help conceptualize potential measures that can be used to demonstrate the impact of accreditation. Table 1 provides a list of example research questions that can serve as a springboard for future discussions and empirical efforts. While this list is not exhaustive, the questions reflect all elements within the logic model, including system inputs, strategies, outputs, and anticipated outcomes. A series of questions related to contextual issues is also provided.
While the questions may appear rudimentary, there is limited research to draw from and, therefore, basic questions require further investigation. For example, what are the incentives for seeking accreditation, why do some agencies appear to be more reluctant to pursue accreditation, what are the system or agency characteristics of those who have achieved successful accreditation, and which standards are linked to anticipated outcomes?
Despite the variability of performance standards and programs currently used across the country, and the challenges associated with securing comparable local-level health outcome data, there are opportunities and existing sources of data that can be used to address many questions. For example, The NACCHO Profile of Local Health Departments21 is a very rich source of detailed information about how local health departments function and about their characteristics. The Community Guide to Preventive Services provides a well-researched body of evidence-based practices to assess the extent of use of such practices in the everyday operations of health departments. The National Public Health Performance Standards Program offers a tool for measuring local public health system-level performance based on a series of model standards. Healthy People 2010 includes objectives to guide future activities, and benchmark data for assessing outcomes. The Behavioral Risk Factor Surveillance System offers data on short-term and intermediate outcomes, and also serves as a potential vehicle for including a limited number of specific survey questions that could be included to help assess the impact of accredited health departments in specific jurisdictions. The 10 Essential Public Health Services framework has recently been used to help shape National Public Health Performance Standards, the NACCHO Operational Definition of Local Health Departments, and the domains of the Voluntary National Accreditation Program for State and Local Health Departments. Given these efforts and continued interest, a universal set of public health standards may eventually emerge. A universal set of standards would help eliminate one of the existing challenges for conducting widespread and generalizable research in this area. Organizing these data sources within the logic model will help guide research and will be an important next step.
Given many of the current research and evaluation challenges, exploratory studies would likely prove useful. These studies can begin to provide information on potential linkages, characteristics, contributions, and relationships. For example, existing national level data (eg, National Public Health Performance Standards Program) could be used to explore the relationship between performance scores and local intermediate community health outcomes known to be sensitive to public health interventions (eg, youth smoking rates). This type of exploratory cross-sectional research design could begin to provide preliminary information on whether greater performance is linked with positive health outcomes, while controlling for population and system characteristics that are known to influence health status. While cross-sectional research designs will provide some insight, longitudinal studies that utilize an agreed-upon core set of standards and outcomes will likely provide valuable information on the relationship between performance/accreditation and outcomes. Finally, systems approaches and simulation models can be utilized to test hypotheses and better understand how the complex public health system works,22 as well as the interaction between system characteristics and outcomes.
If demonstrating a positive change in community health status is the “holy grail” for public health, just as sequencing of the human genome was once considered the grail of biology, then this article is more appropriately focused on the quest than on the final destination. Public health, relative to other fields, has recently begun to impose upon itself internal discipline in the form of accreditation. However, unlike many other fields, public health has demanded that this process not only results in some form of recognition, but that it also improves the quality of health departments, and ultimately improves the health outcomes of a community.
We first looked to other human service industries to review the track record of accreditation activities across a variety of applications. The results of the review might be described as “cautiously optimistic,” using descriptors such as “modest” and “promising.” Motivated, perhaps, on more of an intuitive than sound empirical base, public health practitioners, researchers, and funders have moved forward with accreditation development. While there is much to be learned from the experience of other industries, public health has some unique challenges including the lack of a universally accepted set of outcomes that public health agencies are responsible for.
Given the limitations and complexities, the authors have proposed a guide for the intrepid determined to pursue this important quest. The logic model offered here attempts to illuminate the many mutually influential relationships and components of the public health enterprise from beginning to end, from inputs to outcomes. In addition, it sets forth a structure to test each step in the journey. It is hoped that the research questions suggested will advance the existing body of public health systems research and ultimately lead us to the grail we seek. If so, this framework may confirm our twin hypotheses that public health does result in health status improvement, and that accreditation will both demonstrate and enhance this result.
The danger I see is that we cannot let people think they can get measurably improved health results just through accreditation or measurement. I believe any efforts to improve the quality of practice are good---and having some way to measure and demonstrate improvements is necessary if you undertake that aim.
However, there is not a common understanding of what outcomes can actually be expected vis-à-vis public health and accreditation of agencies.
Public health efforts are drastically underfunded, poorly organized, and fragmented. No amount of “quality improvement” will make a dent in the health outcomes we experience today unless those efforts are coupled with increased investments. And, it will take years to empirically link the two.
Public Health Practitioner
Local health directors reported that their agencies were directly responsible for contributing an average of 67% of the total effort devoted to the 20 public health activities in their jurisdictions,… the remaining one third of the community public health effort was contributed by other than the local health department.
Mays et al (2004)
We know, for example, that if a health department is accredited, this may improve their relationship with their board and the city governance. This improved relationship, and perhaps improved visibility, could lead to acceptance of written ordinances and opening the lines of communication. This happened in one accredited county in Missouri. Food vendors (eg, tamale stands, bar-b-cue stands) became a major problem in the city conceivably placing the public at risk of food-borne illness. The County Health Department wrote an ordinance after accreditation to ban nonlicensed food stands in the city limits and the ordinance passed. The health department spent many hours educating the city officials, attending meetings, and educating on epidemiological principals of food preparation. The health department attributes this policy action to an improved relationship between the city council and the health department.
Public Health Practitioner
2. Task Force on Community Preventive Services. The Guide to Community Preventive Services: What Works to Promote Health. New York: Oxford University Press; 2005.
4. Shaw C. Evaluating accreditation. Int J Qual Health Care. 2003;15(6):455–456.
5. Scutchfield FD, Knight EA, Kelly AV, Bhandari MW, Vasilescu IP. Local public health agency capacity and its relationship to public health system performance. J Public Health Manag Pract. 2004;10(3):204–215.
6. Turnock B, Handler A, Hall W, Lenihan DP, Vaughn E. Capacity-building influences on Illinois local health departments. J Public Health Manag Pract. 1995;1(3):50–58.
7. Beitsch L, Thielen L, Mays G, et al. The multistate learning collaborative, states as laboratories: informing the national public health accreditation dialogue. J Public Health Manag Pract. 2006;1293:217–231.
8. Hutchison KD, Turnock BJ. Feasibility of Linking Core Function-related Performance Measures and Community Health Outcomes. Chicago: University of Illinois at Chicago, Center for Public Health Practice; 1999.White Paper commissioned by Public Health Futures Illinois.
9. Public Health Foundation. Turning Point Performance Management Collaborative Survey on Performance Management in States. Seattle, WA: Turning Point National Program Office at the University of Washington; 2002.
10. Beitsch LM, Grigg CM, Mason K, Brooks RG. Profiles in courage: evolution of Florida's quality improvement and performance measurement systems. J Public Health Manag Pract. 2000;6(5):31–41.
12. Institute of Medicine, Committee on Assuring the Health of the Public in the 21st Century. The Future of the Public's Health in the 21st Century. Washington, DC: National Academy Press; 2003.
13. Beitsch L, Borrks RG, Manachemi N, Libby P. Public health at center stage: new roles, old props. Health Aff. 2006;25(4):911–922.
14. Institute of Medicine. Who Will Keep the Public Healthy? Educating Public Health Professionals for the 21st Century. Washington, DC: National Academy Press; 2003.
17. Lenaway D, Halverson P, Sotniknov S, Tilson H, Corso L, Millington W. Public health systems research: setting a national agenda. Am J Public Health. 2006;96(3):410–413.
18. Tilson H, Berkowitz B. The public health enterprise: examining out twenty-first-century policy challenges. Health Aff. 2006;25(4):900–910.
20. Handler A, Issel M, Turnock B. Conceptual framework to measure performance of the public health system. Am J Public Health. 2001;91(8):1235–1239.
22. Sterman JD. Learning from evidence in a complex world. Am J Public Health. 2006;96(3):505–514.
© 2007 Lippincott Williams & Wilkins, Inc.