Share this article on:

Making the Most of Our Community Health Assessment by Developing a Framework for Evaluation

Gutilla, Margaret J. MS; Hewitt, Susan J. MS; Cooper, Bruce MD, MSPH

Journal of Public Health Management & Practice: July/August 2017 - Volume 23 - Issue - p S34–S38
doi: 10.1097/PHH.0000000000000594
Practice Article

The Health District of Northern Larimer County is a special district delivering community health programs and services in Northern Colorado. Since 1995, we have conducted a community health assessment (CHA) every 3 years. In 2015, we looked at our CHA process to see whether improvements could be made. We gathered information to understand recommended processes and potential impacts of CHA. We then developed a logic model to describe our CHA process and intended outcomes. The model outlined our CHA activities and impacts and informed our evaluation questions. We identified areas for process improvement, including our ability to analyze health inequities and effectively disseminate CHA data and findings. We recommend that others conducting a CHA evaluate their processes and impacts. Developing a logic model can be a first step toward creating an evaluation. While evaluation requires resources, we believe that the practice of CHA could greatly benefit from additional evaluation efforts.

Assessment, Research, and Evaluation, Health District of Northern Larimer County, Fort Collins, Colorado.

Correspondence: Margaret J. Gutilla, MS, Health District of Northern Larimer County, 120 Bristlecone Dr, Fort Collins, CO 80524 (mgutilla@healthdistrict.org).

The authors thank James N. Burdine and Micheal R. J. Felix for their assistance in the creation of our first community health assessment process in 1995 and their ongoing support since. They also thank Tom Chapel, Chief Evaluation Officer, Centers for Disease Control and Prevention, for his assistance in developing our logic model.

The authors declare no conflicts of interest.

Community health assessments (CHAs) have become increasingly common as a result of several drivers, including Public Health Accreditation Board requirements, the Affordable Care Act (ie, the IRS Community Benefit Rule), and other types of mandates.1–3 While some agencies may be just beginning to integrate a CHA process into their work, the Health District of Northern Larimer County (Health District) has conducted a triennial CHA process since 1995. The purpose of the Health District's CHA is to assess community health status and identify and prioritize local health needs. Our CHA process has consistently used an original population-based survey of adults, has facilitated in-person discussion forums, and has included analyses of secondary data to describe the health status, needs, and concerns of the residents of our community. For 20 years, the CHA has been used by our board of directors to prioritize the services and programs provided by our agency. In addition, findings from the CHA are shared with partners and program planners in Northern Colorado, including the local public health department and the hospital system.

It has long been assumed that CHAs lead to improved health and well-being of populations4; however, evidence that CHAs lead to improved community health is lacking. Starting in 2006, the American Public Health Association, the Council of State and Territorial Epidemiologists, and others released calls to action for increased evaluation of the impact of CHAs.5–8 Despite this, 10 years later, there are still few published evaluations examining outcomes and impact of CHAs.

In 2015, after 7 CHA cycles that used a consistent process, we decided to evaluate our CHA effort to determine whether our approach was still relevant and effective and to uncover opportunities to improve our methods for assessing community health in the future. The purpose of this article is to describe the Health District's approach to developing a logic model to describe our CHA, to share how this model informed evaluation questions, and to promote the need for additional CHA evaluation.

Back to Top | Article Outline

Methods

Setting

The Health District of Northern Larimer County, Colorado, is a special tax district that provides affordable dental, mental health, health care access, and health promotion services. The mission of the Health District is to enhance the health of the community. Formerly a hospital district (1960-1994), in 1995, the transition to a Health District meant that the elected board of directors was charged with creating new services and programs that would improve the health of the community. We developed a CHA, which gave us timely and local data, to help prioritize needs.

In 2016, the Health District conducted its eighth triennial CHA. The methods and process used have changed only in small ways since first implemented in 1995. We used 2015 as a planning year to reflect on both the strengths and the weaknesses of our past processes and to guide us in decision making about our CHA methods in 2016 and beyond.

Back to Top | Article Outline

Information gathering

Community health assessment involves activities such as public health surveillance, identifying needs, analyzing the cause of problems, and collecting and interpreting population health data.4 To aid us in understanding best practices for conducting an effective CHA, we examined peer-reviewed and gray literature that described evaluation of CHA or community health improvement planning. In addition, we also talked with 6 experts about best practices in CHA. These included academic researchers and authors of articles we identified during our reading.

Back to Top | Article Outline

Development of our logic model

Following our reading and a reflection on how our 20-year CHA process fits with recommendations for CHA best practices, in February of 2015, staff members of the Health District research and evaluation group and the manager of our local health department's community health improvement program held a day-long retreat with the following goals: 1. Describe our CHA activities, outputs, intended outcomes, and their causal relationship. 2. Discuss the political, economic, social, and technological modifying factors that facilitate or hinder our work. 3. Create a logic model for our CHA that would guide our evaluation. The retreat was facilitated by a national expert in public health evaluation, logic modeling, and strategic planning.

Back to Top | Article Outline

Results

Findings from information gathering

Following our review of literature and conversations with experts, we identified several factors related to the quality and impact of CHAs. First, we discussed multiple methods of local data collection and determined that our approach was appropriate. During these conversations, experts continually mentioned the importance of strong community engagement throughout the CHA process. In addition, these key themes that emerged from our readings were considered as we worked on the logic model design:

* Ongoing community engagement is recommended to create a high-quality CHA.1,9,10

* The use of quality data (locally collected, if possible), including data that can describe health inequities.9–11

* The creation of clear and understandable CHA reports, documentation, and other materials.10–15

* Strong communication and marketing about the CHA can increase dialogue in the community and promote action on specific health issues.16–20

In addition, our review of published literature revealed 3 logic models that could inform the development of our own.10,21,22

Back to Top | Article Outline

Logic model

The Washington State Logic Model for Community Health Assessment was used as a starting point to develop our logic model10; we chose to start with this model for its simplicity. Referencing the model's 4 steps, our goal was to further develop the CHA activities and outcomes/impacts from the perspective of a team of local health planners and analysts.

We started by placing our organizational mission, “to enhance the health of the community” as the distal outcome of our CHA logic model. We listed our CHA activities (engage stakeholders, collect data, analyze data) on the left side. Then, we connected our CHA process activities, outcomes, and impact (Figure). We identified areas related to action outside our organization (dotted lines), actions that relied on our internal organization (solid lines), and key areas that required internal and external collaboration (gray boxes). Our major activities included engaging stakeholders; collecting data (population-based survey, discussion groups, other secondary data); preparing high-quality, useful products; and disseminating the CHA results internally to our agency and to external stakeholders, including the public. If we succeeded in our CHA activities, we hoped that the data products would be used by both internal and external stakeholders to inform decision making. Then, the use of CHA information could result in stronger policies, programs, and services, as well as increased funding, ultimately helping to achieve our organizational mission of enhancing community health. The creation of the logic model allowed us to frame conversations with stakeholders to discuss our effectiveness along each path.

Back to Top | Article Outline

Evaluation questions

Following the development of the logic model, we drafted evaluation questions that correspond with our logic model. To help evaluate our activities we asked the following:

* Which, if any, of our current CHA data collection and assembling activities are effective for describing and understanding community health?

* Are there other data sources (eg, electronic health records, administrative data) or techniques we should consider and if so, would the findings look different using these different tools?

* Is our CHA process evolving and making use of technology (eg, social media) to understand and share information regarding our community's health?

* Are there gaps in our CHA process or important information we may be missing?

To evaluate our outcomes and impact, we asked the following:

* How has our CHA information and prioritization/strategic planning shaped the services and programs we have provided at the Health District?

* Has our CHA information served as a resource to support decision making elsewhere? If so, are there any parts of our CHA that were more or less helpful?

* Can we more effectively leverage community partners to contribute to the planning, funding, implementation, and use of our CHA findings?

We discussed these evaluation questions with internal teams and external stakeholders (including local public health agency and community hospital representatives). Following these meetings, we made 3 adjustments to our 2016 process to improve our process and impact. First, we noted that one of our most frequent types of data requests was for survey data stratified by race and ethnicity. In previous years, we had been unable to produce health status estimates for the Hispanic/Latino community due to a small sample size of this subgroup. Therefore, we made changes to our survey sampling methods to increase representativeness from this subpopulation. Second, internal users of the data told us that they were satisfied with our dissemination processes. However, external users felt that a faster and broader dissemination effort would increase the utilization of CHA data. To address this, we created a more extensive dissemination plan that included a series of topical issue fact sheets and a searchable data dashboard Web site. Finally, we realized that we need to set up systems to track requests and document impacts of our future CHA.

Back to Top | Article Outline

Lessons Learned

In the past, we had informally examined the CHA process and made small changes to our activities. Developing our logic model brought to the forefront our intended outcomes, impacts, and modifying factors, and also provided us a framework from which to develop evaluation questions. This experience also reiterated the complexity of designing a CHA and emphasized the need to include adequate time and resources to embed evaluation within our CHA efforts.23

Developing the logic model also allowed us to broaden our thinking. During our work, we considered political, economic, social, and technological factors that could influence our CHA. Political modifiers included the Affordable Care Act, Colorado's Public Health Act of 2008, the privatization of health care, and the political relationship of public and private sectors. Economic influences included our current budget for CHA and available staff for doing assessment work. We also acknowledged the increase in the number of organizations that are engaging in CHA and future opportunities for collaboration. We reflected on social factors that could influence our ability to conduct a CHA, including the public's desire for privacy, a growing distrust of government, and the ability of our team to use CHA data to provide information and insights that anticipate the needs of decision makers. Finally, we noted that the technology used to collect and disseminate data is rapidly changing, and the methods used in the past may no longer be cost-effective.

For future CHAs, we will improve our methods to gather feedback from those who use our data, including developing user surveys similar to those used by Stoto et al24 and following up with data requestors to find out how our CHA data were used and how they contributed to decision making. To fully understand the value of a CHA, though, additional investigations comparing the impacts between CHA-supported decisions and non-CHA-supported decisions are needed.25

Back to Top | Article Outline

Conclusions

We explored how our CHA process could contribute to our organization's mission. We identified areas for process improvement, including our ability to analyze health inequities and more effectively disseminate CHA data and findings. In our experience, the impacts from CHAs are rarely measured and reported systematically. This may be due in part to the number of resources that developing a formal evaluation requires and may point to the difficulty in measuring outcomes and impacts of a CHA process.8

Creating a logic model directed our attention to the desired outcomes and impacts of our CHA process. Moreover, it has inspired us to plan for further evaluation. For other organizations that may be developing a CHA process, our logic model illustrates the steps we identified to link CHA data to health impacts. Other organizations may consider our work as starting place to begin to construct their own logic model and corresponding evaluation plan. Additional evaluations of CHAs are needed to further understand the impact of this work in regard to achieving improved community health.

Back to Top | Article Outline

Implications for Policy & Practice

* Community health assessment (CHA) is a recommended process to assist in developing strategies to improve public health. Despite the increase in CHA activity, evaluations exploring how this practice is impacting community health are limited.

* As an initial step toward the creation of an evaluation, logic models can be developed to help understand and communicate intended impacts. Logic models can also inform evaluation approaches and provide guidance in the development of relevant evaluation questions.

* Our logic model outlined our CHA activities and impacts and informed our evaluation questions. We identified areas for process improvement, including our ability to analyze health inequities and effectively disseminate CHA data and findings.

* The practice of CHA could greatly benefit from evaluation efforts and the dissemination of lessons learned. While evaluation requires additional resources, we suggest that investments are worthwhile to improve CHA processes and reach intended impact.

Back to Top | Article Outline

References

1. PHAB Standards and Measures. Public Health Accreditation Board Standards & Measures Version 1.5. PHABOARD.org. http://http://www.phaboard.org/accreditation-process/public-health-department-standards-and-measures/. Accessed July 31, 2014.
2. Colorado Senate Bill 08-194. https://http://www.colorado.gov/pacific/sites/default/files/OPP_Senate-Bill-08-194_0.pdf. Published 2008. Accessed July 31, 2014.
3. IRS. New requirements for 501(c)(3) hospitals under the Affordable Care Act. http://http://www.irs.gov/Charities-&-Non-Profits/Charitable-Organizations/New-Requirements-for-501(c)(3)-Hospitals-Under-the-Affordable-Care-Act. Accessed April 19, 2015.
4. Institute of Medicine. The Future of Public Health. Vol. 88. Washington, DC: The National Academies Press; 1988.
5. Friedman DJ, Parrish RG. Is community health assessment worthwhile? J Public Health Manag Pract. 2009;15(1):3–9.
6. American Public Health Association. Conduct research to build an evidence-base of effective community health assessment practice—policy statement 20066. https://http://www.apha.org/policies-and-advocacy/public-health-policy-statements/policy-database/2014/07/07/13/06/conduct-research-to-build-an-evidence-base-of-effective-community-health-assessment-practice. Published 2006. Accessed November 20, 2015.
7. Council of State and Territorial Epidemiologists. Position Statement: Conduct Crosscutting Research to Build an Evidence Base for Effective Community Health Assessment in Practice. Atlanta, GA: Council of State and Territorial Epidemiologists; 2006. http://c.ymcdn.com/sites/http://www.cste.org/resource/resmgr/PS/06-EC-04.pdf. Accessed April 12, 2017.
8. Martin R. Examining the impact of public health assessments. J Public Health Manag Pract. 2009;15(1):1–2.
9. National Association of County & City Health Officials (NACCHO). Recommendations on Characteristics for High-Quality Community Health Assessments and Community Health Improvement Plans. Washington, DC: National Association of County & City Health Officials; 2013.
10. Spice C, Snyder K. Reviewing self-reported impacts of community health assessment in local health jurisdictions. J Public Health Manag Pract. 2009;15(1):18–23.
11. Behringer BA, Omohundro E, Boswell D, Evans D, Ferranti LB. The Tennessee Department of Health workshops on use of secondary data for community health assessment, 2012. Prev Chronic Dis. 2014;11:130206.
12. Myers S, Stoto M. Criteria for Assessing the Usefulness of Community Health Assessments. A Literature Review. Santa Monica, CA: RAND Corporation; 2006.
13. Byrne C, Crucetti JB, Medvesky MG, Miller MD, Pirani SJ, Irani PR. The process to develop a meaningful community health assessment in New York State. J Public Health Manag Pract. 2002;8(4):45–53.
14. Dunět DO, Butterfoss FD, Hamre R, Kuester S. Using the state plan index to evaluate the quality of state plans to prevent obesity and other chronic diseases. Prev Chronic Dis. 2005;2(2):.
15. Pennel CL, McLeroy KR, Burdine JN, Matarrita-Cascante D. Nonprofit hospitals' approach to community health needs assessment. Am J Public Health. 2015;105(3):e103–e113.
16. Irani P, Bohn C, Halasan C, Landen M, McCusker D. Community health assessment: driving the need for current, easily accessible population health data. J Public Health Manag Pract. 2006;12(2):113–118.
17. Luck J, Chang C, Brown ER, Lumpkin J. Using local health information to promote public health. Health Aff (Millwood). 2006;25(4):979–991.
18. Rohan AM, Booske BC, Remington PL. Using the Wisconsin County Health Rankings to catalyze community health improvement. J Public Health Manag Pract. 2009;15(1):24–32.
19. Peppard PE, Kindig DA, Dranger E, Jovaag A, Remington PL. Ranking community health status to stimulate discussion of local public health issues: the Wisconsin County Health Rankings. Am J Public Health. 2008;98(2):209.
20. Remington PL, Booske BC. Measuring the health of communities—how and why? J Public Health Manag Pract. 2011;17(5):397–400.
21. Kuehnert P, Graber J, Stone D. Using a Web-based tool to evaluate a collaborative community health needs assessment. J Public Health Manag Pract. 2014;20(2):175–187.
22. Barnett K. Best practices for community health needs assessment and implementation strategy development: a review of scientific methods, current practices, and future potential. http://http://www.phi.org/uploads/application/files/dz9vh55o3bb2×56lcrzyel83fwfu3mvu24oqqvn5z6qaeiw2u4.pdf. Published 2012. Accessed October 26, 2015.
23. Solet D, Ciske S, Gaonkar R, et al Effective community health assessments in King County, Washington. J Public Health Manag Pract. 2009;15(1):33–40.
24. Stoto MA, Straus SG, Bohn C, Irani P. A Web-based tool for assessing and improving the usefulness of community health assessments. J Public Health Manag Pract. 2009;15(1):10–17.
25. Rabarison K, Timsina L, Mays GP. Community health assessment and improved public health decision-making: a propensity score matching approach. Am J Public Health. 2015;105(12):2526–2533.

community health planning; evaluation; needs assessment

Copyright © 2017 Wolters Kluwer Health, Inc. All rights reserved.