Secondary Logo

Share this article on:

Core Metrics Pilot Project: A Case Study

Ryan-Ibarra, Suzanne, PhD, MPH; Pearlman, Dana, MSc; Grinnell, Sue, MPH; Hanni, Krista, PhD; Islas, Genoveva, MPH; Teutsch, Steven, MD, MPH

Journal of Public Health Management and Practice: November/December 2018 - Volume 24 - Issue 6 - p 578–585
doi: 10.1097/PHH.0000000000000715
Case Study

Survey Research Group, Public Health Institute, Sacramento, California (Dr Ryan-Ibarra); Independent Consultant, Santa Rosa, California (Ms Pearlman); Public Health Institute, Oakland, California (Ms Grinnell); County of Monterey Health Department, Salinas, California (Dr Hanni); Cultiva La Salud, Fresno, California (Ms Islas); and Public Health Institute, Oakland, California (Dr Teutsch).

Correspondence: Suzanne Ryan-Ibarra, PhD, MPH, Survey Research Group, Public Health Institute, 1825 Bell Street, Suite 102, Sacramento, CA 95825 (sryan@s-r-g.org;sue.grinnell@phi.org).

S.T. is the principal investigator.

The authors thank numerous stakeholders from communities in Fresno and Monterey Counties who engaged in the pilot project and provided valuable feedback. The authors also thank Giselle Chow, who provided graphic facilitation during the convenings. The authors are grateful for the support of numerous staff from Cultiva La Salud and the Monterey County Health Department, including Liliana Gomez from Cultiva La Salud and Ana Reyna-Leyva from the Monterey County Health Department. The authors particularly thank their sponsor, the Blue Shield of California Foundation, for its generous support. Richard Thomason and Rachel Wick provided wise guidance that greatly enhanced the quality of the work.

Steven Teutsch and Sue Grinnell conceptualized and designed the study. Suzanne Ryan-Ibarra and Dana Pearlman wrote the first draft of the manuscript. Sue Grinnell, Krista Hanni, Genoveva Islas, and, Steven Teutsch reviewed and revised the manuscript for intellectual content. All authors approved the final manuscript content for submission.

This study was funded by Blue Shield of California Foundation grant no. 24020379.

The authors declare no conflicts of interest.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's Web site (http://www.JPHMP.com).

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

Back to Top | Article Outline

Background: Setting for the Case Study

The National Academies of Sciences, Engineering, and Medicine, formerly known as the Institute of Medicine, (IOM), released a report on Vital Signs in April 2015 and presented a parsimonious set of Core Metrics that could be implemented and adopted by health care organizations and local communities.1 The concepts for the report were generated during an IOM workshop sponsored by the Blue Shield of California Foundation. The measures and recommendations were the subsequent work of an IOM committee. The report identifies national metrics in 4 domains: healthy people (also referred to as population health), quality of health care, cost of health care, and engagement. The report provides national estimates for each domain's indicators. However, these metrics were not estimated at the state or local level, nor was the feasibility of doing so assessed.

What distinguishes the Core Metrics from other sets of measures, such as Healthy People and the County Health Rankings, is the imprimatur of the IOM and the broad group of experts who wrote the report. One of the major challenges of the report, however, is that while general measures were recommended, they were not specified sufficiently to be used in a consistent fashion. For example, while high school graduation was a recommended measure, it was not clear how it would be measured. It could be operationalized as the proportion of ninth graders graduating in 4 years or people aged 25 years who have completed high school equivalency, or any of several other measures. In addition, communities may be interested in additional topics not in the core measure set, for example, housing or access to healthy food.

The value of measures is not in creating them but in using them to drive changes that lead to health improvement and health equity. The Core Metrics focus heavily on clinical issues and are, therefore, most important for clinical care systems. However, the public health community also finds clinical data useful to guide its work. We wanted to see how the Core Metrics could be implemented in community settings. To demonstrate how the Vital Signs recommendations could be operationalized at the community level and how the measures can be used by key stakeholders to shape planning processes that lead to health improvement and health equity, we conducted a pilot project in 2 California communities—Fresno and Monterey Counties. Fresno County is located in California's San Joaquin Valley, and Monterey County is located on the Central Coast of California. These pilot project communities had both rural and urban areas, with significant portions of their economics based in agriculture and high percentages of Latino residents (Table 1).2

TABLE 1

TABLE 1

Back to Top | Article Outline

Recruitment

To identify and recruit 2 communities for our pilot project, we used the following criteria: (1) having an existing multistakeholder collaborative with representation from both the public health and health care sectors; (2) current ability to collect and use metrics, including their engagement in related population health activities; (3) representation of different types of communities, for example, urban or rural; (4) geographic and ethnic diversity; (5) current engagement of Public Health Institute's (PHI) staff within the community; (6) how well the goals of our pilot project match the interests and needs of the community; and (7) how well identification of a core set of population health metrics could advance their community's priorities.

California's San Joaquin Valley counties (Fresno, Kern, Kings, Madera, Merced, San Joaquin, Stanislaus, and Tulare) were particularly attractive because PHI's Cultiva La Salud is based in the San Joaquin Valley and focuses on advancing health equity using policy and environmental change approaches. Having a specific healthy equity focus was attractive because of the opportunity to not only look at a geographic area but also to further explore a specific population subgroup. We spoke with the director of Cultiva La Salud who works closely with county governments as well as community organizations and safety net providers. Her program is deeply embedded and trusted within the Latino community and has a long track record of successful projects. The focus on the Latino population presented both a data challenge and a great opportunity. Cultiva La Salud's director was eager to use data-driven decision making and welcomed a set of locally specific and credible indicators. Although the program is based in California's San Joaquin Valley, a region of 8 counties, for demonstration purposes, we recognized that a single county would be both more feasible and have a greater likelihood of influencing decision making. In partnership with Cultiva La Salud, we selected Fresno County as the best site for this work.

In Monterey County, we approached the Central California Alliance for Health and found there was interest in collaboration. We had subsequent discussions with the planning, evaluation, and policy manager of the Monterey County Health Department and director of Community Impact of United Way Monterey County. Impact Monterey County, a collaborative organized by United Way Monterey County, had been working as a broad coalition on a community assessment and was transitioning to a network working on metrics, strategies, and objectives, as well as the creation of a dashboard. They had had initial meetings and identified an initial set of metrics. We did a crosswalk of the indicators they had identified with the Core Metrics and found several areas where the Core Metrics added valuable information to their indicator selection process. Moreover, they felt that our convening process to not only identify measures but also drive action would be particularly useful. After further discussion, it seemed like our time lines would mesh and there was potential for good synergies.

Back to Top | Article Outline

Interviews and Surveys With Stakeholders in Each Community

We identified stakeholders in each community under the guidance of a community leader who was a champion for the Core Metrics work (director of Cultiva La Salud; and the Planning, Evaluation, and Policy Manager of the Monterey County Health Department). To gather stakeholder input, we conducted online surveys. Stakeholders were diverse and included representatives from safety net systems including Federally Qualified Health Centers, not-for-profit hospitals working on their community health needs assessments, county health departments working on their community health assessments, not-for-profit organizations, and stakeholders in other sectors, such as education and housing. Themes discussed included identifying the social determinants of health and health care as the most pressing community health needs and using a shared measure set to move toward collective impact (Table 2).

TABLE 2

TABLE 2

Back to Top | Article Outline

Convening #1, August 2016—Meeting With Community Stakeholders to Plan Story Content

The first convening with community stakeholders was designed to spur collective action among participants and its objectives were designed to do this (Table 3).

TABLE 3

TABLE 3

The community champions set the tone for the convening, explained to the participants why they were invited, and explained their vision for the work and hopes for the day. We used World Café format where rotating small groups of 4 explored questions presented to them. It is a methodology that ensures broad engagement of participants and helps surface themes, patterns, and new ideas. It also helps surface collective wisdom in support of moving toward collective action. The World Café allowed each group of participants to prioritize their topics of interest and any additional topics to add to the Core Metrics best current measures. During the World Café, we asked the groups to discuss the following questions:

  • How might the Core Metrics + LiveStories help us make our community healthier?
  • Are there any critical gaps in the measures themselves?
  • We want to get to a single issue we are going to be working on. If you were to drive a course of action with these tools, where would you focus your attention?
  • What story do you want to tell?

The Convening #1 for Fresno County was held in Fresno, California, on August 23, 2016. Twenty-two stakeholders attended this convening. An overarching theme that emerged during Convening #1 in Fresno County was that the Core Metrics were an opportunity to offer the Latino community a greater voice and inclusion in conversations around health, including social determinants of health. In addition, 4 main findings emerged during Convening #1's World Café (see Figure, Supplemental Digital Content 1, available at http://links.lww.com/JPHMP/A401, graphic facilitation from Convening #1 in Fresno—findings from the World Café):

  1. The Core Metrics should show where disparities/inequities exist.
  2. Many people need to know about Core Metrics, including residents, elders, and decision makers/officials.
  3. The Core Metrics will likely demonstrate what the community has already been saying.
  4. Gaps exist in the Core Metrics primary measure set, and these gaps are primarily related to social determinants of health.

Then, various issues were raised for consideration in future Core Metrics data compilation and storytelling components of the project. Convening #1 concluded in Fresno with a vote on which issue should be added as a supplemental issue to the Core Metrics best current measure set. Active transportation received the most votes (n = 6), so the group decided that this issue would be chosen to add as a supplemental indicator for the Core Metrics best current measure set in the Fresno community.

The Convening #1 for Monterey County was held in Monterey, California, on August 22, 2016. Twelve stakeholders attended this convening. Details about this convening are included in Appendix 6, Supplemental Digital Content 2, available at http://links.lww.com/JPHMP/A421. Four main findings emerged during Convening #1's The World Café:

  1. Need more information about the Core Metrics and how they are defined/operationalized.
  2. The Core Metrics can help us work together across organizations by providing a shared purpose.
  3. Gaps exist in the Healthy People primary measure set because the set did not include all indicators that were important to the community.
  4. Core Metrics provide structure against which to evaluate other indicator sets that the county is currently using.

The gaps identified during Monterey's Convening #1 were mostly in the Healthy People/population health domain. Participants also discussed how the Core Metrics could provide structure for Impact Monterey4 if appropriate steps were taken to compare the measures in each set. At the end of Monterey's Convening #1, participants discussed ways to move forward with the Core Metrics work and how it could complement the work that was already occurring in the County, particularly with Impact Monterey. The group reached consensus that the Core Metrics team would add safety-related measures as a supplemental issue to the Core Metrics best current measure set for Monterey County.

Back to Top | Article Outline

Data Compilation

Based on the information gathered at the first convenings, the Core Metrics indicators were compiled for each county. After examining the measure sets currently in use in each pilot site, existing data sources were examined and used as much as possible. We focused our attention on the best current measures and incorporated only the related priority measures if they were of particular interest to the individual communities. In addition, we worked with each community to select one additional high-priority topic to include in the Core Metrics for that community.

The following approach was used to gather and report data:

  • Use publicly accessible data from reports (eg, health care–acquired infections) or dashboard Web sites (eg, California Health Interview Survey [AskCHIS], County Health Rankings, Dartmouth Atlas).
  • Present data that are statistically stable.
  • Compare to state and national data when possible.
  • Drill down by geography, race/ethnicity, age, and gender when data are available and statistically stable.
  • Present statistics to community stakeholders before finalizing them to be certain that they accurately represent the community.

Data were available for 14 of the 15 Core Metrics indicators at the county level and 5 of the 15 at the subcounty level (Table 4). Of the 14 Core Metrics available, 12 were identical to those included in the national IOM report and 2 were included as proxies when the national measures did not exist at the local level (ie, health literacy, social support). All of the measures in the Healthy People and Care Quality domains were available at the county level. Three of the 6 measures in the Healthy People domain were available at the subcounty level. Two of the 5 measures in the Care Quality domain were available at the subcounty level. One of the 2 measures in the Care Cost domain was available at the county level. The other (high spending relative to income) was not available. None of the 2 recommended primary measures in the Engaged People domain were available at the county or state level, so other measures were substituted for these (English language literacy, voter participation). Many of the community stakeholders were very interested in data on social support and health literacy but they were not available.

TABLE 4

TABLE 4

Back to Top | Article Outline

Building Core Metrics Web Sites

Between Convening #1 and Convening #2, the PHI team built custom Web sites featuring the Core Metrics indicators using LiveStories, a Web-based data storytelling platform that combines images, charts, maps, videos, and rich text to tell a story. We built 4 online “Vital Signs report card” interactive Web sites, 2 each (1 in English and 1 in Spanish) for Fresno and Monterey (see Figures, Supplemental Digital Content 2, available at http://links.lww.com/JPHMP/A402, and Supplemental Digital Content 3, available at http://links.lww.com/JPHMP/A403, screenshots of custom-built Web sites).

These Web sites displayed the Core Metrics indicators at the county and subcounty levels compared with state and national data.5 , 6 The charts and graphs were supplemented with photographs and text to tell the full story of the indicators in each community. Each Web site was designed to be a sustainable reporting tool that each community could update and use to assess progress. Additional Web sites were built to share additional indicators selected by consensus during Convening #1 (active transportation in Fresno7, safety in Monterey8). The additional indicators were chosen because they were important for current work in each community-based site. In addition to data, the report included photographs provided by our partners in each community and text written by our partners to describe their current efforts, tell their story in their own words, and indicate the future direction of their work.

After the second convening, the PHI team made additional changes to the LiveStories sites so that they were meaningful and useful to the community. Mostly, this meant revising text regarding recommendations for future action to make sure that it accurately reflected what community-based groups were committed to acting on. Then, all the content of the LiveStories sites was transferred to a representative from each of the community-based pilot sites.

Back to Top | Article Outline

Convening #2, December 2016—Meeting With Community Stakeholders to Present Draft Stories

A second, final convening was held to review the Core Metrics displayed in interactive, storytelling data visualization Web sites in LiveStories, present the results, and discuss possible actions moving forward. The objectives for the second convening are listed in Table 3. Again, we used a World Café format to harness the collective wisdom of the group. The questions explored were as follows:

  1. How might we use these tools (Core Metrics and LiveStories) for advocacy/driving change?
  2. How might we work together with this tool to move actions forward? How can we maintain it in the future?
  3. Commitments/Next steps

During Convening #2, preliminary versions of the LiveStories sites5 , 6 were presented to the participants. Participants had a chance to share their impressions of the sites and suggestions for improvement. Then, the facilitators led the participants in a World Café and group discussion of how to use the Core Metrics work.

The Convening #2 for Fresno County was held in Fresno, California, on December 7, 2016, and 11 stakeholders attended. Three main themes emerged during the World Café:

  1. Involve the community.
  2. Use the Core Metrics and LiveStories sites to secure funding to improve public health and the social determinants of health.
  3. People and efforts should be organized to drive action using the Core Metrics and LiveStories.

At the end of the convening, the participants committed to use the Core Metrics LiveStories sites to accomplish the following:

  1. Gather and share more data by asking government officials to share their data and by collecting additional needed data using teams of volunteers.
  2. Build capacity and leadership.
  3. Create economic opportunities.

The Convening #2 for Monterey County was held in Monterey, California, on December 8, 2016, and 15 stakeholders attended (see Figure, Supplemental Digital Content 4, available at http://links.lww.com/JPHMP/A404, graphic facilitation from Convening #2 in Monterey—Using Custom-Built Web Sites to Drive Action).

Three main themes emerged during the World Café:

  1. Growing together as we work together.
  2. Participating in mutually reinforcing activities.
  3. The Core Metrics and LiveStories Web sites make it easier for us to invite the media into our work.

In addition, convening participants voted on the Core Metrics that were most important for their community. During the convening, opportunities were identified that could drive the Core Metrics work (so-called “champion initiatives”) in the focus areas of Impact Monterey County: economic self-sufficiency, education, health, and safety.

Back to Top | Article Outline

Lessons Learned

The community engagement process we used increased familiarity with the Core Metrics while fostering collaboration and common understanding among the members of the Core Metrics implementation team (stakeholders, facilitators, data analysts, and project leadership). Community engagement is critical to determine how to share data that can inform current work occurring in communities. Participants and the pilot project communities found the community engagement process very beneficial, and it stimulated conversations that needed to happen, as reported in postconvening surveys. The communities defined for themselves actionable metrics, and we helped them put in place a process to set goals and track progress that engaged them for long-term success. When our work began, there were extensive activities already directed toward improving social determinants of health in each of the pilot project communities. They fit well with the Core Metrics because the Core Metrics measure set includes social determinants of health. The community groups we worked with were also active in working toward health equity. We found that the communities believed that the Core Metrics could drive further data analysis to support equity issues. For example, the engagement process in our pilot project augmented, but did not supplant, existing indicators projects in the communities where we worked. In addition, our work informed community groups of the criteria used to define and select the Core Metrics, which enhanced credibility to these groups that later considered adopting it for their indicator selection processes. It was during the community engagement process that our implementation team learned that both community stakeholder groups believed that it was critical to build all interactive data visualization sites in English as well as Spanish. The community engagement process also included challenges; such as, it was difficult to gather people and have the community champion lead this effort because they have so many other commitments. The community engagement process was highly resource intensive and required many more hours of staff time than originally planned.

Regarding data compilation, we found that it is possible to compile the Core Metrics data at the local level and compare them to state and national measures. However, one of the major challenges with implementing the measures in the Vital Signs report was the lack of specificity of the recommendations. Recommended measures are more “topics” than actual measures. The data presented in the Vital Signs report are national, which are of only marginal use to local communities. Thus, it was important to identify the information that can be used at the local level, and, to the extent possible, use measures that have the same specifications in all communities. However, one challenge was that sometimes, national measures (such as health literacy) were not available at the local level, so we used proxies (English language literacy, voter turnout).

Back to Top | Article Outline

Epilogue

Our pilot project suggests that the Core Metrics may be a useful indicator set for local jurisdictions to adopt and track. Our project is distinguished from other health indicator projects because of its focus on community engagement and sharing the indicators via custom-built Web sites focused on combining data with storytelling. Our work with 2 California communities found that the Core Metrics can add value to existing data projects, and building the custom Web sites facilitated easy sharing of the Core Metrics with partners, government officials, and policy makers. Each site reported that the Core Metrics added value to its existing work. As a result of the Core Metrics pilot project in Fresno, the Core Metrics were shared via custom-built Web sites with policy makers and city council members to increase awareness and drive action related to active transportation. In Monterey, additional measures were added to increase the breadth and diversity of the set of indicators being tracked by the Impact Monterey County initiative. In addition, participation in the Core Metrics pilot project diversified the types of stakeholders participating in the Impact Monterey County network and increased involvement in the network by members from health collaboratives.

Many indicator sets exist; no matter which indicator set the community chooses to adopt (eg, County Health Rankings, 500 Cities, America's Health Rankings, Core Metrics), a shared indicator set can help communities compile and present data to drive action. Virtually, every community in the United States has at least 1 indicator project underway. Our project's findings demonstrate that the Core Metrics can fit in with existing efforts (such as with the community health needs assessment in Fresno County or Impact Monterey County), and the Core Metrics can be adapted so that they are meaningful for communities and provide indicators that are important for their work. The question we are left with is as follows: Do the Core Metrics add value to existing data projects? We observed in this pilot project that they do. In Fresno, they provided data to a community-based group that did not have access to the health data they craved for use in their advocacy efforts. In Monterey, they lent credibility to processes, identified some missing partners and some gaps, underscored a focus on health equity, and provided a much needed opportunity for an alignment of effort between several indicator selection projects that were being conducted simultaneously.

There is statewide and national interest in expanding our work with the Core Metrics. Future opportunities may arise to develop custom Web sites that combine data with storytelling for each county in California and to pilot additional sites outside of California. These opportunities include work with the National Academies of Science, Engineering, and Medicine, and the Accountable Communities for Health. It is becoming increasingly recognized that sharing data in ways that many people, not only those with expertise in data interpretation, can understand and find meaning in is a critical way to advance public health. Sharing the Core Metrics in a way that is easy to understand via custom Web sites, and designing the indicator set to include input from the community so that the metrics are accepted, is important and critical for the community's health priorities.

Back to Top | Article Outline

Questions for the Reader:

  1. Identify the reasons that the 2 community champions were motivated to participate in the Core Metrics Pilot Project.
  2. Each of the 2 community champions had a different goal for participating in the Core Metrics Pilot Project. Comment on the differences among their agendas and motivating factors.
  3. During the Core Metrics Pilot Project, community engagement techniques were used to make sure that the project met the needs of the pilot communities. What were some of the barriers to using community engagement techniques during this project and how could those be overcome in future projects?
  4. What was the benefit of asking for additional measures of importance to each community? Why do you think they chose the topics that they did?
  5. How well did the communities appreciate the importance of nonclinical factors in the health and well-being of their communities? The importance of clinical factors?
  6. If you were going to implement the Vital Signs in a community, what would you do differently? The same?
Back to Top | Article Outline

References

1. Institute of Medicine. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press; 2015. http://nap.edu/19402. Accessed July 5, 2017.
2. US Census Bureau. DP-1 profile of general population and housing characteristics 2010. http://factfinder.census.gov/. Published 2010. Accessed June 30, 2017.
3. US Census Bureau. Place of Birth by Nativity and Citizenship Status: 2011-2015 American Community Survey 5-Year Estimates. Suitland, MD: US Census Bureau; 2015.
    4. Impact Monterey County. http://www.impactmontereycounty.org/. Published 2017. Accessed September 5, 2017.
    5. Cultiva la Salud, Public Health Institute. Fresno County Core Metrics Web site. https://insight.livestories.com/s/v2/fresno-county-core-metrics-overview/18574b4e-a48c-4537-adb5-3dae7b63f59f. Published 2017. Accessed September 5, 2017.
    6. Monterey County Health Department, Public Health Institute. Monterey County Core Metrics Web site. https://insight.livestories.com/s/v2/monterey-county-core-metrics-overview/c3f8ed0a-87ec-4267-92fd-6e9cb4a56b8b/. Published 2017. Accessed September 5, 2017.
    7. Cultiva La Salud, Public Health Institute. Fresno County Active Transportation Web site. https://insight.livestories.com/s/v2/fresno-active-transportation-overview/d3571c3e-a515-4ff8-9193-447a05dd8bc4/. Published 2017. Accessed September 5, 2017.
    8. Monterey County Health Department, Public Health Institute. Monterey County Safety and Health Web site. https://insight.livestories.com/s/v2/safety-and-health-in-monterey/d3947d2c-6804-4438-879c-17677d5d904d/. Published 2017. Accessed September 5, 2017.

    Supplemental Digital Content

    Back to Top | Article Outline
    Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved.