Secondary Logo

Journal Logo

Articles

Teaching Public and Population Health in Medical Education: An Evaluation Framework

Johnson, Sherese B. MPH; Fair, Malika A. MD, MPH; Howley, Lisa D. MEd, PhD; Prunuske, Jacob MD, MSPH; Cashman, Suzanne B. ScD; Carney, Jan K. MD, MPH; Jarris, Yumi Shitama MD; Deyton, Lawrence R. MSPH, MD; Blumenthal, Daniel MD, MPH; Krane, N. Kevin MD; Fiebach, Nicholas H. MD; Strelnick, Alvin H. MD; Morton-Eggleston, Emma MD, MPH; Nickens, Chloe; Ortega, LaVonne MD, MPH

Author Information
doi: 10.1097/ACM.0000000000003737
  • Free

Abstract

During the past several decades, recommendations to improve the integration of public and population health (PPH) in medical education have been widely emphasized as necessary to address population health needs in a rapidly changing U.S. health and health care landscape.1–3 The U.S. population is becoming increasingly diverse, which requires a physician workforce that is culturally responsive to patient needs.4,5 Additionally, physicians must be equipped to address persistent health and health care disparities.5 Physicians’ increased understanding of social and economic factors is a necessity given that social determinants of health (SDOH) account for approximately 80 percent of health outcomes.6,7 The Association of American Medical Colleges (AAMC) has supported the inclusion of population health perspectives in physician education since 1939, when the Committee on the Teaching of Preventive Medicine and Public Health was established.8 Since 2000, the AAMC has engaged in activities funded by a cooperative agreement with the Centers for Disease Control and Prevention (CDC) to enhance PPH education and strengthen public health workforce capacity.9,10 The CDC and AAMC partnership advances a shared vision for training a health workforce that understands the broader context of health and is adept in improving the health of individuals and populations to advance health equity. The AAMC, along with its intra- and interprofessional partners, continues to promote collaborations and educational initiatives that support developing a diverse and culturally responsive workforce. This workforce should be equipped with the knowledge, attitudes, and skills necessary not only to effectively address social inequities, meet patient needs, and eliminate health disparities but also to respond to existing and emerging PPH crises, including the COVID-19 pandemic.

As the U.S. health care system evolves, many medical schools and teaching hospitals are training physicians with an expanded scope of practice that includes PPH. Although medical schools across the United States have worked to improve learning and practical experiences in the PPH sciences, no standard set of desired outcomes exists to measure the effectiveness of PPH training in medical education programs. In this article, we describe the development of a PPH educational program evaluation framework adapted from the New World Kirkpatrick Model, a training evaluation model used across disciplines in higher education.11,12 This framework is designed to help institutional leaders create a locally relevant evaluation strategy that they can use to optimize PPH curricular integration into medical education programs.

Establishing an Expert Panel on PPH in Medical Education

In July 2015, the AAMC identified 20 U.S. medical school faculty members with PPH expertise and established the Expert Panel on Public and Population Health in Medical Education (hereafter, Expert Panel). Members were asked to identify general principles and projects that should be considered to promote and improve the integration of PPH content and educational experiences across the medical education continuum. The culmination of the Expert Panel’s work was an internal report that included a ranked list of 10 project ideas to improve PPH educational programming in physician training. One of the proposed projects was to develop measures to help institutions assess their overall efforts in PPH education and identify areas for further investment. The AAMC engaged a subset of the Expert Panel to design an evaluation framework as a response to this recommendation.

Developing a PPH Evaluation Framework

The purpose of the framework is to assist medical faculty and educational leaders in developing locally relevant evaluations related to PPH educational programming at their institutions. In this article, we define public health as the science and art of preventing disease, prolonging life, and promoting health through the organized efforts and informed choices of society, organizations, public and private communities, and individuals.13 We define population health as the health outcomes of a group of individuals, including the distribution of such outcomes within the group. The field of population health includes health outcomes, patterns of health determinants, and policies and interventions that link these 2 elements.14 Health systems often engage in population health activities ranging from those limited to patient populations (e.g., patients with diabetes who are seen at a particular clinic) to those inclusive of the local geographic region (e.g., patients in the neighborhood, city, or county where the medical school is located). These activities—and both the terms “population health” and “public health”—are sometimes used interchangeably with “community health.” While community health has been defined as focusing on collective efforts of individuals and organizations who work to promote health within a geographically or culturally defined group, population health stresses an outcome-driven approach.15 Population and community health intersect, especially regarding community engagement, a foundational element of advancing any population’s health. Community engagement requires understanding local community needs and assets, as well as developing intentional partnerships with community members and organizations.16 Population, public, and community health all have the overarching goal of pursuing health equity, which strives for the highest possible standard of health for all people, giving special attention to the needs of those at greatest risk of poor health, based on social conditions.17

This article focuses on the educational opportunities and roles physicians may take related to PPH within medical schools, teaching hospitals, health systems, and communities. We define educational programs as all related didactic, experiential, required, and elective components of an institution’s undergraduate and graduate medical education (GME) curriculum.

The Expert Panel convened in April 2017 for a one-day meeting to draft the evaluation framework. This was accomplished through a series of small-group exercises focused on identifying, prioritizing, and ranking the proposed list of desired outcomes and areas of measurement. The group also identified measurement examples and tools that institutions have used for assessment.

The final evaluation framework, as presented and discussed in this article, is predicated on the 4-level New World Kirkpatrick Model that is recognized as a structure for evaluating training effectiveness.11,12 That framework specifies the following levels: (1) reaction, (2) learning, (3) behavior, and (4) results. Panel members added a fifth level, systems-based outcomes, to the model. This fifth level is adapted from published work on return-on-investment methodology.18 The model is similar to Moore and colleagues’ expanded continuing medical education framework for assessing physician competence, performance, and population health outcomes.19,20 Thus, the Expert Panel developed the following 5 levels of evaluation to measure the outcomes of PPH curricular content in physician education: (1) reaction, (2) learning accomplished, (3) application of knowledge and skills to practice, (4) outcomes achieved as a result of PPH educational programs and practice, and (5) systemic change that improves outcomes for relevant stakeholders. The group identified outcomes metrics at each Kirkpatrick evaluation level and then provided practical curricular or institutional examples to illustrate how a PPH educational program might be evaluated at those levels.

First level: Measuring reaction

Most educational programming evaluations begin with surveys of the learners and instructors in an effort to glean their reaction to, including satisfaction with, a program. Although faculty may be learners, the term “learner” as used in this article refers to premedical and medical students, residents, and fellows. Within the context of PPH, reactions of relevant community partners, who are often key contributors in the educational program, are also important. Community partners—defined here as individuals or groups of individuals who reflect the interest of the local community—include community residents or neighbors, community-based organizations, the local business community and employers, the local department of health, and other relevant governmental agencies. Ideally, program evaluations also assess community partners to help determine if these stakeholders find the experience of educational partnering beneficial to their communities. The Community-Campus Partnerships for Health Board of Directors has articulated guiding principles of authentic partnerships for service learning and participatory research, as well as for community-engaged research and services.21

Table 1 (row 1) highlights outcomes metrics at the reaction level that would show the degree to which learners, faculty, and community partners report positive experiences with a PPH educational program. Specifically, a successful educational program would have stakeholders reporting positive experiences, defining areas in need of improvement, recognizing opportunities for collaboration in the educational process, expressing interest and readiness to continue participating in the work, and valuing the community–academic collaboration. Measuring faculty reaction may be particularly important for an educational program’s continuous quality improvement; in one evaluation, survey respondents from 8 academic health centers identified inadequate faculty roles and rewards as a significant barrier to successful PPH programming.22 Involving community organizations in the evaluation of ongoing relationships is vital for building and sustaining partnerships that support PPH educational programs.23–25 Information gathered from this phase of the evaluation can help programs evolve, assist leaders with determining overall satisfaction with activities and effectiveness of partnerships, and guide future improvements.

T1
Table 1:
Outcomes Metrics for and Practical Curricular or Institutional Illustrations of PPH Content in Medical Education by Kirkpatrick Training Evaluation Level11 , 12

Second level: Measuring learning

Assessment of learning (i.e., accomplishment of program objectives) is essential for gauging curricular effectiveness and relevance to learners, the systems in which they work, and the populations and communities they will serve. This level of outcomes is best described as the degree to which learners, faculty, and community partners acquire the knowledge, skills, attitudes, confidence, and commitment they were intended to acquire based on their participation in a given educational program.26Table 1 (row 2) highlights outcomes metrics at this second evaluation level.

A goal of many PPH educational initiatives is to instill in learners the foundational knowledge and skills needed for a greater scope of practice, expanded beyond individual patient care. The expanded scope emphasizes physicians’ roles in identifying, preventing, or remediating PPH issues. This work includes striving toward health systems improvement, addressing the health care needs of underserved populations, and collaborating with community partners as part of the professional social contract.27,28 Professional social contract in medicine is defined here as society’s expectations of physicians to serve as healers; have guaranteed competence; provide altruistic services; promote the public good; and act with morality, integrity, and transparency.29 This includes advocating policies to improve health, providing care to underserved populations, and addressing health disparities within one’s practice and community. A successful educational program would promote learners’, faculty members’, and community partners’ commitments to learning and modeling PPH principles, as well as to integrating them into their practice and teaching activities, which, in turn, will help nurture the community–academic collaboration.

A learner who participates in a PPH educational program should be able to identify leading health indicators, specific SDOH, relevant national and local health statistics, and the policies that affect them. A learner’s ability to access and interpret local health statistics, familiarity with population health assessment tools, and ability to identify vulnerable populations may be assessed through standardized exams, structured exercises, and surveys.30–32 Surveys can also be used to determine interest in working with medically underserved populations, as well as learners’ confidence in, and commitment to, applying PPH principles in practice. For example, national surveys such as the AAMC’s Matriculating Student Questionnaire and the Graduation Questionnaire can be used to measure medical students’ perceptions of and interest in working to improve population health, working in underserved areas, or caring for underserved populations.33–35

PPH education requires faculty who have population health knowledge and skills to optimally teach and assess learners. Historically, PPH competencies have not been emphasized in physician training. A mismatch between educational need and faculty skill set may create challenges and require significant new academic collaborations, curricular approaches, and faculty development. While working with nonphysician experts and educators and emphasizing interprofessional education will help, having strong physician role models is ideal. The presence of committed and qualified faculty who can provide PPH teaching and training for learners is associated with student satisfaction.36 Although expecting all faculty to effectively design and deliver PPH educational programs may be unrealistic, frameworks and resources are available to engage select faculty in teaching these expanding health care constructs. For example, the Association for Prevention Teaching and Research and the Association of Academic Health Centers convened the Healthy People Curriculum Task Force to develop a framework that health professions educators can use to design a relevant and appropriate population health curriculum.37Also, faculty from Duke University’s Department of Community and Family Medicine developed a population health competency roadmap that emphasizes the importance of highly collaborative interprofessional teams.38 More recently, the American Hospital Association released a 5-module SDOH curriculum for clinicians.39

While faculty development is underway, integrating PPH educational objectives into relevant existing basic and clinical science course content, case-based learning, and clerkship assignments can be accomplished by inviting known resident PPH experts (e.g., existing faculty with PPH expertise/experience; public health faculty; experts from health departments, professional associations, health systems, or advocacy organizations) to suggest specific PPH enhancements to existing course materials or student assignments. For example, The George Washington University School of Medicine and Health Sciences has created a corps of clinician faculty with PPH expertise to work with other faculty, to lead coursework, and to facilitate student PPH activities.40 An increasing number of medical schools also offer degree programs that create opportunities for faculty to pursue graduate education in PPH and, in turn, teach PPH concepts at their own institution. Assessments in continuing education, faculty development programs, and faculty surveys about career interest, planning, and intent can measure faculty knowledge and skills.

Third level: Measuring application

Educators are encouraged to evaluate the degree to which learners, faculty, and community partners apply what they have learned as a result of a PPH educational program. Table 1 (row 3) highlights outcomes metrics that may indicate learners’ abilities to employ skills and enact behaviors that address PPH needs. In many instances when educators ask the question, “Toward what end are we providing an education?” or “What do we want to accomplish through the education experience?”, the answer extends beyond simply improving knowledge to ensuring that the knowledge gained by learners is appropriately applied in education and practice. Encouraging learners to leverage PPH knowledge, attitudes, and skills can be accomplished and assessed by integrating structured opportunities for them to apply their PPH education. Potential examples include participating in organized projects targeted to health system, patient experience, and quality improvement activities, or working with faculty leaders, local community-based service organizations, governments, or advocacy groups to address community health problems. For example, The George Washington University School of Medicine and Health Sciences creates topic-specific “Summits” through which students, in collaboration with community organizations, government officials, and health system leaders, apply their scientific and clinical knowledge to address local public health or community health priorities or quality improvement activities in the health system.41–44 Using a sustained community–academic partnership, the Larner College of Medicine at the University of Vermont provides measurable benefits to community health and helps meet social needs using structured medical student projects.45 Several schools train students as health navigators to help them understand the health care system.46–48 Students enrolled in Georgetown University School of Medicine’s Population Health Scholar Track partner with health systems, public health institutions, and policy institutes on research projects that entail population-based interventions.49 Other medical schools are assessing the application of PPH principles by incorporating content related to health disparities and local health concerns into clinical rotations50 or by providing culturally responsive care content in objective structured clinical examinations (OSCEs) or standardized patient scenarios.51 Assessing outcomes can help educators determine when an anticipated effect on learners is not realized, suggesting that the educational content or approach needs modification.

This third level of outcomes evaluation requires assessing not only whether learners and faculty are applying what they learn but also whether they are actively engaging in activities that address health systems improvement and community needs. Such activities could include collaborating with health system leaders, public health professionals, and local community organizations; participating in community health needs assessments; or engaging in community-based participatory research projects, clinical outreach, and population health management and quality improvement initiatives. Notably, the Accreditation Council for Graduate Medical Education provides national guidance on expected outcomes or competencies in PPH through its Clinical Learning Environment Review (CLER) program and Common Program Requirements.52,53 One of the 6 CLER focus areas is health care quality, which includes a pathway for GME engagement in reducing health care disparities for the patient populations trainees serve. In addition, the AAMC’s Quality Improvement and Patient Safety Competencies Across the Learning Continuum outlines cross-continuum competencies for medical students, residents, and experienced faculty that advance health care quality and patient safety, including PPH.54

Fourth level: Measuring results

Level-4 outcomes refer to metrics that show the degree to which targeted outcomes are achieved as a result of the educational program. Targeted PPH outcomes may include (1) the degree to which learners and faculty incorporate PPH into their careers either formally or informally and (2) the degree to which learners, faculty, and community partners advance scholarship and disseminate findings with the aim of improving the public’s health. Surveys of alumni could capture graduates’ career choices, the populations they serve, and their formal or informal participation in PPH activities.

An additional outcome metric is increasing the number of students practicing in underserved areas.55 Mapping tools may help institutions understand the opportunities to build and maintain physician capacity in rural and underserved areas.56 The University of California, Los Angeles (UCLA)/Charles R. Drew University Medical Education Program focuses on training physicians for practice in underserved areas. The program’s results show that its graduates have higher odds of practicing in underserved areas (e.g., a Health Professional Shortage Area; a rural, high minority area; a high-poverty area) compared with other UCLA medical students who did not participate in the program.57 Similarly, Montefiore Medical Center’s Residency Program in Social Medicine, located in the Bronx, New York, has achieved its mission since 1970: more than two-thirds of all its graduates enter practice in medically underserved communities—a portion significantly higher than a sample of its applicants who trained elsewhere.58 Such integrated models require not only a supportive institutional infrastructure that includes selective admissions criteria focused on recruiting students who are more likely to practice in underserved areas but also incentives for faculty and community preceptors aimed at achieving systematic and lasting community engagement.59,60 In the academic setting, systemic change requires including criteria for academic review, promotion, and tenure that explicitly value PPH as measured by the results and effect of faculty leadership and activities in these areas.61–64

As outlined in Table 1 (row 4), assessing Level-4 progress could include determining the number of community–academic partnerships that support educational experiences. These could be measured by formal memoranda of agreement, joint grants and projects, or board member exchanges. Such partnerships could also be defined and included in an academic organization’s strategic plan. Measuring the duration of curricula or partnerships would gauge project longevity or sustainability. Patient and community satisfaction surveys could be used to assess how well or how appropriately learners and faculty respond to a community’s culture, what skills learners apply, and the outcomes of community–academic partnerships. These outcomes are also measured as part of the anchor institution movement, which aims to improve socioeconomic conditions in the communities surrounding institutions of higher education.65 Anchor institutions intentionally commit to a mission of collaboration with community partners through economic partnerships, capacity building, and other support. Health systems, as anchor institutions, have an opportunity to engage with and effect positive change in the neighborhoods where they are located.66 For example, Rush University Medical Center in Chicago has developed the Anchor Mission Strategy that outlines its commitment to channeling its economic power to improve health outcomes for Chicago’s West Side residents through addressing social conditions. West Side community-based organizations have partnered with Rush to support an anchor mission employment preference initiative and expect to realize their goal of ensuring that West Side residents acquire jobs that provide economic value.67 Similarly, the Stanford Collective Impact model68,69 has been adopted by the federal Health Resources and Services Administration (HRSA) for its Healthy Start programs. Since 2014, HRSA has required all program grantees—including the local health departments; community foundations and nonprofits; federally qualified health centers, hospitals, and medical schools that serve as the “backbone support” or anchor institutions for Healthy Start—to apply the model in their effort to reduce infant and maternal mortality.70 As medical schools, teaching hospitals, and health systems incorporate an anchor mission into their strategic plans, establishing a framework of metrics will contribute to collecting data that tell their stories and capture the results of embracing PPH curricula.

As future physician leaders, learners should be able to apply a comprehensive, holistic approach to clinical care that extends beyond the health care setting. This approach would incorporate systems thinking and awareness of the complex web of interdependencies that contribute to population health.71 PPH advocacy and policy initiatives focused on community health that emanate from PPH educational programs may serve as markers of a program’s success. For example, in California, Kaiser Permanente conducted a review of its assets to determine how it could use them to study and improve health conditions. Its results and subsequent investments have focused on developing a local workforce, procuring goods from local and diverse sources, and embracing a policy role.72–74 In Massachusetts, Boston Medical Center conducted an analysis to determine where its initiatives could be most effective and then, based on the results, developed various partnerships to fund housing development initiatives that addressed identified housing needs.72

Programs could also measure their success by their development of physician leaders who advocate health system and community health improvements. Program assessment might include determining changes in clinical referral patterns to encompass public health partners or social service agencies addressing SDOH75; measuring the number and type of advocacy-related contacts with policymakers; and gathering qualitative and quantitative descriptions of community partnerships that might support teaching, research, and service.

Fifth level: Measuring systems-based outcomes

The fifth level of outcomes measurement, systems-based outcomes, is essential for determining improvements in population health and for gaining institutional, political, and financial support for PPH educational programs.66 Calculating a return on investment requires data collection and analyses, efforts to isolate the effects of a given initiative, consideration of both the tangible and intangible benefits and costs, and the reporting and dissemination of outcomes to stakeholders. Previous studies of mobile health care interventions in underserved communities serve as an example.76,77 As presented in Table 1 (row 5), the outcomes metrics of PPH educational programs should include both (1) a measurable improvement in the health of communities and populations and (2) a measurable reduction of health disparities along with an associated decrease in health care costs. Ideally, evaluation of health in target communities and populations would demonstrate that improved morbidity and mortality and reduced disparities were related to an institution’s PPH education curricula and programs. These benefits could be the direct result of the educational program itself. For example, student or faculty projects, in partnership with community organizations, might lead to improved community health or health care. Likewise, other activities, including advocacy initiatives that program graduates and faculty lead, might result in improved housing or after-school programming for local residents. Of course, countless confounding factors contribute to changes in health outcomes; nonetheless, medical schools and teaching hospitals should, at a minimum, track measures and quantify local and regional trends that detail how their work likely contributed to overall changes in the public’s health. Institutions should also consider measuring the results of their interprofessional approaches and collective impact strategies to improve PPH outcomes.78,79

In addition to the collecting and recording of health data, evaluation efforts can help identify leading indicators that contribute to improved PPH. Potential examples include access to medical care, which could be improved if graduates practice in underserved communities, health promotion programs established through community–academic partnerships developed with input from faculty and/or graduates, or new laws and policies that promote health established through program graduates’ and faculty members’ advocacy efforts (e.g., tobacco control or childhood vaccination regulations). Initiatives to evaluate PPH educational programming could work in tandem with community- or state-level health assessments, public health improvement plans, or national goals and objectives, such as those outlined in Healthy People 2030.80

Effects of PPH education may be reflected in process and outcome measures over short- and long-term time frames. Measures should be developed collaboratively with members of the target populations and include perspectives from communities, patients, caregivers, employers, and health care teams. Level-5 outcomes metrics should also reflect advocacy and policy initiatives and include program costs and benefits. Community health needs assessments and similar community assessments may provide baseline information and may also be useful for assessing changes longitudinally. Specific examples of measurements of community-led institutional initiatives include large-scale program evaluations, community satisfaction surveys, and outcomes data collected from electronic medical records. This information can provide context and focus for improving patient outcomes data, population health, and health disparities. Measurement of the effects of educational programs on trainees might be accomplished using graduate surveys or employer questionnaires. A comprehensive review of student, trainee, or faculty roles in local, state, and national policies to determine whether they advance health equity could indicate successful advocacy efforts. Program costs can be assessed against measurable changes in preventive screening, access to care, morbidity, and mortality in the communities where trainees serve.

Discussion

Institutions that train physicians to improve the health of individuals and populations should measure the benefit of that education to the patient, health system, learner, faculty, and specific population. The SDOH and their influence on PPH are increasingly recognized in national policy recommendations. The American College of Physicians and National Academies of Sciences, Engineering, and Medicine have each recently published policy recommendations to improve patient care and health outcomes.81,82 This article applies an expanded New World Kirkpatrick Model of training evaluation11,12 to the integration of PPH education in U.S. medical education programs in an effort to examine such outcomes. This work contributes to the existing literature by describing a structured framework that can be adapted and applied to locally developed PPH educational programs in medical schools and teaching hospitals. It integrates the expert consensus of medical faculty across the United States with peer-reviewed literature describing medical education programs that seek to reinforce PPH education approaches. This framework was developed to assist institutional leaders with identifying curricular goals, areas for improvement, and opportunities for promoting investment in PPH educational programming. It may serve as a starting point for leaders seeking to demonstrate effects to funders and stakeholders; for curricular deans seeking to identify curricular gaps; and to course, clerkship, or program directors and designated institutional officials seeking to evaluate the effects and successes of their respective curricula.

The challenge of designing and evaluating educational programs that prepare physicians to expand their scope of practice and contribute to improving PPH can seem daunting. Many factors affect PPH outcomes, including the economy, the built and natural environment, the education of and educational opportunities afforded to local populations, housing, and other social determinants. The actions of public health leaders, as highlighted in the recent Public Health 3.0 model,83 must incorporate expertise from local health departments, faith-based communities, nonprofit organizations, schools, neighboring health systems, governments, and other groups to effectively improve community and population health. These actions and collaborations will, in turn, influence assessments of educational interventions designed to prepare clinicians for their roles in improving population health. Given this complexity, it is important to create a common framework for evaluating, at different levels, the intended effects on learners, faculty, institutions, and communities. As outlined in this framework, descriptive measures may provide valuable qualitative perspectives and supplement available quantitative data found in County Health Rankings and Roadmaps,84 America’s Health Rankings,85 CDC Wide-ranging Online Data for Epidemiologic Research (i.e., WONDER),86 CDC Health, United States annual reports,87 Healthy People 2030,80 and other quantitative reports. Descriptive measures also provide another way of examining and sharing community and population-level effects of an institution—beyond its social mission ranking.88

Satisfaction surveys are common in the medical education literature. Although satisfaction surveys do not demonstrate curriculum effectiveness, having student support may help educators initiate curricular change and move toward higher-level outcomes assessments. Notably, learner and faculty input alone are insufficient. We want to emphasize the importance of community members’, local leaders’ and agency collaborators’ perspectives in assessing the success of educational initiatives.

The framework as presented has some limitations. While the professional and demographic diversity of the Expert Panel’s members increases the framework’s validity, implementing the framework by accounting for context across a diverse set of institutions may affect ecologic validity. The methodology for developing this framework strengthens its content validity, but its construct validity has not yet been tested. Although the framework was developed by expert consensus and is grounded in the existing literature, no formal systematic review was undertaken. While we believe this framework to be broadly applicable in the context of U.S. medical education, it was not designed to furnish comparisons across institutions. Relatedly, the scope and scale of the framework as a tool to assess PPH educational programs at the institutional level may limit its capacity to discriminate between and compare institutions for some unique curricular areas, such as the care of local Indigenous communities, the health of refugee populations, or other specific topics.

Another challenge associated with this framework is finding time and space to include PPH concepts in already crowded medical education curricula. Medical schools and teaching hospitals around the country are restructuring with a general trend toward increased integration of content within and across the medical education curriculum. As education deans and other medical education leaders continually assess and improve their curricular practices, they should work to identify opportunities to emphasize current PPH concepts and practices. In some cases, PPH integration may require intentional curriculum mapping; for example, relating a single, specific clinical or educational practice (e.g., teaching human papillomavirus [HPV] testing techniques) to the larger public health context (e.g., HPV immunization, cervical cancer epidemiology, health disparities). In other cases, PPH integration may require a more significant addition or restructuring (e.g., developing a Health Systems Sciences course, integrating a longitudinal biostatistics and epidemiology curriculum). Regardless, educators should systematically apply this framework to identify gaps and seek opportunities for new or enhanced curriculum development across the continuum of medical education. In addition, medical curriculum leaders and deans should conduct a careful review of the physician skill set needed for current and future pandemics.

Conclusions and Future Research

An expanded PPH education and practice model designed to improve health in the United States through strengthening curricula and community–academic partnerships is achievable. Future research should explore applying this framework to the assessment and evaluation of institutional and programmatic PPH curricula. Additional efforts should also capture the challenges associated with identifying and defining the resources necessary to apply the framework within an institution. As U.S. medical schools and teaching hospitals seek ways to develop or enhance their PPH education, this framework may assist institutional leaders as they design an evaluation that is applicable to their PPH activities, offerings, and partnerships. It may also assist institutional leaders with the critical work of preparing physicians to understand their role in improving health outcomes in the communities they serve.

Acknowledgments:

The authors thank all members of the Expert Panel of Public and Population Health in Medical Education for their participation in the framework development activities, including Luther Brewster, PhD, Scott Frank, MD, MS, Arthur Kaufman, MD, Benjamin Lebwohl, MD, MS, Shilpa Murthy, MD, MPH, Elizabeth Nelson, MD, Jennifer Newberry, MD, JD, Ann-Gel Palmero, DrPH, Christine Seibert, MD, and Janet Townsend, MD. The authors also thank David Acosta, MD, Rika Maeshiro, MD, MPH, and Alison Whelan, MD, for their conceptual input to earlier drafts and Katy Carkuff-Corey for administrative assistance with this article.

References

1. Institute of Medicine. Training Physicians for Public Health Careers. Washington, DC: National Academies Press, 2007. https://doi.org/10.17226/11915. Accessed August 6, 2020
2. Institute of Medicine. Who Will Keep the Public Healthy? Educating Public Health Professionals for the 21st Century. Washington, DC: National Academies Press, 2003. https://doi.org/10.17226/10542. Accessed August 6, 2020
3. Koo D, Thacker SB. The education of physicians: A CDC perspective. Acad Med. 2008; 83:399–407
4. Colby SL, Ortman JM. Projections of the Size and Composition of the U.S. Population: 2014 to 2060: Population Estimates and Projections. Current Population Reports, P25-1143. Washington, DC: U.S. Census Bureau; March 2015. https://www.census.gov/content/dam/Census/library/publications/2015/demo/p25-1143.pdf. Accessed August 6, 2020
5. Cohen JJ, Gabriel BA, Terrell C. The case for diversity in the health care workforce. Health Aff (Millwood). 2002; 21:90–102
6. Magnan S. Social Determinants of Health 101 for Health Care: Five Plus Five. NAM Perspectives. Discussion Paper. Washington, DC: National Academy of Medicine, 2017. https://doi.org/10.31478/201710c. Accessed August 6, 2020
7. Hood CM, Gennuso KP, Swain GR, Catlin BB. County health rankings: Relationships between determinant factors and health outcomes. Am J Prev Med. 2016; 50:129–135
8. Final report of the Committee on the Teaching of Preventive Medicine and Public Health. J Assoc Am Med Coll. 1945; 20:152–165
9. Maeshiro R, Koo D, Keck CW. Integration of public health into medical education: An introduction to the supplement. Am J Prev Med. 2011; 414 suppl 3S145–S148
10. Maeshiro R, Johnson I, Koo D, et al. Medical education for a healthier population: Reflections on the Flexner Report from a public health perspective. Acad Med. 2010; 85:211–219
11. Kirkpatrick Partners. The New World Kirkpatrick Model. https://www.kirkpatrickpartners.com/Our-Philosophy/The-New-World-Kirkpatrick-Model. Accessed August 6, 2020
12. Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s Four Levels of Training Evaluation: Results – Behavior – Learning – Reaction. Alexandria, VA: ATD Press, 2016
13. Winslow CE. The Untilled Fields of Public Health. Science. 1920; 51:23–33
14. Kindig D, Stoddart G. What is population health? Am J Public Health. 2003; 93:380–383
15. Maraccini AM, Galiatsatos P, Harper M, Slonim AD. Creating clarity: Distinguishing between community and population health. Am J Accountable Care. 2017; 5
16. Bodenheimer T, Syer S, Fair M, Shipman S. Teaching Residents Population Health Management. Washington, DC: Association of American Medical Colleges, 2019. https://store.aamc.org/teaching-residents-population-health-management.html. Accessed August 6, 2020
17. Braveman P. What are health disparities and health equity? We need to be clear. Public Health Rep. 2014; 129suppl 25–8. https://doi.org/10.1177/00333549141291S203. Accessed September 1, 2020
18. Phillips J, Buzachero VV, Phillips PP, Phillips ZL. Measuring ROI in Healthcare: Tools and Techniques to Measure the Impact and ROI in Healthcare Improvement Projects and Programs. New York, NY: McGraw Hill Education, 2013
19. Moore DE Jr, Cervero RM, Fox R. A conceptual model of CME to address disparities in depression care. J Contin Educ Health Prof. 2007; 27suppl 1S40–S54
20. Moore DE Jr, Green JS, Gallis HA. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009; 29:1–15
21. Community-Campus Partnerships for Health Board of Directors. Position statement on authentic partnerships. http://www.ccphealth.org/principles-of-partnership. Published 2013 Accessed August 6, 2020
22. Calleson DC, Seifer SD, Maurana C. Forces affecting community involvement of AHCs: Perspectives of institutional and faculty leaders. Acad Med. 2002; 77:72–81
23. Center for Health Care Strategies, Inc. Partnership assessment tool for health. https://www.chcs.org/resource/partnership-assessment-tool-health. Published October 2017 Accessed August 6, 2020
24. National Collaborating Centre for Methods and Tools. Partnership evaluation: The Partnership Self-Assessment Tool. Hamilton, ON: McMaster University, 2008. https://www.nccmt.ca/registry/resource/pdf/10.pdf. Updated January 4, 2018 Accessed September 1, 2020
25. Centers for Disease Control and Prevention. Evaluation Guide: Fundamentals of Evaluating Partnerships. Atlanta, GA: U.S. Department of Health and Human Services, 2008. https://www.cdc.gov/dhdsp/evaluation_resources/guides/evaluating-partnerships.htm. Accessed August 6, 2020
26. Kirkpatrick DL. Evaluating Training Programs: The Four Levels. 2nd edSan Francisco, CA: Berrett-Koehler Publishers, Inc., 1998
27. Gruen RL, Pearson SD, Brennan TA. Physician-citizens–public roles and professional obligations. JAMA. 2004; 291:94–98
28. Gadon M. Revisiting the social contract: Physicians as community health promoters. Prev Chronic Dis. 2007; 4:A39
29. Cruess SR, Cruess RL. Professionalism and medicine’s social contract with society. Virtual Mentor. 2004; 6:185–188
30. Prunuske J, Remington PL. A community health assessment curriculum to develop population health competencies. PRiMER. 2017; 1:1. https://doi.org/10.22454/PRiMER.2017.1.1. Accessed September 1, 2020
31. University of Washington School of Medicine. Curriculum: Curriculum phases and supplemental programs. https://www.uwmedicine.org/school-of-medicine/md-program/curriculum. Accessed September 1, 2020
32. The George Washington University School of Medicine and Health Sciences. Patients, populations, and systems courses. https://smhs.gwu.edu/academics/md-program/curriculum/clinical-public-health/patients-populations-and-systems-courses. Accessed August 6, 2020
33. Association of American Medical Colleges. Matriculating student questionnaire (MSQ). https://www.aamc.org/data-reports/students-residents/report/matriculating-student-questionnaire-msq. Accessed August 6, 2020
34. Association of American Medical Colleges. Graduation questionnaire (GQ). https://www.aamc.org/data-reports/students-residents/report/graduation-questionnaire-gq. Accessed August 6, 2020
35. Boscardin CK, Grbic D, Grumbach K, O’Sullivan P. Educational and individual factors associated with positive change in and reaffirmation of medical students’ intention to practice in underserved areas. Acad Med. 2014; 89:1490–1496
36. Blumenthal DS, Jones A, McNeal M. Evaluating a community-based multiprofessional course in community health. Educ Health (Abingdon). 2001; 14:251–255
37. Allan J, Barwick TA, Cashman S, et al. Clinical prevention and population health: Curriculum framework for health professions. Am J Prev Med. 2004; 27:471–476
38. Kaprielian VS, Silberberg M, McDonald MA, et al. Teaching population health: A competency map approach to education. Acad Med. 2013; 88:626–637
39. American Hospital Association. Social determinants of health curriculum for clinicians. https://www.aha.org/physicians/SDOH-Main. Accessed August 6, 2020
40. The George Washington University School of Medicine and Health Sciences. Clinical public health faculty and clinical public health mentors. https://smhs.gwu.edu/academics/md-program/curriculum/clinical-public-health/clinical-public-health-faculty-and-clinical. Accessed August 6, 2020
41. Otto L. Creating an AIDS-free generation: First-year GW medical students present recommendations to improve the HIV Care Continuum during White House event. https://smhs.gwu.edu/news/creating-aids-free-generation. December 22, 2014 Accessed August 6, 2020
42. Otto L. Eliminating childhood asthma in D.C. May 21, 2015. https://smhs.gwu.edu/news/eliminating-childhood-asthma-dc. Accessed August 6, 2020
43. The George Washington University School of Medicine and Health Sciences. Tackling obesity: Medical students propose public health solutions. https://smhs.gwu.edu/news/tackling-obesity-medical-students-propose-public-health-solutions. Published September 6, 2016 Accessed August 6, 2020
44. Trent-Gurbuz CJ. Health care beyond the exam room. https://smhs.gwu.edu/diversity/news/health-care-beyond-exam-room. Published March 29, 2018 Accessed September 24, 2020
45. Carney JK, Maltby HJ, Mackin KA, Maksym ME. Community-academic partnerships: How can communities benefit? Am J Prev Med. 2011; 414 suppl 3S206–S213
46. American Medical Association. Creating a community of innovation. https://www.ama-assn.org/sites/ama-assn.org/files/corp/media-browser/public/about-ama/ace-monograph-interactive_0.pdf. Published 2017 Accessed August 6, 2020
47. White F. Medical students walk in patients’ shoes through new OHSU program. https://news.ohsu.edu/2018/07/30/medical-students-walk-in-patients-shoes-through-new-ohsu-program. Published July 30, 2018 Accessed August 6, 2020
48. Blair J. Patient navigators address barriers to health care. https://medicine.yale.edu/news-article/15965. Published October 11, 2017 Accessed August 6, 2020
49. Georgetown University School of Medicine. Population health scholar track. https://som.georgetown.edu/medicaleducation/longitudinal-academic-track/phst. Accessed August 6, 2020
50. Wennerstrom A, Gibson JW, Krane NK. From classroom to community: The impact of a non-clinical clerkship on fourth-year medical students’ ability to address social determinants of health. Med Sci Educ. 2018; 28:381–387
51. Miller E, Green AR. Student reflections on learning cross-cultural skills through a ‘cultural competence’ OSCE. Med Teach. 2007; 29:e76–e84
52. CLER Evaluation Committee. CLER Pathways to Excellence: Expectations for an Optimal Clinical Learning Environment to Achieve Safe and High-Quality Patient Care, Version 2.0. Chicago, IL: Accreditation Council for Graduate Medical Education, 2019. https://doi.org/10.35425/ACGME.0003. Accessed August 6, 2020
53. Accreditation Council for Graduate Medical Education. ACGME Common Program Requirements (Residency), ACGME-approved focus revision: February 3, 2020. Chicago, IL: Accreditation Council for Graduate Medical Education, 2020. https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2020.pdf. Accessed August 26, 2020
54. Association of American Medical Colleges. New and Emerging Areas in Medicine Series: Quality Improvement and Patient Safety Competencies Across the Learning Continuum. Washington, DC: AAMC, 2019. https://store.aamc.org/quality-improvement-and-patient-safety-competencies-across-the-learning-continuum.html. Accessed August 6, 2020
55. Xierali IM, Maeshiro R, Johnson S, Arceneaux T, Fair MA. Public health and community medicine instruction and physician practice location. Am J Prev Med. 2014; 475 suppl 3S297–S300
56. Robert Graham Center. Med School Mapper. https://www.graham-center.org/rgc/maps-data-tools/interactive/med-school-mapper.html. Accessed September 1, 2020
57. Ko M, Heslin KC, Edelstein RA, Grumbach K. The role of medical education in reducing health care disparities: The first ten years of the UCLA/Drew Medical Education Program. J Gen Intern Med. 2007; 22:625–631
58. Strelnick AH, Swiderski D, Fornari A, et al. The residency program in social medicine of Montefiore Medical Center: 37 years of mission-driven, interdisciplinary training in primary care, population health, and social medicine. Acad Med. 2008; 83:378–389
59. Rabinowitz HK, Diamond JJ, Markham FW, Hazelwood CE. A program to increase the number of family physicians in rural and underserved areas: Impact after 22 years. JAMA. 1999; 281:255–260
60. Wilkins CH, Alberti PM. Shifting academic health centers from a culture of community service to community engagement and integration. Acad Med. 2019; 94:763–767
61. Boyer EL. A Special Report. Scholarship Reconsidered: Priorities of the Professoriate. Princeton, NJ: The Carnegie Foundation for the Advancement of Teaching, 1990. https://www.umces.edu/sites/default/files/al/pdfs/BoyerScholarshipReconsidered.pdf. Accessed August 6, 2020
62. Maurana CA, Wolff M, Beck BJ, Simpson DE. Community-Campus Partnerships for Health. Working with our communities: Moving from service to scholarship in the health professions. Educ Health (Abingdon). 2001; 14:207–220
63. Calleson DC, Jordan C, Seifer SD. Community-engaged scholarship: Is faculty work in communities a true academic enterprise? Acad Med. 2005; 80:317–321
64. Register SJ, King KM. Promotion and tenure: Application of scholarship of teaching and learning, and scholarship of engagement criteria to health professions education. Health Prof Educ. 2018; 4:39–47
65. Sladek E. Higher education’s anchor mission: Measuring place-based engagement. The Democracy Collaborative. https://democracycollaborative.org/higher-ed-anchor-mission. Published October 5, 2017 Accessed September 1, 2020
66. Hacke R, Deane KG. Improving community health by strengthening community investment: Roles for hospitals and health systems. Robert Wood Johnson Foundation. https://www.rwjf.org/en/library/research/2017/03/improving-community-health-by-strengthening-community-investment.html. Published March 15, 2017 Accessed January 12, 2020
67. Rush University Medical Center. Rush’s anchor mission strategy: Leveraging our strength to benefit our community. https://www.rush.edu/about-us/rush-community/rushs-anchor-mission-strategy. Accessed August 6, 2020
68. Kania J, Kramer M. Essentials of social innovation: Collective impact. Stanford Social Innovation Review. https://ssir.org/articles/entry/collective_impact. Published Winter 2011 Accessed August 6, 2020
69. Hanleybrown F, Kania J, Kramer M. Channeling change: Making collective impact work. Stanford Social Innovation Review. https://ssir.org/articles/entry/channeling_change_making_collective_impact_work. Published January 26, 2012 Accessed August 6, 2020
70. Turner S, Merchant K, Kania J, Martin E. Understanding the value of backbone organizations in collective impact: Part 1. Stanford Social Innovation Review. https://ssir.org/articles/entry/understanding_the_value_of_backbone_organizations_in_collective_impact_1. Published July 17, 2012 Accessed August 6, 2020
71. Gonzalo JD, Dekhtyar M, Starr SR, et al. Health systems science curricula in undergraduate medical education: Identifying and defining a potential curricular framework. Acad Med. 2017; 92:123–131
72. Rosenberg J. Health systems take on role as anchor institutions, enhance community development. https://www.ajmc.com/newsroom/health-systems-take-on-role-as-anchor-institutions-enhance-community-development. Published July 25, 2018 Accessed August 6, 2020
73. Parker K. Growing healthcare’s anchor mission. https://community-wealth.org/content/growing-healthcare-s-anchor-mission. Published February 18, 2016 Accessed August 6, 2020
74. Kaiser Permanente. Improving community conditions. https://about.kaiserpermanente.org/community-health/improving-community-conditions. Accessed August 27, 2020
75. Sutphen SM, Cibula DA, Morrow CB, Epling JW, Novick LF. Evaluation of a preventive medicine curriculum: Incorporating a case-based approach. Am J Prev Med. 2003; 24suppl 490–94
76. Oriol NE, Cote PJ, Vavasis AP, et al. Calculating the return on investment of mobile healthcare. BMC Med. 2009; 7:27
77. Song Z, Hill C, Bennet J, Vavasis A, Oriol NE. Mobile clinic in Massachusetts associated with cost savings from lowering blood pressure and emergency department use. Health Aff (Millwood). 2013; 32:36–44
78. Institute of Medicine. Measuring the Impact of Interprofessional Education on Collaborative Practice and Patient Outcomes. Washington, DC: National Academies Press, 2015. https://doi.org/10.17226/21726 Accessed August 30, 2020.
79. Parkhurst M, Preskill H. Learning in action: Evaluating collective impact. Stanford Social Innovation Review. https://ssir.org/articles/entry/evaluating_collective_impact. Published Fall 2014 Accessed August 6, 2020
80. Office of Disease Prevention and Health Promotion. Healthy People 2030 Framework: What is the Healthy People 2030 Framework? https://www.healthypeople.gov/2020/About-Healthy-People/Development-Healthy-People-2030/Framework. Accessed August 6, 2020
81. Daniel H, Bornstein SS, Kane GC. Health and Public Policy Committee of the American College of Physicians. Addressing social determinants to improve patient care and promote health equity: An American College of Physicians Position Paper. Ann Intern Med. 2018; 168:577–578
82. The National Academies of Sciences, Engineering, and Medicine. Integrating Social Care into the Delivery of Health Care: Moving Upstream to Improve the Nation’s Health. Washington, DC: National Academies Press, 2019. https://doi.org/10.17226/25467 Accessed August 30, 2020.
83. DeSalvo KB, Wang YC, Harris A, Auerbach J, Koo D, O’Carroll P. Public Health 3.0: A call to action for public health to meet the challenges of the 21st century. Prev Chronic Dis. 2017; 14:170017
84. University of Wisconsin Population Health Institute. 2020 County Health Rankings State Reports. https://www.countyhealthrankings.org. Accessed August 27, 2020
85. United Health Foundation. America’s health rankings. https://www.americashealthrankings.org. Accessed August 6, 2020
86. Centers for Disease Control and Prevention. CDC WONDER. https://wonder.cdc.gov. Reviewed July 24, 2020 Accessed August 6, 2020
87. Centers for Disease Control and Prevention. Health, United States. https://www.cdc.gov/nchs/hus/index.htm. Reviewed October 30, 2019. Accessed August 6, 2020
88. Mullan F, Chen C, Petterson S, Kolsky G, Spagnola M. The social mission of medical education: Ranking the schools. Ann Intern Med. 2010; 152:804–811
Copyright © 2020 by the Association of American Medical Colleges