Rapid transformation of the U.S. health care system in recent decades has resulted in marked changes in health care delivery and an expansion of the clinician’s focus from individual patients to populations and communities. 1 Coupled with this attention to health care delivery is the increasing recognition that social determinants of health have a larger effect than clinical care on the health of people and populations. 2–4 The national reckoning with racial inequities intersecting with the evolving COVID-19 pandemic further underscores the importance of equipping future clinicians with skills to catalyze systems change while also caring for individuals in context. 5,6 New generations of physicians must be equipped to understand and modify health systems while seamlessly integrating basic and clinical sciences in an effort to achieve equity, the opportunity for all members of society to reach their full potential. 1,6,7 Indeed, multiple global accreditation systems have charged medical educators with addressing these new competencies. 8
Health data in the United States highlight the failure of medical education to improve population outcomes. As of 2016, the United States has a 31% higher disease burden, more frequent hospital admissions for preventable diseases, and poorer rates of amenable mortality than comparable developed countries. 9 Furthermore, inequities in population health persist for communities of color. The COVID-19 pandemic has starkly illuminated these fault lines in U.S. health and health care systems; data show grave inequities for populations of color in outcomes ranging from lower testing and higher positivity rates to increased illness severity requiring intensive care unit intervention and resulting in more fatalities. 5,10–12 These facts call for a transformation in medical education that will enable physicians to understand structural determinants of health and prepare them to meaningfully improve the health of communities. 13,14 Medical education reform has, however, fallen short, as noted in 2013 by Lucey, who proposed new competencies for the 21st-century physician graduate, including expertise in patient experience, population health outcomes, quality, safety, and costs. 15
Recently, the term “health systems science” (HSS) has emerged as a unifying label for these competencies. HSS is defined as “the study of how health care is delivered, how health care professionals work together to deliver care, and how the health system can improve patient care and health delivery.” 16 There are many reasons for the lack of robust integration of these competencies into medical education curricula to date. In addition to the classic problem of limited time in a crowded curriculum, HSS is hampered by its own lack of a cohesive data-driven developmental framework, a paucity of expertise, too few resources for teaching, no universally accepted competencies, the absence of an ideal pedagogy, and a dearth of assessment and evaluation tools. 17 Further, learners and faculty alike have resisted the importance of HSS. 18 Whether perceived as a needed modernization of the clinical sciences or as a “third pillar” of medical education, HSS is vital, and the depth and breadth of this content are substantial. 17,19,20
An Elaborated Medical Education Framework
Successful HSS learning extends the physician’s professional identity such that physicians are not only expert in treating individual episodes of illness, but also prepared to champion prevention, to lead health system transformation through quality improvement (QI), and to advocate for health equity in the community. 7 A major challenge of integrating HSS into medical curricula is that doing so requires recognition that the learning environment extends beyond the confines of the medical school and hospital or clinical affiliates. Including HSS in curricula is, thus, a more complex endeavor than previous reform efforts. Figure 1 shows the relationship of HSS to the traditional biomedical science curricula and demonstrates that considerations of health care delivery and population health require expanded physical and conceptual space. HSS does not replace, but rather builds upon and extends beyond the traditional scope of medicine. To build this elaborated medical education framework, educators must harness resources that exist outside the walls of the medical school and its traditional affiliated clinics.
;)
Figure 1: Elaborated educational framework. The figure details unique educational opportunities provided by various learning environments for a health systems science curriculum.
Such a complex endeavor calls for a systematic approach. In this article, we apply Kern’s 6-step approach for curriculum development (CD) in medical education both to support the successful integration of HSS in undergraduate medical education (UME) and to begin building the much-needed evidence base to support its dissemination. 21 To illustrate the application of these steps, we have shared 2 examples from Case Western Reserve University School of Medicine: (1) a QI module from the longitudinal Systems and Scholarship course that addresses health care delivery themes (Figure 2) and (2) the introduction of the structural determinants of health in a year 1 block course, Block One: Becoming a Doctor, that addresses population and community health, determinants of health, health systems, patient-centered care, and health equity (Figure 3).
Figure 2: Kern’s 6-step approach to curriculum development
21 applied to a module for teaching quality improvement (QI), which is related to health care delivery, an element of health systems science (HSS). Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; AAMC, Association of American Medical Colleges; IRB, institutional review board; QIKAT-R, Quality Improvement Knowledge Application Tool-revised
52; QIPAT-7, Quality Improvement Proposal Assessment Tool.
54Kern’s 6-step approach to curriculum development
21 applied to a course on population health and health equity, which are elements of health systems science. Abbreviation: NBME, National Board of Medical Examiners.
The 2006 Western Reserve2 curriculum reform launched Block One: Becoming a Doctor to provide medical students with a concentrated focus on population medicine starting on Day 1 of medical school. Further HSS enhancements were added in 2013. 22 In 2018, Block One was incorporated as the foundational course for a 4-year, integrated HSS curriculum. In 2019, a new, longitudinal HSS preclerkship curriculum, Systems and Scholarship, which included Block One, was implemented. Systems and Scholarship covers research skills, QI, population health, and community engagement.
In addition to these 2 examples, Supplemental Digital Appendix 1 at https://links.lww.com/ACADMED/B106 presents the key differences in CD processes between traditional biomedical science and HSS curricula, outlines resources and recommendations for a health care delivery curriculum and a population health/health equity curriculum, and provides additional resources and examples for CD.
Step 1: Problem identification and general needs assessment
The first step in the CD process is to create a succinct problem statement by completing a rigorous analysis of the current state of medical practice and medical education. The goal is to create an evidence-based argument for the need to create a new curriculum and to establish generalizability beyond the local institution. After collecting information from the literature and databases, the CD team should identify gaps between current practice and ideal practice in health care and medical education. Traditionally, the gaps have highlighted health providers’ deficits in knowledge, skills, and attitudes (KSA).
For HSS, we recommend that educators use systems thinking in analyzing the educational gaps to go beyond the individual health provider and understand the systems and determinants of health that affect patient outcomes. 23 Abundant, if not overwhelming, resources are available for the background information needed to describe the state of a health problem. The National Academies of Sciences, Engineering, and Medicine have published numerous referenced reports on the various domains within HSS (e.g., population health, health care quality, health and health care disparities). Other resources are suggested in Supplemental Digital Appendix 1 at https://links.lww.com/ACADMED/B106. A challenge in working with this information is that the analysis invariably identifies societal, behavioral, policy, and environmental factors not traditionally prioritized in medical CD. 24,25
In the backward design of the 6-step process, the CD team should next identify the KSA, as well as the competencies, that health professionals should possess to address the identified gaps in health systems and population health. The published literature on these competencies indicates that multiple conceptual frameworks are relevant. 15,16,20,26–28 A 2016 study of published recommendations for patient safety and QI education alone reported 22 papers describing, collectively, 59 competencies. 29 Researchers and educators are just now constructing the evidence base for these models, so ascertaining the validity of one framework over another is difficult; however, some consensus work, aligning competencies to training points along the medical education continuum, exists. 30 Completing Step 1 requires awareness of these expert-derived models, their underlying evidence, and how they fit with existing program competencies.
Lastly, Step 1 includes conducting a survey of best practices in medical education that address the health problem of interest. This step yields not only program descriptions, but often validated instruments that can be adopted for assessment and evaluation (Step 6). In addition to searching the literature, exploring both the Curriculum Inventory maintained by the Association of American Medical Colleges (AAMC) and MedEdPORTAL’s Diversity, Inclusion and Health Equity Collection may yield helpful information.
Step 2: Targeted needs assessment
Step 2 requires the CD team to analyze the local learning environment and the specific learners for whom the curriculum will be implemented so as to understand the local factors that will impede or facilitate addressing the gaps identified in Step 1. Barriers for integrating HSS likely include learners’ resistant attitudes to the content. Entering medical students expect deep learning in the biomedical sciences so they can achieve competence in patient care; thus, they may perceive HSS as an irrelevant distraction. 18,31 Although learners may have had experiences in health care systems in premedical education, few have borne the responsibility of patient outcomes, employed systems thinking, or reflected on systems issues before entering medical school. Most medical students have strong backgrounds in STEM (science, technology, engineering, and medicine), are used to applying reductionist approaches, and have often relied on themselves in problem situations. Many of these students may experience cognitive dissonance when they encounter HSS curricula, which typically encourage systems thinking and the appreciation of diverse teams to improve health care delivery. Furthermore, population health, public health, and health equity content can be perceived as politically charged and/or outside the scope of medicine, which can make it particularly challenging for students. One successful strategy is to include learners at this early stage of the CD process. Students are particularly effective in revealing learner perceptions and attitudes, facilitating focus groups, designing and implementing surveys, and suggesting learner-centered approaches.
Another potential challenge to integrating an HSS curriculum locally involves extending learning beyond the medical school and hospital affiliates. Many stakeholders (health systems leaders, local government officials, and community leaders) have a vested interest in health outcomes and in medical education and its contribution to health care workforce development. Early engagement of these stakeholders to understand the local nature of the problem builds support for the curriculum, leads to the identification of new educational partners, and underscores the CD team’s commitment to community. Association with an academic medical center or other institution with a stated mission to improve the health of the local community is helpful for earning the trust of local leaders.
One more potential challenge to local integration of an HSS curriculum is the availability (or lack thereof) of faculty with expertise in health care delivery, population health, and health equity. 23 Most medical schools have departments of biostatistics, of epidemiology, and sometimes, of community health, but expertise in public health, leadership, health policy, and advocacy may be available only outside the academic medical center or university. Similarly, interprofessional education may require new affiliations with health professional schools not embedded in the home university. Clinicians with training and experience in patient safety and QI are available in most health systems and can be engaged to teach and mentor students. Community engagement is vital but can be challenging, particularly for large classes of students, and additional faculty development may be required to prepare community sites for student learning.
Step 3: Goals and objectives
Having gathered the information about what is needed in Steps 1 and 2, the next step requires a refined goal statement and specific measurable learning objectives. The goal statement generally describes the content and the purpose of the curriculum. The learning objectives clearly communicate the specific knowledge and skills learners will achieve, and they form the basis of learner assessment and curriculum evaluation (Step 6). The Kern approach advises using the question, “Who will do how much of what by when?” to facilitate writing a specific measurable learning objective. 21 Importantly, Step 3 requires recognizing that HSS goals may arise from systems’ gaps and will require working backward to identify what learner competencies are needed to help address these gaps. This process is different from traditional medical education in which competencies are more learner-centric and not always explicitly connected to systems or community needs. As stated above, there is a lack of consensus on what HSS competencies are achievable in UME. One approach is to use the 6 core domains of HSS (see Supplemental Digital Appendix 1 at https://links.lww.com/ACADMED/B106) to describe competency goals for students experiencing an HSS curriculum. 16 These competencies may then be dissected into the KSA that collectively support competency. Finally, the CD team should develop learning objectives for the curricular events that both promote learner achievement and, if possible, align with identified system needs.
Step 4: Educational strategies
Step 4 focuses on describing the educational methods that will be used to deliver the content of the curriculum. The CD team will have already detailed the content, the “How much of what,” by writing specific measurable and achievable learning objectives in Step 3. The choice of educational methods to impart this content is critical in HSS since the method directly addresses a significant barrier: student attitudes to learning HSS. For instance, students who have previous experience applying the scientific method in laboratory research may find continuous QI processes less stringent. Exposure to QI that results in improved systems, community, or patient outcomes may change their attitudes (Figure 2).
The opportunity to contribute to authentic clinical work and build longitudinal relationships with patients and populations is particularly effective in changing student attitudes toward health systems, population health, and health equity content. Initial work in this area 32 has led many programs to develop interactive learning experiences in which students’ participation adds value to the system. Examples include embedding students in practices to serve as patient navigators or health coaches. 33,34 A 2018 multi-institutional survey exploring medical students’ attitudes found that students were most eager to participate in roles similar to those filled by traditional physicians and to make systems improvements. 35 Activities such as shadowing, interacting with patients electronically, or completing repetitive tasks were least likely to engage students. 35,36
The power of authentic community engagement, especially with vulnerable populations, can be transformative. Community-based service learning has been broadly implemented for decades. Reviewing the past 20 years, researchers concluded that service learning generally falls into 3 types of programming: (1) health behavior interventions in communities and schools, (2) training in community health clinics, and (3) training through social justice and philanthropic endeavors. 36Critical service learning adds a social justice orientation to the experience; students are charged with addressing or responding to injustice in the communities they serve either through advocacy or by initiating or completing concrete projects. 37,38 Whether instituting patient navigation or other community-based learning experiences, significant preparation is needed to develop trusting relationships with community partners and to engage them as valued contributors to the educational process. True community-engaged medical education aligns the student learning objectives with the community’s health care needs and requires ongoing discussion and deliberation between the academic and community partners. 39,40 Although building such a longitudinal, bidirectional relationship may appear overwhelming to the CD team, the alignment and purpose-driven education are truly rewarding in the long term. The key for HSS learning, regardless of venue, is that students have appropriately supervised, active roles coupled with feedback, discussion, and reflection (Figures 2 and 3).
Addressing sensitive topics, such as hidden biases, structural and social determinants of health, health inequities, and health policy, requires methods that challenge values, encourage students’ personal and professional development, and provide an environment of psychological safety amenable to growth. 41–43 A common approach is using the humanities (e.g., short stories, poetry, narratives, books, film) to open these conversations. 13,44
While fundamental skills, such as engaging in continuous QI, could be contained in a single course (Figure 2), other domains, such as change agency, leadership, and advocacy, are developmental in nature and imply a longitudinal or spiral curriculum, ideally in clinical and community settings (Figure 1).
Step 5: Implementation
This step entails several tasks necessary for producing a high-quality educational intervention while setting the stage for its ongoing continuous improvement. These tasks include obtaining support, identifying resources, anticipating barriers, planning for administration, and introducing the curriculum. 21 Obtaining support may begin with engaging key stakeholders (e.g., deans, health systems leaders, community leaders) and highlighting the shared missions of the curriculum and relevant institutions. Engaged leaders can champion the curriculum and identify currently available resources (such as patient safety and quality officers) and potential resources (such as funding opportunities). Educators, including nontraditional faculty such as community members and leaders, should be identified and prepared for teaching HSS. The broad nature of HSS and the different types of expertise needed to deliver this curriculum set the stage for an inclusive faculty development approach and require shedding traditional hierarchies associated with the term “faculty.” Resources to support the inclusion of other expert teachers should be identified, and efforts to engage these teachers should be prioritized. Factors that might impede and facilitate the inclusion of nontraditional teachers may exist not only within students and current faculty but also in the health system and community contexts. For example, while community faculty provide immense expertise in how to manage structural barriers to health care, this strength may be offset by academic centers not recognizing nontraditional faculty or by concerns of reduced clinical productivity when students are placed in community practices.
Any such barriers and facilitators should be explored and anticipated before launching the curriculum. Unlike the biomedical disciplines, HSS rarely has one departmental home, meaning that a new administrative structure will be necessary to support the needs of the curriculum. Notably, creating the structures to support interprofessional endeavors can be especially challenging. Given all these novel considerations, piloting the curriculum with volunteer learners and a robust evaluation plan is ideal. Facilitating rich qualitative feedback from all stakeholders and using a continuous QI approach are critical to implementing a new curriculum within this extended educational environment.
Educators can adapt frameworks from implementation science 45,46 to plan for and work toward successful outcomes such as high rates of acceptability and adoption; low costs; and high feasibility, fidelity, and sustainability. Considering learner and community outcomes (e.g., satisfaction, engagement or contributions, KSA) is also necessary to establish effectiveness and support the generalizability of the curriculum.
Step 6: Evaluation and assessment
Learner assessment for the HSS curriculum should be embedded in the specific measurable learning objectives detailed in Step 3. Echoing these objectives, the evaluation should answer the question “How much was learned by the targeted learners by completion of the curriculum?” In addition, the evaluation should measure the curriculum with respect to relevant implementation outcomes (e.g., satisfaction, feasibility, acceptability; see Step 5).
The Kirkpatrick framework for program evaluation maps learning objectives to 4 levels: (1) learner satisfaction, (2) changes in KSA, (3) changes in behavior, and (4) program outcomes in context. 47,48 Since these are innovative curricula, targeting all levels in the evaluation plan is appropriate; learner satisfaction alone is insufficient. Level 2 outcomes, specifically changes in knowledge, can be measured with multiple choice tests such as the new National Board of Medical Examiners HSS subject examination. 49 Higher-order (Level 3) objectives, such as demonstrating structural competency and meeting behavioral science objectives, require validated instruments. 43,50 Relevant clinical skills and behaviors such as cultural humility and patient safety, as well as team skills, can be assessed in simulation exercises or through examining the practice environment. 51 In addition, instruments such as the Quality Improvement Knowledge Application Tool-revised (QIKAT-R) 52 and specific rubrics that assess individual QI projects are available to help educators assess whether learners have met the stated QI learning objectives. 53,54
Since HSS curricula are often longitudinal and most HSS competency domains relate to professional identity, they lend themselves well to both reflective writing and a longitudinal learning portfolio wherein learners document their personal growth and how they have applied HSS in their practice environments. 55 Mapping the HSS curriculum to the AAMC or Accreditation Council for Graduate Medical Education competencies is a useful step in designing the portfolio structure. 56–59 Competencies such as Systems-Based Practice, Practice-Based Learning Improvement, and Interpersonal and Communication Skills typically map to most HSS curricular learning goals.
Level 4 on the Kirkpatrick model of learner evaluation is change in practice and benefits to patients. The increasing interest in understanding the effect of learners on local health systems and communities (a Level 4 outcome) is relevant to HSS. As part of Step 2, the targeted needs assessment, educators can work with health systems and community partners to understand what patient and population outcomes they value and how these outcomes can be tracked to measure the effect of HSS activities. Student-run clinics with systems that monitor successful follow-up visits, vaccination rates, and health system cost savings are evaluating programs at Level 4. 60 Whether a school’s entire curriculum has the effect envisioned may be more challenging to evaluate; nonetheless, the AAMC Mission Management Tool benchmarks U.S. and Canadian medical schools’ social mission performances. 61 Several global examples illustrate how to measure the social accountability of medical schools. 62–64 Linking such measures to learner activities not only ensures learner professional development but also demonstrates the added value of including even early learners at the clinical and community level.
Conclusion
Incorporating HSS into medical education poses unique challenges, including working with an incomplete understanding of the ideal timing and pedagogy for conveying information on health equity, population health, and health care delivery. Additionally, as of yet, little is known about the effect of this teaching on population outcomes—which are largely determined by strong societal and economic factors. The recent focus on racial inequities, coupled with the global COVID-19 pandemic, underscores the timely imperative to implement HSS across the medical education continuum. HSS curricula uniquely provide opportunities to expand the clinical learning environment beyond the academic medical center walls and to align education with efforts aimed at effecting more equitable communities and clinical outcomes. In addition to developing broader competencies in learners (e.g., systems thinking, team skills, change agency), HSS curricula expand the definition of faculty to include nontraditional instructors, and they widen the scope of influence that clinical care and advocacy can have beyond the individual patient to include the larger community. Using a systematic approach, CD teams can develop, implement, and evaluate an HSS curriculum and leverage the expanded educational environment and the potential opportunities inherent in it. Any resultant HSS curriculum facilitates education, develops competencies previously siloed or left out of medical education, builds much-needed evidence for the curriculum, transforms the complex systems in which we live, and, ultimately, improves the health of our communities.
Acknowledgments:
The authors wish to thank the American Medical Association Accelerating Change in Medical Education Consortium, which has supported the authors’ institutional innovations, facilitated national discussions among like-minded colleagues, and developed tools for curriculum development in health systems science.
References
1. Berwick DM, Nolan TW, Whittington J. The triple aim: Care, health, and cost. Health Aff (Millwood). 2008;27:759–769.
2. Braveman P, Egerter S, Williams DR. The social determinants of health: Coming of age. Annu Rev Public Health. 2011;32:381–398.
3. Adler NE, Cutler DM, Fielding JE, et al. Addressing social determinants of health and health disparities: A vital direction for health and health care. NAM Perspectives. Discussion Paper. 2016Washington, DC: National Academy of Medicine;
https://doi.org/10.31478/201609t. Accessed April 22, 2021.
4. Daniel H, Bornstein SS, Kane GC; Health and Public Policy Committee of the American College of Physicians.. Addressing social determinants to improve patient care and promote health equity: An American College of Physicians position paper. Ann Intern Med. 2018;168:577–578.
5. Chen JT, Krieger N. Revealing the unequal burden of COVID-19 by income, race/ethnicity, and household crowding: US county vs ZIP code analyses. Harvard Center for Population and Development Studies Working Paper Series.
https://www.hsph.harvard.edu/population-development/research/working-papers/harvard-pop-center-working-paper-series. Published April 21, 2020 Accessed April 22, 2021.
6. Quinn A, Laws M. Addressing community needs and preparing for the secondary impacts of Covid-19 [published online ahead of print June 25, 2020]. NEJM Catal Innov Care Deliv. doi:10.1056/CAT.20.0186
7. Berwick DM, Finkelstein JA. Preparing medical students for the continual improvement of health and health care: Abraham Flexner and the new “public interest.” Acad Med. 2010;85(9 suppl):S56–S65.
8. Boelen C, Blouin D, Gibbs T, Woollard R. Accrediting excellence for a medical school’s impact on population health. Educ Health (Abingdon). 2019;32:41–48.
9. GBD 2016 Healthcare Access and Quality Collaborators. Measuring performance on the Healthcare Access and Quality Index for 195 countries and territories and selected subnational locations: A systematic analysis from the Global Burden of Disease Study 2016. Lancet. 2018;391:2236–2271.
10. Lieberman-Cribbin W, Tuminello S, Flores RM, Taioli E. Disparities in COVID-19 testing and positivity in New York City. Am J Prev Med. 2020;59:326–332.
11. Raifman MA, Raifman JR. Disparities in the population at risk of severe illness from COVID-19 by race/ethnicity and income. Am J Prev Med. 2020;59:137–139.
12. Maani N, Galea S. COVID-19 and underinvestment in the health of the US population. Milbank Q. 2020;98:239–249.
13. Wear D, Zarconi J, Aultman JM, Chyatte MR, Kumagai AK. Remembering Freddie Gray: Medical education for social justice. Acad Med. 2017;92:312–317.
14. Bailey ZD, Krieger N, Agenor M, Graves J, Linos N, Bassett MT. Structural racism and health inequities in the USA: Evidence and interventions. Lancet. 2017;389:1453–1463.
15. Lucey CR. Medical education: Part of the problem and part of the solution. JAMA Intern Med. 2013;173:1639–1643.
16. Skochelak SE, Hammoud MM, Lomis KD, et al, eds. Health Systems Science. 20202nd ed. Philadelphia, PA: Elsevier; 1.
17. Gonzalo JD, Caverzagie KJ, Hawkins RE, Lawson L, Wolpaw DR, Chang A. Concerns and responses for integrating health systems science into medical education. Acad Med. 2018;93:843–849.
18. Gonzalo JD, Ogrinc G. Health systems science: The “broccoli” of undergraduate medical education. Acad Med. 2019;94:1425–1432.
19. Gonzalo JD, Haidet P, Papp KK, et al. Educating for the 21st-century health care system: An interdependent framework of basic, clinical, and systems sciences. Acad Med. 2017;92:35–39.
20. Gonzalo JD, Chang A, Dekhtyar M, Starr SR, Holmboe E, Wolpaw DR. Health systems science in medical education: Unifying the components to catalyze transformation. Acad Med. 2020;95:1362–1372.
21. Thomas PA, Kern DE, Hughes MT, Chen BY, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2016.3rd ed. Baltimore, MD: Johns Hopkins University Press;
22. Ornt DB, Aron DC, King NB, et al. Population medicine in a curricular revision at Case Western Reserve. Acad Med. 2008;83:327–331.
23. Gonzalo JD, Ahluwalia A, Hamilton M, Wolf H, Wolpaw DR, Thompson BM. Aligning education with health care transformation: Identifying a shared mental model of “new” faculty competencies for academic faculty. Acad Med. 2018;93:256–264.
24. Westerhaus M, Finnegan A, Haidar M, Kleinman A, Mukherjee J, Farmer P. The necessity of social medicine in medical education. Acad Med. 2015;90:565–568.
25. Holmes SM, Hansen H, Jenks A, et al. Misdiagnosis, mistreatment, and harm—When medical care ignores social forces. N Engl J Med. 2020;382:1083–1086.
26. Interprofessional Education Collaborative.Core Competencies for Interprofessional Collaborative Practice: 2016 update. 2016Washington, DC: Interprofessional Education Collaborative;
https://hsc.unm.edu/ipe/resources/ipec-2016-core-competencies.pdf. Accessed April 22, 2021.
27. Guralnick S, Ludwig S, Englander R. Domain of competence: Systems-based practice. Acad Pediatr. 2014;14(suppl 2):S70–S79.
28. Frank JR, Snell L, Sherbino J, eds. CanMEDS 2015 Physician Competency Framework. 2015Ottawa, ON, Canada: Royal College of Physicians and Surgeons of Canada;
http://canmeds.royalcollege.ca/uploads/en/framework/CanMEDS%202015%20Framework_EN_Reduced.pdf. Accessed April 22, 2021.
29. Moran KM, Harris IB, Valenta AL. Competencies for patient safety and quality improvement: A synthesis of recommendations in influential position papers. Jt Comm J Qual Patient Saf. 2016;42:162–169.
30. Kruszewski BD, Spell NO III. A consensus approach to identify tiered competencies in quality improvement and patient safety. J Grad Med Educ. 2018;10:646–650.
31. Gonzalo JD, Davis C, Thompson BM, Haidet P. Unpacking medical students’ mixed engagement in health systems science education. Teach Learn Med. 2020;32:250–258.
32. O’Connell MT, Rivo ML, Mechaber AJ, Weiss BA. A curriculum in systems-based care: Experiential learning changes in student knowledge and attitudes. Fam Med. 2004;36(suppl):S99104.
33. Curry RH. Medical students as health coaches, and more: Adding value to both education and patient care. Isr J Health Policy Res. 2017;6:65.
34. Gonzalo JD, Graaf D, Ahluwalia A, Wolpaw DR, Thompson BM. A practical guide for implementing and maintaining value-added clinical systems learning roles for medical students using a diffusion of innovations framework. Adv Health Sci Educ Theory Pract. 2018;23:699–720.
35. Leep Hunderfund AN, Starr SR, Dyrbye LN, et al. Value-added activities in medical education: A multisite survey of first- and second-year medical students’ perceptions and factors influencing their potential engagement. Acad Med. 2018;93:1560–1568.
36. Stewart T, Wubbena ZC. A systematic review of service-learning in medical education: 1998-2012. Teach Learn Med. 2015;27:115–122.
37. Mitchell TD. Traditional vs. critical service learning: Engaging the literature to differentiate two models. Mich J Community Serv Learn. 2008;14:50–65.
38. Gillis A, Mac Lellan MA. Critical service learning in community health nursing: Enhancing access to cardiac health screening. Int J Nurs Educ Scholarsh. 2013.
https://doi.org/10.1515/ijnes-2012-0031.
39. Strasser R, Worley P, Cristobal F, et al. Putting communities in the driver’s seat: The realities of community-engaged medical education. Acad Med. 2015;90:1466–1470.
40. Wilkins CH, Alberti PM. Shifting academic health centers from a culture of community service to community engagement and integration. Acad Med. 2019;94:763–767.
41. Petty J, Metzl JM, Keeys MR. Developing and evaluating an innovative structural competency curriculum for pre-health students. J Med Humanit. 2017;38:459–471.
42. Donald CA, DasGupta S, Metzl JM, Eckstrand KL. Queer frontiers in medicine: A structural competency approach. Acad Med. 2017;92:345–350.
43. Metzl JM, Petty J, Olowojoba OV. Using a structural competency framework to teach structural racism in pre-health education. Soc Sci Med. 2018;199:189–201.
44. Buttress A, German D, Holtgrave D, Sherman SG. The Wire and urban health education. J Urban Health. 2013;90:359–368.
45. Albers B, Pattuwage L. Implementation in Education: Findings From a Scoping Review. 2017.Victoria, TX: Centre for Evidence and Implementation;
46. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38:65–76.
47. Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE guide no. 67. Med Teach. 2012;34:e288–e299.
48. Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s Four Levels of Training Evaluation. 2016.Alexandria, VA: Association for Talent Development Press;
49. American Medical Association in collaboration with National Board of Medical ExaminersAt-A-Glance: Health Systems Science (HSS) Examination.
https://nbme.org/sites/default/files/2020-01/HSS-Exam-At-A-Glance.pdf. Accessed April 22, 2021.
50. Carney PA, Palmer RT, Fuqua Miller M, et al. Tools to assess behavioral and social science competencies in medical education: A systematic review. Acad Med. 2016;91:730–742.
51. Reid J, Stone K, Brown J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation. 2012;83:879–886.
52. Singh MK, Ogrinc G, Cox KR, et al. The Quality Improvement Knowledge Application Tool Revised (QIKAT-R). Acad Med. 2014;89:1386–1391.
53. Dumenco L, Monteiro K, George P, Rougas S, Dollase R. Outcomes of a longitudinal quality improvement and patient safety preclerkship curriculum. Acad Med. 2019;94:1980–1987.
54. Leenstra JL, Beckman TJ, Reed DA, et al. Validation of a method for assessing resident physicians’ quality improvement proposals. J Gen Intern Med. 2007;22:1330–1334.
55. Van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE guide no. 45. Med Teach. 2009;31:790–801.
56. Englander R, Cameron T, Ballard AJ, Dodge J, Bull J, Aschenbrener CA. Toward a common taxonomy of competency domains for the health professions and competencies for physicians. Acad Med. 2013;88:1088–1094.
57. Eckstrand KL, Potter J, Bayer CR, Englander R. Giving context to the physician competency reference set: Adapting to the needs of diverse populations. Acad Med. 2016;91:930–935.
58. Swing SR. The ACGME Outcome Project: Retrospective and prospective. Med Teach. 2007;29:648–654.
59. Carraccio C, Englander R. Evaluating competence using a portfolio: A literature review and web-based application to the ACGME competencies. Teach Learn Med. 2004;16:381–387.
60. Arenas DJ, Lett LA, Klusaritz H, Teitelman AM. A Monte Carlo simulation approach for estimating the health and economic impact of interventions provided at a student-run clinic. PLoS One. 2017;12:e0189718.
61. Kirch DG, Prescott JE. From rankings to mission. Acad Med. 2013;88:1064–1066.
62. Rourke J. Social accountability: A framework for medical schools to improve the health of the populations they serve. Acad Med. 2018;93:1120–1124.
63. Reeve C, Woolley T, Ross SJ, et al. The impact of socially-accountable health professional education: A systematic review of the literature. Med Teach. 2017;39:67–73.
64. Woolley T, Clithero-Eridon A, Elsanousi S, Othman AB. Does a socially-accountable curriculum transform health professional students into competent, work-ready graduates? A cross-sectional study of three medical schools across three countries. Med Teach. 2019;41:1427–1433.
