Secondary Logo

Journal Logo

Health Systems Science Curricula in Undergraduate Medical Education

Identifying and Defining a Potential Curricular Framework

Gonzalo, Jed D. MD, MSc; Dekhtyar, Michael; Starr, Stephanie R. MD; Borkan, Jeffrey MD, PhD; Brunett, Patrick MD; Fancher, Tonya MD, MPH; Green, Jennifer MD, MPH; Grethlein, Sara Jo MD; Lai, Cindy MD; Lawson, Luan MD, MAEd; Monrad, Seetha MD; O’Sullivan, Patricia EdD; Schwartz, Mark D. MD; Skochelak, Susan MD, MPH

doi: 10.1097/ACM.0000000000001177
Research Reports
Free
SDC

Purpose The authors performed a review of 30 Accelerating Change in Medical Education full grant submissions and an analysis of the health systems science (HSS)-related curricula at the 11 grant recipient schools to develop a potential comprehensive HSS curricular framework with domains and subcategories.

Method In phase 1, to identify domains, grant submissions were analyzed and coded using constant comparative analysis. In phase 2, a detailed review of all existing and planned syllabi and curriculum documents at the grantee schools was performed, and content in the core curricular domains was coded into subcategories. The lead investigators reviewed and discussed drafts of the categorization scheme, collapsed and combined domains and subcategories, and resolved disagreements via group discussion.

Results Analysis yielded three types of domains: core, cross-cutting, and linking. Core domains included health care structures and processes; health care policy, economics, and management; clinical informatics and health information technology; population and public health; value-based care; and health system improvement. Cross-cutting domains included leadership and change agency; teamwork and interprofessional education; evidence-based medicine and practice; professionalism and ethics; and scholarship. One linking domain was identified: systems thinking.

Conclusions This broad framework aims to build on the traditional definition of systems-based practice and highlight the need for medical and other health professions schools to better align education programs with the anticipated needs of the systems in which students will practice. HSS will require a critical investigation into existing curricula to determine the most efficient methods for integration with the basic and clinical sciences.

J.D. Gonzalo is assistant professor of medicine and public health sciences and associate dean for health systems education, Penn State College of Medicine, Hershey, Pennsylvania.

M. Dekhtyar is senior research assistant, Medical Education Outcomes, American Medical Association, Chicago, Illinois.

S.R. Starr is assistant professor of pediatric and adolescent medicine and director of science of health care delivery education, Mayo Medical School, Mayo Clinic College of Medicine, Rochester, Minnesota.

J. Borkan is chair and professor of family medicine and assistant dean for primary care–population health program planning, Alpert Medical School of Brown University, Providence, Rhode Island.

P. Brunett is clinical professor, Department of Emergency Medicine, and associate dean for graduate medical education, Oregon Health & Science University School of Medicine, Portland, Oregon.

T. Fancher is associate professor, Department of Internal Medicine, University of California Davis School of Medicine, Sacramento, California.

J. Green is assistant professor of medicine and pediatrics, Vanderbilt University School of Medicine, Nashville, Tennessee.

S.J. Grethlein is professor of clinical medicine, Department of Medicine, Indiana University School of Medicine, Indianapolis, Indiana.

C. Lai is professor of medicine, University of California San Francisco School of Medicine, San Francisco, California.

L. Lawson is assistant dean for curriculum, assessment, and clinical academic affairs and assistant professor of emergency medicine, Brody School of Medicine at East Carolina University, Greenville, North Carolina.

S. Monrad is clinical assistant professor, Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, Michigan.

P. O’Sullivan is professor of medicine and director of research and development in medical education, University of California San Francisco School of Medicine, San Francisco, California.

M.D. Schwartz is professor of population health and medicine and vice chair for education and faculty affairs, Department of Population Health, New York University School of Medicine, New York, New York.

S. Skochelak is group vice president of medical education, American Medical Association, Chicago, Illinois.

Funding/Support: This project was performed with financial support from the American Medical Association as part of the Accelerating Change in Medical Education Initiative.

Other disclosures: T. Fancher is also funded by the Health Resources & Services Administration. To our knowledge, no conflict of interest, financial or other, exists for any author. The views expressed in this paper reflect the views of the authors and do not necessarily represent the views of the American Medical Association or other participants in this initiative.

Ethical approval: The University of Illinois at Chicago, the central institutional review board for the Accelerating Change in Medical Education initiative, determined that this study did not necessitate oversight as the research activity did not involve human subjects.

Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A341.

Correspondence should be addressed to Jed D. Gonzalo, Division of General Internal Medicine, Penn State Hershey Medical Center, HO34, 500 University Dr., Hershey, PA 17033; telephone: (717) 531-8161; e-mail: jgonzalo@hmc.psu.edu.

Health care delivery in the United States is fragmented, costly, and inefficient.1–5 Faced with increasing accountability for quality of care, payment reform, and the need to transform delivery processes, health care systems are transforming practice environments.6 New care delivery models focus increasingly on interprofessional care teams to achieve the goals of the Institute for Healthcare Improvement’s Triple Aim—to improve the patient experience and population health while reducing costs.7 These changes bring new role expectations for physicians and, therefore, new expectations for medical education programs training the next generation of collaboratively effective, systems-based physicians.8–13

To effectively prepare medical students to thrive in a 21st-century health care system, undergraduate medical education (UME) and graduate medical education (GME) programs must address trainees’ knowledge, attitudes, and skills in systems-based practice (SBP).11,14–19 The medical education community is working to address the transition from UME to GME in this facet of physician development via the Association of American Medical Colleges’ Core Entrustable Professional Activities for Entering Residency (CEPAERs), which include the identification of system failures and contributing to a culture of safety and improvement, and the Accreditation Council for Graduate Medical Education’s (ACGME’s) inclusion of SBP as one of its six core competency domains.20,21 U.S. medical students report inadequate education in health care systems, economics, managed care, and practice management.22,23 Additionally, the literature regarding SBP-related curricula is fragmented and limited despite unified calls for expanded curricula in content areas that are broader than the current ACGME definitions of SBP and practice-based learning (PBL).20 Several independent sources identify critical pieces of this systems-related content, including care transitions, teamwork and coordination, patient safety, policy, cost-conscious care, and public health.6,18,21,24–27

Aligning expanded SBP and PBL content with the CEPAERs and current gaps in curricular content will help define the content of health systems science (HSS).20 Viewed by medical educators as a potential “third science” that complements the basic and clinical sciences, HSS can be considered as the methods and principles for improving quality, outcomes, and costs of health care delivery for patients and populations within medical care systems.13 An inclusive HSS curricular framework describing relevant domains does not currently exist in the literature but is necessary to ensure that this critical content does not continue to be marginalized.28 Closing the educational gap noted above requires identifying a clear HSS framework that can lead to the development of educational methods and evaluation measures to ensure that trainees are adequately prepared for medical practice.

In 2013, the American Medical Association solicited grant proposals for their Accelerating Change in Medical Education (ACE) initiative.29 A primary goal for this program was to enhance the systems thinking competencies of medical school graduates. Leveraging the resources of the ACE initiative, we performed a detailed review of 30 full grant submissions and an in-depth analysis of the HSS-related curricula at the 11 ACE grant recipient schools to develop one potential comprehensive HSS curricular framework with domains and subcategories. This report describes the development of a comprehensive HSS curricular framework and its application to build a comprehensive list of curricular content.

Back to Top | Article Outline

Method

Design

This study was done in two phases: (1) a content analysis of proposed HSS innovations to develop a curricular framework (identifying domains) and (2) an application of the framework to build a list of curricular content (identifying subcategories). We chose this approach because we could not find a source identifying new and cutting-edge topics in HSS and systems thinking applicable to UME or GME. The lead investigators (J.D.G., M.D.) had expertise in designing and implementing an HSS curriculum and/or the research methods used in this work.13,30,31 The University of Illinois at Chicago, the central institutional review board for the ACE initiative, determined that this study did not necessitate oversight because the research activity did not involve human subjects.

Back to Top | Article Outline

Study setting and approach

In 2013, the American Medical Association received 30 full grant submissions (preceded by 119 letters of intent) from medical schools, and funded 11 schools as part of the ACE initiative.29 The 11 schools to receive funding were the Warren Alpert Medical School of Brown University; Brody School of Medicine at East Carolina University; University of California, San Francisco, School of Medicine; University of California, Davis, School of Medicine; Indiana University School of Medicine; Mayo Medical School; University of Michigan Medical School; New York University School of Medicine; Oregon Health & Science University School of Medicine; Pennsylvania State University College of Medicine; and Vanderbilt University School of Medicine. These 11 schools formed a consortium in June 2013 to collaborate and share resources and experiences, and most schools proposed new HSS curricular components. The consortium’s HSS interest group launched this study after discussions identified the absence of a comprehensive HSS curricular framework.

To address our research objective and advance understanding of HSS curricular domains and subcategories, we used an inductive content analysis of all 30 full grant submissions to the ACE initiative, as well as all 11 grantee schools’ syllabi and curriculum documents.32–36 We chose this study design and these data sources, rather than a literature review or interviews with education or systems leaders, because we wanted to identify and synthesize already- or soon-to-be-implemented content applicable to the HSS framework, rather than recommended or conceptualized content.

Back to Top | Article Outline

Data sources and collection

We pursued a two-phase approach to this work (see above). In phase 1, we sought to identify a comprehensive list of curricular domains that could be included in an HSS curricular framework. To do this, we planned to analyze all full grant submissions to the ACE initiative (n = 30). Once curricular domains were identified, the lead investigators developed or modified published definitions for each domain (see Table 1).

Table 1

Table 1

In phase 2, we sought to identify a comprehensive list of curricular subcategories that could be included in the framework as curricular content. To do this, a site coinvestigator (all authors expect M.D., P.O’S., S.S.) at each consortium school (n = 11) performed a detailed review of all existing and planned (within one year) syllabi and curriculum documents from all four years of the UME curriculum required of ≥ 75% of students. Using the domains identified and defined in phase 1 as a guide, the lead investigators extracted data applicable to each curriculum domain from the syllabi and curriculum documents (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A341). Coinvestigators reviewed curricular content with educators at their institution to ensure completeness of data collection and preliminary categorization.

Back to Top | Article Outline

Conceptual frameworks

In our approach to the initial data retrieval, coding, and framing of results, we were guided by several conceptual frameworks. For phase 1, George Engel’s biopsychosocial model allowed for the application of a broad-based and holistic empirical lens to identify HSS curriculum areas beyond biomedicine that could potentially be classified in social or systems domains.37,38 The biopsychosocial model draws on systems theory, with a hierarchy spanning the spectrum from subatomic particles to the person (organismic hierarchy), and from the person to the biosphere (social hierarchy).39 We operationalized the biopsychosocial model by excluding content limited solely to biomedicine or physician–patient interactions (i.e., any issue that would relate to a physician and patient interaction). We focused on subsequent strata in the biopsychosocial hierarchy spanning the psychological to systems levels, enabling us to identify a comprehensive HSS curricular framework. For example, curricular content that educated student physicians about how to communicate with patients about weight loss would not have been included in the coding. However, curricular content that sought to educate student physicians about the social factors that influence a patient’s ability to lose weight, such as access to walking paths or their social support structure, would have been coded as population and public health (domain). In phase 2, we further organized the content in each core curricular domain using existing frameworks from the literature that were the most appropriate for the content items retrieved (e.g., in the example above, the subcategory would have been determinants of health). Coinvestigators collaboratively reviewed content, discussed potential coding schemes, and resolved disagreements via group discussion during workgroup phone calls.

Back to Top | Article Outline

Data analysis

For phase 1, the lead investigators used constant comparative analyses to jointly code 2 full grant submissions and generate a preliminary codebook to facilitate analysis. Codes reflected the domains of curricular content, as shown in Table 1. Investigators then analyzed subsequent full grant submissions, with regular adjudication sessions to compare codes for inconsistency and agreement, and updated and modified the codebook. On the basis of the initial codebook and regular discussions, data saturation was reached after 12 grant submissions. However, a total of 15 submissions were coded to ensure saturation.

We used the results from phase 1 to create a framework to use in phase 2; although some specific curricular content within each domain was identified in phase 1, the majority of these initial data were not specific enough to identify subcategories or specific topic areas. With data obtained from each school’s detailed curricular review in phase 2, and informed by applicable frameworks, the lead investigators coded content in the core curricular domains into subcategories in collaboration with coinvestigators. All investigators contributing to content categorization within each core curricular domain were actively practicing physicians and/or educators with interest in particular curricular domains and, therefore, had a working knowledge of HSS.

Analysis was performed with data manage ment support from the programs ATLAS.ti 6.0 (Scientific Software Development GmbH, Berlin, Germany) and NVivo 10 (QSR International Pty Ltd., Burlington, Massachusetts).33,35 The lead investigators reviewed and discussed selected drafts of the categorization scheme, collapsed and combined domains and subcategories, and resolved disagreements via group discussion during workgroup phone calls. We conducted member checking, a technique used to support the validity of content analysis, via telephone calls with the larger research team, which consisted of educators, administrators, and eval uators at the 11 grantee schools.32,33

Back to Top | Article Outline

Results

Our analysis yielded three types of curricular domains: (1) core, (2) cross-cutting, and (3) linking (see Table 1). Figure 1 illustrates the relationship between these three types of domains. Subcategories (and subtopics) for the core curricular domains are organized using previously published conceptual frameworks (Table 2). Content representative of each domain is shown in Tables 2 (core domains) and 3 (cross-cutting and linking domains).

Table 2

Table 2

Table 3

Table 3

Figure 1

Figure 1

Core curricular domains are content areas that align directly with HSS. These six domains include health care structures and processes; health care policy, economics, and management; clinical informatics and health information technology; population and public health; value-based care; and health system improvement. Content was clustered into a cross-cutting domain if it was identified as transcending multiple core curricular domains. Many UME curricula traditionally address many of these domains, but in this analysis, these domains were emphasized within the context of HSS. These five domains include leadership and change agency; teamwork and interprofessional education; evidence-based medicine and practice; professionalism and ethics; and scholarship. Finally, content that unifies or links the core curricular or cross-cutting domains to other core curricular or cross-cutting domains (internal linking) and to other areas of the curriculum, such as the basic and clinical sciences (external linking), was assigned to the single linking domain: systems thinking. The content of systems thinking allows students to be cognizant of and apply a comprehensive holistic approach to medical care and health care issues. For example, systems thinking skills allow a student to understand the influence of the Affordable Care Act on the determinants of health within a community, and as a result, the ability for his/her patient to access health care and adhere to care plans.

Back to Top | Article Outline

Discussion

Educators have previously proposed a new paradigm for UME that augments the traditional basic and clinical sciences to include a triad of sciences—basic, clinical, and HSS.11,13 The ACGME defines SBP as “an awareness of and responsiveness to the larger … system of health care” and “the ability to call effectively on other resources in the system to provide optimal health care.”20 In this context, resident physicians are expected to function in various health care delivery settings, incorporate cost awareness into their decision making, advocate for optimal care systems, and work in interprofessional teams to improve patient care quality. The proposed new UME paradigm requires an evolution of our understanding of the SBP competency and an evaluation framework, which we described in this report as a comprehensive HSS curricular framework.

Our analysis supports that UME educators have implemented or are implementing HSS curricula for a majority of their students that are broader in scope than the ACGME definition of SBP, and con tain pertinent topic areas addressed in the CEPAERs.20,21 The domains, working defin i tions for curricular content, conceptual frameworks, and subcategories (and subtopics) reported here are a starting point for the development of HSS compe tencies and entrustable professional activities.

Several studies highlight existing and potential HSS topics or concepts that should be considered in such a framework. These include patient safety, coordination of care, multidisciplinary teams, cost awareness, quality improvement, practice management, medical record keeping, managed care, health care systems, and medical economics.23–25 Others allude to the implementation of health-care-related sciences, including global and public health and the skills required for 21st-century providers, such as interprofessional collaboration, quality improvement, clinical informatics, and social medicine.6,16,18,19,26,40–45 Our results synthesize these topics and concepts into one comprehensive HSS curricular framework and lay the groundwork for a critical review of current curricula, program design, and instructional methods, as well as programmatic development and assessment by medical education programs.

We identified several cross-cutting domains of knowledge and skills that traditionally may have been included in UME programs but have a new emphasis in this HSS curricular framework. These cross-cutting domains highlight the overlap and application of the proposed HSS domains with traditional areas of the curriculum. For example, leadership and change agency skills may have been included in some traditional UME programs. In our analysis, though, leadership and change agency knowledge and skills were specifically focused on health systems improvement projects or learner roles on interprofessional care teams. With the transformation into an expanded content and competencies set required for the HSS curricular framework, cross-cutting domains require UME programs to reevaluate their curriculum and assess areas in need of updating or modification. Curricula related to leadership and change agency, professionalism and ethics, and evidence-based medicine and practice, for example, may require modification to better contextualize these areas for 21st-century health systems education, even for schools that do not plan to have a comprehensive HSS curriculum.

Although they have previously been proposed as a component of public health competencies, explicit systems thinking skills appear to be new to UME.46,47 The proposed skill set of systems thinking is a critical piece to include in curricula because it reflects the evolving landscape of health systems toward learning organizations, where the health system and providers become more interconnected and continuously transform systems to improve outcomes.48–50

The potential HSS curricular framework reported here is built from a practical perspective and reflects curricular topics that are currently or will soon be required content for most students taught in UME programs (i.e., it does not reflect recommendations alone). Additionally, this broad conceptualization of HSS reframes the view of educating future physicians. Although our results originate from U.S. UME programs, we believe this HSS curricular framework and content are applicable to GME and continuing medical education, as well as to many allied health professions educational programs, in the United States and abroad. Like many medical schools, other health professional programs, including nursing, pharmacy, and physician assistant programs, are incorporating HSS topics into their curricula. Several authorities (e.g., the Interprofessional Education Collaborative and CanMEDS) increasingly emphasize interprofessional competencies, specifically in quality improvement and patient safety.51,52 In addition, health care transformation to achieve the Triple Aim applies to all educational programs fostering the development of future team members in these care delivery models.7 Our results have the potential to provide an HSS curricular framework for all health professions programs.

We believe this work informs potential next steps related to the HSS. First, these results can and should be used in subsequent work to develop educational methods and evaluation measures (e.g., milestones, competencies, entrustable professional activities) and to design and implement curricula that enable undergraduate trainees to be effective health care professionals. Second, future work should assess these domains for their prevalence and perceived importance in educational programs, the practicality of applying them in interprofessional contexts, and their alignment with the needs of the health system. Specifically, our results provide a static, two-dimensional perspective of the HSS curriculum, but it would be important to present the right educational intervention at the relevant time and in the relevant setting. The continuum between UME, GME, and continuing medical education programs requires a clear road map of trainee progression, specifically in these areas. Future work will be needed to determine when and where along the training spectrum these topics should be introduced. For example, a curriculum inventory mapping tool, such as the Tool for Assessing Cultural Competence Training, could be developed and applied across the educational continuum to assess and target the incorporation of HSS in individual medical education programs.53

There are several limitations to this work. First, although curriculum design ideally starts with learner outcomes rather than content, given the lack of consensus-driven HSS learner outcomes in the literature, we believe the identification of a potential comprehensive curricular framework with domains and subcategories was a required first step to inform the development of learner outcomes.12,54 We considered existing resources (e.g., the Association of American Medical Colleges’ entrustable professional activities and the ACGME’s core requirements), but none adequately provided a comprehensive framework and assembly of content for HSS.20,21,52 Until medical school accreditation is based on more detailed content requirements, UME educators seek a broad menu of curricular items that are grounded in UME experience. We additionally acknowledge that a comprehensive list of curricular topics is less useful than a consensus-driven list of minimum UME learner outcomes; many of the consortium schools are concurrently working toward this goal, informed in part by these results. Next, individual school assessments were limited to the 11 consortium schools (a group of schools seeking to innovate in these areas) rather than a larger sample of schools; although including more schools in the assessments might have captured new curricular content, we found considerable overlap within these 11 schools. We reached data saturation before the completion of phase 1, and this intensive and resource-laden investigation at each consortium school provided a depth that would not have been achievable through a broader or more remote data collection. We considered alternative study designs, such as a Delphi process, whereby experts in education and the health systems would contribute their views about curricular content that needed to be required in a UME program. Therefore, our results have some shortcomings in that the curricular content may not represent a fully exhaustive curricular framework. We also recognize that there is significant conceptual overlap across the core curricular domains (e.g., value-based care and health system improvement) and subcategories within domains (e.g., health care disparities/vulnerable patient populations and determinants of health) and that consensus building by other authors may have yielded different results. Despite these limitations, we believe the method and results are a critical call to action for the medical education community and contribute to potential subsequent work dedicated to elaborating on this initial HSS framework.

In conclusion, we leveraged the work of the ACE initiative to identify six core domains with subcategories (and subtopics), five cross-cutting domains, and one linking domain for an HSS curriculum. This broad framework aims to build on the traditional definition of SBP to better address the needs for 21st-century physician training and to highlight the need for medical and other health professions schools to better align education programs with the anticipated needs of the health care delivery systems in which students will practice. This framework can catalyze the national conversation regarding the knowledge and skills required for physician trainees to become collaboratively effective, systems-based physicians. This third science—HSS—will require a critical investigation into existing curricula to determine the most efficient methods for integration with the basic and clinical sciences. Future work should explore comprehensive learning objectives; identify ideal educational methodologies, such as authentic experiential learning opportunities, for engaging in HSS content; and provide curricular examples that emphasize active learning. In addition, we advocate for the development of valid assessments that test student outcomes, along with robust investigations of the value added to the health system through the incorporation of HSS. Though the conceptualization of HSS curricular domains is just at the beginning, these efforts have the potential to affect medical education for decades to come.

Back to Top | Article Outline

References

1. Berwick DM, Hackbarth AD. Eliminating waste in US health care. JAMA. 2012;307:1513–1516.
2. Hood VL, Weinberger SE. High value, cost-conscious care: An international imperative. Eur J Intern Med. 2012;23:495–498.
3. Millman ML; Institute of Medicine Committee on Monitoring Access to Personal Health Care Services. Access to Health Care in America. 1993.Washington, DC: National Academy Press.
4. Institute of Medicine Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. 2001.Washington, DC: National Academy Press.
5. Institute for Healthcare Improvement. The triple aim. Optimizing health, care and cost. Healthc Exec. 2009;24:64–66.
6. Crosson FJ, Leu J, Roemer BM, Ross MN. Gaps in residency training should be addressed to better prepare doctors for a twenty-first-century delivery system. Health Aff (Millwood). 2011;30:2142–2148.
7. Berwick DM, Nolan TW, Whittington J. The triple aim: Care, health, and cost. Health Aff (Millwood). 2008;27:759–769.
8. Thibault GE. Reforming health professions education will require culture change and closer ties between classroom and practice. Health Aff (Millwood). 2013;32:1928–1932.
9. Skochelak SE. A decade of reports calling for change in medical education: What do they say? Acad Med. 2010;85(9 suppl):S26–S33.
10. Skochelak SE. Commentary: A century of progress in medical education: What about the next 10 years? Acad Med. 2010;85:197–200.
11. Lucey CR. Medical education: Part of the problem and part of the solution. JAMA Intern Med. 2013;173:1639–1643.
12. Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376:1923–1958.
13. Gonzalo JD, Haidet P, Papp KK, et al. Educating for the 21st-century health care system: An interdependent framework of basic, clinical, and systems sciences [Epub ahead of print October 16, 2015]. Acad Med. doi: 10.1097/ACM.0000000000000951.
14. Gonzalo JD, Haidet P, Wolpaw DR. Authentic clinical experiences and depth in systems: Toward a 21st century curriculum. Med Educ. 2014;48:104–105.
15. Chen HC, Sheu L, O’Sullivan P, Ten Cate O, Teherani A. Legitimate workplace roles and activities for early learners. Med Educ. 2014;48:136–145.
16. Berwick DM, Finkelstein JA. Preparing medical students for the continual improvement of health and health care: Abraham Flexner and the new “public interest.” Acad Med. 2010;85(9 suppl):S56–S65.
17. Kebede S, Pronovost P. It is time to reinvent the wheels of medical training. Acad Med. 2015;90:126.
18. Combes JR, Arespacochaga E. Physician competencies for a 21st century health care system. J Grad Med Educ. 2012;4:401–405.
19. Smith CD, Levinson WS; Internal Medicine HVC Advisory Board. A commitment to high-value care education from the internal medicine community. Ann Intern Med. 2015;162:639–640.
20. Accreditation Council for Graduate Medical Education. Outcomes project. 1999. http://www.acgme.org/outcome/comp/compFull.asp. Accessed March 30, 2013. [No longer available.].
21. American Association of Medical Colleges.Core Entrustable Professional Activities for Entering Residency: Curriculum Developers’ Guide. 2014.Washington, DC: American Association of Medical Colleges.
22. Patel MS, Lypson ML, Davis MM. Medical student perceptions of education in health care systems. Acad Med. 2009;84:1301–1306.
23. Association of American Medical Colleges. Medical School Graduation Questionnaire: 2013 All Schools Summary Report. 2013. Washington, DC: American Association of Medical Colleges; https://www.aamc.org/download/350998/data/2013gqallschoolssummaryreport.pdf. Accessed February 4, 2016.
24. Cordasco KM, Horta M, Lurie N, Bird CE, Wynn BO. How Are Residency Programs Preparing Our 21st Century Internists? A Review of Internal Medicine Residency Programs’ Teaching on Selected Topics [working paper]. 2009. Santa Monica, Calif: Rand Health for the Medicare Payment Advisory Commission; http://www.rand.org/pubs/working_papers/WR686.html. Accessed March 15, 2016.
25. Chaudhry SI, Lien C, Ehrlich J, et al. Curricular content of internal medicine residency programs: A nationwide report. Am J Med. 2014;127:1247–1254.
26. Chase J, Evert J; Global Health Education Consortium. Global Health Training in Graduate Medical Education: A Guidebook. 2011.2nd ed. Bloomington, Ind: iUniverse.
27. Dyne PL, Strauss RW, Rinnert S. Systems-based practice: The sixth core competency. Acad Emerg Med. 2002;9:1270–1277.
28. Gonzalo JD, Haidet P, Blatt B, Wolpaw D. Identifying challenges in implementing systems-based curriculum: A qualitative assessment of medical student perspectives. Paper presented at: National Society of General Internal Medicine Conference; April 23, 2015; Toronto, Ontario, Canada.
29. American Medical Association. Accelerating change in medical education. http://www.ama-assn.org/sub/accelerating-change/index.shtml. Accessed February 4, 2016.
30. Gonzalo JD, Dekhtyar M, Starr SR, et al. Identifying and defining content domains for a science of healthcare delivery curriculum: Results from the American Medical Association Accelerating Change in Medical Education Systems-Based Practice Working Group. Paper presented at: Society of General Internal Medicine National Conference; April 23, 2015; Toronto, Ontario, Canada.
31. Gonzalo JD, Heist BS, Duffy BL, et al. Identifying and overcoming the barriers to bedside rounds: A multicenter qualitative study. Acad Med. 2014;89:326–334.
32. Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. 2009.3rd ed. Thousand Oaks, Calif: Sage Publications.
33. Patton MQ. Qualitative Research and Evaluation Methods. 2002.3rd ed. Thousand Oaks, Calif: Sage Publications.
34. Glaser BG, Strauss AL. The Discovery of Grounded Theory: Strategies for Qualitative Research. 1967.Chicago, Ill: Aldine Pub. Co.
35. Corbin JM, Strauss AL. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. 2008.3rd ed. Los Angeles, Calif: Sage Publications.
36. Watling CJ, Lingard L. Grounded theory in medical education research: AMEE guide no. 70. Med Teach. 2012;34:850–861.
37. Engel GL. The clinical application of the biopsychosocial model. Am J Psychiatry. 1980;137:535–544.
38. Flexner A. Medical education in the United States and Canada. From the Carnegie Foundation for the Advancement of Teaching, Bulletin Number Four, 1910. Bull World Health Organ. 2002;80:594–602.
39. von Bertalanffy L. General System Theory: Foundations, Development, Applications. 1973.New York, NY: G. Braziller.
40. Hersh WR, Gorman PN, Biagioli FE, Mohan V, Gold JA, Mejicano GC. Beyond information retrieval and electronic health record use: Competencies in clinical informatics for medical education. Adv Med Educ Pract. 2014;5:205–212.
41. Otero P, Hersh W, Jai Ganesh AU. Big data: Are biomedical and health informatics training programs ready? Contribution of the IMIA Working Group for Health and Medical Informatics Education. Yearb Med Inform. 2014;9:177–181.
42. Westerhaus M, Finnegan A, Haidar M, Kleinman A, Mukherjee J, Farmer P. The necessity of social medicine in medical education. Acad Med. 2015;90:565–568.
43. Drain PK, Primack A, Hunt DD, Fawzi WW, Holmes KK, Gardner P. Global health in medical education: A call for more training and opportunities. Acad Med. 2007;82:226–230.
44. Association of American Medical Colleges. Behavioral and Social Science Foundations for Future Physicians. 2011. Washington, DC: Association of American Medical Colleges; https://www.aamc.org/download/271020/data/behavioralandsocialsciencefoundationsforfuturephysicians.pdf. Accessed February 4, 2016.
45. Combes JR, Arespacochaga E. Lifelong Learning—Physician Competency Development. 2014. Chicago, Ill: American Hospital Association’s Physician Leadership Forum; http://www.ahaphysicianforum.org/files/pdf/physician-competency-development.pdf. Accessed February 4, 2016.
46. Public Health Foundation. PHF, ACHI exploring population health competencies. http://www.phf.org/phfpulse/Pages/PHF_ACHI_Exploring_Population_Health_Competencies.aspx. Accessed February 4, 2016.
47. Public Health Foundation. Core competencies for public health professionals. http://www.phf.org/resourcestools/Documents/Core_Competencies_for_Public_Health_Professionals_2014June.pdf. Accessed February 4, 2016.
48. Senge PM. The Fifth Discipline: The Art and Practice of the Learning Organization. 2006.Rev and updated ed. New York, NY: Doubleday/Currency.
49. Nelson EC, Batalden PB, Huber TP, et al. Microsystems in health care: Part 1. Learning from high-performing front-line clinical units. Jt Comm J Qual Improv. 2002;28:472–493.
50. Mabry PL, Olster DH, Morgan GD, Abrams DB. Interdisciplinarity and systems science to improve population health: A view from the NIH Office of Behavioral and Social Sciences Research. Am J Prev Med. 2008;35(2 suppl):S211–S224.
51. Frank JR, Danoff D. The CanMEDS initiative: Implementing an outcomes-based framework of physician competencies. Med Teach. 2007;29:642–647.
52. Interprofessional Education Collaborative. Core Competencies for Interprofessional Collaborative Practice: Report of an Expert Panel. May 2011. Washington, DC: Association of American Medical Colleges; https://www.aamc.org/download/186750/data/core_competencies.pdf. Accessed February 4, 2016.
53. Lie DA, Boker J, Crandall S, et al. Revising the Tool for Assessing Cultural Competence Training (TACCT) for curriculum evaluation: Findings derived from seven US schools and expert consensus. Med Educ Online. 2008;13:1–11.
54. Kern DE, Thomas PA, Hughes MT. Curriculum Development for Medical Education: A Six-Step Approach. 2009.2nd ed. Baltimore, Md: Johns Hopkins University Press.
Back to Top | Article Outline

References cited in tables only

55. World Health Organization. Everybody’s Business—Strengthening Health Systems to Improve Health Outcomes: WHO’s Framework for Action. 2007. Geneva, Switzerland: World Health Organization; http://www.who.int/healthsystems/strategy/everybodys_business.pdf. Accessed February 22, 2016.
    56. Altarum Institute. Health economics. http://altarum.org/areas-of-expertise/health-economics. Accessed February 4, 2016.
      57. American Medical Informatics Association. Clinical informatics. https://www.amia.org/applications-informatics/clinical-informatics. Accessed February 4, 2016.
        58. Young L. Population health: Concepts and methods. Am J Hum Biol. 2006;18:159–160.
        59. Graban M. Lean Hospitals: Improving Quality, Patient Safety, and Employee Engagement. 2012.2nd ed. Boca Raton, Fla: CRC Press.
          60. Conger JA, Kanungo RN. Toward a behavioral-theory of charismatic leadership in organizational settings. Acad Manage Rev. 1987;12:637–647.
          61. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: What it is and what it isn’t. BMJ. 1996;312:71–72.
          62. Boyer EL. Scholarship Reconsidered: Priorities of the Professoriate. 1990.Princeton, NJ: Carnegie Foundation for the Advancement of Teaching.
            63. Foster-Fishman PG, Behrens TR. Systems change reborn: Rethinking our theories, methods, and efforts in human services reform and community-based change. Am J Community Psychol. 2007;39:191–196.
            64. Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q. 2005;83:691–729.
            65. McDonald KM; Agency for Healthcare Research and Quality; Stanford–UCSF Evidence-Based Practice Center; National Library of Medicine. Closing the Quality Gap: Vol. 7, Care Coordination—A Critical Analysis of Quality Improvement Strategies. 2007.Rockville, Md: Agency for Healthcare Research and Quality.
              66. World Health Organization. Health impact assessment (HIA): The determinants of health. http://www.who.int/hia/evidence/doh/en/. Accessed February 4, 2016.
                67. Diez Roux AV. Conceptual approaches to the study of health disparities. Annu Rev Public Health. 2012;33:41–58.
                68. Glass TA, McAtee MJ. Behavioral science at the crossroads in public health: Extending horizons, envisioning the future. Soc Sci Med. 2006;62:1650–1671.
                69. Kilbourne AM, Switzer G, Hyman K, Crowley-Matoka M, Fine MJ. Advancing health disparities research within the health care system: A conceptual framework. Am J Public Health. 2006;96:2113–2121.
                70. Porter ME. What is value in health care? N Engl J Med. 2010;363:2477–2481.

                Supplemental Digital Content

                Back to Top | Article Outline
                © 2017 by the Association of American Medical Colleges