Secondary Logo

Journal Logo

Articles

Priority Areas and Potential Solutions for Successful Integration and Sustainment of Health Systems Science in Undergraduate Medical Education

Gonzalo, Jed D. MD, MSc; Baxley, Elizabeth MD; Borkan, Jeffrey MD, PhD; Dekhtyar, Michael; Hawkins, Richard MD; Lawson, Luan MD, MAEd; Starr, Stephanie R. MD; Skochelak, Susan MD, MPH

Author Information
doi: 10.1097/ACM.0000000000001249
  • Free

Abstract

Educators, policy makers, and health systems leaders are calling for significant reform of undergraduate medical education (UME) and graduate medical education (GME) programs to meet the evolving needs of the health care system.1–3 One critical component of recommended reforms is physician education in health systems science (HSS), which includes topics such as value-based care, health system improvement, clinical informatics, and population and public health.4,5 Considered the “third science” that integrates with the traditional basic and clinic sciences, HSS can be viewed as the methods and principles of improving quality, outcomes, and costs of health care delivery for patients and populations of patients. Education in HSS has the potential to develop a more broadly prepared physician workforce that is better able to lead the health care system to meet the needs of patients and society.5

Despite increasing awareness about the importance of HSS in medical education, most medical schools have not substantially integrated HSS-related competencies into their curricula, resulting in a mismatch between medical education and the needs of the evolving health care system and the patients cared for within this system.2,3,6,7 Nationally, several medical schools have initiated HSS-related innovative curricula in both classroom and workplace learning environments, and recent work has created a framework for HSS curricula.5,8–10 However, given the relatively nascent stage of integration within UME, the successful large-scale implementation of HSS curricula across medical schools is challenged by issues of curriculum design, assessment, culture, and accreditation, among others. These challenges require thoughtful planning and evaluation by the medical education and health systems communities to facilitate the sustainability of HSS in medical education.

Between 2013 and 2015, the American Medical Association’s (AMA’s) Accelerating Change in Medical Education (ACE) consortium of 11 U.S. medical schools (see below) chose to specifically address the challenges of implementing an HSS curriculum to advance progress in this area of UME. In this report of a working conference using thematic analysis of workshop recommendations and experiences from the 11 ACE consortium schools, we describe seven priority areas for the successful integration and sustainment of HSS in educational programs, and their associated challenges and potential solutions.

ACE Consortium Work

In 2013, the ACE consortium of 11 U.S. medical schools undertook a broad-based investigation to identify a comprehensive framework for HSS training. The investigation spanned multiple levels from the individual learner, to medical-school-specific programs, to collaborative national efforts.

The 11 ACE consortium schools were the Warren Alpert Medical School of Brown University; Brody School of Medicine at East Carolina University; University of California, San Francisco, School of Medicine; University of California, Davis, School of Medicine; Indiana University School of Medicine; Mayo Medical School; University of Michigan Medical School; New York University School of Medicine; Oregon Health & Science University School of Medicine; Pennsylvania State University College of Medicine; and Vanderbilt University School of Medicine.11

HSS workgroup

Between October 2013 and August 2015, a workgroup consisting of representatives from each of the consortium schools and AMA staff (educators and curriculum leaders) held monthly conference calls to plan and develop projects related to HSS. Discussion topics during the calls included challenges participants encountered at their home institutions, collaboration on how to improve educational methods or assessment, and overarching themes related to the challenges of incorporating HSS in medical education. Telephone calls were 60 minutes in length, carefully documented by an AMA staff member, and were attended by 5–18 HSS workgroup members.

ACE consortium meeting

In April 2015, a consortium-wide education meeting was held to identify and develop a plan for advancing HSS integration into UME. At least four educators or systems leaders and one medical student from each ACE school (n = 120 participants total) gathered at the two-day meeting to contribute to the ongoing dialogue regarding HSS in UME. The AMA ACE leadership team, advisory board members, and additional educators from Oregon Health & Science University also participated. A plenary session provided an overview of HSS in medical education and updates on the HSS workgroup’s efforts to develop a curricular framework that included the following domains: health care structures and processes; health care policy, economics, and management; clinical informatics and health information technology; population and public health; value-based care; health system improvement; leadership and change agency; teamwork and interprofessional education; evidence-based medicine and practice; professionalism and ethics; scholarship; and systems thinking.12 Next, smaller groups were specifically asked to discuss and provide three suggestions for the following questions: “What would it take on the local and national level for medical schools to become successful and sustain success regarding HSS?” and “What can ACE working groups and the consortium do to advance the national effort?” Lastly, the whole group reconvened to hear reports from each small group, followed by a large-group discussion of barriers and facilitators to advancing HSS in medical education. Field notes from the large-group discussions were recorded by three investigators (J.D.G., J.B., S.R.S.), and all small groups submitted written responses.

Data analysis

Following the consortium meeting, two investigators (J.D.G., M.D.) employed a thematic content analysis and constant comparative analyses to review and code written responses and field notes.13,14 The investigators independently analyzed a portion of the written responses and field notes, then compared their codes for inconsistency or agreement. Using this initial codebook, investigators then independently coded the remaining data. Through regular adjudication sessions, the investigators identified the general categories and themes—that is, the priority areas and potential solutions for advancing HSS. To enhance the trustworthiness of the results, the technique of member checking was performed with members of the HSS workgroup to support the validity of the content analysis.15–17 The workgroup then discussed findings, challenges, and potential solutions and agreed on the final priority areas and potential solutions.

Priority Areas for Advancing HSS

The analysis identified seven priority areas: (1) partner with licensing, certifying, and accrediting bodies; (2) develop comprehensive, standardized, and integrated curricula; (3) develop, standardize, and align assessments; (4) improve the UME to GME transition; (5) enhance teachers’ knowledge and skills, and incentives for teachers; (6) demonstrate value added to the health system; and (7) address the hidden curriculum.

For each of the identified priority areas, two components were considered: (1) the specific challenge (or challenges) to implementing and sustaining HSS in medical education; and (2) the potential solutions, or interventions, to advance the field. Figure 1 is a key driver diagram for advancing HSS in medical education programs that characterizes outcomes, key drivers, and potential solutions (some of these solutions are discussed below); several of the identified solutions are supported by policy recommendations for medical education.18

Figure 1
Figure 1:
Key driver diagram for advancing HSS in medical education programs. The figure demonstrates the relationships between the outcomes (proximal and distal), key drivers (i.e., priority areas), and potential solutions, or interventions, that could potentially influence the successful integration and sustainment of HSS in medical education. Abbreviations: HSS indicates health systems science; UME, undergraduate medical education; GME, graduate medical education; CME, continuing medical education; LCME, Liaison Committee on Medical Education; ACGME, Accreditation Council for Graduate Medical Education; AAMC, Association of American Medical Colleges.

Partner with licensing, certifying, and accrediting bodies

Challenges.

Consensus on national standards and curricular content for medical education emerges from the education community, and when widespread acceptance has occurred, these standards become codified by accrediting bodies, such as the Liaison Committee on Medical Education (LCME) and the Accreditation Council for Graduate Medical Education (ACGME). Curriculum content that is relevant to safe and effective physician practice is then incorporated into national assessment procedures, such as the United States Medical Licensing Examination Step exams or resident training exams. The process for emerging consensus, including the widespread development and adoption of new curricular content and eventual promulgation of new national standards, can require a lengthy cycle time (often a decade or longer). Recent examples of national consensus in medical education, in various stages of adoption by accrediting and assessment agencies, include cultural competence (e.g., the Tool for Assessing Cultural Competence Training),19 integration of behavioral science,20 and use of entrustable professional activities (EPAs) to track student performance.21 This process and the associated delays can create particular challenges for new and innovative educational programs seeking to be adaptive with the rapidly evolving health care landscape.

Potential solutions.

To mitigate the barriers and challenges to introducing innovative educational programs, curriculum change leaders should seek the recommendations and guidance of all stakeholders involved to implement sustainable reform. These stakeholders include national licensing and certifying bodies and accrediting organizations across the continuum from UME, to GME, to continuing medical education (CME). Licensing, certifying, and accrediting agencies could catalyze the collaborative processes that would align guidelines, competencies, and exams to necessitate the learning of HSS-related content. Once curricular reforms have been implemented and evaluated, opportunities should enable clinical faculty and practicing physicians to develop competence in the defined HSS knowledge and skills. These changes may lead to reforms in physician licensure, certification, and maintenance of certification requirements for both current and incoming physicians. Without such systemic change from accrediting organizations, the long-term success of HSS integration will be limited.

Develop comprehensive, standardized, and integrated curricula

Challenges.

Ensuring appropriate education in HSS requires a compre hensive content framework that is integrated with the basic and clinical sciences at the appropriate time in students’ developmental trajectory. To meet this end, some level of stan dardization between medical schools, particularly given the dispersion of graduates across the country as they transition to residencies and practice, would be ideal. Unfortunately, few medical schools have comprehensive HSS curricula, even fewer of these curricula are integrated, and many schools lack faculty members with the ability to teach HSS content. Additionally, there are currently no standards or mechanisms allowing schools to reach agreement on which of the HSS domains (see above) should be included in the curriculum. The medical education community, along with the organizations responsible for segments of medical training and regulation (such as the LCME, ACGME, and United States Medical Licensing Examination), has not yet developed a coordinated approach to this issue. Although medical schools have added HSS content to their curricula, these additions have largely been developed for and tailored to local needs and curricula, with few collaborations across UME programs.20,22 Likewise, the medical education literature on HSS has been fragmented, with no known attempts to provide a comprehensive strategy for developing and executing effective HSS curricula across the whole of medical training.22

Potential solutions.

A critical first step toward comprehensive, standardized, and integrated HSS curricula will be for health professions schools to recognize the importance and necessity of expanding these competencies. Local and national discussion forums dedicated to HSS can raise awareness of this need and promote discourse around the core content and concepts to be included in such curricula.12 Inclusion of both allopathic and osteopathic UME and GME programs, as well as CME accrediting bodies and specialty boards, would best inform this work and allow the medical education continuum to be addressed from beginning to end. Efforts to achieve standardization will need to be sufficiently rigorous to ensure that all trainees gain competencies in HSS domains, but not so inflexible that they discourage innovation or the ongoing evaluation of topics as health care realities change. Leadership in this area may come from medical education organizations, health care organizations, or trainees, with collaborative ventures among all stakeholders having the greatest chance of success.

Develop, standardize, and align assessments

Challenges.

High-quality assessment methods are essential to ensure learner progress in achieving HSS competencies. In addition, assessment drives student learning, and facilitates skill development in workplace-based settings.23 There is a relative paucity of validated assessment methods targeting HSS domains; for example, HSS does not have a robust framework of milestones, competencies, and/or EPAs. One challenge to developing and implementing HSS assessment methods in UME is the lack of a clear developmental paradigm for mastery in these domains. Foundational HSS knowledge is currently being defined, and the potential roles and expected level of performance that students should demonstrate in HSS domains have not yet been elucidated. Furthermore, it would be difficult to construct high-quality multiple-choice items to assess knowledge in key HSS domains with or without a synthetic HSS curricular framework (see previous priority area). Assessments of more practical aspects of HSS will require the development and implementation of observational methods in the workplace, where approaches to ensuring high-quality, reliable, and valid assessment outcomes are evolving.24,25

Potential solutions.

Ongoing work to define the core curriculum, objectives, and content in HSS is a critical step in understanding the types of new assessment methods that are needed.12 Collaboration with licensing, certifying, and accrediting bodies is essential for developing a comprehensive assessment of HSS knowledge and skills along the continuum of education.26 As core knowledge, skills, and competencies are defined, a matrix outlining validated assessments should be developed and aligned with the HSS domains. Fortunately, existing assessment methods could be used within this matrix and provide an initial educator tool kit. For example, assessment methods related to teamwork, quality, safety, and evidence-based practice include multiple-choice exams, knowledge application tests, work product ratings, simulation-based tools, indirect observation methods (e.g., peer assessment, multisource feedback), and direct observation of clinical activities.27–41 Unfortunately, most assessment methods involve locally developed tools or modifications to existing national tools using small samples of students with variable psychometric characteristics. This forbids the use of existing validity data in informing the use of the tool, prevents comparisons across studies, and hinders rigorous analysis of new tools.42,43 Future work should encourage collaborative research using existing and yet-to-be-developed validated tools to ensure that the assessment matrix contains trustworthy information regarding each tool’s quality and recommendations for effective use.

Improve the UME to GME transition

Challenges.

Ensuring learners’ smooth transition from UME to GME and beyond is not a challenge unique to HSS.2,3,7,44 While GME is using the language of milestones as part of the ACGME Next Accreditation System and the Association of American Medical Colleges has offered EPAs as outcome goals for graduating medical students, the LCME does not stipulate a preference for either.21,45,46 Although EPAs and milestones (being written by many schools) can be used in a complementary manner, ideal educational handoffs are hindered by the lack of a consistent assessment language.47 The ACGME Clinical Learning Environment Review Program and its mission to improve care via a reimagined integration of clinical practice and education strengthens the importance of the UME to GME transition in HSS.48 Lastly, variation across GME programs’ expectations of graduating medical student competence in HSS and their assessment and prioritization of these domains in the residency selection process further reinforce gaps in the UME to GME transition.

Potential solutions.

Educators at all levels need a common language to guide learning and assessment, specifically for HSS, to reliably ensure that physicians are prepared to meaningfully participate in complex, evolving, team-based care models. Medical schools should pilot the Association of American Medical Colleges’ EPAs and locally written milestones (grounded in existing GME milestones) that pertain to HSS. Residency programs should examine whether the breadth and depth of the HSS domains are satisfactory or whether these need to evolve over time. Together, UME and GME educator partnerships across disciplines can define identifying features of ideal educational handoffs and implement assessment tools that can be used across the UME-GME continuum. Program director confidence in the description of a student in the dean’s letter could be targeted as one potential outcome of these ideal educational handoffs.

Enhance teachers’ knowledge and skills, and incentives for teachers

Challenges.

Identifying faculty who understand, practice, and effectively teach HSS content is key to integrating HSS into existing curricula. Traditionally trained clinical faculty typically need to enhance their own HSS expertise.2,7 This means that clinical teachers are challenged to teach HSS while they are learning it and simultaneously working in a continuously changing environment.49 Competing priorities for time and inadequate resource allocation, coupled with increased emphasis on clinical productivity, make it difficult for faculty to undertake intensive training or integrate newly developed skills into clinical practice and teaching. Without in-house expertise and institutional infrastructure to educate faculty, external programs are sometimes used to fill this gap. However, these programs are costly and do not provide an opportunity for faculty to understand the content within the context of the daily work environment. Collaborative efforts across health professions’ disciplines are required to consistently integrate newly learned concepts into practice, improve outcomes, and role model new competencies to trainees and colleagues.

Potential solutions.

Training and support of faculty capabilities in care improvement, educational innovation, and dissemination of HSS scholarship facilitates a sustainable clinical learning environment, and collaboration with the health care system provides the hands-on learning opportunities required to integrate new concepts and practices into the daily work environment. Institutional faculty development programs and learning communities can be created using a mixture of online, face-to-face, and experiential learning that is supported by a shared repository of educational modules and materials. Institutions can support faculty by facilitating honorific recognition of professional development, structural support for maintenance of certification and CME, streamlined institutional review board processes to encourage quality improvement interventions, protected time to advance learning and teaching, leadership and career opportunities in related clinical and educational environments, assistance with scholarly productivity, and inclusion of HSS scholarship in promotion guidelines. A national certificate program could promote the dissemination of information to a broad base of medical educators while allowing institutions to focus on local integration and faculty support for continued learning and improvement.

Demonstrate value added to the health system

Challenges.

Traditionally, UME has focused primarily on the development of students’ knowledge and skills in the basic and clinical sciences, with the goal of caring for patients in the future. And clinical training experiences continue to link students directly with resident and attending physicians during clinical care duties.50 This apprenticeship model requires time to mentor and educate students, which often decreases efficiency and negatively impacts physician productivity and the profitability of the health system.51–55 The increasing need for physicians and care delivery models to optimize efficiency and quality while minimizing cost, and the added work of mentoring medical students in this apprenticeship model, needs to be reexamined. Faculty and schools have traditionally presumed that students cannot add value to patient care while they are students. However, recommendations have been made for increased education and research into further integrating medical schools and student activities with academic health centers and community health programs so that students could add value to the health system while they learn.56,57

Potential solutions.

Educators have recommended an increased focus on identifying and providing value-added roles for medical students to “share the care” of health care delivery.58,59 The application of HSS in experiential roles within the health care system can often be lower-stakes (e.g., health coaching) compared with traditional biomedical decisions (e.g., ordering medications). This key difference opens up several opportunities for medical students to engage with the health system by performing authentic systems-based tasks that can add value and improve patient outcomes, while also learning HSS.5,10,59 Students can add value by serving as patient navigators and health coaches, facilitating effective care transitions, and assisting with medication reconciliation and education. These roles align with the clinical care needs of the health system, specifically focusing on outcomes such as reducing readmissions, improving care transitions, and improving patient satisfaction.60 These new student roles can lessen the “burden” on the system and mentors, enhance student education in HSS, and potentially improve health outcomes.

Address the hidden curriculum

Challenges.

The hidden curriculum is the influence of institutional structure and culture on the learning environment61 and often reinforces the notions of physician autonomy and authority, influencing trainees’ perceptions of patient worth and the roles of health care team members as they model faculty behaviors.62–64 Similarly, policies, the formal curriculum, exams, and professional development of faculty reflect institutional goals and values, which, in turn, affect the learning environment.65–67 Although trainees have identified gaps in their HSS education, this content is assigned a lower priority because it is not included in licensing and board exams or residency placement criteria.68–71 Important, emerging evidence suggests that students who train in clinical environments with lower resource use are more likely to practice similar methods in the future, suggesting that role modeling during training is critical to learner development.72,73 If role models do not demonstrate HSS-informed clinical practice, learners will be less likely to incorporate these behaviors into their own practice.

Potential solutions.

Creating initiatives to introduce HSS curricula will require a change in institutional values and culture. As such, implementation and evaluation of specific curricular changes at each institution will need to model the expected value changes for the rest of the medical education community. Because perceptions of learning environments vary between institutions, efforts to evaluate the effects of the hidden curriculum must be directed toward each specific locale.74 Understanding each community’s readiness for educational change will assist that institution’s leadership in understanding the barriers and tensions to implementing a formal HSS curriculum and allow them to devise incentive structures for faculty (via resources and promotion) and students (via exams) accordingly. Increasing students’ recognition of the importance of HSS to their careers could be addressed by exposing students to integrated, longitudinal, and meaningful patient-centered experiences. Aligning their HSS education with positive experiences in health systems improvement efforts may reduce gaps in the curriculum and create a “fluid” learning environment. Evolving discourse on HSS education at the national level should include conversations about physician account ability in espousing HSS tenets in their practice and teaching.

Conclusions

Despite recommendations calling for curricular reform in health professions schools, there remains a significant mismatch between what is taught and measured and the knowledge and skills that are required for the next generation of physicians. HSS is one pivotal piece in education reform, and medical schools are designing and implementing learning activities focused on related areas. In the next wave of medical education reform, HSS will be a primary focus of innovation, requiring strategic thinking and action by the medical education community to positively influence large-scale, sustainable reform. The achievement of success with regard to HSS will occur most expeditiously through new and evolving collaborative partnerships between medical schools; health systems; licensing, accrediting, and regulatory bodies; and funding partners. The priority areas and potential solutions described here can be used by individual schools and national HSS education collaboratives to further outline and delineate the steps needed to create, deliver, study, and sustain effective HSS curricula with an eye toward integration with the basic and clinical sciences curricula. We believe our results can accelerate the positive impact of medical education on improving patient health and outcomes.

Acknowledgments: The authors would like to thank all contributing members of the Health Systems Science workgroup, as well as all participants at the American Medical Association’s (AMA’s) Accelerating Change in Medical Education (ACE) consortium meeting in Portland, Oregon (April 2015).

References

1. Skochelak SE. Commentary: A century of progress in medical education: What about the next 10 years? Acad Med. 2010;85:197200.
2. Crosson FJ, Leu J, Roemer BM, Ross MN. Gaps in residency training should be addressed to better prepare doctors for a twenty-first-century delivery system. Health Aff (Millwood). 2011;30:21422148.
3. Thibault GE. Reforming health professions education will require culture change and closer ties between classroom and practice. Health Aff (Millwood). 2013;32:19281932.
4. Lucey CR. Medical education: Part of the problem and part of the solution. JAMA Intern Med. 2013;173:16391643.
5. Gonzalo JD, Haidet P, Papp KK, et al. Educating for the 21st-century health care system: An interdependent framework of basic, clinical, and systems sciences [published online ahead of print October 16, 2015]. Acad Med. doi: 10.1097/ACM.0000000000000951.
6. Berwick DM, Finkelstein JA. Preparing medical students for the continual improvement of health and health care: Abraham Flexner and the new “public interest.” Acad Med. 2010;85(9 suppl):S56S65.
7. Combes JR, Arespacochaga E. Physician competencies for a 21st century health care system. J Grad Med Educ. 2012;4:401405.
8. Jube N. The Albany Medical College patient navigator project: Medical students as patient navigators. Poster presented at: Association of American Medical Colleges 2012 OSR Annual Meeting; November 2–7, 2012; San Francisco, Calif. https://www.aamc.org/download/309968/data/2012postersessionsummaries.pdf. Accessed September 18, 2013.
9. Rennie W. First year medical students as EMTs: Skill building, confidence, and professional formation. Poster presented at: Association of American Medical Colleges Annual Meeting; November 2–7, 2012; San Francisco, Calif.
10. Gonzalo JD, Haidet P, Wolpaw DR. Authentic clinical experiences and depth in systems: Toward a 21st century curriculum. Med Educ. 2014;48:104105.
11. American Medical Association. Accelerating change in medical education. http://www.ama-assn.org/sub/accelerating-change/index.shtml. Accessed March 24, 2016.
12. Gonzalo JD, Dekhtyar M, Starr SR, et al. Health systems science curricula in undergraduate medical education: Identifying and defining a potential curricular framework. [published online ahead of print April 5, 2016]. Acad Med. doi:10.1097/ACM.0000000000001177.
13. Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. 2009.3rd ed. Thousand Oaks, Calif: Sage Publications.
14. Boyatzis RE. Transforming Qualitative Information: Thematic Analysis and Code Development. 1998.Thousand Oaks, Calif: Sage Publications.
15. Fraenkel JR, Wallen NE. How to Design and Evaluate Research in Education. 2009.7th ed. New York, NY: McGraw-Hill.
16. O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: A synthesis of recommendations. Acad Med. 2014;89:12451251.
17. Shenton A. Strategies for ensuring trustworthiness in qualitative research projects. Educ Inf. 2004;22:6375.
18. Cooke M, Irby DM, O’Brien BC; Carnegie Foundation for the Advancement of Teaching. Educating Physicians: A Call for Reform of Medical School and Residency. 2010.San Francisco, Calif: Jossey-Bass.
19. Lie DA, Boker J, Crandall S, et al. Revising the Tool for Assessing Cultural Competence Training (TACCT) for curriculum evaluation: Findings derived from seven US schools and expert consensus. Med Educ Online. 2008;13:111.
20. Satterfield JM, Carney PA. Kaplan RM, Spittel ML, David DH. Aligning medical education with the nation’s health priorities. In: Population Health: Behavioral and Social Science Insights. 2015. Rockville, Md: Agency for Healthcare Research and Quality; http://www.ahrq.gov/professionals/education/curriculum-tools/population-health/satterfield.html. Accessed April 6, 2016.
21. Association of American Medical Colleges. Core Entrustable Professional Activities for Entering Residency: Curriculum Developers’ Guide. 2014. Washington, DC: Association of American Medical Colleges; https://members.aamc.org/eweb/upload/Core%20EPA%20Curriculum%20Dev%20Guide.pdf. Accessed March 24, 2016.
22. Association of American Medical Colleges. Behavioral and Social Science Foundations For Future Physicians: Report of the Behavioral and Social Science Expert Panel. November 2011. Washington, DC: Association of American Medical Colleges.
23. Larsen DP, Butler AC, Roediger HL 3rd. Test-enhanced learning in medical education. Med Educ. 2008;42:959966.
24. van der Vleuten CP, Schuwirth LW. Assessing professional competence: From methods to programmes. Med Educ. 2005;39:309317.
25. Schuwirth L, Ash J. Assessing tomorrow’s learners: In competency-based education only a radically different holistic method of assessment will work. Six things we could forget. Med Teach. 2013;35:555559.
26. Baker DP, Salas E, King H, Battles J, Barach P. The role of teamwork in the professional education of physicians: Current status and assessment recommendations. Jt Comm J Qual Patient Saf. 2005;31:185202.
27. Aboumatar HJ, Thompson D, Wu A, et al. Development and evaluation of a 3-day patient safety curriculum to advance knowledge, self-efficacy and system thinking among medical students. BMJ Qual Saf. 2012;21:416422.
28. Daud-Gallotti RM, Morinaga CV, Arlindo-Rodrigues M, Velasco IT, Martins MA, Tiberio IC. A new method for the assessment of patient safety competencies during a medical school clerkship using an objective structured clinical examination. Clinics (Sao Paulo). 2011;66:12091215.
29. Dory V, Gagnon R, De Foy T, Duyver C, Leconte S. A novel assessment of an evidence-based practice course using an authentic assignment. Med Teach. 2010;32:e65e70.
30. Hughes C, Toohey S, Velan G. eMed Teamwork: A self-moderating system to gather peer feedback for developing and assessing teamwork skills. Med Teach. 2008;30:59.
31. Meier AH, Boehler ML, McDowell CM, et al. A surgical simulation curriculum for senior medical students based on TeamSTEPPS. Arch Surg. 2012;147:761766.
32. Mookherjee S, Ranji S, Neeman N, Sehgal N. An advanced quality improvement and patient safety elective. Clin Teach. 2013;10:368373.
33. Olupeliyawa AM, O’Sullivan AJ, Hughes C, Balasooriya CD. The Teamwork Mini-Clinical Evaluation Exercise (T-MEX): A workplace-based assessment focusing on collaborative competencies in health care. Acad Med. 2014;89:359365.
34. Olupeliyawa A, Balasooriya C, Hughes C, O’Sullivan A. Educational impact of an assessment of medical students’ collaboration in health care teams. Med Educ. 2014;48:146156.
35. Paige JT, Garbee DD, Kozmenko V, et al. Getting a head start: High-fidelity, simulation-based operating room team training of interprofessional students. J Am Coll Surg. 2014;218:140149.
36. Sharma N, Cui Y, Leighton JP, White JS. Team-based assessment of medical students in a clinical clerkship is feasible and acceptable. Med Teach. 2012;34:555561.
37. Singh MK, Ogrinc G, Cox KR, et al. The Quality Improvement Knowledge Application Tool Revised (QIKAT-R). Acad Med. 2014;89:13861391.
38. Tartaglia KM, Walker C. Effectiveness of a quality improvement curriculum for medical students. Med Educ Online. 2015;20:27133.
39. White JS, Sharma N. “Who writes what?” Using written comments in team-based assessment to better understand medical student performance: A mixed-methods study. BMC Med Educ. 2012;12:123.
40. Wright MC, Segall N, Hobbs G, Phillips-Bute B, Maynard L, Taekman JM. Standardized assessment for evaluation of team skills: Validity and feasibility. Simul Healthc. 2013;8:292303.
41. Havyer RD, Wingo MT, Comfere NI, et al. Teamwork assessment in internal medicine: A systematic review of validity evidence and outcomes. J Gen Intern Med. 2014;29:894910.
42. Oates M, Davidson M. A critical appraisal of instruments to measure outcomes of interprofessional education. Med Educ. 2015;49:386398.
43. Gillan C, Lovrics E, Halpern E, Wiljer D, Harnett N. The evaluation of learner outcomes in interprofessional continuing education: A literature review and an analysis of survey instruments. Med Teach. 2011;33:e461e470.
44. Hirsh DA, Ogur B, Thibault GE, Cox M. “Continuity” as an organizing principle for clinical education reform. N Engl J Med. 2007;356:858866.
45. Accreditation Council for Graduate Medical Education. Outcomes Project. 1999. http://www.acgme.org/outcome/comp/compFull.asp. Accessed March 30, 2013. [No longer available.].
46. Accreditation Council for Graduate Medical Education; American Board of Internal Medicine. The Internal Medicine Milestone Project: A Joint Initiative of the Accreditation Council for Graduate Medical Education and the American Board of Internal Medicine. July 2015. Chicago, Ill: Accreditation Council for Graduate Medical Education; http://www.acgme.org/portals/0/pdfs/milestones/internalmedicinemilestones.pdf. Accessed April 6, 2016.
47. Hawkins RE, Welcher CM, Holmboe ES, et al. Implementation of competency-based medical education: Are we addressing the concerns and challenges? Med Educ. 2015;49:10861102.
48. Accreditation Council for Graduate Medical Education. Clinical Learning Environment Review Program. http://www.acgme.org/acgmeweb/tabid/436/Prog ramandInstitutionalAccreditation/NextAccreditationSystem/ClinicalLearningEnvironmentReviewProgram.aspx. Accessed March 24, 2016.
49. Clay MA 2nd, Sikon AL, Lypson ML, et al. Teaching while learning while practicing: Reframing faculty development for the patient-centered medical home. Acad Med. 2013;88:12151219.
50. Ludmerer KM. Time to Heal: American Medical Education From the Turn of the Century to the Era of Managed Care. 1999.Oxford, England: Oxford University Press.
51. Jones RF, Korn D. On the cost of educating a medical student. Acad Med. 1997;72:200210.
52. Shea S, Nickerson KG, Tenenbaum J, et al. Compensation to a department of medicine and its faculty members for the teaching of medical students and house staff. N Engl J Med. 1996;334:162167.
53. Baldor RA, Brooks WB, Warfield ME, O’Shea K. A survey of primary care physicians’ perceptions and needs regarding the precepting of medical students in their offices. Med Educ. 2001;35:789795.
54. Chandra A, Khullar D, Wilensky GR. The economics of graduate medical education. N Engl J Med. 2014;370:23572360.
55. Wynn BO, Smalley R, Cordasco KM; Rand Corporation; Medicare Payment Advisory Commission. Does It Cost More to Train Residents or to Replace Them? A Look at the Costs and Benefits of Operating Graduate Medical Education Programs. 2013.Santa Monica, Calif: Rand Corporation.
56. Clancy GP. Good neighbors: Shared challenges and solutions toward increasing value at academic medical centers and universities. Acad Med. 2015;90:16071610.
57. Walsh K. Oxford Textbook of Medical Education. 2013.Oxford, England: Oxford University Press.
58. Lin SY, Schillinger E, Irby DM. Value-added medical education: Engaging future doctors to transform health care delivery today. J Gen Intern Med. 2015;30:150151.
59. Gonzalo JD, Thompson B. Value-added student roles that align education and health system needs. Presented at: IAMSE Webinar Series; September 10, 2015; Phoenix, Ariz.
60. Gonzalo JD, Graaf D, Johannes B, Blatt B, Wolpaw DR. Adding value to the healthcare system: Identifying value-added systems roles for medical students. [published online ahead of print April 26, 2016]. Am J Med Quality. doi:10.1177/1062860616645401.
61. Hafferty FW. Beyond curriculum reform: Confronting medicine’s hidden curriculum. Acad Med. 1998;73:403407.
62. Michalec B, Hafferty FW. Stunting professionalism: The potency and durability of the hidden curriculum within medical education. Soc Theory Health. 2013;11:388406.
63. Karnieli-Miller O, Vu TR, Frankel RM, et al. Which experiences in the hidden curriculum teach students about professionalism? Acad Med. 2011;86:369377.
64. Higashi RT, Tillack A, Steinman MA, Johnston CB, Harper GM. The “worthy” patient: Rethinking the “hidden curriculum” in medical education. Anthropol Med. 2013;20:1323.
65. Moriates C, Arora V, Shah N. Understanding Value-Based Healthcare. 2015.New York, NY: McGraw-Hill.
66. Hafferty FW, O’Donnell JF. The Hidden Curriculum in Health Professional Education. 2014.Hanover, NH: Dartmouth College Press.
67. Hafler JP, Ownby AR, Thompson BM, et al. Decoding the learning environment of medical education: A hidden curriculum perspective for faculty development. Acad Med. 2011;86:440444.
68. Patel MS, Lypson ML, Davis MM. Medical student perceptions of education in health care systems. Acad Med. 2009;84:13011306.
69. Brooks KC. A piece of my mind. A silent curriculum. JAMA. 2015;313:19091910.
70. Garvey KC, Kesselheim JC, Herrick DB, Woolf AD, Leichtner AM. Graduate medical education in humanism and professionalism: A needs assessment survey of pediatric gastroenterology fellows. J Pediatr Gastroenterol Nutr. 2014;58:3437.
71. Gonzalo JD, Haidet P, Blatt B, Wolpaw DR. Exploring challenges in implementing a health systems science curriculum: A qualitative analysis of student perceptions. Med Educ. 2016;50:523531.
72. Chen C, Petterson S, Phillips R, Bazemore A, Mullan F. Spending patterns in region of residency training and subsequent expenditures for care provided by practicing physicians for Medicare beneficiaries. JAMA. 2014;312:23852393.
73. Sirovich BE, Lipner RS, Johnston M, Holmboe ES. The association between residency training and internists’ ability to practice conservatively. JAMA Intern Med. 2014;174:16401648.
74. Skochelak SE, Stansfield RB, Dunham L, et al. Medical student perceptions of the learning environment at the end of the first year: A 28-medical school collaborative [published online ahead of print March 8, 2016]. Acad Med. doi:10.1097/ACM.0000000000001137.
Copyright © 2016 by the Association of American Medical Colleges