The Lancet Global Commission on Education of Health Professionals for the 21st Century report outlined a vision for medical education in the 21st century and reinforced the primary outcomes for competency-based medical education.1 Importantly, this report emphasized the improvement of health and health care outcomes as the ultimate mission of all medical education programs—from undergraduate medical education, to graduate medical education (GME), to continuing medical education and faculty development. As such, there is an urgent need to align educational and clinical outcomes in GME and refocus attention on health and health care outcomes to prepare physicians for 21st-century health care practice and ongoing learning and development.
Despite an increased focus on improving patient safety and quality of care over the past decade, recent estimates still suggest that the current health care delivery model fails to achieve the desired outcomes for our patients.2 Preventable adverse event rates in hospitals do not appear to have decreased.3,4 Patients receive the basic elements of health care inconsistently.5 And perhaps most concerning, more Americans suffer from poor health and are dying younger when compared with other developed countries,6 despite the United States spending the highest proportion of the nation’s gross domestic product to fund health care.7,8
Until recently, educational leaders and regulators have tried to address this concern internationally by focusing their efforts on developing competency frameworks for the individual physician, such as the Accreditation Council for Graduate Medical Education (ACGME) Core Competencies,9 CanMEDS,10 and the Good Medical Practice frameworks.11 To a variable degree, each of these frameworks includes newer competencies needed by physicians to deliver high-quality, safe, patient-centered care, such as care coordination, patient safety, quality improvement, stewardship and cost consciousness, interprofessional teamwork, shared decision making, and use of health information technology and data (List 1).12,13 These frameworks also explicitly recognize that these competencies had previously been missing from physician training—meaning, by extension, that most of today’s academic faculty were not trained in these competencies.
List 1 Examples of Newer Competencies Needed by Physicians to Deliver High-Quality, Safe, Patient-Centered Care Cited Here
- Care coordination
- Patient safety
- Quality improvement
- Stewardship and cost consciousness
- Interprofessional teamwork
- Shared decision making
- Use of health information technology and data
Unfortunately, GME programs have faced challenges teaching and assessing these newer competencies, so even recent physician graduates may lack the full complement of skills necessary to address patient and health care system needs.13,14 The resulting widespread concern has led to the proposed Graduate Medical Education Reform Act, which links GME program funding to its ability to meet specific performance standards,15 as well as to the recent publication of the Institute of Medicine report Graduate Medical Education That Meets the Nation’s Health Needs, which concluded that there is “an unquestionable imperative to assess and optimize the effectiveness of the public’s investment in GME.”16
What has been missing from these physician-based competency frameworks is an explicit acknowledgment of the role and influence of the clinical learning environment, embedded within the institutional context and culture, where the finishing phases of physicians’ professional development occur. To this end, the ACGME, through its Clinical Learning Environment Review (CLER) program,14 now focuses on several dysfunctional aspects of the clinical learning environment.17–19
But is there support for the belief that what trainees experience with regard to quality, safety, and value will affect how they ultimately practice? Asch and colleagues20 found that the major obstetrical complication rates of practicing obstetricians in two U.S. states were correlated with the complication rates of the institutions where the physicians had trained. The relative risk difference between the top and bottom quintile was 32%, representing substantial and real harm to patients. Similarly, Chen and colleagues found that health care cost patterns within a hospital referral region were associated with the future cost patterns of physicians who had completed their residency within that hospital referral region.21 These examples demonstrate that imprinting can start early in training and can persist long into practice.
In this Perspective, we explore the implications of the insufficient inte gration of education about patient safety and quality improvement by academic faculty into the clinical learning environment in many GME programs, and the important role that academic faculty need to play to better align the educational and clinical contexts to improve both learner and patient outcomes.
Current Academic Faculty Perspective on GME
What, then, are the implications of aligning the educational and clinical contexts for academic faculty, especially those working as frontline supervisors in the clinical setting? Unfortunately, the traditional faculty perspective and the current educational design of GME programs often center educational outcomes around learners and clinical outcomes around patients (Figure 1). As a result, academic faculty may be less inclined to align these two types of outcomes in the context of their roles as teachers and care providers. This concern is supported by the fact that academic faculty insufficiently integrate learning about patient safety and quality improvement into clinical training.19 For example, the most recent update from the CLER program revealed recurring instances of
learning environment[s] in which faculty and staff generally lacked the ability to recognize instances of compromised patient safety, in which policies and mechanisms for reporting such problems were unclear, and in which there was no process for analyzing such events and preventing them in the future.19
Furthermore, most academic faculty, in their role as clinical supervisors, tend to focus on learner–patient interactions and much less on learner–patient–clinical microsystem interactions (Figure 1). And yet, the clinical microsystem is the “local milieu in which patients, providers, support staff, information, and processes converge for the purpose of providing care to meet health needs,”22 and it provides a useful framework to organize, measure, and improve the delivery of care. Many academic faculty lack a basic awareness of clinical microsystems science, theories, and principles, and of the important role that clinical microsystems play in influencing both educational and clinical outcomes.22
However, learning in the context of the clinical microsystem (often referred to as learning through the informal curriculum23), when properly and intentionally designed, is where important education about patient safety and quality improvement should take place.24 Take, for example, hospital discharge as a critical systems-based competency, which notably involves care coordination and interprofessional teamwork. Many of the residents we work with view completing discharge summaries and scheduling follow-up tests and appointments as having little educational value and frequently refer to these as scut work. Yet, many tests ordered in the hospital prior to discharge require urgent action and are left without proper follow-up.25 Additionally, hospital readmissions have garnered such widespread attention as a marker of care quality that the Centers for Medicare and Medicaid Services now tie hospital funding directly to a hospital’s ability to reduce its readmission rates.26 In this example, when academic faculty do not frame work carried out by residents at the time of hospital discharge as having an educational value around key system-based competencies, they lose an opportunity to properly prepare residents to improve care transitions for patients and learn core coordination skills needed for future practice.
Add to this the fact that most residency training programs still rely on a rotational model, whereby residents parachute in and out of the inpatient ward or ambulatory clinic on the basis of their rotational schedule and are not entrenched in the clinical microsystem,27 and one can easily see why a misalignment between educational and clinical contexts exists. Yet, optimal patient safety and quality improvement outcomes require the full engagement of all academic faculty to improve both educational processes and systems performance.28 Therefore, academic faculty need to expand their focus within the educational context beyond learner needs and training program goals, and consider how these align with patient needs within the clinical context.
Transforming the Academic Faculty Perspective in GME
Ensuring that faculty achieve a newer set of patient-centered competencies (List 1) so that they can support the integration of patient safety and quality improvement into clinical learning environments will require major investments in continuing medical education and faculty development.29 The Association of American Medical Colleges Teaching for Quality (Te4Q) initiative recently summarized many of these faculty competencies in its expert panel report, calling out the need for all faculty to gain proficiency across these domains (e.g., demonstrating systems and team leadership in a manner which creates educational experiences in systems improvement, and recognizing that quality patient care only occurs in the context of the interprofessional team).30 Although traditional faculty development models effectively improve teaching performance, most are workshops, seminars, or short courses delivered to faculty participants outside of the clinical setting.31 Furthermore, the majority of faculty development programs focus primarily on more concrete teaching skills, like delivering a lecture or providing feedback, and less on the integrative skills that academic faculty now require to acknowledge, and ultimately to enhance, the bidirectional nature of learner and patient outcomes.31
National efforts are now under way to enhance faculty skills in patient safety and quality improvement.30,32 A train-the-trainer approach has shown early promise in building faculty capacity for teaching patient safety even among more senior faculty members.33 Some institutions, such as the Mayo Clinic with their bronze-, silver-, and gold-level quality improvement certification system, make basic quality improvement training mandatory for all staff.34 Others advocate for the integration of quality improvement content in traditional continuing medical education activities as a strategy for ensuring broad exposure to basic quality improvement concepts and tools.35
Although the emergence of these faculty development efforts is encouraging, they may not sufficiently address our urgent need to rapidly increase faculty competence with respect to integrating patient safety and quality improvement into both teaching and clinical activities. One potential approach to rapidly scale up the integration of patient safety and quality improvement into faculty’s teaching and clinical activities would be to treat faculty and trainees as co-learners (see below). The truth is that many of our trainees are at least on equal footing with, if not further ahead than, many of our academic faculty when it comes to patient safety and quality improvement.36–38
A co-learning approach, which involves educating faculty alongside trainees with the same curriculum, offers a number of potential advantages over other more traditional approaches to faculty development. First, programs do not require a critical mass of experts to get started. A single expert can lead a group of faculty and trainee learners to start building capacity from within a program. Second, because the faculty (who are typically the trainers in a traditional train-the-trainer model) do not receive their training in a separate time and space from trainees, programs do not need to develop expertise among faculty trainers before trainees can be taught, avoiding the potential delays in translating knowledge and skills to the “end-learner” that are commonly associated with other forms of faculty development. Perhaps most appealing is that this model can naturally bring the faculty closer to the learner, patient, and clinical microsystem and might work toward promoting the integration of educational and clinical outcomes. However, to achieve this level of integration, co-learning must occur in the context of patient care.
Patient safety morbidity and mortality rounds (M&Ms) offer a potential venue for faculty–resident co-learning to occur around systems improvement. From an educational perspective, many training programs use M&Ms as a way to teach system-based practice competencies, such as systems thinking and adverse event analysis.39 Discussions between faculty and residents to deconstruct patient safety incidents allow faculty to draw on their clinical expertise and role model a just and fair approach to handling patient safety incidents, but also offer faculty opportunities for personal learning about systems improvement. The fact that M&Ms link back to inform local quality improvement efforts demonstrates the advantage of linking educational and systems improvement efforts.40,41
Some programs are now exploring the use of faculty–trainee co-learning. For example, the American Board of Internal Medicine developed practice improvement modules as a way of promoting practice audit and self-reflection to engage practicing physicians in testing small-scale quality improvements.42 Although initially designed as a tool for continuing medical education, some programs have taken to engaging faculty and residents together to collaboratively complete a practice improvement module. And at the University of Toronto, one of us has implemented a faculty–resident co-learning quality improvement curriculum, whereby faculty supervisors, program directors, and residents attend face-to-face interactive workshops and work together on team-based quality improvement projects.43
Refocusing GME on the Patient
We propose a framework that closely aligns the educational and clinical contexts, such that both educational and clinical outcomes are centered around the patient. Making the patient the subject of the learning, rather than the object, requires a reorganization of academic faculty perspective and educational design of GME training programs that recognizes that (1) the dynamic interplay between the faculty, learner, training program, and clinical microsystem ultimately influences the quality of physician that emerges from the training program and the environment, and (2) patient outcomes relate to the quality of education and the success of clinical microsystems (see Figure 2).
How might academic faculty approach their clinical supervision differently within this reimagined organization of GME? Ideally, academic faculty would role model a commitment to align educational and clinical outcomes (see Table 1) and reframe some patient care activities typically viewed as scut work as important educational opportunities. So, in our prior example of hospital discharge, faculty need to emphasize that efforts to ensure a safe, reliable discharge are important from the patient’s perspective. Faculty members could observe a discharge conversation and review discharge summaries prepared by residents and provide them with feedback. Or faculty could emphasize the importance of confirming follow-up appointments prior to discharge as a strategy for reducing avoidable hospital readmissions. How might faculty track how well trainees discharge patients? One option would be to use patient-reported outcomes to measure the patient’s satisfaction with the discharge process, which would reflect the effectiveness of both the educational and the clinical process. The Care Transition Measure, a validated assessment method developed by Coleman and colleagues44 to measure quality of care from the patient’s perspective, could be used concomitantly as an educational assessment that provides learners with feedback.
The organization of residency educa tional programs and curricula would also need to change within the reimagined organization of GME. The majority of internal medicine and pediatric training programs rely heavily on rotations and the inpatient setting despite evidence of the dysfunction of frequent rotations17 and the fact that the vast majority of patients are now cared for in outpatient settings.16 The rotational model impedes learners’ meaningful integration into the clinical microsystem, as seen in the early findings of the CLER program, which showed that the majority of residents and fellows are still not meaningfully integrated into hospitals’ patient safety and quality improvement initiatives.14
Although there are clearly major barriers to moving away from the rotational model, such as scheduling challenges, there are some smaller-scale changes that can achieve better learner–patient–clinical microsystem integration. For example, the Beth Israel Deaconess Medical Center’s internal medicine residency program reorganized the clinical care delivery model and inpatient medical teaching service to assign faculty members and residents to geographically localized clinical microsystems in an effort to more effectively integrate patient safety and quality improvement principles into the everyday clinical practice of all residents.45
There are some other intriguing exam ples where residency programs have reimagined the relationship between the faculty, learner, training program, and clinical microsystem when designing and delivering educational activities. The White River Junction VA Medical Center, a Veterans Affairs hospital affiliated with the Geisel School of Medicine at Dartmouth, integrated systems improvement into the daily work of the internal medicine clinical teaching unit by assigning residents responsibility over both a group of admitted patients as well as a ward-based quality improvement initiative.46 Ogrinc and colleagues,46 who reported on this system, cited important educational system factors (e.g., quality improvement whiteboard, expert quality improvement faculty, access to performance data) that influenced the success of the integrated quality improvement experience.
Similarly, the University of Cincinnati Department of Medicine completely reorganized the ambulatory experience in their internal medicine residency program using the Chronic Care Model as a guiding framework.47 Residents participated in interprofessional team quality improvement meetings, received quarterly performance measures that reported their personal performance benchmarked relative to other residents in the clinic, and actively engaged in clinic-based quality improvement activities. In both of these examples, integrating and embedding the learners and training program (i.e., the general medicine teaching service) into the clinical microsystem resulted in the broad engagement of residents in quality improvement activities directed at clinically relevant problems and led to important changes that benefited learners, patients, and systems alike.
There is also a need for the alignment of educational and clinical outcomes, such that training programs aim to achieve improvements for both learners and patients alike, while the clinical microsystem embraces the learner and nurtures his/her professional development in its care delivery models. This is not a new conceptualization of educational outcomes. The Kirkpatrick model, one of the most recognized frameworks for program evaluation, emphasizes clinical outcomes as the highest level of impact attainable by educational interventions.48 Yet, few educational programs are able to successfully deliver on improving patient outcomes.49,50 Engaging trainees in quality improvement, a natural bridge between education and clinical care, may enable the alignment of the training program and clinical microsystem that is needed to bring about improvements in both educational and clinical outcomes.
Are there any encouraging examples to suggest that such an alignment of educational and clinical outcomes is possible? Recently, a group of nine pediatric hospitals implemented the I-PASS (Initiative for Innovation in Pediatric Education–Pediatric Research in Inpatient Settings Accelerating Safe Sign-outs) handoff bundle with the dual goal of improving resident team communication skills and reducing medical errors and preventable adverse events.51,52 Critical to the intervention was the alignment of the training program (e.g., structured teamwork training) and clinical microsystem (e.g., providing dedicated space for handoffs and modifying the electronic medical record to embed a computerized signout tool that included specific fields, such as “To Do List” and “Contingency Planning,” to prompt use of the I-PASS framework). More important, there was explicit oversight by faculty, who directly observed at least one handoff encounter per month to provide direct feedback. The result? A significant reduction in medical errors and preventable adverse events without any increase in resident workload or time spent performing handoffs.52
We have an urgent need to engage academic faculty in patient safety and quality improvement practices to support the GME mandate to create competency-based training programs that focus on patient-centered outcomes. Professional development programs are urgently needed to help faculty acquire newer patient-centered competencies they were not explicitly exposed to during their own training. But this will also require us to look at how we might reorganize the way in which we deliver and structure physician training within the clinical learning environment (i.e., clinical microsystem). Beyond our training programs demonstrating a return on investment to the financial sponsors of GME, our patients expect and deserve a health care system that delivers high-quality, safe, and appropriate care, and training programs that produce physicians that are ready, able, and willing to support, nurture, and continuously improve this system. To enable this evolution, we will need to revisit the core competencies expected of academic faculty, implement innovative faculty development strategies, examine closely faculty’s current clinical supervision practices, and establish a training environment that supports bridging from clinician to educator, training program to clinical microsystem, and educational outcomes to clinical outcomes that benefit our patients. Placing the patient at the center will remind us of this urgent need and guide our efforts.
Acknowledgments: The authors thank Wendy Levinson, MD, and Kevin Imrie, MD, for their helpful comments on an earlier draft of this manuscript.
1. Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376:1923–1958
2. Wachter RM.. Patient safety at ten: Unmistakable progress, troubling gaps. Health Aff (Millwood). 2010;29:165–173
3. Classen DC, Resar R, Griffin F, et al. “Global trigger tool” shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood). 2011;30:581–589
4. Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ.. Temporal trends in rates of patient harm resulting from medical care. N Engl J Med. 2010;363:2124–2134
5. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–2645
6. Woolf SH, Aron LYNational Academies Panel on Understanding Cross-National Health Differences Among High-Income Countries, Institute of Medicine Board on Population Health and Public Health Practice. U.S. Health in International Perspective: Shorter Lives, Poorer Health. 2013 Washington, DC National Academies Press
7. Berwick DM, Hackbarth AD.. Eliminating waste in US health care. JAMA. 2012;307:1513–1516
8. Keehan SP, Sisko AM, Truffer CJ, et al. National health spending projections through 2020: Economic recovery and reform drive faster spending growth. Health Aff (Millwood). 2011;30:1594–1605
9. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S.. General competencies and accreditation in graduate medical education. Health Aff (Millwood). 2002;21:103–111
10. Frank JR, Danoff D.. The CanMEDS initiative: Implementing an outcomes-based framework of physician competencies. Med Teach. 2007;29:642–647
11. General Medical Council. Good Medical Practice. http://www.gmc-uk.org/guidance/good_medical_practice.asp
. Accessed September 24, 2015
12. Medicare Payment Advisory Commission. Chapter 4: Graduate medical education financing: Focusing on educational priorities. June 2010 http://www.medpac.gov/documents/reports/Jun10_Ch04.pdf
. Accessed September 24, 2015
13. Crosson FJ, Leu J, Roemer BM, Ross MN.. Gaps in residency training should be addressed to better prepare doctors for a twenty-first-century delivery system. Health Aff (Millwood). 2011;30:2142–2148
14. Weiss KB, Bagian JP, Nasca TJ.. The clinical learning environment: The foundation of graduate medical education. JAMA. 2013;309:1687–1688
15. Graduate Medical Education Reform Act of 2012, S 3201, 112th Cong, 2nd Sess. (2012)
16. Institute of Medicine. Graduate Medical Education That Meets the Nation’s Health Needs. 2014 Washington, DC National Academies Press
17. Bernabeo EC, Holtman MC, Ginsburg S, Rosenbaum JR, Holmboe ES.. Lost in transition: The experience and impact of frequent changes in the inpatient learning environment. Acad Med. 2011;86:591–598
18. Hafferty FW, Franks R.. The hidden curriculum, ethics teaching, and the structure of medical education. Acad Med. 1994;69:861–871
19. Nasca TJ, Weiss KB, Bagian JP.. Improving clinical learning environments for tomorrow’s physicians. N Engl J Med. 2014;370:991–993
20. Asch DA, Nicholson S, Srinivas S, Herrin J, Epstein AJ.. Evaluating obstetrical residency programs using patient outcomes. JAMA. 2009;302:1277–1283
21. Chen C, Petterson S, Phillips R, Bazemore A, Mullan F.. Spending patterns in region of residency training and subsequent expenditures for care provided by practicing physicians for Medicare beneficiaries. JAMA. 2014;312:2385–2393
22. Nelson EC, Batalden PB, Huber TP, et al. Microsystems in health care: Part 1. Learning from high-performing front-line clinical units. Jt Comm J Qual Improv. 2002;28:472–493
23. Hafferty FW.. Beyond curriculum reform: Confronting medicine’s hidden curriculum. Acad Med. 1998;73:403–407
24. Kesselheim JC, Cassel CK.. Service: An essential component of graduate medical education. N Engl J Med. 2013;368:500–501
25. Roy CL, Poon EG, Karson AS, et al. Patient safety concerns arising from test results that return after hospital discharge. Ann Intern Med. 2005;143:121–128
26. Berenson RA, Paulus RA, Kalman NS.. Medicare’s readmissions-reduction program—A positive alternative. N Engl J Med. 2012;366:1364–1366
27. Holmboe E, Ginsburg S, Bernabeo E.. The rotational approach to medical education: Time to confront our assumptions? Med Educ. 2011;45:69–80
28. Batalden PB, Davidoff F.. What is “quality improvement” and how can it transform healthcare? Qual Saf Health Care. 2007;16:2–3
29. Cook DA, Holmboe ES, Sorensen KJ, Berger RA, Wilkinson JM.. Getting maintenance of certification to work: A grounded theory study of physicians’ perceptions. JAMA Intern Med. 2015;175:35–42
30. Headrick LA, Baron RB, Pingleton SK, et al. Teaching for Quality: Integrating Quality Improvement and Patient Safety Across the Continuum of Medical Education: Report of an Expert Panel. 2013 Washington, DC Association of American Medical Colleges
31. Steinert Y, Mann K, Centeno A, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME guide no. 8. Med Teach. 2006;28:497–526
32. Royal College of Physicians and Surgeons of Canada. ASPIRE: Advancing Safety for Patients in Residency Education. http://www.royalcollege.ca/portal/page/portal/rc/events/aspire
. Accessed September 24, 2015
33. Ahmed M, Arora S, Baker P, Hayden J, Vincent C, Sevdalis N.. Building capacity and capability for patient safety education: A train-the-trainers programme for senior doctors. BMJ Qual Saf. 2013;22:618–625
34. Majka AJ, Cook KE, Lynch SL, et al. Teaching quality essentials: The effectiveness of a team-based quality improvement curriculum in a tertiary health care institution. Am J Med Qual. 2013;28:214–219
35. Shojania KG, Silver I, Levinson W.. Continuing medical education and quality improvement: A match made in heaven? Ann Intern Med. 2012;156:305–308
36. Charlson ME, Karnik J, Wong M, McCulloch CE, Hollenberg JP.. Does experience matter? A comparison of the practice of attendings and residents. J Gen Intern Med. 2005;20:497–503
37. Hess BJ, Johnston MM, Lynn LA, Conforti LN, Holmboe ES.. Development of an instrument to evaluate residents’ confidence in quality improvement. Jt Comm J Qual Patient Saf. 2013;39:502–510
38. Holmboe ES, Hess BJ, Conforti LN, Lynn LA.. Comparative trial of a Web-based tool to improve the quality of care provided to older adults in residency clinics: Modest success and a tough road ahead. Acad Med. 2012;87:627–634
39. Gonzalo JD, Yang JJ, Huang GC.. Systems-based content in medical morbidity and mortality conferences: A decade of change. J Grad Med Educ. 2012;4:438–444
40. Calder LA, Kwok ES, Adam Cwinn A, et al. Enhancing the quality of morbidity and mortality rounds: The Ottawa M&M model. Acad Emerg Med. 2014;21:314–321
41. McDonnell C, Laxer RM, Roy WL.. Redesigning a morbidity and mortality program in a university-affiliated pediatric anesthesia department. Jt Comm J Qual Patient Saf. 2010;36:117–125
42. Bernabeo E, Hood S, Iobst W, Holmboe E, Caverzagie K.. Optimizing the implementation of practice improvement modules in training: Lessons from educators. J Grad Med Educ. 2013;5:74–80
43. Wong BM, Goguen J, Shojania KG.. Building capacity for quality: A pilot co-learning curriculum in quality improvement for faculty and resident learners. J Grad Med Educ. 2013;5:689–693
44. Coleman EA, Smith JD, Frank JC, Eilertsen TB, Thiare JN, Kramer AM.. Development and testing of a measure designed to assess the quality of care transitions. Int J Integr Care. 2002;2:e02
45. Tess AV, Yang JJ, Smith CC, Fawcett CM, Bates CK, Reynolds EE.. Combining clinical microsystems and an experiential quality improvement curriculum to improve residency education in internal medicine. Acad Med. 2009;84:326–334
46. Ogrinc G, Ercolano E, Cohen ES, et al. Educational system factors that engage resident physicians in an integrated quality improvement curriculum at a VA hospital: A realist evaluation. Acad Med. 2014;89:1380–1385
47. Warm EJ, Schauer DP, Diers T, et al. The ambulatory long-block: An Accreditation Council for Graduate Medical Education (ACGME) Educational Innovations Project (EIP). J Gen Intern Med. 2008;23:921–926
48. Kirkpatrick DL. Evaluating Training Programs: The Four Levels. 1994 San Francisco, Calif Berrett-Koehler
49. Yardley S, Dornan T.. Kirkpatrick’s levels and education “evidence.” Med Educ. 2012;46:97–106
50. Patow CA, Karpovich K, Riesenberg LA, et al. Residents’ engagement in quality improvement: A systematic review of the literature. Acad Med. 2009;84:1757–1764
51. Starmer AJ, O’Toole JK, Rosenbluth G, et al.I-PASS Study Education Executive Committee. Development, implementation, and dissemination of the I-PASS handoff curriculum: A multisite educational intervention to improve patient handoffs. Acad Med. 2014;89:876–884
52. Starmer AJ, Spector ND, Srivastava R, et al.I-PASS Study Group. Changes in medical errors after implementation of a handoff program. N Engl J Med. 2014;371:1803–1812