The Development of Entrustable Professional Activities for Internal Medicine Residency Training: A Report From the Education Redesign Committee of the Alliance for Academic Internal Medicine : Academic Medicine

Secondary Logo

Journal Logo

Advanced Search
Research Reports

The Development of Entrustable Professional Activities for Internal Medicine Residency Training

A Report From the Education Redesign Committee of the Alliance for Academic Internal Medicine

Caverzagie, Kelly J. MD; Cooney, Thomas G. MD; Hemmer, Paul A. MD, MPH; Berkowitz, Lee MD

Author Information

Dr. Caverzagie is associate professor and associate dean for educational strategy, College of Medicine, University of Nebraska Medical Center, Omaha, Nebraska.

Dr. Cooney is professor and vice chair for educational programs, Department of Medicine, Oregon Health & Science University, Portland, Oregon.

Dr. Hemmer is professor and vice chair for educational programs, Department of Medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland.

Dr. Berkowitz is professor and associate chair for education, Department of Medicine, University of North Carolina School of Medicine, Chapel Hill, North Carolina.

Funding/Support: The AAIM covered the travel costs related to in-person meetings of the AAIM Education Redesign Committee.

Other disclosures: None reported.

Ethical approval: Reported as not applicable.

Disclaimer: The views expressed in this article are those of the authors and do not necessarily represent the views of the Uniformed Services University of the Health Sciences, the Department of Defense, or other federal agencies.

Previous presentations: The AAIM End-of-Training entrustable professional activities were presented as part of a workshop at the Association of Program Directors of Internal Medicine meeting on Friday, October 12, 2012, and made available on the AAIM Web site following that meeting.

Correspondence should be addressed to Dr. Caverzagie, 986430 Nebraska Medical Center, Omaha, NE 68198-6430; telephone: (402) 559-7502; e-mail: [email protected].

Academic Medicine 90(4):p 479-484, April 2015. | DOI: 10.1097/ACM.0000000000000564
  • Free

Abstract

With the implementation of the Next Accreditation System (NAS),1 the Accreditation Council for Graduate Medical Education (ACGME) will require graduate medical education (GME) training programs to report residents’ outcomes using developmentally based milestones.2,3 To achieve this goal, many program directors will need to restructure their assessment systems to meet new challenges.4–7 In this article, we describe the work of the Education Redesign Committee of the Alliance for Academic Internal Medicine (AAIM), including the development of entrustable professional activities (EPAs)8,9 for internal medicine training and strategies for assessing those EPAs, to help facilitate this transition for internal medicine program directors, faculty, and residents.

Although not new, competency-based medical education (CBME) has gained favor in recent decades.10–13 Frank and colleagues12 defined CBME as “an outcomes-based approach to the design, implementation, assessment, and evaluation of a medical education program using an organizing framework of competencies.” Embedded within CBME is the concept of competence, defined as:

The array of abilities across multiple domains or aspects of physician performance in a certain context. Statements about competence require descriptive qualifiers to define the relevant abilities, context, and stage of training. Competence is multidimensional and dynamic. It changes with time, experience, and setting.12

Put simply, competence is synthetic (it requires the bringing together, simultaneously, of multiple abilities) and contextual (it varies with, and is influenced by, circumstances, settings, and culture and requires one to know what to bring to a given situation).14,15 For example, to demonstrate competence in leading and working within an interprofessional health care team, a physician must demonstrate skills in interpersonal communication, a professional attitude, and an intimate knowledge of the roles and responsibilities of individuals and structures within the local health care delivery system (i.e., the synthetic aspects of competence) and variably apply those abilities to meet the needs of a homeless male with recurrent substance abuse problems in addition to an elderly female who has recurrent falls (i.e., the contextual aspects of competence).

In 1999, the ACGME embraced an outcomes-based approach to medical education with the release of its six core competencies: patient care, medical knowledge, interpersonal communication and skills, professionalism, practice-based learning and improvement, and systems-based practice. We see similar trends worldwide, including in Canada with the development of the CanMEDS framework16 and in the United Kingdom with the creation of Tomorrow’s Doctors.17 In 2002, the ACGME required all GME training programs to use the core competencies as their framework for the assessment of residents in the United States.

By 2005, the ACGME recognized that the core competencies framework had a number of limitations1: Some concepts were abstract and thus not only difficult to teach but also difficult to evaluate; the analytic nature of the competencies framework implied an independence among the competencies that was difficult to establish18,19; and the framework did not reflect the synthetic nature of competence and the activities of physicians caring for patients.20 Thus, a revision to the framework was in order.1,21

Starting in 2007, leaders in the internal medicine community, in conjunction with representatives of the ACGME and the American Board of Internal Medicine (ABIM), began to develop a set of milestones.22 The goal of this effort was to “explicate the six ACGME general competencies by describing a developmental progression of observable behaviors.”22 Using the Dreyfus developmental framework (novice, advanced beginner, competent, proficient, expert, master),23 a task force developed a set of 142 discrete and observable behaviors that residents should demonstrate as they progress through training. The task force organized the internal medicine milestones within the ACGME competencies framework to give more practical and clinical relevance to the six core competencies.

Although the internal medicine milestones provide a detailed roadmap describing “success” during residency training, the level of detail was, admittedly, daunting. The milestones also fell prey to what researchers24 have warned is a potential risk of using competency frameworks—that their use chances “atomizing competencies, increasing bureaucracy, and moving away from expert opinion and from what really matters in day-to-day clinical practice.”25 Another challenge in using the internal medicine milestones has been to demonstrate that the assessment of one or more discrete behaviors actually equates to competence.18,20,25 Likened to a medical student who can successfully complete all items on a history and physical examination checklist yet is unable to identify the central clinical problems, the successful completion of a detailed operational checklist may not equate to competence. The 142 internal medicine milestones in and of themselves do not provide the context that is vitally important to a work-based assessment of competence.26–28 Finally, faculty are key in CBME because of their relationship with residents and their role in direct observation and in the overall assessment process.29,30 As such, the internal medicine milestones risk deemphasizing the wisdom of faculty in reaching synthetic judgments based on their daily interactions with residents, a feature that is the hallmark of GME training.

In 2011, the ACGME began to clarify how milestones would be incorporated into the NAS reporting, requiring the consolidation and revision of the existing internal medicine milestones into a structure amenable to reporting the educational outcomes of individual residents. Accordingly, the Reporting Milestones for internal medicine, which are based on, but distinct from, the original 142 internal medicine milestones (which are now called the Curricular Milestones),3 were published in 2013 and became a required part of the NAS reporting for internal medicine programs in June 2014.2

Method

The AAIM is a consortium of five academically focused specialty organizations, including the Association of Program Directors of Internal Medicine (APDIM), representing academic internal medicine departments at medical schools and teaching hospitals in the United States and Canada. The mission of the AAIM is to provide leadership and direction for these departments in education, research, and patient care. Since 2006, the AAIM has supported an Education Redesign Task Force that has provided guidance to members regarding critical issues in medical education.31,32 Given the rapidly evolving landscape and context of GME, the AAIM Board of Directors in 2011 identified education redesign as its top policy priority and elevated the Education Redesign Task Force to that of a standing committee within the organization. Since this restructuring, the committee has held in-person meetings and monthly conference calls to complete the work described in this article in addition to that related to other initiatives. All funding for this project was provided by the AAIM. We, the authors of this article, were all members of the Education Redesign Committee.

As part of our initial charge, we focused on furthering the evolution toward competency-based assessment by addressing the challenges that program directors face in using the ACGME core competencies and internal medicine milestones. We adopted the work-based assessment framework of EPAs as the foundation for our efforts.8,9,25,33–37 Our rationale for using EPAs was twofold: (1) EPAs identify those critical activities that, when taken together, constitute a specialty; and (2) EPAs are framed around recognizable professional activities that can be readily assessed by faculty. The concept of EPAs also recognizes both that activities based in patient care typically involve multiple “competencies” or observable behaviors simultaneously—that competence is synthetic and contextual—and that many activities may incorporate multiple ACGME core competency domains.8,27 Finally, faculty and residents are more likely to recognize the EPAs as readily familiar actions (e.g., lead and work within an interprofessional team)37 rather than as abstract concepts (e.g., systems-based practice). Using this background, we worked to establish a set of EPAs that would embrace the core of the internal medicine profession and serve as a framework for developing a system to assess internal medicine residents.

We used the defining attributes of an EPA to develop and define the internal medicine EPAs (see List 1).8 Recognizing that many varied interpretations and applications of the EPA framework exist, we decided to focus our work on the EPAs expected of a resident who is ready to enter unsupervised practice. In doing so, our goal was to provide a manageable and feasible starting point from which training programs could develop assessments and curricula. Accordingly, we did not intend to capture every possible entrustment decision that occurs over the course of residency training but, instead, to focus on those core activities that define the profession. We believed that, when considered together, the EPAs would constitute the core of the profession36; this belief was the guiding principle for our work.

List 1 Defining Attributes of Entrustable Professional Activities8 Cited Here

  1. Is part of essential professional work in a given context
  2. Must require adequate knowledge, skills, and attitude
  3. Must lead to a recognized output of professional labor
  4. Should be confined to qualified personnel
  5. Should be independently executable
  6. Should be executable within a time frame
  7. Should be observable and measureable in its process and outcome (well done or not well done)
  8. Should reflect one or more competencies

First, we reviewed existing and emerging literature on competency-based training and EPAs.8,10,18,25,26,30,34,36,38–49 We held consultations with leading experts in the field of evaluation and assessment in internal medicine and with program directors and other AAIM members who had begun the process of implementing the internal medicine Curricular Milestones into their training programs. Through a series of in-person meetings, conference calls, and e-mail exchanges, we used an iterative process to develop a draft list of EPAs, modeling it on earlier work describing the development of specialty-specific EPAs.36

Leaders from the AAIM provided feedback on this initial list of the End-of-Training EPAs, which we then revised on the basis of their feedback. Next, we introduced the revised list to the APDIM membership at their spring 2012 conference. We requested that they provide formal written feedback by way of a specific Web page on the AAIM Educational Affairs Web site. The AAIM also issued a call for feedback and disseminated the draft list of EPAs to other internal medicine stakeholder communities.

In addition, we presented the draft list of EPAs to the Internal Medicine Education Redesign Advisory Board, a steering committee of internal medicine stakeholders consisting of representatives from the ACGME, ABIM, AAIM, the American College of Physicians, the Society of General Internal Medicine, and the Society of Hospital Medicine.50 The advisory board discussed and debated the EPA framework and specific EPAs and provided verbal feedback to us. Their feedback focused primarily on defining each EPA more specifically to include the activities residents performed rather than the training settings in which residents performed those activities. For example, draft EPAs that focused on managing the care of patients in the ambulatory or critical care setting were edited to focus on managing the care of acute or chronic diseases.

In total, we received feedback from 18 different sources, representing individuals, training institutions, medical organizations, and specialty societies. Using this detailed feedback, we made extensive revisions to the draft list of EPAs, including clarification of individual existing EPAs and the addition of three new EPAs.

Results

In List 2, we present the 16 AAIM End-of-Training EPAs that together constitute the core of the internal medicine profession. Residents may demonstrate competence in these EPAs at different times throughout their training. In addition, they may demonstrate competence in some EPAs prior to the completion of residency; however, all residents should demonstrate competence in all EPAs prior to the end of training. In October 2012, we added this list to the AAIM Academic Affairs Web site so the internal medicine community could begin to use it.51

List 2 End-of-Training Entrustable Professional Activities Developed by the Education Redesign Committee of the Alliance for Academic Internal Medicine, 2012 Cited Here

  1. Manage care of patients with acute common diseases across multiple care settings.
  2. Manage care of patients with acute complex diseases across multiple care settings.
  3. Manage care of patients with chronic diseases across multiple care settings.
  4. Provide age-appropriate screening and preventative care.
  5. Resuscitate, stabilize, and care for unstable or critically ill patients.
  6. Provide perioperative assessment and care.
  7. Provide general internal medicine consultation to nonmedical specialties.
  8. Manage transitions of care.
  9. Facilitate family meetings.
  10. Lead and work within interprofessional health care teams.
  11. Facilitate the learning of patients, families, and members of the interdisciplinary team.
  12. Enhance patient safety.
  13. Improve the quality of health care at both the individual and systems level.
  14. Advocate for individual patients.
  15. Demonstrate personal habits of lifelong learning.
  16. Demonstrate professional behavior.

To help program directors and faculty begin to apply these EPAs in a meaningful fashion, we also developed a companion document entitled “Building Assessments for the End-of-Training EPA,” which is available with the EPAs on the AAIM Academic Affairs Web site. This document describes a three-step process for connecting an EPA to the Curricular Milestones as one potential method for developing synthetic, contextual, work-based assessments (see List 3). In addition, we provided completed examples of this type of work.22,52 See Chart 1 for an example of EPA #8: manage transitions of care.

List 3 Three-Step Process for Connecting an End-of-Training Entrustable Professional Activity to the Curricular Milestones to Build a Meaningful Assessment Cited Here

Step 1: Describe the activity and the tasks required of the resident for a faculty member/program director to entrust this activity to a trainee.

Step 2: Identify those milestones that best inform assessment of the description and tasks identified in Step 1.

Step 3: Apply the Curricular Milestones to a particular assessment method or tool.

T1-26
Chart 1 Example of the Three-Step Process Connecting an End-of-Training Entrustable Professional Activity (EPA) to the Curricular Milestones

Our work does not map the End-of-Training EPAs to the Reporting Milestones because these milestones had not yet been released when we completed this work. We presented the final list of EPAs to the academic internal medicine community at the APDIM fall meeting in October 2012. At the same time, we presented an overview of the NAS, the EPA development process, and an illustration of the three-step process for connecting an EPA to the Curricular Milestones.

Discussion

Although the ACGME implemented their core competency framework in 2002, medical educators and residents continued to struggle to understand the utility of the competencies in learning, designing curricula, assessing competence, and providing meaningful feedback.31,40 Recognizing this, the ACGME, in collaboration with the internal medicine community, developed and introduced milestones as a way to outline a trajectory of growth toward independent practice for residents and to document their achievement of educational outcomes, with the goal that these milestones would facilitate the development and implementation of more robust outcomes-based programs of assessment. Nevertheless, both the Curricular and Reporting Milestones for internal medicine are elaborate. Although such detail may be appropriate for managers of academic programs and regulatory bodies to use in designing curriculum and assessment programs, that same detail can be overwhelming to faculty and residents as they engage in the daily tasks of learning and assessment in the context of caring for patients.25,26,29,30,53 As written, both sets of milestones do not necessarily describe the context of assessment and, if taken in isolation, may seem disengaged from the reality of training and patient care. We believe that a framework which embeds the milestones in authentic work-based activities, such as an EPA model, will help program directors, faculty, and residents alike to provide meaningful assessment and feedback on performance.

We propose using this list of EPAs as a beginning strategy for conducting competency-based assessment in internal medicine residency training. When paired with and mapped to the Curricular Milestones, which will be necessary for full implementation, these EPAs allow for detailed assessment and feedback on performance within a meaningful, work-based context (i.e., the activities that define our profession).8,27,28 In aggregate, these EPAs can assist program directors with determining an individual resident’s development of competence and progression towards unsupervised practice.54–56 Similar efforts to develop and implement EPAs in medical education training are taking place around the world and across disciplines.10,33,43,57–59

We recognize that a limitation in using these EPAs is that some are very broad. This decision was intentional as the overall activity described is considered to be critical to the profession and as more narrow EPAs may be “nested” within the broader ones.60 Program directors and other groups may wish to develop a series of more “discrete” EPAs that, in aggregate, inform a more fully developed End-of-Training EPA. For example, a program may elect to use End-of Training EPA #8 (manage transitions of care) and then define component EPAs, such as “develop a safe discharge plan” and “ensure safe handoffs between caregivers,” that are more easily assessable, given local program resources and existing rotations.

We also believe that individual training programs should have the flexibility to determine when, where, how, and even if they assess a particular EPA. Recognizing that work-based assessments are context specific, we believe that the assessment of an individual EPA should be too. Because EPAs are synthetic, they will require multiple methods of assessment that are authentic to the included activities.5,6 Accordingly, we encourage program directors to consider their local resources, rotational structure, and other context-specific opportunities when building assessments for an EPA. Programs may wish to use one or more of the EPAs, but their use is not mandatory. Furthermore, programs may wish to integrate the End-of-Training EPAs with existing developmental assessment frameworks, such as the RIME (Reporter, Interpreter, Manager, Educator) or the similar LMT (Learner, Manager, Teacher) models.15,61–63

With regard to assessments using EPAs, the academic medicine community must determine the correct balance between individual program flexibility and guidance or direction from oversight bodies. On one hand, allowing flexibility and autonomy for programs to implement assessments is crucial because of the varied contexts, resources, and missions of internal medicine training programs in the United States. On the other hand, the outcomes of training (i.e., a physician who is ready for unsupervised practice) must be standardized for all programs. Unlike the core competency domains or the individual milestones, we believe that EPAs provide the necessary degree of flexibility for individual programs to obtain meaningful assessments of residents while still informing the overall developmental progression of competence.

The use and application of these EPAs is not mandatory, and they are not part of the formal reporting structure in the NAS. Instead, these EPAs are intended to serve as a starting point or guide for program directors to begin developing meaningful assessments that inform the evaluation of residents’ competence. When linked to specific Curricular and Reporting Milestones, they provide valuable context and meaning to work-based assessments, facilitate purposeful and specific feedback, and offer direction to residents regarding the next steps in their developmental progression. The pace of the development of EPAs has been remarkably fast in the past few years, and already multiple sets of EPAs have been defined both within internal medicine46,47,57 and among other disciplines.58,59,64 The internal medicine community must work collaboratively, establishing a common understanding and language of competency-based assessment and of EPAs, as well as common EPAs, so as not to unnecessarily complicate the job of program directors and faculty.9,35 This complexity and the evolving common language present other limitations to the implementation of EPAs for assessment. Our next steps in conjunction with the AAIM are to work with the internal medicine community to develop and share assessments of EPAs, to develop a strategy to study the effectiveness of EPAs as a framework for the assessment of competence, and to build on the strong relationships established through the Internal Medicine Education Redesign Advisory Board.

Acknowledgments: The authors wish to thank Margie Breida and Michael Meirovitz of the Alliance for Academic Internal Medicine (AAIM) for their ongoing support of the AAIM Education Redesign Committee and their dedication to helping complete this work.

References

1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366:1051–1056
2. . The Internal Medicine Milestone Project: A Joint Initiative of the Accreditation Council for Graduate Medical Education and the American Board of Internal Medicine. 2013 http://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/InternalMedicineMilestones.pdf. Accessed September 17, 2014
3. Caverzagie KJ, Iobst WF, Aagaard EM, et al. The internal medicine reporting milestones and the next accreditation system. Ann Intern Med. 2013;158:557–559
4. Dijkstra J, Galbraith R, Hodges BD, et al. Expert validation of fit-for-purpose guidelines for designing programmes of assessment. BMC Med Educ. 2012;12:20
5. Dijkstra J, Van der Vleuten CP, Schuwirth LW. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. 2010;15:379–393
6. van der Vleuten CP, Schuwirth LW. Assessing professional competence: From methods to programmes. Med Educ. 2005;39:309–317
7. van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Med Teach. 2012;34:205–214
8. ten Cate O. Entrustability of professional activities and competency-based training. Med Educ. 2005;39:1176–1177
9. Ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ. 2013;5:157–158
10. Carraccio C, Burke AE. Beyond competencies and milestones: Adding meaning through context. J Grad Med Educ. 2010;2:419–422
11. Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. Toward a definition of competency-based education in medicine: A systematic review of published definitions. Med Teach. 2010;32:631–637
12. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: Theory to practice. Med Teach. 2010;32:638–645
13. McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based curriculum development on medical education: An introduction. Public Health Pap. 1978;68:11–91
14. Hemmer PA. Professor and vice chair for educational programs, Department of Internal Medicine, Uniformed Services University of the Health Sciences. Personal communication with L.N. Pangaro 2012
15. Pangaro L, ten Cate O. Frameworks for learner assessment in medicine: AMEE guide no. 78. Med Teach. 2013;35:e1197–e1210
16. Royal College of Physicians and Surgeons of Canada. . The CanMEDS Framework. 2005 http://www.royalcollege.ca/portal/page/portal/rc/canmeds/framework. Accessed September 17, 2014
17. General Medical Council. . Tomorrow’s doctors: Outcomes and standards for undergraduate medical education. 2009 http://www.gmc-uk.org/Tomorrow_s_Doctors_0414.pdf_48905759.pdf. Accessed September 30, 2014
18. Lurie SJ, Mooney CJ, Lyness JM. Commentary: Pitfalls in assessment of competency-based educational objectives. Acad Med. 2011;86:412–414
19. Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: A systematic review. Acad Med. 2009;84:301–309
20. Lurie SJ. History and practice of competency-based assessment. Med Educ. 2012;46:49–57
21. Arrighi J, Vasilias J Talk entitled “ACGME Update”.presented at: 2012 Association of Program Directors in Internal MedicineOctober 2012Phoenix, Ariz http://www.im.org/Meetings/Past/2012/AIMW12/Presentations/Documents/2012%20APDIM%20Fall%20Meeting/APDIM%20Wksp%20101%20and%20201_Arrighi.pdf. [No longer available.] Accessed March 21, 2014
22. Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the road to competence: Developmental milestones for internal medicine residency training. J Grad Med Educ. 2009;1:5–20
23. Dreyfus HL, Dreyfus DS, Athanasiou T Mind Over Machine. 1986 New York, NY Simon & Schuster
24. Huddle TS, Heudebert GR. Taking apart the art: The risk of anatomizing clinical competence. Acad Med. 2007;82:536–541
25. ten Cate O, Snell L, Carraccio C. Medical competence: The interplay between individual ability and the health care environment. Med Teach. 2010;32:669–675
26. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32:676–682
27. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE guide no. 31. Med Teach. 2007;29:855–871
28. Norcini JJ. Work based assessment. BMJ. 2003;326:753–755
29. Dath D, Iobst W. The importance of faculty development in the transition to competency-based medical education. Med Teach. 2010;32:683–686
30. Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment: The missing link in competency-based medical education. Acad Med. 2011;86:460–467
31. Weinberger SE, Pereira AG, Iobst WF, Mechaber AJ, Bronze MSAlliance for Academic Internal Medicine Education Redesign Task Force II. . Competency-based education and training in internal medicine. Ann Intern Med. 2010;153:751–756
32. Weinberger SE, Pereira AG, Iobst WF, Mechaber AJ, Bronze MS. Internal medicine residency redesign. Ann Intern Med. 2011;154:442–443
33. Sterkenburg A, Barach P, Kalkman C, Gielen M, ten Cate O. When do supervising physicians decide to entrust residents with unsupervised tasks? Acad Med. 2010;85:1408–1417
34. ten Cate O. Trust, competence, and the supervisor’s role in postgraduate training. BMJ. 2006;333:748–751
35. Ten Cate O. Competency-based education, entrustable professional activities, and the power of language. J Grad Med Educ. 2013;5:6–7
36. ten Cate O, Scheele F. Competency-based postgraduate training: Can we bridge the gap between theory and clinical practice? Acad Med. 2007;82:542–547
37. ten Cate O, Young JQ. The patient handover as an entrustable professional activity: Adding meaning in teaching and practice. BMJ Qual Saf. 2012;21(suppl 1):i9–i12
38. Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: Translating the Dreyfus developmental model to the learning of clinical skills. Acad Med. 2008;83:761–767
39. Hicks PJ, Englander R, Schumacher DJ, et al. Pediatrics milestone project: Next steps toward meaningful outcomes assessment. J Grad Med Educ. 2010;2:577–584
40. Iobst WF, Sherbino J, Cate OT, et al. Competency-based medical education in postgraduate medical education. Med Teach. 2010;32:651–656
41. Jones MD Jr, Rosenberg AA, Gilhooly JT, Carraccio CL. Perspective: Competencies, outcomes, and controversy—linking professional activities to competencies to improve resident education and practice. Acad Med. 2011;86:161–165
42. MacDonald J, Ellis PM. Supervision in psychiatry: Terra incognita? Curr Opin Psychiatry. 2012;25:322–326
43. Mulder H, Ten Cate O, Daalder R, Berkvens J. Building a competency-based workplace curriculum around entrustable professional activities: The case of physician assistant training. Med Teach. 2010;32:e453–e459
44. Rushton JL, Hicks PJ, Carraccio CL. The next phase of pediatric residency education: The partnership of the Milestones Project. Acad Pediatr. 2010;10:91–92
45. Schumacher DJ, Slovin SR, Riebschleger MP, Englander R, Hicks PJ, Carraccio C. Perspective: Beyond counting hours: The importance of supervision, professionalism, transitions of care, and workload in residency training. Acad Med. 2012;87:883–888
46. Hauer KE, Kohlwes J, Cornett P, et al. Identifying entrustable professional activities in internal medicine training. J Grad Med Educ. 2013;5:54–59
47. Hauer KE, Soni K, Cornett P, et al. Developing entrustable professional activities as the basis for assessment of competence in an internal medicine residency: A feasibility study. J Gen Intern Med. 2013;28:1110–1114
48. Hauer KE, Ten Cate O, Boscardin C, Irby DM, Iobst W, O’Sullivan PS. Understanding trust as an essential element of trainee supervision and learning in the workplace. Adv Health Sci Educ Theory Pract. 2014;19:435–456
49. Wijnen-Meijer M, van der Schaaf M, Nillesen K, Harendza S, Ten Cate O. Essential facets of competence that enable trust in medical graduates: A ranking study among physician educators in two countries. Perspect Med Educ. 2013;2:290–297
50. Alliance for Academic Internal Medicine. . Redesigning Internal Medicine Education. 2013 http://connect.im.org/p/cm/ld/fid=199. Accessed September 17, 2014
51. Alliance for Academic Internal Medicine. . Internal Medicine End of Training EPAs. 2013 http://connect.im.org/p/cm/ld/fid=639. Accessed September 17, 2014
52. Alliance for Academic Internal Medicine. . Building Assessments for an EPA in Three Steps. 2013 http://dev.im.org/AcademicAffairs/milestones/Pages/BuildingAssessmentsforanEPAinThreeSteps.aspx. Accessed September 17, 2014.
53. Dijksterhuis MG, Voorhuis M, Teunissen PW, et al. Assessment of competence and progressive independence in postgraduate clinical training. Med Educ. 2009;43:1156–1165
54. Wijnen-Meijer M, Burdick W, Alofs L, Burgers C, ten Cate O. Stages and transitions in medical education around the world: Clarifying structures and terminology. Med Teach. 2013;35:301–307
55. Wijnen-Meijer M, Kilminster S, Van Der Schaaf M, Ten Cate O. The impact of various transitions in the medical education continuum on perceived readiness of trainees to be entrusted with professional tasks. Med Teach. 2012;34:929–935
56. Wijnen-Meijer M, ten Cate OT, van der Schaaf M, Borleffs JC. Vertical integration in medical school: Effect on the transition to postgraduate training. Med Educ. 2010;44:272–279
57. Chang A, Bowen JL, Buranosky RA, et al. Transforming primary care training—patient-centered medical home entrustable professional activities for internal medicine residents. J Gen Intern Med. 2013;28:801–809
58. Shaughnessy AF, Sparks J, Cohen-Osher M, Goodell KH, Sawin GL, Gravel J Jr.. Entrustable professional activities in family medicine. J Grad Med Educ. 2013;5:112–118
59. Boyce P, Spratt C, Davies M, McEvoy P. Using entrustable professional activities to guide curriculum development in psychiatry training. BMC Med Educ. 2011;11:96
60. Hemmer PA. Professor and vice chair for educational programs, Department of Internal Medicine, Uniformed Services University of the Health Sciences. Personal communication with O.T. ten Cate 2012
61. Pangaro L. A new vocabulary and other innovations for improving descriptive in-training evaluations. Acad Med. 1999;74:1203–1207
62. Rodriguez RG, Pangaro LN. AM Last Page. Mapping the ACGME competencies to the RIME framework. Acad Med. 2012;87:1781
63. Varney A, Todd C, Hingle S, Clark M. Description of a developmental criterion-referenced assessment for promoting competence in internal medicine residents. J Grad Med Educ. 2009;1:73–81
64. Burke AE, Hicks PJ, Carraccio C. Towards meaningful outcomes assessment: Collaborative efforts in pediatric medical education. Acad Pediatr. 2012;12:79–80
© 2015 by the Association of American Medical Colleges