To bolster public trust that our graduating trainees have the knowledge, skills, and attitudes required for independent practice in their respective fields, the Accreditation Council for Graduate Medical Education (ACGME) initiated the process of standardizing the Subspecialty Reporting Milestones in 2012. This concept had been advancing in graduate medical education since the development of the 6 core competencies in 1999: Patient Care (PC), Medical Knowledge (MK), Interpersonal and Communication Skills (ICS), Professionalism (PROF), Systems-Based Practice (SBP), and Practice-Based Learning and Improvement (PBLI).1 For internal medicine subspecialties, creation of the Reporting Milestones in 2013 was accomplished by convening a large group of experienced educators from all internal medicine subspecialties. After completion of that effort, the Oversight Working Network, with representation and input from the major gastrointestinal (GI) and hepatology societies (American Association for the Study of Liver Disease, American College of Gastroenterology, American Gastroenterological Association, American Neurogastroenterology and Motility Society, American Society for Gastrointestinal Endoscopy, and North American Society for Pediatric Gastroenterology Hepatology and Nutrition) and with representation from the GI Program Directors’ Caucus, developed a companion document of Entrustable Professional Activities (EPAs) for gastroenterology (GE) (Table 1). These 13 EPAs were linked to observable behaviors and objectives along with a toolbox that accompanied each EPA.2,3
Over time, the ACGME monitored, assessed, and analyzed the Reporting Milestones (Milestones 1.0) for all specialties, soliciting feedback from the graduate medical education community. Many believed there were too many subcompetencies that were too general, that the language was often complex, and the milestones were cumbersome.4 A review of 26 core specialty milestones conducted by the ACGME revealed 144 different ways to describe ICS and 200 ways to describe PROF. This degree of variation contributed to dissatisfaction with the use of ICS, PROF, SBP, and PBLI milestones. The GE milestones lacked specialty-specific language to guide assessment, making implementation of the milestones difficult. This prompted the ACGME to make significant changes, many of which have already been reviewed and published after public comment.5
The first major change for internal medicine subspecialties involved the development of a set of harmonized milestones for ICS, PROF, SBP, and PBLI. These milestones focus on essential skills required of all trainees across every field of internal medicine, such as helping a patient navigate the complex healthcare system, serving patients and colleagues in the most professional manner, using clear and understandable communication through all levels of health care (patient to colleague to the system itself), and enabling learners to be responsible for their own professional development.6 The second major effort, presented in this document, involved development of specialty-specific milestones for GE and transplant hepatology.
This set of milestones does not have the former “Critical Deficiencies” section. This designation was rarely used in fellowship and resulted in a limited range of reporting by programs. All writing groups also used positive language and removed negative language (eg, “does not perform”) from milestone descriptors.
The Dreyfus model was used to develop Milestones 2.0, using the 5 stages of development from novice to expert.7 Based on feedback from Milestones 1.0, Milestones 2.0 was created with specialty-specific PC and MK milestones. Each subcompetency within the core competencies was developed with up to 3 themes (rows) that each completed a developmental trajectory using positive descriptive language.
Each of 4 GE societies (American Association for the Study of Liver Disease, American College of Gastroenterology, American Gastroenterological Association, and American Society for Gastrointestinal Endoscopy) sent 4 representatives to the Milestones 2.0 working group. The group consisted of these representatives, a fellow, and a public member. The group met twice in person in February and August 2019. Starting with PC and MK milestones, the group created subcompetency themes. Small groups developed up to 3 narrative threads for each subcompetency along with descriptive language for each of the 5 developmental stages. These were then reviewed and edited by the entire group. The harmonized milestones (SBP, PBLI, PROF, and ICS) were similarly reviewed and edited.
Creation of MK and PC milestones
In response to feedback from the learner and program director communities, the ACGME asked GE and transplant hepatology to create specialty-specific PC and MK milestones. The group was asked to envision the “gastroenterologist of 2025” at the March 2019 kick-off meeting. This brainstorming activity resulted in over 30 descriptors and informed the construction of the PC and MK milestones. Participants created descriptions of each subcompetency that were reviewed for consensus. Small groups developed themes for each subcompetency, which were then reviewed and edited by the entire group until consensus was reached. Milestones 2.0 for transplant hepatology was developed in conjunction with the GE milestones and led by the transplant hepatology members of the working group using a similar process.
Creation of harmonized milestones
Because the skills embodied by ICS, PROF, SBP, and PBLI are common across specialties, subcompetencies for these 4 areas were drafted as a set of harmonized milestones with common language available for use by every specialty. The ACGME convened interprofessional groups, including content experts, program directors, interprofessional team members, other faculty, and the public, to create draft documents for each of these 4 competencies using the best available evidence and expert opinion. Each draft was vetted through open, public review and comment.6 The final drafts were made available to the GE and transplant hepatology milestone working group for review and editing. Although subcompetencies within the harmonized milestones needed to remain the same across specialties, individual specialties were permitted to edit the narratives within each subcompetency. Proposed edits were reviewed by the entire group, including edits that were specific to transplant hepatology.
Supplemental Guide creation
The Supplemental Guide was designed as a companion document to Milestones 2.0. It includes the overall intent of what is being evaluated, level-specific examples, assessment models, and resources. During development of Milestones 2.0, discussion among working group members revolved around individual experiences or cases, which facilitated consensus on the milestone language. These conversation topics, along with ideas from the original brainstorming, were retained as examples in the Supplemental Guide. Small groups worked together to draft the language for the Supplemental Guide that was reviewed and edited by the entire group.
After completion of the draft Milestone 2.0 document and Supplemental Guide, the ACGME facilitated a 1-month public comment period from middle to late January 2020. A public comment survey was sent directly to program directors and coordinators, posted on the ACGME website, and highlighted in the ACGME weekly eCommunication. Members of the working group shared the link with faculty, fellows, and specialty societies. Respondents were asked to rate each milestone on a 4-point Likert scale (1 = strongly disagree, 4 = strongly agree) according to whether the milestone represented a realistic progression of knowledge, skills, and behaviors; whether the milestone discriminated between meaningful levels of competency; whether the respondent knew how to assess the milestone effectively; and whether the Supplemental Guide was a useful resource in understanding the milestone. Respondents were also able to provide free-text comments.
Milestones 2.0 contains 17 milestones for GE and 16 for transplant hepatology (Table 2), which are reduced from 23 in Milestones 1.0.
Milestones 2.0 contain 4 PC subcompetencies. PC1 (Data Gathering and Nonprocedural Diagnostic Testing) describes the development of a trainee’s ability to conduct a specialty-focused history and physical exam and order appropriate diagnostic testing. PC2 (Patient Management in Gastrointestinal and Liver Disease) describes the development of a trainee’s ability to develop a comprehensive specialty-specific care plan based on disease presentation and urgency.
Because GE is a procedural-based medical subspecialty, the group included 2 subcompetencies related to endoscopy: PC3 (Procedures: Cognitive Components) and PC4 (Procedures: Technical Components). This distinction reflects the need for both cognitive and technical skills for competence and eventual mastery of endoscopy. PC3 describes the development of a trainee’s understanding of the indications and contraindications for endoscopic procedures and the interpretation of normal and abnormal findings and therapeutic options. PC4 focuses on the psychomotor development necessary to perform endoscopy and includes pre- and postprocedural assessments, therapeutic interventions, and follow-up. For the transplant hepatology milestones, PC3 and PC4 were replaced with a single subcompetency (PC3: Technical Procedures – Liver Biopsy), representing the only procedure currently required in transplant hepatology training.
The MK milestones, MK1 (Clinical Knowledge of Gastrointestinal and Liver Diseases: Non-Procedural) and MK2 (Clinical Reasoning), are directly related to subspecialty GE or transplant hepatology care. They are independent of the internal medicine milestones (expected competence of a graduating resident), thereby highlighting the unique knowledge necessary for GE or transplant hepatology competence. MK1 describes the development of knowledge of specialty-specific disorders and of diagnostic, therapeutic, and pharmacologic options for prevention and treatment of diseases. MK2 focuses on the development of differential diagnoses and the role of cognitive bias, a new concept introduced in Milestones 2.0.
The core harmonized milestones were used as a foundation and tailored to meet the needs of the specialty. The working group recognized the unique aspects of SBP in transplant hepatology and developed narratives to reflect this. SBP1 (Patient Safety and Quality Improvement) highlights unique liver transplant regulatory requirements, whereas SBP2 (System Navigation for Patient-Centered Care) emphasizes organ allocation policies and Model for End-Stage Liver Disease (MELD) score exceptions.
Public comment survey
Forty-eight respondents provided 96 entries with free-text comments. Survey results are presented in Table 2. All milestones had a score of 3.0 or greater for each of the items assessed. Respondents believed the milestones represented a realistic progression (mean, 3.49) and were capable of discriminating between meaningful levels of competency (mean, 3.41). Most respondents believed they could assess the milestones (mean, 3.43). ICS1, SBP2, and SBP3 had the lowest ratings among respondents when asked about assessment. Overall, the respondents found the Supplemental Guide useful for each milestone (mean, 3.42). ACGME staff (L.E.) reviewed the survey results and found no trends that required action for any of the comments.
The GE and transplant hepatology Milestones 2.0 work group developed a set of milestones containing specialty-specific PC and MK milestones and harmonized ICS, PROF, SBP, and PBLI milestones. The primary goal for this effort was to simplify milestone reporting to the ACGME. However, the milestones were designed not simply for reporting to ACGME but also to aid in trainee and program assessment, curriculum development, and self-directed learning.
Across the curriculum, each clinical rotation and learning activity (including conferences) should be identified to achieve specific learning objectives, which can be mapped to the subcompetencies. The training program should focus on the milestones that are best learned and evaluated in a particular rotation. For example, milestones assigned to an inpatient consult rotation could include SBP2, SBP3, PBLI1, and PC1 to PC3. This would promote consistency with respect to expectations across the program and a means to evaluate progress along the milestone trajectory for individual trainees.
To achieve a more learner-centered approach, a program can identify the subcompetencies that a specific trainee will want or need to work on during a particular rotation or learning activity. These subcompetencies (and the learning objectives derived from them) should be shared with the trainee and the supervising faculty in advance of the rotation or learning activity, and assessment tools should be designed to evaluate those specific objectives.
The development of assessment tools that together accurately identify trainee progression toward competence in the milestones is an ongoing challenge.8 Although each learning activity generally involves multiple competencies, the milestones as a whole should not be used as an assessment tool. Instead, assessment tools should be used to inform the work of the Clinical Competency Committee (CCC). If existing assessment tools are judged to be inadequate for the task, new tools should be obtained or designed. Rather than developing new tools from scratch, we encourage programs to share their assessment tools to facilitate a standardized approach to trainee assessment across programs within the specialty. The Milestones 2.0 Supplemental Guide can be used to identify the types of assessment tools ideally suited to assessing particular subcompetencies.
Although the harmonized milestones contain the same subcompetencies across specialties, some of the transplant hepatology narratives within SBP1 and SBP2 are unique to this specialty, as discussed above. This facilitates specific assessment of SBP in transplant hepatology trainees, and programs should tailor the systems-based practice curriculum to this context.
Role of the CCC
For trainee assessment, the milestones are a tool best used by the CCC rather than as an assessment method used by individual supervising clinical faculty, whose often brief interactions with trainees make meaningful milestone evaluation difficult. Indeed, a criticism of the milestones is that competencies designed as an educational framework have often been misused as evaluation tools.9 We do not recommend assigning individual faculty to rate trainees on all competencies using the narratives provided in the milestones. Multiple existing assessment tools should be used by the CCC as a window into the developmental level achieved by each trainee at a given time in training. Assessment tools can include some language from the narratives of specific subcompetencies appropriate for the learning objectives of the rotation/learning activity. Using a variety of assessment methods, including direct observation, EPAs, chart-stimulated recall, multisource evaluations, and so on, the CCC synthesizes assessments of trainee performance from multiple sources and determines each trainee’s progress based on the competencies. Objective assessment of the developmental milestones achieved by each trainee is accomplished by the CCC by aggregating assessment tools and developing a shared mental framework that allows for consistent assessment.
Mapping a program’s curriculum to the milestones enables program directors to identify gaps in the curriculum and make necessary adjustments as part of a program’s continuous quality improvement. For example, a program director may recognize the curriculum has limited opportunities for learners to demonstrate competence in SBP1 (Patient Safety and Quality Improvement) and could use this observation to identify key faculty with expertise or interest in quality improvement to work with trainees to create and conduct quality improvement initiatives. In this context, milestone-directed curriculum development could be generalized or shared across programs depending on program needs and strengths.
The milestones are designed to help assess the development of competence across the continuum of training. Rotations that are repeated in subsequent training years (eg, inpatient GE consultation) can be used to assess the same subcompetencies at different points in the trajectory toward unsupervised practice.
Individual learners progress at different rates, and competency-based medical education should accommodate these differences. One of the main benefits of milestones-based assessment is to be able to identify those who are not on a trajectory toward competence early in their training and offer remediation opportunities. Expectations should be set for each year of training to identify where a trainee should fall along the narrative trajectory of each subcompetency. These expectations may be specialty-wide, program-specific, or based on individual trainee goals. Those who exceed expectations can be offered a flexible curriculum that permits achievement of aspirational milestones or allows them to focus on unique learning objectives that align with their career goals.
The GE–transplant hepatology dual-certification pathway (formerly known as the “pilot”) is 1 example of how a curriculum can be redesigned to meet a trainee’s learning needs and career goals (in this case, clinical transplant hepatology).10 This dual-certification pathway requires a unique use of specialty-specific milestones in 2 specialties and a close collaboration between GE and transplant hepatology program directors and the CCC. It is a clear example of how development of competence can be objectively measured based on the milestones rather than duration of training.
An individual trainee can use the milestones to self-assess progress and direct his or her own learning. The harmonized milestones contain similar language from the internal medicine milestones to the specialty milestones, allowing trainees to continue building their skills from residency. Trainees should work with their program director and clinical faculty to identify learning needs and focus on areas of deficiency. These learning objectives can be specific to a rotation or a point in time during training and may thus be matched to the program’s learning objectives for a particular rotation or learning activity. Ideally, trainees could select learning experiences or rotations that will meet their needs and goals, although this is not often practical or desired early in training. During the final year of training, when most trainees have largely achieved a satisfactory level of competence (level 4), they should be able to individualize their learning to start their journey to achieving aspirational (often level 5) milestones and other goals specific to their learning needs and career objectives.
The milestones have been developed as a means to describe the developmental trajectory of learners on the path to competence, independent practice, and expertise. Evaluating competencies separately from one another has been criticized as reductionist and has generally not been intuitive to most clinical faculty.11 Competencies can be difficult to separate when assessing specific tasks. This is precisely why the milestones themselves should not be used as assessment tools. The granular nature of the milestones makes their use as an assessment tool challenging and unrealistic, especially by individual faculty evaluating learners after brief interactions. Other assessment tools should be used to evaluate learners, and those assessments should be used by the CCC to inform the milestones.
It is hoped that Milestones 2.0, which contains specialty-specific narratives, in conjunction with the Supplemental Guide that contains specific examples and suggested assessment methods, will assist with trainee assessment in each of the competencies. The greater specificity of the language for PC and MK milestones should make assessment and program development more meaningful. The new harmonized milestones will allow for continued development of the learner toward independent practice that encompasses the entirety of graduate medical education, not just fellowship. The community of program directors and clinician educators in GE and transplant hepatology will need to work together to develop new curricular tools and assessments to aid in the implementation of these milestones.
All authors disclosed no financial relationships.
We thank Sydney McClean, Bincy Abraham, Jean-Paul Achkar, Jonathan Cohen, Sarah Diamond, Margarita German, Ashley Grantham, Dina Halegoua-De Marzio, Arnab Mitra, and Robert Sedlack. Access to Research Electronic Data Capture (RedCap) was made possible due to the Research Computing Facility grant support (UL1TR002377).
1. Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach 2007;29:648–654.
2. Rose S, Fix O, Shah BJ, et al. Entrustable professional activities for gastroenterology fellowship training. (Published in multiple journals simultaneously as an effort of the 5 GE societies). Gastroenterology 2014;147:233-242, Hepatology 2014;60:433-43, Gastrointest Endosc 2014;80:16-27.
3. Oversight Working Network. Available at: http://www.ownyourfellowship.org/
. Accessed May 19, 2021.
4. Edgar L, Roberts S, Yaghmour NA, et al. Competency crosswalk: a multispecialty review of the Accreditation Council for Graduate Medical Education milestones across four competency domains. Acad Med 2018;93:1035–1041.
6. Edgar L, Roberts S, Holmboe E. Milestones 2.0: a step forward. J Grad Med Educ 2018;10:367–369.
7. Dreyfus SE. The five-stage model of adult skill acquisition. Bull Sci Technol Soc 2004;24:177–181.
8. Iobst WF, Caverzagie KJ. Milestones and competency-based medical education. Gastroenterology 2013;145:921–924.
9. Ginsburg S, McIlroy J, Oulanova O, et al. Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Acad Med 2010;85:780–786.
11. Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review. Acad Med 2009;84:301–309.