The Accreditation Council for Graduate Medical Education (ACGME) began to move to an outcomes-based accreditation framework with the creation of the six core competency domains in 1999.1 The launch of the Outcomes Project in 2001 was the first effort to incorporate these domains into educational practice.2 Early on, programs struggled to implement the competencies into their curricula and assessment systems.3–6 In 2009, the first set of milestones was published in internal medicine as a possible framework to help programs implement the competencies more effectively.7,8 Others also found this developmental model useful in training.9 On the basis of these early experiences, milestones became the outcomes-based assessment framework for all specialties accredited by the ACGME in July 2013 as part of the Next Accreditation System.1
Each specialty, with support from the ACGME and the American Board of Medical Specialties (ABMS), coproduced its own milestones. Specialty working groups included program directors, faculty, residents, and other stakeholders (e.g., medical educators, nurses, allied health professionals). These working groups first determined the subcompetencies most relevant to the education of residents in their respective specialties. Working groups then used program requirements, certification examination outlines, curricula, national competency statements, literature reviews, and results from national consensus-building exercises to guide the milestone development process.10,11
Each specialty independently created milestones for subcompetencies within each of the six core competencies: medical knowledge (MK), patient care (PC), interpersonal and communication skills (ICS), practice-based learning and improvement (PBLI), professionalism (PROF), and systems-based practice (SBP). The rationale for this flexibility was to create a sense of ownership and engagement in the milestone development process and to account for differences among specialties. During the design phase, working groups recognized that specialty milestones would be reviewed periodically using current documents (e.g., specialty-specific competency guidelines, program requirements, ABMS and American Osteopathic Association Board certification documents), research, and surveys of specialty communities to ensure that the milestones continue to meet the needs of residents, training programs, and society.
As part of the initial rollout of the milestones, ACGME milestones staff (L.E., S.J.H., and E.S.H.) also routinely collected feedback on early stakeholder experiences with the milestones.12,13 This was accomplished by conducting focus groups with program directors, faculty, designated institutional officials, and residents at conferences, specialty society meetings, and institutional visits. These stakeholders frequently commented on the high degree of variability across specialties and overlap in content across core competency domains. The variability and overlap complicated efforts to share assessment tools across programs and provide comprehensive faculty development across specialties.
In undertaking this study, we aimed to identify common and overlapping themes among the ICS, PBLI, PROF, and SBP milestones of the transitional year and 26 core specialties. Identifying areas of overlap across competencies will allow the medical education community to organize themes more consistently within the core competencies framework (e.g., whether milestones related to care coordination should be placed under ICS or SBP). Identifying areas of overlap across specialties will highlight potential opportunities for convergence in order to reduce variability in milestone content across training programs. We focused on the four crosscutting competencies (ICS, PBLI, PROF, and SBP) because they are less specialty specific than MK and PC milestones and hence may offer more opportunities for meaningful convergence.
Our data source was the ICS, PBLI, PROF, and SBP milestones for the transitional year and the 26 core specialties (those leading to an ABMS member board primary certification).14 Transitional year programs do not represent a core specialty but provide education in multiple clinical disciplines to prepare residents for a specific specialty. The 26 core specialties were allergy and immunology; anesthesiology; colon and rectal surgery; dermatology; emergency medicine; family medicine; internal medicine; medical genetics and genomics; neurological surgery; neurology; nuclear medicine; obstetrics–gynecology; ophthalmology; orthopedic surgery; otolaryngology; pathology; pediatrics; physical medicine and rehabilitation; plastic surgery; preventive medicine; psychiatry; radiation oncology; radiology; surgery; thoracic surgery; and urology. In May 2017, milestones were accessed from the ACGME specialties website.14
We analyzed these milestones using qualitative thematic analysis for several reasons.15 First, as described by Braun and Clarke,16 thematic analysis is “a method for identifying, analysing and reporting patterns (themes) within data.” Second, this study is using existing narratives previously developed by groups through an iterative process. As such, these narratives constitute the source of the “interview” data for analysis, unlike a typical qualitative study. Third, although content analysis and thematic analysis share many similarities, thematic analysis typically involves a deeper level of interpretation of the data (i.e., both deductive and inductive), a necessary step in this project because of the complexity of the milestone language across the 27 different sets.17
Two members of the study team (L.E. and S.R.) first conducted coding of all ICS, PBLI, PROF, and SBP milestones, using keywords derived from the ACGME Common Program Requirements as initial guides to coding.18 Through multiple rounds of analysis, the authors reconciled differences, developed a final list of codes, and combined codes into groups sharing a common theme (overarching ideas or concepts that appeared frequently in the milestones of one or more competency domains).
To assess the reproducibility of this analysis, two different authors (A.L.H. and L.C.) independently applied the coding taxonomy to the milestones of seven specialties (representing select surgical, medical, and hospital-based programs). The codes assigned by these two authors matched those of the initial coders, supporting the validity and reproducibility of the coding scheme.
Overall, we identified 22 themes among the ICS, PBLI, PROF, and SBP milestones of the transitional year and the 26 core specialties. Fifteen themes (68%) appeared only in milestones related to one of the four core competencies (3 related to ICS, 4 related to PBLI, 5 related to PROF, and 3 related to SBP) and hence were unique to a given competency (Table 1). The remaining 7 themes (32%) appeared in the milestones of more than one core competency (Table 2). We describe each of these 22 themes in more detail below. Chart 1 demonstrates which themes were included in each set of milestones, and Table 3 provides a specific example of each theme.
Interpersonal and communication skills
Three themes were unique to ICS milestones (Table 1). The most common was “communication with patients, families, and the public,” followed by “communication with teams,” which included use of proactive communication skills, negotiation, and team building (among others). The third unique ICS theme was “informed consent,” which often included education of patients and families as part of the informed consent process.
Practice-based learning and improvement
Four themes were unique to PBLI milestones (Table 1). The most common was “self-directed learning.” This theme included assessment of educational needs or educational gap analysis, journal club participation, creation and alteration of learning plans, self-assessment, and literature reviews. The second unique PBLI theme was “evidence-based patient care,” which focused on the use of evidence-based medicine (EBM), scholarly activity with EBM (e.g., applying knowledge), or the use of national guidelines and often related directly to a program requirement within the specialty. The third unique PBLI theme was “teaching,” which addressed education of patients, families, medical students, and peers, either on a one-to-one basis or during conferences. The fourth unique PBLI theme was “scholarly activity,” which focused on the completion of scholarly projects, study design, and other research skills.
Five themes were unique to PROF milestones (Table 1). The most common were “attitudes and behaviors” and “accountability.” The “attitudes and behaviors” theme included the demonstration of integrity, truthfulness, empathy, sensitivity, compassion, responsibility, respect, cultural competence, responsiveness, and managing personal beliefs. The “accountability” theme went beyond compliance with requirements to also include fatigue management, physician wellness (physical, emotional, and mental), and patient advocacy. The third unique PROF theme was “ethics,” which incorporated general ethical behavior and recognition of ethical violations such as conflict of interest or boundary breaches. The remaining two unique PROF themes were “limits” and “administrative tasks.” “Limits” incorporated recognizing personal limits and seeking help, whereas “administrative tasks” included topics related to the timely completion of administrative responsibilities such as medical records and documentation.
SBP milestones contained three unique themes (Table 1). The most common was “health economics,” which included cost–benefit considerations to treatment, proper and complete coding, insurance, cost-effective care, utilization and review, and practice management. The second most common SBP theme was “health care delivery settings,” which addressed residents’ ability to understand and navigate various practice settings (e.g., clinic vs. community hospital vs. Veterans Affairs facility). The third unique SBP theme was “communities,” which addressed the identification and understanding of the importance of community resources for patients.
Nearly one-third of the 22 themes (7/22; 32%) appeared in milestones of more than one core competency domain (Table 2). Themes related to SBP demonstrated the greatest degree of overlap. Examples of milestones illustrating these overlapping themes are presented in Table 3.
The most common area of overlap was “medical records and technology.” This theme addressed residents’ ability to utilize the electronic medical record and other technology to provide safe patient care. Other common overlapping themes were “coordination and transitions of care,” “feedback,” and “patient safety and errors.” The “coordination and transitions of care” theme included coordination within and across health care systems and transitions between health care teams. The “feedback” theme addressed residents’ ability to solicit and incorporate feedback as well as provide feedback. The “patient safety and errors” theme addressed topics such as error avoidance, error reporting, and practice safety (e.g., radiation safety). A fifth area of overlap was “quality assurance/improvement.” This theme included quality improvement principles and projects, as well as laboratory proficiency testing. The last two overlapping themes were “confidentiality” and “teamwork.” The “confidentiality” theme included the Health Information Portability and Accountability Act specifically and patient confidentiality generally. The “teamwork” theme addressed the roles of various team members and being a team player.
Areas of overlap across specialties
As shown in Chart 1, we identified substantial variability in themes across specialties. This chart lists themes in descending order, with themes used by the most specialties at the top of the chart and those used by the fewest specialties at the bottom. Eleven of the 22 themes we identified (50%) were used by 20 or more specialties. These are “self-directed learning,” “communication with patients,” “attitudes and behaviors,” “accountability,” “medical records and technology,” “health economics,” “communication with teams,” “coordination and transitions of care,” “feedback,” “patient safety and errors,” and “ethics.” Six of the 22 themes (27%) appeared in the milestones of 10 or fewer specialties: “teaching,” “teamwork,” “health care delivery settings,” “informed consent,” “scholarly activity,” and “community.”
This study identified 22 themes within the milestones of four crosscutting competencies (ICS, PBLI, PROF, and SBP) for the transitional year and 26 core specialties. References supporting the relevance and importance of these themes are available in Supplemental Digital Appendix 1, at https://links.lww.com/ACADMED/A511. The results demonstrate areas of overlap among the ICS, PBLI, PROF, and SBP milestones and substantial variability across specialties.
With respect to overlap among competencies, we found that approximately one-third of the themes we identified appeared in the milestones of more than one of the four crosscutting competencies. It is not surprising to find overlapping themes among the ICS, PBLI, PROF, and SBP milestones as the competencies do not discreetly manifest in practice. Specialties have differing perspectives on the primary core competency reflected by a given theme (e.g., the degree to which “giving and receiving feedback” represents PBLI vs. PROF) or emphasize different aspects of a given theme (e.g., the PROF aspect of teamwork vs. the role of teamwork in SBP). The results of this analysis highlight areas where harmonization could strengthen shared understanding across specialties to enhance educational and clinical practice.
We also observed variability in themes across specialties in that some themes appeared in the milestones of most specialties, whereas others appeared only in a minority. Some of this variability is not necessarily a problem given that some milestones may be of particular importance for select specialties, such as community in family medicine. Themes appearing in the milestones of the smallest number of specialties are not necessarily less important than those appearing in the milestones of more specialties. Rather, they represent themes that may have the greatest potential for convergence across specialties. However, some of the variability is concerning when it involves abilities considered crucial for 21st-century practice, such as evidence-based patient care and health care delivery systems.19 As the specialties prepare for revision of their milestones, it will be important for them to reconsider some of these important competency areas.
The original milestones created a natural experiment for how each specialty views and interprets the core competencies in the context of their discipline.12 Although our findings are encouraging in terms of what was included across all specialties, the differences provide an opportunity for greater interspecialty work on the competencies moving forward. For example, “accountability” and “safety (patient) and errors” are generalizable across all specialties. Those responsible for developing the next version of the milestones will be able to use and expand these findings to determine those of most importance to all specialties. Our findings suggest that a core set of ICS, PBLI, PROF, and SBP themes could be similar across all specialties. Each specialty will need to customize the milestones to fit specialty-specific needs (e.g., radiation safety in radiology), including additional themes if necessary. Through coproduction with and inclusion of additional community and public members, we can improve our understanding and assessment of these themes, subcompetencies, and competency domains.
Another important decision moving forward is determining which essential, or core, subcompetency themes programs should report as part of accreditation and continuous quality improvement. The Next Accreditation System has embraced the need to use meaningful data for continuous quality improvement of educational programs. National milestone reporting has already enabled analysis of competencies other than MK within several disciplines and has provided a rich source of feedback for those disciplines to guide changes in curriculum and assessment.20–23
The intent of this work is not to dictate content to future milestone developers but, rather, to demonstrate what was important to each of the specialties in the original milestones and equip specialties with this information as they begin to consider their next steps. The ACGME has also recently appointed four interdisciplinary milestone development groups. These groups will work toward consensus on a set of draft milestones for the ICS, PBLI, PROF, and SBP competencies that each specialty can consider using in future versions. We fully expect the application of these consensus milestones to be variable across specialties because of practice differences.
This study has some limitations. First, this study did not include a systematic review of what important themes might be missing from existing milestones. Second, we did not perform a gap analysis between specialties. Third, we did not perform a specific analysis of the quality of the actual language and construction of the milestones. Finally, this study did not include data on how specialty programs interpret and use the milestones.
Future directions for this work include additional research to more fully explore each competency domain, harmonize curricular constructs across specialties, and determine whether there is sufficient representation of integral aspects of medical education and physicians’ expected role in society. Graduate medical education and the specialties must work together to determine what is necessary for effective training in all competencies, including how milestones can best help drive meaningful change nationally and what should be left for institutions and programs to customize locally. Finally, we need research for national improvement in the way we teach and train residents to ensure that they are equipped to work in the health care system of today and tomorrow.
Acknowledgments: The authors wish to thank all of the volunteers who contributed their time and efforts to the milestones development.
1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The Next GME Accreditation System—Rationale and benefits. N Engl J Med. 2012;366:10511056.
2. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Aff (Millwood). 2002;21:103111.
3. Swing SR. The ACGME Outcome Project: Retrospective and prospective. Med Teach. 2007;29:648654.
4. Lee AG, Oetting T, Beaver HA, Carter K; Task Force on the ACGME Competencies at the University of Iowa Department of Ophthalmology. The ACGME Outcome Project in ophthalmology: Practical recommendations for overcoming the barriers to local implementation of the national mandate. Surv Ophthalmol. 2009;54:507517.
5. Schumacher DJ, Lewis KO, Burke AE, et al. The pediatrics milestones: Initial evidence for their use as learning road maps for residents. Acad Pediatr. 2013;13:4047.
6. Korte RC, Beeson MS, Russ CM, Carter WA, Reisdorff EJ; Emergency Medicine Milestones Working Group. The emergency medicine milestones: A validation study. Acad Emerg Med. 2013;20:730735.
7. Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the road to competence: Developmental milestones for internal medicine residency training. J Grad Med Educ. 2009;1:520.
8. Nabors C, Peterson SJ, Forman L, et al. Operationalizing the internal medicine milestones—An early status report. J Grad Med Educ. 2013;5:130137.
9. Meade LB, Borden SH, McArdle P, Rosenblum MJ, Picchioni MS, Hinchey KT. From theory to actual practice: Creation and application of milestones in an internal medicine residency program, 2004–2010. Med Teach. 2012;34:717723.
10. Holmboe ES, Edgar L, Hamstra SJ. The Milestones Guidebook. 2016. Chicago, IL: Accreditation Council for Graduate Medical Education; http://www.acgme.org/Portals/0/MilestonesGuidebook.pdf
. Accessed October 21, 2017.
11. Swing SR, Beeson MS, Carraccio C, et al. Educational milestone development in the first 7 specialties to enter the next accreditation system. J Grad Med Educ. 2013;5:98106.
12. Holmboe ES, Yamazaki K, Edgar L, et al. Reflections on the first 2 years of milestone implementation. J Grad Med Educ. 2015;7:506511.
13. Holmboe ES, Call S, Ficalora RD. Viewpoint: Milestones and competency-based medical education in internal medicine. JAMA Intern Med. 2016;176:16011602.
14. Accreditation Council for Graduate Medical Education. Specialty milestones. http://www.acgme.org/Specialties
. Published 2015. Accessed October 21, 2017.
15. Marks D, Yardley L. Research Methods for Clinical and Health Psychology. 2004.Thousand Oaks, CA: SAGE.
16. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77101.
17. Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study. Nurs Health Sci. 2013;15:398405.
18. Accreditation Council for Graduate Medical Education. Common program requirements. http://www.acgme.org/What-We-Do/Accreditation/Common-Program-Requirements
. Published September 2013. Accessed October 21, 2017.
19. Institute of Medicine. Graduate Medical Education That Meets the Nation’s Health Needs. 2014.Washington, DC: National Academies Press.
20. Beeson MS, Holmboe ES, Korte RC, et al. Initial validity analysis of the emergency medicine milestones. Acad Emerg Med. 2015;22:838844.
21. Hauer KE, Clauser J, Lipner RS, et al. The internal medicine reporting milestones: Cross-sectional description of initial implementation in U.S. residency programs. Ann Intern Med. 2016;165:356362.
22. Hauer KE, Vandergrift J, Hess B, et al. Correlations between ratings on the resident annual evaluation summary and the internal medicine milestones and association with ABIM certification examination scores among U.S. internal medicine residents, 2013–2014. JAMA. 2016;316:22532262.
23. Li ST, Tancredi DJ, Schwartz A, et al. Competent for unsupervised practice: Use of pediatric residency training milestones to assess readiness. Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group. Acad Med. 2017;92:385393.