Secondary Logo

Journal Logo

Using Longitudinal Milestones Data and Learning Analytics to Facilitate the Professional Development of Residents

Early Lessons from Three Specialties

Holmboe, Eric S. MD; Yamazaki, Kenji PhD; Nasca, Thomas J. MD; Hamstra, Stanley J. PhD

doi: 10.1097/ACM.0000000000002899
Research Report: PDF Only

Purpose: To investigate the effectiveness of using national, longitudinal milestones data to provide formative assessments to identify residents at risk of not achieving recommended competency milestone goals by residency completion. The investigators hypothesized that specific, lower milestone ratings at earlier time points in residency would be predictive of not achieving recommended Level (L) 4 milestones by graduation.

Method: In 2018, the investigators conducted a longitudinal cohort study of emergency medicine (EM), family medicine (FM), and internal medicine (IM) residents who completed their residency programs from 2015 to 2018. They calculated predictive values (PVs) and odds ratios (ORs), adjusting for nesting within programs, for specific milestone rating thresholds at 6-month intervals for all subcompetencies within each specialty. They used final milestone ratings (May/June 2018) as the outcome variables, setting L4 as the ideal educational outcome.

Results: The investigators included 1,386 (98.9%) EM residents, 3,276 (98.0%) FM residents, and 7,399 (98.0%) IM residents in their analysis. The percentage of residents not reaching Level 4 by graduation ranged from 11-31% in EM, 16-53% in FM, and 5-15% in IM. Using a milestone rating of Level 2.5 or lower at the end of PGY2, the predictive probability of not attaining the L4 milestone graduation goal ranged from 32-56% in EM, 32-67% in FM, and 15-36% in IM.

Conclusions: Longitudinal milestones ratings may provide educationally useful, predictive information to help individual residents address potential competency gaps, but the predictive power of the milestones ratings varies by specialty and subcompetency within these three adult care specialties.

E.S. Holmboe is chief research, milestones development and evaluation officer, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID:

K. Yamazaki is senior analyst, Milestones, Accreditation Council for Graduate Medical Education, Chicago, Illinois.

T.J. Nasca is president and chief executive officer, Accreditation Council for Graduate Medical Education, Chicago, Illinois; professor of medicine and molecular physiology, Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, Pennsylvania; and senior scholar, Department of Education University of Illinois at Chicago School of Medicine, Chicago, Illinois; ORCID:

S.J. Hamstra is vice-president, Milestones Research and Evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois; adjunct professor, Faculty of Education, University of Ottawa, Ottawa, Ontario, Canada; and adjunct professor, Department of Medical Education, Feinberg School of Medicine, Northwestern University, Chicago, Illinois; ORCID:

Supplemental digital content for this article is available at

Funding/Support: The authors acknowledge the support of the Accreditation Council for Graduate Medical Education.

Other disclosures: The authors are employees of the Accreditation Council for Graduate Medical Education. Dr. Holmboe receives royalties from Elsevier for a textbook on assessment.

Ethical approval: The American Institutes for Research reviewed the research protocol reported herein and deemed it exempt.

Previous presentations: Portions of this research were presented at the Association of Medical Education in Europe (AMEE), August 2018, Basel, Switzerland; and at the International Conference on Residency Education (ICRE), October 2018, Halifax, Nova Scotia, Canada.

Data: No external data outside of the Accreditation Council for Graduate Medical Education was used for this study.

Correspondence should be addressed to Eric S. Holmboe, ACGME, 401 North Michigan Avenue, Chicago, IL 60611; telephone (312) 755-7034; email:; Twitter: @acgme.

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

© 2019 by the Association of American Medical Colleges