Secondary Logo

Journal Logo

Do End-of-Rotation Evaluations Adequately Assess Readiness to Operate?

Ahle, Samantha L. MD, MHS; Schuller, Mary MSEd; Clark, Michael J. PhD; Williams, Reed G. PhD; Wnuk, Gregory MHSA; Fryer, Jonathan P. MD, MHPE; George, Brian C. MD, MAEd

doi: 10.1097/ACM.0000000000002936
Research Report: PDF Only
Free
SDC
PAP

Purpose: Medical educators have developed no standard way to assess the operative performance of surgical residents. Most residency programs use end-of-rotation (EOR) evaluations for this purpose. Recently some programs have implemented workplace-based “micro-assessment” tools that faculty use to immediately rate observed operative performance. The authors sought to determine (1) the degree to which EOR evaluations correspond to workplace-based micro-assessments and (2) which factors most influence EOR and directly observed workplace-based performance ratings and how the influence of those factors differs for each assessment method.

Method: In 2017, the authors retrospectively analyzed EOR evaluations and immediate postoperative assessment ratings of surgical trainees from a university-based training program from the 2015-2016 academic year. A Bayesian multivariate mixed model was constructed to predict operative performance ratings for each type of assessment.

Results: Ratings of operative performance from EOR evaluations vs. workplace-based micro-assessment ratings had a Pearson correlation of 0.55. Postgraduate year (PGY) of training was the most important predictor of operative performance ratings on EOR evaluations: model estimates ranged from 0.62 to 1.75 and increased with PGY. For workplace-based assessment, operative autonomy rating was the most important predictor of operative performance (coefficient = 0.74).

Conclusions: EOR evaluations are perhaps most useful in assessing the ability of a resident to become a surgeon compared to other trainees in the same PGY of training. Workplace-based micro-assessments may be better for assessing a trainee’s ability to perform specific procedures autonomously, therefore, perhaps providing more insight into a trainee’s true readiness for operative independence.

S.L. Ahle is a general surgery resident, Yale School of Medicine, New Haven, Connecticut.

M. Schuller is manager, Surgical Education, Department of Surgery, Northwestern University, Chicago, Illinois.

M.J. Clark is lead statistician, Consulting for Statistics, Computing and Analytics Research, University of Michigan, Ann Arbor, Michigan.

R.G. Williams was, at the time of the research reported here, adjunct research professor, Indiana University School of Medicine, Indianapolis, Indiana.

G. Wnuk is program manager, Center for Surgical Training and Research, University of Michigan, Ann Arbor, Michigan.

J.P. Fryer is vice chair of education, Department of Surgery, Northwestern University, Chicago, Illinois.

B.C. George is assistant professor and director, Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan; ORCID: https://orcid.org/0000-0002-9404-5255.

Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A728.

Dedication: Dr. Reed Williams unexpectedly passed away during this study. In this project, as in so many others, Reed’s contributions were substantial. He leaves a hole in the authors’ hearts, and the authors greatly miss his humility, kindness, and generosity.

Acknowledgements: The authors would like to thank the Surgical Education Research Fellowship of the Association of Surgical Education, as well as the many institutional members of the Procedural Learning and Safety Collaborative (PLSC) for their support of this project.

Funding/Support: None reported.

Other disclosures: None reported.

Ethical approval: The institutional review board of Yale School of Medicine has granted exemption for this study.

Correspondence should be addressed to Samantha L. Ahle, Department of Surgery, Yale University School of Medicine, 330 Cedar Street, FMB 107, New Haven, CT 06519; telephone: (203) 785-7890; email: samantha.ahle@yale.edu; Twitter: @YaleSurgery

© 2019 by the Association of American Medical Colleges