To establish the number of operative performance observations needed for reproducible assessments of operative competency.
Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown.
Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement.
For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings.
The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.
*Department of Surgery, Indiana University, Indianapolis, IN
†Academic Programs and Services, American Board of Medical Specialties, Chicago, IL
‡Department of Surgery, Northwestern University, Evanston, IL
§Department of General and Thoracic Surgery, Northwestern University, Evanston, IL
¶Department of General Surgery, Massachusetts General Hospital, Boston, MA
||Department of General Surgery, Brigham and Women's Hospital, Boston, MA
**Department of Surgery, University of Michigan, Ann Arbor, MI.
Reprints: Reed G. Williams, PhD, Department of Surgery, Indiana University, 108 Maple Grove, Springfield, IL 62712. E-mail: email@example.com.
The initial development of SIMPL was funded by grants from Massachusetts General Hospital, Northwestern University, and Indiana University. Later development was funded by contributions from the 25 institutional members of PLSC, grants from the American Board of Surgery, and from the Association of Program Directors in Surgery.
The participating investigators were instrumental in creating the conditions necessary to acquire the data but bear no responsibility for analysis or interpretation of the data or for conclusions and recommendations presented in this manuscript.
Participating Investigators: Laura Torbeck, PhD1, John T. Mullen, MD4, Edward Auyang, MD7, Jeffrey G. Chipman, MD8, Jennifer Choi, MD1, Michael Choti, MD9, Eric Endean, MD10, Eugene F. Foley, MD11, Samuel Mandell, MD12, Andreas H. Meier, MD, MEd13, Douglas S. Smink, MD, MPH5, Kyla P. Terhune, MD14, Paul Wise, MD15, Debra DaRosa, PhD3, Nathaniel Soper, MD3, Joseph B. Zwischenberger, MD10, Keith D. Lillemoe, MD4, Chandrakanth Are, MD, MBA16 on behalf of the Procedural Learning and Safety Collaborative.
1Indiana University, Surgery, Indianapolis, IN, USA, 2American Board of Medical Specialties, Chicago, IL, USA, 3Northwestern University, Surgery, Chicago, IL, USA, 4Massachusetts General Hospital, Surgery, Boston, MA, USA, 5Brigham And Women's Hospital, Surgery, Boston, MA, USA, 6University Of Michigan, Surgery, Ann Arbor, MI, USA 7University Of New Mexico, Surgery, Albuquerque, NM, USA 8University Of Minnesota, Surgery, Minneapolis, MN, USA 9University Of Texas Southwestern, Surgery, Dallas, TX, USA 10University Of Kentucky, Surgery, Lexington, KY, USA 11University Of Wisconsin, Surgery, Madison, WI, USA 12University Of Washington, Surgery, Seattle, WA, USA 13State University of New York, Surgery, Syracuse, NY, USA 14Vanderbilt University Medical Center, Surgery, Nashville, TN, USA 15Washington University, Surgery, Saint Louis, MO, USA 16University of Nebraska, Surgery, Omaha, NE, USA.
Reprints will not be available from the authors.
The authors report no conflicts of interest.