Objective: This report describes the development, initial implementation, and reliability of American College of Surgeons Resident Objective Structured Clinical Examination (ACS OSCE).
Background: Variability in clinical knowledge and skills of entering surgery residents has been demonstrated. The ACS OSCE was developed to evaluate and help remediate residents' knowledge and skills in managing patients with life-threatening conditions.
Methods: A task force of surgeons and professional educators developed 10 standardized clinical case stations, evaluation checklists, and rating scales. Standardized patients (SPs) evaluated each resident's clinical skills (history taking, physical examination, communication, and SP-global scores). Residents completed checklists on diagnosis and management. Coefficient alpha and item-total correlations were used, respectively, to assess internal consistency of metrics and station validity. The resident's overall performance for each station was calculated by combining scores of the individual skills. Analysis of variance compared performance across different institutions.
Results: A total of 103 postgraduate year 1 residents from 7 institutions completed the OSCE. Reliability coefficients of skills ranged from 0.38 for diagnosis to 0.68 for global scores. For overall performance on individual stations, the reliability coefficients ranged from 0.51 to 0.82. Using total percent correct scores from highly reliable stations (α > 0.8), wide variability in resident performance was demonstrated within and between the 7 institutions.
Conclusions: The ACS OSCE was successfully implemented across diverse institutions. It had moderate reliability and demonstrated variability among entering surgery residents. The ACS OSCE is now available for broader implementation. It should help reduce resident variability and address the requirements of Accreditation Council for Graduate Medical Education for resident supervision.
This report describes the development, initial implementation and reliability testing of American College of Surgeons Objective Structured Clinical Examination in a multi-institutional setting with the aim to evaluate residents' knowledge and skills to manage patients with life threatening conditions, develop remediation strategies, and define appropriate supervision levels.
*American College of Surgeons, Chicago, IL
†Duke University Medical Center, Durham, NC
‡University of Nebraska Medical Center, Omaha, NE
§Veterans Administration Nebraska Western Iowa Health Care System, Omaha, NE
¶New York Medical College, Valhalla, NY; and
‖New York Methodist Hospital, Brooklyn, NY.
Reprints: Ranjan Sudan, MD, Division of Education, American College of Surgeons, 633 North St Clair Street, Chicago, IL 60611. E-mail: Ranjan.Sudan@duke.edu.
Disclosure: This study was supported by the Agency for Healthcare Research and Quality Grant U18 HS 12021 (principal investigator: A.K.S.). The authors declare no conflicts of interest.