Core Entrustable Professional Activities for Entering Residency: A National Survey of Graduating Medical Students’ Self-Assessed Skills by Specialty : Journal of the American College of Surgeons

Journal Logo

Original Scientific Articles

Core Entrustable Professional Activities for Entering Residency: A National Survey of Graduating Medical Students’ Self-Assessed Skills by Specialty

Grbic, Douglas PhD; Gielissen, Katherine A MD, MHS; Obeso, Vivian MD; Amiel, Jonathan M MD; Jayas, Amy MPH; Andriole, Dorothy A MD, FACS

Author Information
Journal of the American College of Surgeons: December 2022 - Volume 235 - Issue 6 - p 940-951
doi: 10.1097/XCS.0000000000000395
  • Open
  • SDC
  • Infographic

In 2014, the Association of American Medical Colleges (AAMC) released a list of 13 Core Entrustable Professional Activities (EPAs; see Table 1) for entering residency as a set of tasks that all graduating medical students should be prepared to perform under indirect supervision on day one of residency, regardless of specialty.1 At that time, a national survey of members of the Association of Program Directors in Surgery (APDS) was conducted to determine the extent to which APDS members agreed that they had confidence that their entering interns were competent to perform each of the 13 Core EPAs without direct supervision.2 Results indicated that, although respondents generally endorsed all 13 Core EPAs as being important for a trustworthy physician, there were only 3 of the EPAs (EPA1: Gather a history and perform a physical examination; EPA5: Document a clinical encounter in the patient record; and EPA9: Collaborate as a member of an interprofessional team) for which > 50% of respondents agreed or strongly agreed that they were confident their entering interns were competent to perform under indirect supervision.2

Table 1. - The Association of American Medical Colleges’ Core Entrustable Professional Activities for Entering Residency44
The 13 Core Entrustable Professional Activities for entering residency
1: Gather a history and perform a physical examination
2: Prioritize a differential diagnosis following a clinical encounter
3: Recommend and interpret common diagnostic and screening tests
4: Enter and discuss orders and prescriptions
5: Document a clinical encounter in the patient record
6: Provide an oral presentation of a clinical encounter
7: Form clinical questions and retrieve evidence to advance patient care
8: Give or receive a patient handover to transition care responsibility
9: Collaborate as a member of an interprofessional team
10: Recognize a patient requiring urgent/emergent care and initiate evaluation/management
11: Obtain informed consent for tests and/or procedures
12: Perform general procedures of a physician
13: Identify system failures and contribute to a culture of safety and improvement

Amid growing concerns regarding the preparedness of incoming surgery interns for the responsibilities they assume at the start of PGY-1 of training, the American Board of Surgery, the American College of Surgeons (ACS), the APDS, and the Association for Surgical Education (ASE) released a joint consensus statement regarding a national surgery residency preparation curriculum,3 and the ACS, APDS, and ASE collaborated to develop a standardized national curriculum (the ACS/APDS/ASE Resident Prep Curriculum).4 This curriculum, administered by the ACS, includes 11 scenarios that map to many of the Core EPAs for Entering Residency; >100 institutions in the US are included on a list of pilot institutions.5,6 In the 2020 Consensus statement released by the APDS regarding ideal medical student experiences in preparing for the PGY-1 year of training in general surgery, recommendations include readiness for entrustment in all 13 Core EPAs.5

As the undergraduate medical education (UME) and graduate medical education (GME) communities work toward more effective and safe transitions to residency, understanding differences in preparedness for residency training in relation to intended specialty is an integral first step in addressing training gaps.7 Although studies have investigated learners’ preparedness for residency training from the perspective of the learner,8,9 medical school,10,11 and residency program,12,13 few have focused on specialty-specific aspects of preparedness.14 Given the heterogeneous nature of educational experiences during the final (“fourth”) year of medical school15,16 (during which students can tailor much of their fourth year curriculum depending on their intended specialty), students may be differentially prepared to perform Core EPA tasks depending on their planned specialty. Medical students generally seek the advice and direction of program directors in their chosen specialty regarding how to optimally prepare for the start of residency training in the specialty. In the context of recommending readiness for entrustment in all 13 Core EPAs for their entering PGY-1 residents among surgery program directors, we hypothesized that students’ self-assessed skills in the Core EPAs would differ at graduation based on specialty, with students planning to enter surgery self-assessing their skills in the Core EPAs more highly than students planning to enter various other specialties. To examine the relationship between intended specialty and self-assessed skills in the Core EPAs, we used national-level data from several sources, collected from and about students who completed the AAMC’s annual Graduation Questionnaire (GQ),17 to determine the extent to which there was an independent relationship between graduating students’ intended specialties and self-assessed skills in the Core EPAs.

METHODS

Setting and participants

Our study-eligible population included all US medical students at Liaison Committee for Medical Education (LCME)-accredited schools who had matriculated in 2015 or later and graduated in academic year 2018 to 2019. We included selected items from the 2019 GQ and AAMC Student Records System (SRS), and students’ first-attempt results on the National Board of Medical Examiners US Medical Licensing Examination Step 2 Clinical Skills (CS) and Step 2 Clinical Knowledge (CK), released with permission from the National Board of Medical Examiners. We also included 2 school-level indicators derived from data collected on the LCME Annual Medical School Questionnaire Part II 2018 to 2019.

Association of American Medical Colleges Graduation Questionnaire items

The GQ is administered for program evaluation purposes on a confidential and voluntary basis to all graduating students at US LCME-accredited medical schools.17 Items included on the GQ evolve somewhat on an annual basis but include a broad array of experiential and career plan items. The EPA-related items we examined were newly added to the 2019 GQ. We derived our outcome variable from the following self-assessed skill item: “Indicate whether you agree or disagree with the following statements about your preparedness for beginning a residency program: I have the skills to...” For each of the 13 activities described as the Core EPAs, respondents could select strongly disagree, disagree, neutral, agree, or strongly agree. It should be noted that the 13 activities described as Core EPAs were not identified as “Core EPAs” on the GQ. For example, the GQ item pertaining to “EPA1: Gather a history and perform a physical examination” appeared on the GQ as “Gather a history and perform a physical examination” (ie, the respondent was asked to indicate their agreement with the statement that, “I have the skills to gather a history and perform a physical examination”). Based on responses for the 13 activities, we computed our outcome measure, “skill acquisition agreement” (SAA) group, which assigns respondents to groups based on their patterns of agreement to having the skill to perform each of the 13 activities (described in more detail in Data Analysis).

We derived specialty, our primary explanatory variable, from responses to the item: “When thinking about your career, what is your intended area of practice?” We grouped respondents into one of the following 13 specialty categories: “anesthesiology,” “emergency medicine,” “family medicine”, “general surgery” (including choices of surgery, vascular surgery, and thoracic surgery), “internal medicine” (IM), “obstetrics and gynecology” (OBGYN), “ophthalmology,” “orthopaedic surgery,” “other surgical specialties” (including choices of neurological surgery, otolaryngology, plastic surgery, and urology), “pediatrics,” “psychiatry,” “radiology,” and “all other specialties” (including choices of dermatology, neurology, child neurology, pathology, physical medicine and rehabilitation, preventive medicine, radiation oncology, and medical genetics).

Because frequency of observation/feedback in practicing a skill is associated with self-assessed skill acquisition,18 we included responses to the item: “In the workplace (clinical setting), how often during medical school have supervising residents or faculty members directly observed you performing the following activity and also provided you with immediate (within 24 hours) verbal or written feedback on your performance of the activity?” For each activity, response options were: never, once, 2 to 5 times, or >5 times. These activities included EPAs 1 to 11, EPA13, and 6 procedural skills: perform basic cardiopulmonary resuscitation, perform bag and mask ventilation, perform sterile technique, perform venipuncture, insert an intravenous line, and place a urinary catheter (ie, instead of a single item for EPA12, “Perform general procedures of a physician,” respondents were asked about each of these 6 specific procedural skills). We also computed an “Extent of observation and feedback across the 13 skills” variable as the number of activities (of 13 possible, using “insert an intravenous line” for EPA12) for which the respondent reported “>5 times.”

Association of American Medical Colleges Student Records System items

The AAMC’s SRS “houses secure, centralized enrollment information on the national medical student population.”19 Based on SRS data, we created a dichotomous variable for gender that we included given the substantial differences in association with gender of specialty.20 We also used SRS data for medical school of enrollment, which we included given school differences in the extent to which curricular content and assessment related to the Core EPAs may have been incorporated in the curriculum (eg, in required clinical clerkships, in school-specific use of the ACS/APDS/ASE Resident Prep Curriculum, and/or in other school-specific required or elective residency preparation courses).21

National Board of Medical Examiners

The National Board of Medical Examiners released first-attempt US Medical Licensing Examination Step 2 CS (since discontinued) and Step 2 CK results for our study,22,23 which we included because these measures, which are not self-reported, align with several Core EPAs’ content and may vary substantially among graduates by specialty.24

Liaison Committee for Medical Education Annual Questionnaire Part II

From the 2018 to 2019 LCME Questionnaire, we created 2 school-level measures (not mutually exclusive measures)—for attendance at a school which indicated that selected training topics were covered within a one-course-for-all-students required fourth-year transition to residency curriculum (“Attendance at a medical school with required fourth-year transition to residency course for all students”), and attendance at a school which indicated that selected training topics were covered within required specialty-specific fourth-year transition to residency curriculum (“Attendance at a medical school with required specialty-specific fourth-year transition to residency course[s]”).

Data analysis

The merged database of individual-level linked data for all variables was de-identified for analysis. This study used latent profile analysis (LPA)25-28 to generate latent profiles, or groups, of respondents based on patterns of responses across the 13 EPA-SAA items. LPA is a probabilistic technique that assigns respondents an estimated probability of belonging to a latent group. The optimal number of groups was identified by assessing model-fit statistics—Bayesian Information Criterion values and average within- and between-group estimated probabilities—from a series of LPA models. From the best fitting model, respondents are assigned probabilities of belonging to each group and each respondent is then assigned to the group for which they have the highest probability.

We used chi-square tests, ANOVA, and t-tests for the difference between means to assess univariate associations between variables. Cohen’s d was used to estimate the effect size for the difference between 2 means; in general, an effect size is considered small if Cohen’s d is ≥0.2 but <0.5, medium if Cohen’s d is ≥0.5 but <0.8, and large if Cohen’s d is ≥0.8.29 Our dataset has a multilevel structure with students nested in 142 US LCME-accredited medical schools. Because shared similarities (eg, exposure to school-specific required and elective curricula or school-specific programmatic objectives for the MD-degree program), can characterize students within a given learning environment, nested data violate the assumption of independence of observations, potentially resulting in incorrect estimates of standard errors. We therefore used multilevel logistic regression to examine the extent to which each measure described in preceding text was independently associated with group membership. Only measures statistically significant in the univariate analysis were included in the regression model. We assessed model fit via the receiver operator characteristic (area under the curve); an area under the curve value >0.7 is generally considered a reasonable model fit.30 We performed all quantitative analyses using Stata 15 (StataCorp, College Station, TX). The AAMC Human Subjects Protection Program staff reviewed this study and determined that it was exempt from further institutional review board review because no human subjects were involved.

RESULTS

Study sample derivation

Of 16,047 students who matriculated to a US LCME-accredited medical school in 2015 or later and graduated in academic year 2018 to 2019, there were 13,813 (86.1% of 16,047) who responded to the 2019 GQ, at least in part. Of these 13,813 GQ respondents, there were 12,308 respondents (89.1% of all 13,813 GQ respondents; 76.7% of all 16,047 study-eligible class of 2019 graduates) with complete data for all variables of interest in our study. Table 2 shows the characteristics of the 16,047 study-eligible class of 2019 graduates, grouped by final study sample inclusion status. As shown, compared with the 3,739 graduates (23.3% of all 16,047 eligible graduates) excluded from our final sample, the 12,308 graduates (76.7% of all 16,047 eligible graduates) included were disproportionately women, had passed Step 2 CS, and had higher mean Step 2 CK scores (each p < 0.001).

Table 2. - Characteristics of All Eligible 2019 Graduates, Grouped by Study Sample Inclusion Status
Characteristic Total
(n = 16,047)
In study sample Not in study sample p Value
All eligible 2019 graduates, n (%) 16,047 12,308 (76.7)* 3,739 (23.3)*
Gender, n (%) 16,044 (100.0) 12,308 (76.7)* 3,736 (23.3)* <0.001
 Men 8,307 (51.8)† 6,215 (50.5)† 2,092 (56.0)†
 Women 7,737 (48.2)† 6,093 (49.5)† 1,644 (44.0)†
USMLE Step 2 CS, n (%) 16,037 (100.0) 12,308 (76.7)* 3,729 (23.3)* <0.001
 Fail, 1st attempt 637 (4.0)† 437 (3.6)† 200 (5.4)†
 Pass, 1st attempt 15,400 (96.0)† 11,871 (96.4)† 3,529 (94.6)†
USMLE Step 2 CK <0.001
 n (%) 16,042 (100.0) 12,308 (76.7)* 3,734 (23.3)*
 Mean ± SD 246.3 ± 14.5 247.1 ± 14.2 243.5 ± 15.0
* Percentage shown is percentage of row total.
† Percentage shown is percentage of column total within the characteristic category.
CK, clinical knowledge; CS, clinical skills; GQ, graduation questionnaire; USMLE, US Medical Licensing Examination.

Outcome derivation

After assessing model fit statistics, LPA results showed that a 2-group model was the best fit for the data (for detail, see Supplemental Digital Content 1, https://links.lww.com/JACS/A136). After each respondent was assigned to the group for which they had the highest probability, one group comprised 7,863 (63.9%) respondents with an average within-group probability of 0.99 and the other group comprised 4,445 (36.1%) respondents with an average within-group probability of 0.98. Figure 1 shows the mean value (0 = “strongly disagree” to 4 = “strongly agree”) for each of the 13 activities by group. For each of the 13 Core EPAs, the mean difference between the 2 groups was significant (p < 0.001) and the effect size was large (Cohen’s d >0.8).

F1
Figure 1.:
Mean value for each of the 13 entrustable professional activities (EPAs) by high (63.9%) vs moderate (36.1%) skills acquisition agreement (SAA) group membership.

We described the group of respondents with mean values for most of the 13 Core EPAs roughly equivalent to “strongly agree” (and none below “agree”) as the “high SAA” group (n = 7,863). We described the group of respondents with mean values for the 13 Core EPAs roughly equivalent to “agree” (including means of 7 of the 13 Core EPAs below “agree”) as the “moderate SAA” group (n = 4,445). Within each SAA group (ie, both within the high SAA group only and within the moderate SAA group only), there were 6 EPAs (EPA4: Enter and discuss orders/prescriptions; EPA8: Patient handover; EPA10: Urgent/emergent care; EPA11: Informed consent; EPA12: Procedures; and EPA13: Patient safety) that had uniformly significantly lower mean values compared with the remaining 7 EPAs (EPA1: History/physical examination; EPA2: Differential diagnosis; EPA3: Diagnostic/screening tests; EPA5: Oral presentation; EPA6: Written documentation; EPA7: Evidence-based medicine; and EPA9: Interprofessional collaboration).

Table 3 presents the associations between explanatory variables and the SAA group outcome measure. As shown, specialty was associated with SAA group distribution (p < 0.001). Respondents in general surgery, other surgical specialties (including neurological surgery, otolaryngology, plastic surgery, and urology), orthopaedic surgery, IM, and emergency medicine specialty categories were slightly overrepresented in the high SAA group, whereas respondents in most other specialty categories (including OBGYN and ophthalmology, among others) were underrepresented in the high SAA group; respondents in the “all other specialties” group (dermatology, neurology, child neurology, pathology, physical medicine and rehabilitation, preventive medicine, radiation oncology, and medical genetics) were similarly represented in both the high SAA and moderate SAA groups. Also shown, men and women respondents who had passed Step 2 CS were more highly represented in the high SAA group (p < 0.001), as were respondents with higher Step 2 CK scores (high SAA group mean score ± SD = 248.0 ± 14.0, moderate SAA group mean score ± SD = 245.7 ± 14.4; p < 0.001), although the effect size was small (Cohen’s d = 0.16). Respondents who attended a medical school with required specialty-specific fourth-year transition to residency course(s) were more highly represented in the high SAA group (p < 0.001), and respondents who attended a medical school with a required fourth-year transition to residency course for all students were similarly represented in both the high SAA and moderate SAA groups. The mean number of EPAs for which respondents reported more frequent observation/feedback (>5 times) was greater among the high SAA group than among the moderate SAA group (mean ± SD: 9.1 ± 2.7) vs mean ± SD: 6.6 ± 3.4, respectively; p < 0.001) and the effect size was large (Cohen’s d = 0.87). Also, not shown in Table 3, school attended by the respondent was associated with SAA group distribution (school mean for percent of respondents in high SAA group = 64%; SD = 11%; range = 16% to 94%; p < 0.001) and with the frequency of observations/feedback (school mean for percent of EPAs reported as receiving 5 or more observations/feedback = 63%; SD = 6%; range = 43% to 85%; p < 0.001).

Table 3. - Univariate Associations with Skills Acquisition Agreement Outcome
Variable Total*
(N = 12,308)
Moderate SAA group
(n = 4,445*)
High SAA group
(n = 7,863*)
p Value
Specialty, n (%) <0.001
 Anesthesiology 777 (6.3) 314 (7.1) 463 (5.9)
 Emergency medicine 1,236 (10.0) 388 (8.7) 848 (10.8)
 Family medicine 1,039 (8.4) 413 (9.3) 626 (8.0)
 General surgery† 943 (7.7) 316 (7.1) 627 (8.0)
 Internal medicine 2,613 (21.2) 914 (20.6) 1,699 (21.6)
 Obstetrics and gynecology 875 (7.1) 327 (7.4) 548 (7.0)
 Ophthalmology 277 (2.3) 107 (2.4) 170 (2.2)
 Orthopaedic surgery 521 (4.2) 155 (3.5) 366 (4.7)
 Other surgical specialties‡ 667 (5.4) 195 (4.4) 472 (6.0)
 Pediatrics 1,228 (10.0) 472 (10.6) 756 (9.6)
 Psychiatry 676 (5.5) 276 (6.2) 400 (5.1)
 Radiology 558 (4.5) 245 (5.5) 313 (4.0)
 All other specialties§ 898 (7.3) 323 (7.3) 575 (7.3)
Gender, n (%) 0.017
 Men 6,215 (50.5) 2,181 (49.1) 4,034 (51.3)
 Women 6,093 (49.5) 2,264 (50.9) 3,829 (48.7)
USMLE Step 2 CS, n (%) <0.001
 Fail, 1st attempt 437 (3.6) 207 (4.7) 230 (2.9)
 Pass, 1st attempt 11,871 (96.4) 4,238 (95.3) 7,633 (97.1)
Attendance at a medical school with required fourth-year transition to residency course for all students, n (%) 0.402
 No 4,201 (34.1) 1,496 (33.7) 2,705 (34.4)
 Yes 8,107 (65.9) 2,949 (66.3) 5,158 (65.6)
Attendance at a medical school with required specialty-specific fourth-year transition to residency course(s), n (%) <0.001
 No 6,859 (55.7) 2,567 (57.8) 4,292 (54.6)
 Yes 5,449 (44.3) 1,878 (42.2) 3,571 (45.4)
USMLE Step 2 CK, mean (SD) 247.1 (14.2) 245.7 (14.4) 248.0 (14.0) <0.001
Extent of observation and feedback across the 13 activities, mean ± SD 8.2 ± 3.2 6.6 ± 3.4 9.1 ± 2.7 <0.001
* Percentages shown are percentages within the column for each variable.
† “General surgery” includes surgery, vascular surgery, and thoracic surgery.
‡ “Other surgical specialties” includes neurological surgery, otolaryngology, plastic surgery, and urology.
§ “All other specialties” includes dermatology, neurology, child neurology, pathology, physical medicine and rehabilitation, preventive medicine, radiation oncology, and medical genetics.
CK, clinical knowledge; CS, clinical skills; USMLE, US Medical Licensing Examination.

Supplemental Digital Content 2 through 5 (https://links.lww.com/JACS/A136) present additional findings. Supplemental Digital Content 2 and 3 present a heat map of the percentage of respondents reporting “>5 times” observations/feedback by specialty for each EPA (for EPA12, “Perform general procedures of a physician,” data are shown for each of 6 specific procedural skills). As shown in Supplemental Digital Content 2 for EPAs 1 through 11 and EPA13, the percentage of all GQ respondents who reported observations/feedback >5 times varied widely on an EPA-specific basis, ranging from 9% (1,112 of 12,308) for EPA13 (Patient safety) to 93% (11,440 of 12,308) for EPA6 (Oral presentation). There were also significant differences in observations/feedback frequency by specialty for EPAs 1, 2, 4, 7, 8, 10, 11, and 13. For instance, for EPA4 (Enter and discuss orders/prescriptions), the percentage of respondents reporting observations/feedback >5 times ranged from 54% (475 of 875) for OBGYN to 70% (1,822 of 2,613) for IM. As shown in Supplemental Digital Content 3 for the 6 procedural skills, the proportions of respondents who reported observation/feedback >5 times in each of these 6 procedural skills varied widely across the 6 skills and also by specialty for each procedural skill. Although 85% (10,499 of 12,308) of all respondents reported observation/feedback >5 times for “Perform sterile technique,” <50% of all respondents reported observation/feedback >5 times for each of the other 5 procedural skills shown.

Supplemental Digital Content 4 (https://links.lww.com/JACS/A136) presents a heat map of the mean value for SAA by specialty for each EPA. Consistent with Figure 1, EPAs 1, 5, 6, and 9 generally had the highest means; EPAs 2, 3, and 7 had intermediate means; and EPAs 4, 8, and 10 to 13 had the lowest means. On an EPA-specific basis, means varied for EPAs from 2.71 for EPA13 (Patient Safety) to 3.76 for EPA1 (History/Physical examination). There were significant differences in mean SAA by specialty across all EPAs. Supplemental Digital Content 5 (https://links.lww.com/JACS/A136) presents a heat map of the mean value for SAA by all “surgical specialties” (general surgery, neurological surgery, OBGYN, ophthalmology, orthopaedic surgery, otolaryngology, plastic surgery, thoracic surgery, urology, vascular surgery) vs “all nonsurgical specialties” (all remaining specialties not included in the “surgical specialties” group) for each EPA. Results showed significantly higher means among “surgical specialties” respondents for EPAs 1 to 3, 6, 7, 10, 12, and 13. For EPA4, “surgical specialties” respondents had a significantly lower mean than “nonsurgical specialties” respondents (2.88 vs 3.00, respectively; p < 0.001). The mean SAA did not significantly differ between the 2 groups for the 4 remaining EPAs: 5, 8, 9, and 11 (p > 0.05).

Regression analysis

Table 4 presents results from the multilevel logistic regression model. Model fit was reasonable (area under the curve = 0.76; 95% CI, 0.75–0.76). To assess our hypothesis that students intending general surgery self-assessed their skills more highly than students intending various other specialties, we treated general surgery as the reference category to which each remaining specialty category was compared. The odds of high SAA group membership were lower for respondents in each of the anesthesiology (adjusted odds ratio [AOR] = 0.61, p < 0.001), ophthalmology (AOR = 0.65; p = 0.006), pediatrics (AOR = 0.82; p = 0.047), psychiatry (AOR = 0.67, p < 0.001), and radiology (AOR = 0.62, p < 0.001) categories compared with those in the general surgery category. The odds of high SAA group membership were similar for respondents in each of the emergency medicine, family medicine, OBGYN, orthopedics surgery, other surgical specialties, internal medicine, and all other specialtiescategories compared with those in the general surgery category.

Table 4. - Multilevel Logistic Regression Results
Variable AOR 95% CI p Value
Specialty
 Anesthesiology 0.61 0.49-0.77 <0.001
 Emergency medicine 0.94 0.77-1.15 0.568
 Family medicine 0.82 0.67-1.01 0.062
 General surgery*
 Internal medicine 0.88 0.74-1.05 0.157
 Obstetrics and gynecology 0.84 0.68-1.04 0.114
 Ophthalmology 0.65 0.48-0.89 0.006
 Orthopaedic surgery 1.06 0.82-1.37 0.678
 Other surgical specialties† 1.21 0.96 -1.54 0.112
 Pediatrics 0.82 0.67-1.00 0.047
 Psychiatry 0.67 0.53-0.84 <0.001
 Radiology 0.62 0.49-0.79 <0.001
 All other specialties‡ 0.84 0.68-1.04 0.118
Gender
 Men Ref Ref
 Women 1.01 0.93-1.10 0.839
USMLE Step 2 CS
 Fail, 1st attempt Ref Ref
 Pass, 1st attempt 1.36 1.09-1.69 0.007
Attendance at a medical school with required specialty-specific fourth-year transition to residency course(s)
 No Ref Ref
 Yes 1.14 0.98-1.31 0.084
USMLE Step 2 CK, per unit increase§ 1.12 1.07-1.69 <0.001
Extent of observation and feedback across the 13 skills, per unit increase§ 2.41 2.30-2.52 <0.001
N = 12,308. Because the data consist of 12,308 respondents nested in 142 US LCME-accredited medical schools, we used multilevel logistic regression to examine the extent to which each measure in the model was independently associated with group membership.
* “General surgery” includes surgery, vascular surgery, and thoracic surgery.
† “Other surgical specialties” includes neurological surgery, otolaryngology, plastic surgery, and urology.
‡ “All other specialties” includes dermatology, neurology, child neurology, pathology, physical medicine and rehabilitation, preventive medicine, radiation oncology, and medical genetics.
§ Units are in standard deviations.
AOR, adjusted odds ratio; CK, clinical knowledge; CS, clinical skills; USMLE, US Medical Licensing Examination.

Results also showed that passing Step 2 CS (AOR = 1.36, p = 0.007), higher Step 2 CK score (per unit increase, AOR = 1.12, p < 0.001), and higher EPA observation/feedback frequency (AOR = 2.41, p < 0.001) were each associated with a greater odds of high SAA group membership. The odds of high SAA group membership were similar for women (compared with men, AOR = 1.01, p = 0.84) and for attendance at a medical school with (compared to without) required specialty-specific fourth-year transition to residency course or courses (AOR = 1.14, p = 0.08). Finally, if data for EPA12: Perform general procedures (example: intravenous line placement) of a physician were excluded from analyses, LPA produced the same 2-group solution and all regression results were essentially unchanged (data not shown).

DISCUSSION

The APDS has recently recommended readiness for entrustment in all 13 Core EPAs (ie, ready to be entrusted to perform the activity under indirect supervision) for graduating students entering general surgery training.5 Our findings represent a first step in providing national-level data regarding the extent to which US LCME-accredited medical school graduates planning to enter surgery training have acquired the skills to perform these activities. Our results suggest that, on the basis of self-assessment data only, those intending general surgery may be relatively more well prepared to perform the Core EPAs compared with their peers intending various other specialties. Findings for each of orthopaedic surgery, “other surgical specialties” (of neurosurgery, plastic surgery, urology, and otolaryngology), OB/GYN, and ophthalmology shared some similarities with findings for general surgery, but there were some specialty-specific differences. However, our findings also highlight “systemic” gaps at a national level in graduating medical students’ self-assessed skills for certain EPAs across all specialties.

Although all 13 EPAs are considered by program directors to be relevant activities for graduates entering general surgical training,2,5 this may not necessarily be so for program directors in other specialties—there may be specialty-specific differences in program directors’ expectations for their entering PGY-1 residents. According to results of a national survey of IM program directors regarding their perspectives on the 13 Core EPAs,31 although most respondents indicated that it was not necessary for incoming interns in IM to be prepared to perform EPA12 (Procedures) under indirect supervision at the start of residency, most respondents indicated that incoming interns must or should be ready to perform all the other Core EPAs under indirect supervision.31 In a separate study focused on competency-based, time-variable education in pediatrics (the Education in Pediatrics Across the Continuum [EPAC] study), readiness to perform all the Core EPAs, except EPA12, under indirect supervision was among the criteria for advancement from UME to GME.32 Thus, although there is likely a role for specialty-specific curricula in UME for select Core EPAs (eg, EPA12), many of the Core EPAs may be broadly applicable across numerous specialties and for substantial proportions of all graduating medical students. Although only 2 national program director organizations (surgery program directors and internal medicine program directors) have taken positions on their expectations for incoming residents regarding the Core EPAs, it is worth noting that program directors in these 2 specialties alone had responsibility for the incoming residents who filled 42% (13,506 of 32,194) of all PGY-1 positions offered in the National Resident Matching Program in 2019.33

Based on response patterns across all 13 EPA-SAA items, our LPA identified 2 groups of respondents: about one-third of respondents in a moderate SAA group and two-thirds in a high SAA group. Across the 13 activities, we observed a common within-group profile of agreement with highest levels of self-assessed skill for EPAs 1, 5, 6, and 9; intermediate levels for EPAs 2, 3, and 7; and relatively low levels for EPAs 4, 8, and 10 to 13. Our study included cross-sectional data collected from students at only one point in time—in spring of the respondents’ final year in medical school—so we do not have information regarding when in the course of their undergraduate education the respondents in our study may have felt that they had acquired skills in these activities. However, insights about when in the course of their medical school education students may acquire the skills to perform various EPAs have emerged from the work of the EPAC consortium.32 In brief, at the 4 institutions that participated in the EPAC study, small numbers of selected students (~4 per year at each school) committed to careers in pediatrics were advanced in a time-variable competency-based program of progression through UME, into GME, and through GME. Readiness for entrustment to perform the Core EPAs (except EPA12) under indirect supervision, as determined by the Clinical Competency Committee, was among the criteria that students had to meet to advance to GME (students also had to meet all other graduation requirements for the MD-degree program at their respective medical schools).32,34 For these students, the Core EPAs were implemented in an EPA framework.35 Assessment data for students from all 4 schools participating in EPAC were pooled to create growth curves predicting students’ achievement of level 3a (as specified by the EPAC investigators, “Allowed to practice EPA only under reactive/on-demand supervision, with supervisor immediately available; all findings double checked”) for each EPA.32 Students were predicted to reach level 3a early in the third year of medical school for EPAs 1, 5, and 6.32 These findings are consistent with the reporter-interpreter-manager-educator model regarding identification of these 3 EPAs as “reporter-level,”36 which are foundational activities well ingrained in the medical education curriculum starting in the preclerkship years. EPAC students were predicted to reach level 3a for EPAs 2 to 4, 7 to 11, and 13 later in the third year and well into the fourth year of medical school. For EPA12, level 3a was predicted not to be reachable by students in EPAC at any point in medical school, which the study authors suggested was due to a lack of procedural practice opportunities for their students.32

We speculate that there may be greater variation in the extent to which medical schools have integrated curriculum content and assessment of the various non-Reporter level activities in the required curriculum for all students. Students may have fewer opportunities to perform and get feedback on these more advanced level EPAs in the clinical workplace, entering residency with relatively lower levels of self-assessed skills in these activities that they may be expected to perform under indirect supervision as new interns. Our findings suggest that EPAs 4, 8, and 10 through 13 may warrant particular attention of medical educators across the UME-GME continuum for joint efforts to optimally prepare graduates for the start of residency.

Our specialty findings may be of interest to surgery program directors and other GME leadership offering not only categorical, but also preliminary PGY-1 positions that may be filled by trainees subsequently entering advanced positions in specialties such as anesthesiology or ophthalmology. Our results suggest that there may be differences in skills at the start of GME training between categorical and preliminary position residents, depending on their ultimate specialty intentions. Therefore, certain trainees may need additional support and supervision at the start of the PGY-1 year. In addition, this information may be helpful for medical educators in UME regarding the need to communicate to students the expectations regarding the Core EPAs if they may enter a PGY-1 year in surgery, regardless of their ultimate specialty. Specific curricular experiences in the fourth year (eg, surgery acting internships/subinternships, the ACS/APDS/ASE Residency Preparation Curriculum), including opportunities to perform Core EPAs (among other aspects) might be particularly useful for their preparedness for the PGY-1 year of surgical training.

We did not have information at the school level regarding approaches taken to the integration of all or some of the Core EPAs in the curriculum, for all of their students or for a selected subset of their students (such as students participating in EPAC32). Thus, although the skills items we analyzed in our study are the skills that have been described as Core EPAs, we cannot make any inferences about how the respondents in our study may have acquired the skills. We speculate that variations in the extent to which the skills described as Core EPAs were integrated into required and elective UME curricula, and approaches taken to do so at the schools attended by students in our study, may have contributed to differences we observed across schools in SAA group membership. We describe a few such examples of different schools’ approaches here. At one school, readiness to perform Core EPAs 1 to 3, 5, and 6 under indirect supervision at the completion of the required clerkships’ curriculum is being piloted as a criterion for advancement to acting internships/subinternships in the final year of medical school.37 Some schools have used targeted approaches for a subset of Core EPAs during subinternships to increase students’ opportunities to perform and get feedback in more advanced, non–reporter-level Core EPAs38 (of note, a single-institution study of entering residents across numerous specialties reported independent associations between acting internship/subinternship participation and self-assessed readiness to perform some Core EPAs8). Finally, some schools have used, in whole or in part, the ACS/APDS/ASE Resident Prep Curriculum that involves 11 scenarios that map to 7 of the Core EPAs.5,6

We did have limited information at the school level regarding required transition to residency courses and observed an association in univariate analysis only between attendance at a school with specialty-specific transition to residency course(s) and SAA group membership. However, we did not have any information about the specialties in which these specialty-specific courses were offered and which, if any, such specialty-specific courses were taken by students at those schools that offered them. We did not observe an association in univariate analysis between attendance at a school with a required transition to residency course for all students and SAA group membership. This may be due, at least in part, to the wide variation across medical schools in the content and duration of these required transition to residency courses for all students.39

In considering our self-assessment data for the skills described as Core EPAs in the context of students’ preparation for the transition to residency, it is very important to note that the clinical skills required to begin a residency program extend well beyond those described as the Core EPAs; this caveat equally applies to those specialties for which program directors have endorsed the Core EPAs for their entering residents.5,31 LaFemina and colleagues5 described entrustability in the 13 Core EPAs as only one of a number of recommendations for senior students preparing for general surgery training. Similarly, in their report of the results of a national survey of internal medicine residents regarding important skills for internship, Pereira and colleagues40 observed that, of the 10 skills identified as being very important, only 4 overlapped with the skills described in the AAMC Core EPAs. Thus, skills in Core EPAs can be regarded as, potentially, a subset of all the skills needed in preparing to enter GME, but should not be considered as an exhaustive list encompassing all the skills required for the start of residency in any specialty.

Our study has numerous additional limitations. Because of the relatively small numbers of GQ respondents intending some surgical specialties (each of neurosurgery, urology, otolaryngology, and plastic surgery), we created a single “other surgical specialties” category for these 4 surgical specialties combined. The data included in our study regarding frequency of observation/feedback were subject to recall bias. We note that our outcome of interest was based on self-assessment data only; it is possible that specialty-specific differences in respondents’ overall self-confidence may have contributed to differences in self-assessed skills. Although meta-analysis of 35 published articles of medical student self-assessment suggests that students may be more accurate nearer the end of medical school,41 as were the GQ respondents in our study, these self-assessments of skill may be overestimates; program directors’ assessments of PGY-1 residents generally regarding readiness to perform EPAs may differ from those of graduate cohorts themselves.2,42 Despite limitations of self-assessment data, we do note that studies have shown similarities in patterns across various EPAs—to patterns we observed across EPAs in self-assessed skills in both high SAA and moderate SAA groups—in assessments by others of students’ readiness to perform EPAs under indirect supervision.32,42,43

Although we used a national dataset, there was selection bias in the final sample toward women and graduates who performed more highly on Step 2 CK and passed Step 2 CS. Because these Step results were independently associated with our study outcome, it is likely the distribution of moderate and high SAA is skewed toward the high SAA group in our study sample compared with all US LCME medical school graduates, in general. Finally, because our sample comprised only US LCME-accredited medical school graduates, our findings may not generalize to osteopathic or international medical school graduates.

CONCLUSIONS

In a national sample of sufficient size to examine 13 different specialty categories, we observed that graduating students’ self-assessed Core EPAs skills were higher for those intending general surgery, as well as those intending orthopaedic surgery, and “other” surgical specialties (plastic surgery, neurosurgery, otolaryngology, and urology, combined as a single group for analysis) compared with those intending various nonsurgical specialties. We identified some specialty-specific differences in graduates’ self-assessed skills in Core EPAs both among surgical specialty categories and among all specialty categories included in the study. We also observed similar patterns across all specialties in a group of Core EPAs that graduating medical students (regardless of specialty), in general, felt more well-prepared to perform, based on the self-assessment data we examined (EPAs 1, 5, 6, and 9) and in a group of Core EPAs that graduating medical students (regardless of specialty) generally felt less well prepared, based on the self-assessment data we examined, to perform (EPAs 4, 8, and 10 to 13). These results can inform collaborative efforts across the UME-GME continuum, catalyzed by the recent release of the Coalition for Physician Accountability Undergraduate to Graduate Review Committee report,7 to more explicitly define the skills that will be expected of graduates at the start of PGY-1 training in each specialty and to ensure graduates’ acquisition of these expected skills for the specialties they enter as PGY-1 residents

Author Contributions

Conceptualization: Andriole, Grbic, Gielissen, Obeso, Amiel, Jayas

Data curation: Grbic

Formal analysis: Grbic, Jayas

Methodology: Grbic, Andriole, Gielissen, Obeso, Amiel, Jayas

Project administration: Andriole, Grbic

Visualization: Grbic

Supervision: Andriole

Writing – original draft: Grbic

Writing – review and editing: Andriole, Grbic, Jayas, Gielissen, Obeso, Amiel

Acknowledgment:

We thank the National Board of Medical Examiners for providing access to the US Medical Licensing Examination Step 2 Clinical Skills and Step 2 Clinical Knowledge. We thank David Matthew (Data Steward for the AAMC’s Graduation Questionnaire data) and Diana Tung (AAMC Data Steward for LCME Annual Questionnaire data) for their support and advice.

    REFERENCES

    1. Lomis K, Amiel JM, Ryan MS, et al.; AAMC Core EPAs for Entering Residency Pilot Team. Implementing an entrustable professional activities framework in undergraduate medical education: early lessons from the AAMC Core Entrustable Professional Activities for Entering Residency Pilot. Acad Med. 2017;92:765–770.
    2. Lindeman BM, Sacks BC, Lipsett PA. Graduating students’ and surgery program directors’ views of the Association of American Medical Colleges Core Entrustable Professional Activities for Entering Residency: where are the gaps? J Surg Educ. 2015;72:e184–e192.
    3. American Board of Surgery, American College of Surgeons, Association of Program Directors in Surgery, and Association for Surgical Education. Statement on surgical preresidency preparatory courses. J Surg Educ. 2014;71:777–778.
    4. American College of Surgeons. Educational Programs. ACS/APDS/ASE resident prep curriculum. Available at: https://www.facs.org/for-medical-professionals/education/programs/acs-apds-ase-resident-prep-curriculum. Accessed July 17, 2022.
    5. LaFemina J, Ahuja V, Alseidi A, et al. APDS Consensus Statement: ideal senior medical student experiences for preparedness for general surgery internship. J Surg Educ. 2021;78:69–75.
    6. ACS/APDS/ASE Resident Prep Curriculum. Pilot Institutions. Available at: https://www.facs.org/-/media/files/education/resident-prep/acsapdsasepilotinstitutions.ashx. Accessed January 25, 2022.
    7. Coalition for Physician Accountability. The Coalition for Physician Accountability’s Undergraduate Medical Education-Graduate Medical Education Review Committee (UGRC): recommendations for comprehensive improvement of the UME-GME transition. Available at: https://physicianaccountability.org/wp-content/uploads/2021/08/UGRC-Coalition-Report-FINAL.pdf. Accessed October 14, 2021.
    8. Pearlman RE, Pawelczak MA, Bird JB, et al. Incoming interns perceived preparedness for core entrustable professional activities. Med Sci Educ. 2019;29:247–253.
    9. Smith HL, Craig SR, Yost WJ. Examination of entering residents’ self-reported confidence and supervision needs performing AAMC entrustable professional activities. J Grad Med Educ. 2018;10:474.
    10. Brenner J, Bird J, Ginzburg SB, et al. Trusting early learners with critical professional activities through emergency medical technician certification. Med Teach. 2018;40:561–568.
    11. Eliasz KL, Ark TK, Nick MW, et al. Capturing entrustment: using an end-of-training simulated workplace to assess the entrustment of near-graduating medical students from multiple perspectives. Med Sci Educ. 2018; 28:739–747.
    12. Pearlman RE, Pawelczak M, Yacht AC, Akbar S, Farina GA. Program director perceptions of proficiency in the core entrustable professional activities. J Grad Med Educ. 2017;9:588–592.
    13. CarlLee S, Rowat J, Suneja M. Assessing entrustable professional activities using an orientation OSCE: identifying the gaps. J Grad Med Educ. 2019;11:214–220.
    14. Coates WC, Crooks K, Slavin SJ, et al. Medical school curricular reform: fourth-year colleges improve access to career mentoring and overall satisfaction. Acad Med. 2008;83:754–760.
    15. Elnicki DM, Gallagher S, Willett L, et al.; Clerkship Directors in Internal Medicine Association of Program Directors in Internal Medicine Committee on Transition to Internship. Course offerings in the fourth year of medical school: how U.S. medical schools are preparing students for internship. Acad Med. 2015;90:1324–1330.
    16. Reddy ST, Chao J, Carter JL, et al. Alliance for clinical education perspective paper: recommendations for redesigning the “final year” of medical school. Teach Learn Med. 2014;26:420–427.
    17. Association of American Medical Colleges. Medical school graduation questionnaire. 2019 all schools summary report. Available at: https://www.aamc.org/system/files/2019-08/2019-gq-all-schools-summary-report.pdf. Accessed December 12, 2020.
    18. Kogan JR, Hatala R, Hauer KE, Holmboe E. Guidelines: the do’s, don’ts and don’t knows of direct observation of clinical skills in medical education. Perspect Med Educ. 2017;6:286–305.
    19. Association of American Medical Colleges. Student record system (SRS). Available at: https://www.aamc.org/data-reports/students-residents/report/student-records-system-srs. Accessed June 1, 2021.
    20. Pelley E, Carnes M. When a specialty becomes “women’s work”: trends in and implications of specialty gender segregation in medicine. Acad Med. 2020;95:1499–1506.
    21. McOwen KS, Whelan AJ, Farmakidis AL. Medical education in the United States and Canada, 2020. Acad Med. 2020;95(9S A Snapshot of Medical Student Education in the United States and Canada: Reports From 145 Schools):S2–S4.
    22. United States Medical Licensing Examination. Step 2 CK. Content description. Available at: https://usmle.org/step-2-ck/#contentdescription. Accessed December 12, 2020.
    23. United States Medical Licensing Examination. Work to relaunch USMLE Step 2 CS discontinued. Available at: https://www.usmle.org/work-relaunch-usmle-step-2-cs-discontinued. Accessed July 24, 2022.
    24. Association of American Medical Colleges. Report on residents. Table B2: USMLE Step 1 and Step 2 CK scores of first-year residents, by specialty. Available at: https://www.aamc.org/data-reports/students-residents/interactive-data/table-b2-usmle-step-1-and-step-2-ck-scores-first-year-residents-specialty. Accessed July 23, 2021.
    25. Boscardin CK. Profiling students for remediation using latent class analysis. Adv Health Sci Educ Theory Pract. 2012;17:55–63.
    26. Collins LM, Lanza ST. Latent Class and Latent Transition Analysis: With Applications in the Social, Behavioural, and Health Sciences. New York: Wiley2010.
    27. Silkens MEWM, Chahine S, Lombarts KMJMH, Arah OA. From good to excellent: improving clinical departments’ learning climate in residency training. Med Teach. 2018;40:237–243.
    28. Vermunt JK, Magidson J. Latent class analysis. The Sage Encyclopedia of Social Sciences Research Methods. Thousand Oaks: Sage Publications2004.
    29. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Erlbaum1988.
    30. Hosmer DW Jr, Lemeshow S, Sturdivant RX. Applied Logistic Regression. 3rd ed. Hoboken NJ: John Wiley & Sons2013.
    31. Angus SV, Vu TR, Willett LL, et al. Internal medicine residency program directors’ views of the core entrustable professional activities for entering residency: an opportunity to enhance communication of competency along the continuum. Acad Med. 2017;92:785–791.
    32. Murray KE, Lane JL, Carraccio C, et al.; Education in Pediatrics Across the Continuum (EPAC) Study Group. Crossing the gap: using competency-based assessment to determine whether learners are ready for the undergraduate-to-graduate transition. Acad Med. 2019;94:338–345.
    33. National Resident Matching Program. Results and Data: 2019 Main Residency Match®. Washington, DC: National Resident Matching Program2019.
    34. Andrews JS, Bale JF Jr, Soep JB, et al.; EPAC Study Group. Education in Pediatrics Across the Continuum (EPAC): first steps toward realizing the dream of competency-based education. Acad Med. 2018;93:414–420.
    35. Carraccio C, Martini A, Van Melle E, Schumacher DJ. Identifying core components of EPA implementation: a path to knowing if a complex intervention is being implemented as intended. Acad Med. 2021;96:1332–1336.
    36. Meyer EG, Kelly WF, Hemmer PA, Pangaro LN. The RIME model provides a context for entrustable professional activities across undergraduate medical education. Acad Med. 2018;93:954.
    37. Keeley MG, Gusic ME, Morgan HK, Aagaard EM, Santen SA. Moving toward summative competency assessment to individualize the postclerkship phase. Acad Med. 2019;94:1858–1864.
    38. Garber AM, Ownby AR, Trimble G, et al. Evaluating the association of a core EPA–Oriented Patient Handover Curriculum on Medical Students’ Self-reported Frequency of Observation and Skill Acquisition. Med Sci Educ. 2022;32:855–864. Available at: https://doi.org/10.1007/s40670-022-01578-x Accessed July 26, 2022.
    39. Association of American Medical Colleges. Curriculum Reports. Transition to Residency Courses in Medical School Program. Available at: https://www.aamc.org/data-reports/curriculum-reports/interactive-data/transition-residency-courses-medical-school-programs. Accessed July 21, 2022.
    40. Pereira AG, Harrell HE, Weissman A, et al. Important skills for internship and the fourth-year medical school courses to acquire them: a National Survey of Internal Medicine Residents. Acad Med. 2016;91:821–826.
    41. Blanch-Hartigan D. Medical students’ self-assessment of performance: results from three meta-analyses. Patient Educ Couns. 2011;84:3–9.
    42. Amiel JM, Andriole DA, Biskobing DM, et al.; Association of American Medical Colleges Core EPAs for Entering Residency Pilot Team. Revisiting the core entrustable professional activities for entering residency. Acad Med. 2021;96(7S):S14–S21.
    43. Brown DR, Moeller JJ, Grbic D, et al.; Core Entrustable Professional Activities for Entering Residency Pilot. Entrustment decision making in the core entrustable professional activities: results of a multi-institutional study. Acad Med. 2022;97:536–543.
    44. Englander R, Flynn T, Call S, et al. Toward defining the foundation of the MD degree: core entrustable professional activities for entering residency. Acad Med. 2016;91:1352–1358.

    Supplemental Digital Content

    © 2022 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the American College of Surgeons.