Journal Logo

Research Article

A Cross-sectional Analysis of Minimum USMLE Step 1 and 2 Criteria Used by Orthopaedic Surgery Residency Programs in Screening Residency Applications

Schrock, John B. BA; Kraeutler, Matthew J. MD; Dayton, Michael R. MD; McCarty, Eric C. MD

Author Information
Journal of the American Academy of Orthopaedic Surgeons: June 2017 - Volume 25 - Issue 6 - p 464-468
doi: 10.5435/JAAOS-D-16-00725
  • Free

Abstract

Matching into Accreditation Council for Graduate Medical Education (ACGME)-approved orthopaedic surgery residency positions in the United States has continued to remain competitive for applicants in recent years. Average United States Medical Licensing Examination (USMLE) Step 1 and 2 scores for applicants to orthopaedic surgery residency programs have increased for the past 8 years.1 Applicants who match into orthopaedic surgery residency positions have substantially higher USMLE Step 1 and 2 scores, as well as more research experiences and products (ie, abstracts, posters, presentations, and publications), compared with those of unmatched applicants.1

In 2014, the National Resident Matching Program (NRMP) conducted a general survey of program directors (PDs).2 In this survey, PDs were asked to cite important factors for ranking applicants. The fifth and seventh most cited factors were the USMLE Step 1/COMLEX (Comprehensive Osteopathic Medical Licensing Examination) Level 1 score and the USMLE Step 2 Clinical Knowledge/COMLEX Level 2 Cognitive Evaluation score. Multiple studies have sought to predict applicant factors that correlate with resident success.3-8 In a survey of PDs conducted in 2003, the USMLE Step 1 score was the second most important factor in residency applicant selection.9

Many orthopaedic surgery residency programs use a minimum USMLE Step 1 and/or Step 2 score to screen applicants; however, to our knowledge, detailed information on the use of USMLE minimum scores in screening residency program applicants has not been previously published. This information on minimum scores is important for prospective applicants, especially for those who have borderline USMLE scores. Many applicants are not aware that some programs have a minimum Step 1 and/or Step 2 score that is required for further review of an application. Thus, knowledge of minimum score criteria used by residency programs may provide realistic expectations for medical students hoping for further consideration by these programs. The purpose of this study was to analyze both how and to what extent the selection personnel of orthopaedic surgery residency programs use minimum USMLE Step 1 and 2 score criteria in screening residency applications.

Methods

A survey was sent to each ACGME-approved allopathic orthopaedic surgery residency PD in the United States (Table 1). In the survey, PDs were asked if they currently use a minimum USMLE Step 1 and/or Step 2 score to screen residency applications and how these minimum criteria may or may not have changed in recent years. In addition, PDs were asked to provide as much information as possible regarding their filtering process. Responses and comments were recorded and analyzed. PDs were given the opportunity to state how absolute the USMLE criteria were. If the criteria were not strict, they were then considered a “soft” cutoff although the program was still characterized as having minimum score criteria.

Table 1
Table 1:
Survey on the Use of Minimum USMLE Step 1 and 2 Scores by Orthopaedic Surgery Residency Programsa

Results

During the 2015/2016 application cycle, 151 US ACGME-approved allopathic orthopaedic surgery residency programs were identified by the Electronic Residency Application Service. Survey responses were received between January 2016 and July 2016 from 113 of the 151 programs (75%). One program did not have the information requested in the survey and five programs declined participation, resulting in analysis of 107 responses.

Eighty-nine programs (83%) used a minimum Step 1 score criterion, with 71 of the PDs noting that the minimum Step 1 score was a “hard” cutoff. Eighty-three programs (78%) required a Step 1 score ≥210, 80 (75%) required a score ≥220, 57 (53%) required a score ≥230, and 22 (21%) required a score ≥240 (Figure 1). Of the programs instituting a minimum Step 1 score, the mean and median minimum criteria scores were 230 and 231, respectively.

Figure 1
Figure 1:
The percent of programs represents the percent of all responding programs that use a minimum Step 1 score equal to that shown on the x-axis. A minimum Step 1 score was used by 89 of 107 program directors, with five programs using a different method for determining the minimum score. USMLE = United States Medical Licensing Examination.

Five programs used a different strategic approach for determining a minimum score. Other programs set a minimum score based on the average USMLE scores for all residency applicants, while some averaged the Step 1 and 2 scores of each applicant to determine eligibility for further review. Two PDs indicated the use of geographic considerations to filter applicants in addition to requiring a minimum Step 1 score. One PD used a maximum Step 1 and 2 score criteria in addition to a minimum score, with the filtering out of applicants scoring >270.

Seventeen programs mentioned special considerations in addition to using minimum USMLE score criteria. Ten programs indicated special considerations for medical students who completed a rotation at their home program, five programs mentioned taking into account special recommendations (ie, unsolicited letters, e-mails, or phone calls recommending an applicant to the program), four programs allowed special considerations for an institution’s own medical students, and one gave special consideration to women and/or minorities. Seventy-five programs (70%) did not have a minimum criterion for Step 2 scores.

Of the 83 responders who noted having a minimum score criterion, 49 (59%) have increased the minimum Step 1 and/or 2 score or scores in recent years or plan to do so in coming years. Nineteen PDs (16%) reported an exact minimum score and the year the minimum score was changed. Among these responses, the mean Step 1 minimum score was 223 within the last 3 years. Twelve PDs mentioned the high volume of applications as a reason for using a minimum score and for increasing the minimum score in recent years.

Discussion

The use of USMLE scores in the screening process for orthopaedic surgery residency applicants has not previously been reported in detail. The most significant finding of this study is that a large proportion of orthopaedic surgery residency PDs use a USMLE Step 1 minimum score when screening residency applications. In 2014, the NRMP conducted a survey of PDs.2 Based on these results, the average orthopaedic surgery residency program had five postgraduate year-1 positions available, received 549 applications, and sent 72 interview invitations. In our survey, multiple PDs mentioned using a minimum Step 1 score because of the high number of applications they received. Many orthopaedic surgery residency applicants do not realize that having a Step 1 and/or Step 2 score below the minimum criteria employed by programs may eliminate their application from further review, particularly at programs that employ a “hard” cutoff. Thus, publication of these minimum score criteria is important information for applicants to allow them to estimate their probability of receiving further consideration by programs based on other application material.

Based on survey responses, a variety of methods are used to create minimum USMLE score criteria. One method involved PDs setting a minimum score to allow for review of a limited number of applications, typically between 200 and 500. Another method involved PDs reviewing every application, followed by distributing only those with scores above an established minimum to additional faculty reviewers.

Use of minimum Step 2 scores is highly variable among PDs. Because these scores are not always available at the time of application submission, many PDs do not factor Step 2 scores into the review process. In contrast, some PDs report that Step 2 scores are more important than those from Step 1 in their review of applications. PDs also acknowledge that applicants may compensate for a low Step 1 score with an improved Step 2 score.

Multiple studies have sought to determine correlations between USMLE scores and resident success.3-8 Raman et al3 reported that the Step 2 score, the number of honors grades in medical school clerkships, and membership in Alpha Omega Alpha demonstrated the strongest correlations with resident performance as measured by the American Board of Orthopaedic Surgery Part I scores, Orthopaedic In-Training Examination scores, and subjective ratings by faculty. Egol et al4 reported that a higher number of honors grades on clinical rotations correlated with higher grading of residency interpersonal skills, higher grading of resident knowledge, and higher surgical skills evaluations. The authors also noted that Step 1 scores correlated with high Orthopaedic In-Training Examination scores.4

Opinions from PDs were diverse on the use of minimum scores, with many unique responses. Numerous PDs mentioned that they did not believe that higher scores equate with future highly qualified orthopaedic surgeons, although the high number of applicants necessitates the use of minimum scores. Because of the high number of applications and increasingly higher Step 1 and 2 scores in recent years,1 programs need to set minimums to create a manageable number of applications to be reviewed. Although great importance is placed on high USMLE scores in order for an application to be reviewed, the most important factor when determining rank lists is typically interpersonal characteristics of applicants. In the NRMP’s 2014 survey of orthopaedic surgery PDs, the four most cited factors for determining rank in lists of applicants were interactions with faculty during the interview and visit, interpersonal skills, interactions with house staff during the interview and visit, and feedback from current residents.2

Many PDs mentioned special considerations for applicants who had rotated at the program’s home institution. In these cases, the minimum score either did not apply to these applicants or the minimum score was reduced. Orr et al10 surveyed Army orthopaedic surgery residency PDs who unanimously expressed that performance during the on-site rotation at the PD’s institution was the most important factor, followed by USMLE Step 1 and 2 scores. Baldwin et al11 studied the association between the number of away rotations an applicant chose to perform and his or her odds of matching into an orthopaedic surgery residency program. Two or more rotations increased the match odds of an applicant, and those who did exactly two away rotations tended to be stronger academically than those who completed three or more rotations.11 Bernstein et al9 conducted a survey of orthopaedic surgery residency PDs in 2003 and determined the three most important residency selection criteria were a rotation at the PD’s institution, the USMLE Step 1 score, and rank in medical school.

Based on the survey results of our study, other important factors are involved in determining interview selections in addition to USMLE scores. Applicants with low scores combined with various impressive factors, such as being in the top 25% of their class, Alpha Omega Alpha membership, research, volunteerism, and being an All-American athlete, can warrant review by some PDs. However, Step 1 and/or Step 2 scores still remain a highly important factor in applications submitted to orthopaedic surgery residency programs. When PDs were queried about the reason for the use of a minimum score, the most cited answer was the high number of applicants.

In noting the strengths of this study, it is the first to report detailed information on the use of minimum USMLE score criteria by orthopaedic surgery residency programs. The limitations of this study should also be noted. No response was received from 25% of the programs, thus resulting in unknown minimum score criteria for these programs. Furthermore, the results of this study reflect only the minimum USMLE score criteria used by orthopaedic surgery programs during a cross-sectional period from January 2016 to July 2016, and these criteria are likely to continue to change in future years. The minimum USMLE scores reported in this study do not necessarily correlate with those of matriculated residents, but rather those who receive further consideration by programs in an initial application review. Finally, correlation between USMLE scores and performance as an orthopaedic surgery resident was not determined and is beyond the scope of this study.

Regardless of any change in orthopaedic surgery residency positions in the future, the problem of not matching and its associated mental and financial burdens may possibly be avoided if applicants have realistic expectations when determining their specialty of choice. Of the 48.4% of orthopaedic surgery residency applicants who responded to the 2015 NRMP applicant survey, the median number of applications submitted for matched US seniors was 70.12 This finding would equate to $1,419.00 in application costs, assuming every application was to an orthopaedic surgery residency program. Knowledge of minimum USMLE scores may help applicants avoid paying application fees to programs that have a minimum score above the applicant’s score. LaPaglia et al13 encourage applicants to consider their statistical chances as well as geographic variations in the supply of residency training slots to guide rank order decisions. The data we present on the use of minimum USMLE scores by orthopaedic surgery residency programs should help guide and assist applicants in the pursuit of orthopaedic surgery residency positions.

Conclusion

In an effort to reduce the number of applications to be reviewed, a large proportion of orthopaedic surgery residency PDs use a minimum USMLE Step 1 score when screening applications. In addition, many PDs are planning to raise this minimum score in the coming years because of increasingly higher scores and more applicants. Some PDs will still consider applicants with scores below a minimum criterion if the applicants have other impressive attributes, such as research and successful clinical experience.

References

References printed in bold type are those published within the past 5 years.

1. Schrock JB, Kraeutler MJ, Dayton MR, McCarty EC: A comparison of matched and unmatched orthopaedic surgery applicants from 2006 to 2014: Data from the National Resident Matching Program. J Bone Joint Surg Am 2017;99(1):e1.28060237
2. National Resident Matching Program: Results of the 2014 NRMP Program Director Survey. http://www.nrmp.org/wp-content/uploads/2014/09/PD-Survey-Report-2014.pdf. Accessed March 7, 2017.
3. Raman T, Alrabaa RG, Sood A, Maloof P, Benevenia J, Berberian W: Does residency selection criteria predict performance in orthopaedic surgery residency? Clin Orthop Relat Res 2016;474(4):908-914.25940336
4. Egol KA, Collins J, Zuckerman JD: Success in orthopaedic training: Resident selection and predictors of quality performance. J Am Acad Orthop Surg 2011;19(2):72-80.21292930
5. Crawford CH III, Nyland J, Roberts CS, Johnson JR: Relationship among United States Medical Licensing Step I, orthopedic in-training, subjective clinical performance evaluations, and American Board of Orthopedic Surgery examination scores: A 12-year review of an orthopedic surgery residency program. J Surg Educ 2010;67(2):71-78.20656602
6. Dougherty PJ, Walter N, Schilling P, Najibi S, Herkowitz H: Do scores of the USMLE Step 1 and OITE correlate with the ABOS Part I certifying examination?: A multicenter study. Clin Orthop Relat Res 2010;468(10):2797-2802.20352386
7. Thordarson DB, Ebramzadeh E, Sangiorgio SN, Schnall SB, Patzakis MJ: Resident selection: How we are doing and why? Clin Orthop Relat Res 2007;459(459):255-259.17415012
8. Black KP, Abzug JM, Chinchilli VM: Orthopaedic in-training examination scores: A correlation with USMLE results. J Bone Joint Surg Am 2006;88(3):671-676.16510836
9. Bernstein AD, Jazrawi LM, Elbeshbeshy B, Della Valle CJ, Zuckerman JD: An analysis of orthopaedic residency selection criteria. Bull Hosp Jt Dis 2002-2003;61(1-2):49-57.12828380
10. Orr JD, Hoffmann JD, Arrington ED, Gerlinger TL, Devine JG, Belmont PJ Jr: Army orthopaedic surgery residency program directors’ selection criteria. J Surg Orthop Adv 2015;24(2):120-124.25988694
11. Baldwin K, Weidner Z, Ahn J, Mehta S: Are away rotations critical for a successful match in orthopaedic surgery? Clin Orthop Relat Res 2009;467(12):3340-3345.19582529
12. National Resident Matching Program: Results of the 2015 NRMP Applicant Survey. http://www.nrmp.org/wp-content/uploads/2015/09/Applicant-Survey-Report-2015.pdf. Accessed March 7, 2017.
13. LaPaglia D, Robiner WN, Yozwiak JA, Brosig C, Cubic B, Leventhal G: A shortage of medical residency positions: Parallels with psychology. Acad Psychiatry 2015;39(6):706-712.26022617
Keywords:

orthopaedic surgery residency; United States Medical Licensing Examination; USMLE; Step 1; Step 2

Copyright 2017 by the American Academy of Orthopaedic Surgeons.