Specialties in which the majority of positions are filled by graduating seniors of U.S. medical schools may be considered more competitive than specialties that fill with a high percentage of international graduates. The top row of Table 3 indicates the U.S. fill rate for each specialty. These data were derived from published 2007 match statistics for the NRMP, the San Francisco match (neurosurgery and ophthalmology), and the American Urological Association (AUA) match. Orthopedic surgery, plastic surgery, and otolaryngology had more than 90% of their positions filled by U.S. graduating seniors, whereas family medicine, internal medicine, and neurology had much lower percentages of their positions filled by U.S. graduating seniors. To facilitate the analysis of the survey results, we adapted the approach used by Wagoner and Suriano2 and grouped specialties according to the percentages of residency spots that were filled by U.S. graduates. Specialties with U.S. graduate fill rates above 81% were grouped and designated as the “most competitive” residency programs. Specialties with U.S. graduate fill rates between 61% and 80% were designated as being “competitive.” Specialties with U.S. graduate fill rates below 60% were designated as being “less competitive.” For each criterion, the means across all specialties were calculated and are displayed in Table 3. Within specialties, significant differences existed. It should be noted that three criteria (grades in preclinical courses, grades in other senior electives, and usefulness of the MSPE) were never found to be statistically similar to the top criterion group for any specialty. In Table 3, the selection criteria also are ranked by numbers from 1 (highest) to 16 (lowest) to illustrate absolute rank within a specialty.
Table 4 shows the mean USMLE Step 1 and Step 2 CK scores for the groups of specialties that are listed. These data were obtained from the match statistics reported by the NRMP, the San Francisco match, and the AUA match. The mean USMLE Step 1 scores for matched students are significantly higher in the “most competitive” specialties when compared with the other two groups. The USMLE Step 1 scores in the “competitive” specialties are significantly higher than for the “less competitive” group. This trend holds for the “most competitive” specialties with USMLE Step 2 CK scores.
Our survey results are consistent with previously reported studies conducted in only one specialty. Although Otero et al8 used a different survey instrument when reporting on selection criteria for radiology residency programs, the top criteria did not significantly differ from our findings except that they placed a higher value on letters of recommendation. Similar studies in orthopedics and emergency medicine show that the most highly valued selection criteria are consistent with the results of our survey.4,9 In family medicine, former studies have emphasized the importance of more qualitative selection criteria. However, our study indicates that program directors in family medicine are increasingly valuing more objective criteria, such as class rank and USMLE Step 2 scores.14 Studies reporting the relative importance of comprehensive selection criteria for residency programs in other specialties have not been performed.
The results of this survey indicate that, of the top academic selection criteria (Table 2), all are based on clinical performance, with the exception of USMLE Step 1 scores. Not surprisingly, indicators that reflect excellence in clinical performance are valued across the specialties by residency program directors. Among nonacademic selection criteria that rise to the tops of program directors’ lists, letters of recommendation and audition electives figure prominently, again highlighting students’ clinical performance (see Table 3). The availability of USMLE Step 2 results in time for residency application, in addition to the development of the new clinical skills component of the USMLE exam, may have resulted in the relatively high ranking that program directors place on Step 2 results. Given the increasing emphasis that program directors are placing on Step 2 performance, students should be advised to take both components of this exam in time for residency application. Additionally, despite their subjective nature, letters of reference are highly ranked by most specialties, and students should be careful in selecting faculty members who write in support of their applications.
Despite student perceptions that research in medical school is an essential ingredient for a successful application to residency, our findings indicate that research experience ranks low among selection criteria when all specialties are grouped together. The NRMP also found that research experience in medical school did not differ significantly among U.S. seniors who matched in their specialties of choice and those who did not.1 This was consistent across competitive and noncompetitive specialties. It should be noted, however, that in our survey, program directors in particularly competitive specialties (radiation oncology, plastic surgery, neurosurgery, and dermatology) ranked research experience highly (Table 3). It may be that when all other selection criteria are outstanding among applicants to a particular specialty, research experience or research publications may help discriminate candidates.
In our survey, grades in preclinical courses are not highly valued by program directors. This may be because there is considerable variability in the naming and content of courses in medical schools in the preclinical curriculum, perhaps making grades difficult to interpret. The USMLE Step 1 exam is a well-understood and objective means to assess basic science knowledge, and it may act as a substitute for preclinical grades. It could be inferred that schools that have pass/fail grading systems in the preclinical years may not necessarily disadvantage their students in the residency application process.
The MSPE was ranked lowest of all criteria by the program directors. In theory, MSPEs should provide most of the information that program directors value. However, MSPE usefulness may be limited by variability in content and quality across institutions. Additionally, information included in the MSPE can be acquired through the transcript, USMLE score report, ERAS application, or letters of recommendation. Finally, the MSPE release date of November 1 may be too late to be useful to program directors, particularly for early match specialties. Given the enormous effort that goes into the creation of the MSPE, it may be worth further study to assess why program directors do not find it useful and to judge whether the resources required to create it are justified.
The responses from program directors can be interpreted in a variety of ways. There is remarkable consistency in the top six selection criteria between the specialties with >80% U.S. senior fill rates, 60% to 80% U.S. senior fill rates, and <60% U.S. senior fill rates (see Table 3). Among other criteria, however, our study does indicate trends that seem to be based on the competitiveness of the specialties. USMLE Step 2 CK score ranks higher in the specialties with <60% U.S. senior fill rates as compared with specialties with the highest U.S. fill rates. This difference is even more prominent for the Step 2 CS exam. Does performance on these exams reflect resident characteristics that are more highly valued in these specialties? Do students who perform poorly on Step 1 apply more often to the less competitive specialties, thereby raising the relative importance of Step 2? The data from the NRMP show that USMLE scores for both Step 1 and Step 2 are higher in the most competitive specialties (Table 4). Given the strong academic performance of individuals who apply to the most competitive specialties, characteristics that distinguish individuals among this highly accomplished group are difficult to discern. The higher emphasis on class rank, AOA membership, and published medical school research may represent accomplishments that allow program directors to select one student over another.
There are several limitations to our study. First, there was variability in response rates from program directors across specialties and in some specialties this may not be a representative sample. We do not have information about why some specialties had a higher response rate than others. However, this study surveyed a larger and more comprehensive group of program directors from more specialties than has been performed in prior research.2 Our methodology conformed with best practices in survey research.15 Although our response rates varied by specialty, the broad scope of our study and the large number of respondents overall make our results noteworthy and, we believe, a reliable snapshot of widespread practices. Future studies that achieve relatively high response rates across all sampled specialties could further support our conclusions. Second, we chose not to survey community hospitals that did not have university affiliations. The FREIDA database lists relatively few non-university-affiliated community hospital programs in most of the selected 21 specialties. Although in a few specialties the number of programs may have provided an adequate number of responses for statistical comparisons, in the majority of non-university-affiliated programs the likelihood of an adequate number of responses was very low. The limitation is that our results do not provide the perspective of community-based program directors who do not have university affiliations. Third, because of the diverse formats that exist in conducting interviews, we did not explore the relative importance of the interview in the final selection of candidates.
In summary, our study provides important information about the current perspectives of program directors across specialties regarding the relative importance of specific selection criteria. Given the changes that have occurred in information available to program directors, this study may be useful in advising students about career selection. Additional studies are necessary to understand which of these selection criteria predict residents’ performance. This may become particularly important as the NBME considers changing the format and content of licensure examinations, because, after such reforms, data that are currently highly valued by residency program directors (e.g., USMLE Step 1 scores) may no longer be available.
The authors greatly appreciate the assistance of Dr. Norma Wagoner, who worked with us to develop and initiate this study. The authors thank Stephanie Miller for assistance in the submission and collection of the survey results, Dr. Jessica Tooredman Casey and Sarah Umetsu for compiling the program directors list, and Dr. Jenny Huang for her help with the statistical analysis. Finally, the authors thank Dr. William McGaghie for his critique of early drafts of this manuscript.
2 Wagoner NE, Suriano JR. Program directors’ responses to a survey on variables used to select residents in a time of change. Acad Med. 1999;74:51–58.
3 Hemaida RS, Kalb E. Using the analytic hierarchy process for the selection of first-year family practice residents. Hosp Top. 2001;79:11–15.
4 Bernstein AD, Jazrawi LM, Elbeshbeshy B, Della Valle CJ, Zuckerman JD. Orthopaedic resident-selection criteria. J Bone Joint Surg Am. 2002;84:2090–2096.
5 Grantham JR. Radiology resident selection: Results of a survey. Invest Radiol. 1993;28:99–101.
6 Garden FH, Smith BS. Criteria for selection of physical medicine and rehabilitation residents. A survey of current practices and suggested changes. Am J Phys Med Rehabil. 1989;68:123–127.
7 DeLisa JA, Jain SS, Campagnolo DI. Factors used by physical medicine and rehabilitation residency training directors to select their residents. Am J Phys Med Rehabil. 1994;73:152–156.
8 Otero HJ, Erturk SM, Ondategui-Parra S, Ros PR. Key criteria for selection of radiology residents: Results of a national survey. Acad Radiol. 2006;13:1155–1164.
9 Crane JT, Ferraro CM. Selection criteria for emergency medicine residency applicants. Acad Emerg Med. 2000;7:54–60.
10 Bajaj G, Carmichael KD. What attributes are necessary to be selected for an orthopaedic surgery residency position: Perceptions of faculty and residents. South Med J. 2004;97:1179–1185.
11 Turner NS, Shaughnessy WJ, Berg EJ, Larson DR, Hanssen AD. A quantitative composite scoring tool for orthopaedic residency screening and selection. Clin Orthop. 2006;449:50–55.
12 Galazka SS, Kikano GE, Zyzanski S. Methods of recruiting and selecting residents for U.S. family practice residencies. Acad Med. 1994;69:304–306.
13 Taylor CA, Weinstein L, Mayhew HE. The process of resident selection: A view from the residency director’s desk. Obstet Gynecol. 1995;85:299–303.
14 Travis C, Taylor CA, Mayhew HE. Evaluating residency applicants: Stable values in a changing market. Fam Med. 1999;31:252–256.
© 2009 Association of American Medical Colleges
15 Mangione TW. Mail Surveys: Improving the Quality. Thousand Oaks, Calif: Sage Publications; 1995.