Secondary Logo

Journal Logo

Research Reports

Unmatched U.S. Allopathic Seniors in the 2015 Main Residency Match: A Study of Applicant Behavior, Interview Selection, and Match Outcome

Liang, Mei MS; Curtin, Laurie S. PhD; Signer, Mona M. MPH; Savoia, Maria C. MD

Author Information
doi: 10.1097/ACM.0000000000001501

Abstract

Each year, the National Resident Matching Program (NRMP) conducts the Main Residency Match (“the Match”) in which students and graduates from U.S. and international medical schools vie for residency positions in U.S. graduate medical education (GME) programs (“programs”). Participating in the Match is an essential step in the career of young physicians because it places applicants in training programs that qualify them for licensure and eventual specialty board certification. Applicants who do not match suffer a setback that can compromise their considerable personal and financial investment in medical education.1

Senior students from U.S. allopathic medical schools (“U.S. seniors”) are the most successful applicants in the Match. Match outcome data routinely show that U.S. seniors have the highest match rate; historically, about 94% match to postgraduate year one (PGY-1) positions every year, and they are more likely to match into competitive specialties.2,3 In 2015, 18,025 U.S. seniors participated in the Match, and only 1,093 failed to obtain PGY-1 positions. Some organizations believe that unmatched U.S. seniors do not match because of an insufficient number of residency slots,4 and others have lobbied the U.S. Congress to lift the cap on Medicare funding for GME,5,6 but for many years the number of PGY-1 positions in the Match has surpassed by several thousand the number of registered U.S. seniors. Figure 1 illustrates the numbers of PGY-1 positions in the Match, the numbers of U.S. seniors and other applicants, and U.S. senior PGY-1 match rates since 1982. U.S. seniors have maintained a steady match rate even as more independent applicants (primarily students and graduates of osteopathic and international medical schools) have joined the Match.7

Figure 1
Figure 1:
Number of PGY-1 positions, U.S. seniors and independent applicants, and U.S. seniors matched to PGY-1 positions, 1982 to 2015, from a study of unmatched seniors’ behaviors in the 2015 Main Residency Match. Abbreviation: PGY-1 indicates postgraduate year 1. Source: National Resident Matching Program Data Warehouse.

The long-standing success of U.S. seniors in the Match and the surfeit of training positions for that cohort do not negate the stresses and challenges they experience in obtaining a residency position. The rising tide of residency applications makes understanding the profile and Match behaviors of “successful” U.S. seniors crucial for future students and medical school advisors. Since 2006, the NRMP has published Charting Outcomes in the Match,8 an extensive report presenting the characteristics of U.S. seniors and other applicants who match to their preferred specialty, which is the specialty of the first-ranked program on the applicant’s rank order list (ROL). Other studies investigating Match success have found that Match outcome can be affected by metrics (e.g., United States Medical Licensing Examination [USMLE] score), education (e.g., nationally ranked medical school), and personal attributes (e.g., Alpha Omega Alpha honor medical society membership, personal statement).9–11 This study, a companion to an earlier article that examined the interviewing and ranking behaviors of unmatched international medical school graduates,12 analyzes application, interviewing, and ranking behaviors of unmatched U.S. seniors participating in the 2015 Match to better understand factors that affect interview selection and Match outcome.

Method

Data sources and study population

This study is based on results from the 2015 NRMP Applicant Survey (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A411) and Match data extracted from NRMP’s Data Warehouse. The Applicant Survey is an online survey administered biennially, most recently in 2015, to collect information on interview and ranking behaviors of Match applicants. To minimize sampling bias, the survey is sent to all applicants who rank a program. In addition, the survey closes the day before NRMP releases Match results so that responses are not affected by an applicant’s Match outcome. We obtained data on numbers of applications sent, interviews granted and attended, and programs ranked, as well as specific ranking behaviors from survey responses.

The NRMP Data Warehouse stores applicant and program information as submitted through the NRMP’s online Registration, Ranking, and Results (R3) system. The Match registration “Professional Profile” section allows Match applicants to self-report their USMLE Step scores, and medical schools are encouraged to verify the scores entered by their students. In 2015, 84% of reported U.S. seniors’ Step 1 scores were verified by their medical schools. We extracted Step 1 scores, preferred specialties, applicant and program ranking information, and Match outcomes from the Data Warehouse.

The 2015 Applicant Survey was sent to 18,012 U.S. seniors, and 8,754 responded, for a 49.4% response rate. Of the 8,754 respondents, 992 (11%) did not have a preferred specialty or report a USMLE Step 1 score. Because we were interested only in survey respondents with both a USMLE Step 1 score and preferred specialty, the final dataset included 7,762 U.S. seniors, of whom 7,463 (96%) were matched and 299 (4%) were unmatched when the matching algorithm was processed.

Definitions

We defined “matched” U.S. seniors as those who obtained a PGY-1 position and “unmatched” U.S. seniors as those who did not obtain a PGY-1 position when the matching algorithm was processed. A U.S. senior who obtained only an advanced (PGY-2) position would have been classified as “unmatched”; however, no U.S. seniors in the study cohort met that classification. We defined “preferred specialty” as the specialty of the first-ranked program on the applicant’s ROL. Because preliminary positions provide only one or two years of prerequisite training for entry into advanced specialty training, a U.S. senior was treated as not having a preferred specialty (and thus not included in the study) if the first program on the ROL was a preliminary (PGY-1 only) program; only two U.S. seniors met that criterion. U.S. seniors who obtained PGY-1 positions in the Match Week Supplemental Offer and Acceptance Program (SOAP) were not included in the final dataset.

We used USMLE Step 1 scores to measure applicant strength because it is the only standardized quantitative parameter available to all program directors. For the final dataset, we assigned each unmatched U.S. senior to one of four categories based on the perceived competitiveness of their Step 1 scores compared with the mean scores of U.S. seniors who matched in the same preferred specialty: “Strong” applicants had Step 1 scores that were 1 standard deviation (SD) higher than the group mean; “solid” applicants had Step 1 scores between the group mean and 1 SD higher than the group mean; “marginal” applicants had Step 1 scores between the group mean and 1 SD below the group mean; and “weak” applicants had scores more than 1 SD lower than the group mean.

Statistical analysis

Because not all self-reported scores were verified, we compared the self-reported Step 1 scores with school-verified scores using the intraclass correlation coefficient (ICC). Analysis showed that only 1.7% of the scores were entered incorrectly and that self-reported scores were highly reliable: ICC was 0.987 for matched and 1 for unmatched applicants.

We compared matched and unmatched U.S. seniors based on numbers of applications sent, interview invitations received, interviews attended, and programs and specialties ranked. We also calculated number of interviews received as a percentage of applications sent, and interviews attended as a percentage of invitations received. We used the Wilcoxon rank-sum test to study the differences between matched and unmatched applicants for all categories.

We used the Pearson chi-square test to compare matched and unmatched U.S. seniors’ strategies and the Fisher exact test to generate a similar comparison for each of the four categories.

We examined the ROLs of unfilled programs and the 299 unmatched U.S. seniors to determine whether any U.S. seniors failed to rank a program that ranked them. By definition, an unmatched U.S. senior would have matched if the senior had ranked an unfilled program that ranked him.

Statistical analysis was performed using SAS statistical software, version 9.4 (SAS Institute Inc., Cary, North Carolina). The significance level used in this study was .05.

Results

USMLE scores

USMLE Step 1 scores of matched and unmatched U.S. seniors were evaluated, and the mean Step 1 score of the 299 unmatched U.S. seniors was 225.2, lower than the mean of 233.6 for the 7,463 matched U.S. seniors (P < .001). The distribution of matched and unmatched U.S. seniors across the four categories is shown in Table 1. Distributions across the four categories differed for matched and unmatched U.S. seniors (P < .001). More than half (54%) of matched U.S. seniors were strong or solid based on perceived competitiveness of their Step 1 scores, but only one-quarter of unmatched U.S. seniors earned that designation. Forty-two percent (127) of unmatched U.S. seniors were weak compared with only 15% (1,124) of their matched counterparts (see Table 1).

Table 1
Table 1:
Summary Statistics: Distribution of the “Strong,” “Solid,” “Marginal,” and “Weak” Categories Among Matched and Unmatched Applicants in the 2015 Main Residency Match, From a Study of Unmatched Seniors’ Behaviors in the 2015 Main Residency Match

Application and interview

Table 2 compares matched and unmatched U.S. seniors by numbers of applications sent, interviews attended, and programs ranked and shows percentages of interviews obtained and attended. Statistically significant differences were found in the numbers of applications sent by matched and unmatched U.S. seniors in the solid, marginal, and weak categories. Compared with their matched counterparts, unmatched solid, marginal, and weak U.S. seniors applied to more programs (P < .001 for solid and marginal, P = .031 for weak), received fewer interview invitations (P < .001 for all three categories), attended fewer interviews (P < .001), and ranked fewer programs (P < .001). Unmatched seniors in the strong category also applied to more programs than their matched counterparts (P < .001) but did not receive more interview invitations (P = .341). Overall, the rate of return on applications (number of interview invitations obtained as a percentage of applications sent) was lower for unmatched U.S. seniors in all categories (median of 55.6% vs. 14.0%, P < .001); however, with the exception of the strong category (P = .180), unmatched U.S. seniors attended a higher percentage of interviews offered by programs (P < .001) (see Table 2).

Table 2
Table 2:
Applications, Interviews, and Programs Ranked by Matched and Unmatched U.S. Seniors in the 2015 Main Residency Match, From a Study of Unmatched Seniors’ Behaviors in the 2015 Main Residency Match

Ranking strategies

Figure 2 shows percentages of matched and unmatched U.S. seniors using specific ranking strategies, as reported on the 2015 Applicant Survey (n = 7,177; 585 missing). Unmatched U.S. seniors were significantly less likely to rank programs in order of their true preferences (P < .001), rank a mix of competitive and less competitive programs (P < .001), or rank one or more less-competitive programs in their preferred specialty as a “safety net” (P < .001). Unmatched U.S. seniors were significantly more likely to rank programs based on the perceived likelihood of matching (P < .001) and rank programs where they applied but did not interview (P < .001). Although unmatched U.S. seniors were more likely than matched U.S. seniors to rank all the programs at which they interviewed (P < .001), they were less likely to rank all programs they were willing to attend (P < .001).

Figure 2
Figure 2:
Ranking strategies used by 7,177 matched and unmatched U.S. seniors in the 2015 Main Residency Match, from a study of unmatched seniors’ behaviors in the 2015 Main Residency Match. Source: National Resident Matching Program Data Warehouse, National Resident Matching Program 2015 Applicant Survey.

Analysis of the ROLs of unfilled programs showed that 13 unmatched U.S. seniors would have matched if they had ranked unfilled programs that ranked them, including 2 in the strong, 6 in the solid, 2 in the marginal, and 3 in the weak categories. All 13 reported ranking programs in order of preference, and most did not rank programs based on the likelihood of matching; however, those 13 U.S. seniors did not report ranking less competitive programs or alternative specialties as a “fall-back” plan.

Discussion

Because program directors receive many more applications than they can review in-depth, an accurate profile of a “successful” Match applicant is crucial. Findings from this study reveal program director reliance on factors other than metrics in selecting applicants, identify potential missteps in applicant interview and ranking behaviors, and offer direction for additional research on factors affecting applicant Match outcomes.

The profile of a successful Match applicant often includes a competitive USMLE Step 1 score, and the reliance of programs on USMLE scores in applicant selection has been a source of debate and study.13–16 Since the first NRMP Program Director Survey in 2008, the USMLE Step 1 score has been the factor cited most frequently by program directors when asked about the criteria used to assess applications for the purpose of inviting applicants to interview, and it remains an important factor in deciding which applicants to rank.17 Still, findings from the 2014 Program Director Survey show that the USMLE score is not the factor given the most weight. Program directors report that specialty focus (e.g., letters of recommendation in the specialty, perceived commitment to the specialty, clerkship honors and grades in the desired specialty) and professionalism/ethics are more important than test scores when considering which applicants to invite for an interview, and interactions during the interview visit and interpersonal skills are viewed as more important when considering which applicants to rank.18

Reliance on the USMLE Step 1 examination as a screening tool for resident selection has led some in the GME community to call for a more holistic evaluation of candidates’ attributes, skills, and behaviors.19 Findings from this study indicate that some program directors are heeding that call. It is not surprising that our data show that, on average, the USMLE scores of matched applicants were higher than those of unmatched applicants, and that the largest number of unmatched applicants had USMLE scores more than 1 SD below the mean for their preferred specialty. Nor is it surprising that marginal and weak U.S. seniors in this study, regardless of matched status, submitted on average more than twice the number of applications than interviews offered. What is interesting is that strong unmatched U.S. seniors applied to twice as many programs on average as their matched counterparts but received roughly the same number of interviews. Our findings cannot explain why the number of applications sent by strong unmatched applicants was significantly higher or why their yield of interviews per application sent was not significantly better; however, the data suggest that program directors are looking beyond metrics to gauge applicant fit for their programs. That finding is supported by data from the NRMP Program Director Survey showing an increase over time in the frequency with which program directors cite the medical school performance evaluation (MSPE) as a factor when considering applicants for interview, a trend that may have resulted from the earlier MSPE release date and research suggesting that negative evaluations on the MSPE correlate positively with problem behavior among applicants during residency.20 Other research finds that program directors are considering additional measures to determine whether applicants are truly interested in the specialties and programs to which they apply, and that program directors also are looking for indices of maturity, patient commitment, and a sense of team spirit in the residents they select for interview.21,22 A robust test score may not be enough to overcome perceived inconsistencies in an application.

Findings from this study also afford insight into the relationship between ranking behaviors and Match outcomes. Matched applicants were more likely than unmatched applicants to rank a mix of competitive and less competitive programs and to rank programs in order of true preference. Conversely, a higher percentage of unmatched U.S. seniors did not rank programs in order of their true preferences, opting instead to rank programs based on the perceived likelihood of matching. They also were less likely to rank a mix of competitive and less competitive programs, rank all programs they would be willing to attend, or rank a safety net program. Such behaviors run counter to NRMP recommendations,23 and for a small group of 13, they may have contributed to a missed opportunity to match.

This study had limitations. It may be argued that a USMLE Step 1 score is not an appropriate measure of applicant strength; however, it is the only standardized measurement available to program directors and, thus, is a reasonable basis on which to group all U.S. seniors for analyses. In addition, the four categories—strong, solid, marginal, and weak—were designed using SDs in USMLE Step 1 scores in applicants’ preferred specialty. Although SDs were not uniform across the specialties, potentially limiting the robustness of the conclusions, grouping in this manner allowed us to cluster applicants and account for variability among specialties. Survey data and USMLE scores were self-reported, and not all U.S. seniors provided USMLE scores or gave NRMP permission to use their scores, even if reported. In addition, data sources did not include applicants’ applications, thereby limiting the definition of applicant competitiveness to a standardized measure and restricting our ability to examine what insufficiencies, if any, existed in the application. Lastly, the 2015 NRMP Applicant Survey attained a 50% response rate among U.S. seniors, which may not be representative of the entire U.S. senior applicant pool.

The relationship between applicant behavior, interview selection, and Match outcome warrants further evaluation. Findings from this and other studies suggest that applying to more programs does not guarantee an interview, let alone a match24; accordingly, sending more applications should not be considered a panacea for “at-risk” students or an acceptable tactical approach for any applicant. An unsuccessful Match experience for U.S. seniors, even those with strong USMLE Step 1 scores, may begin with characteristics that impede their selection for interview. Additional research on that topic could help medical school advisors engage their students in an honest assessment of their candidacy along with a systematic study of specialties and programs to determine goodness of fit so that they can appropriately target their applications. For U.S. seniors with low Step 1 scores, a deeper understanding of the characteristics that separate matched marginal and weak applicants from unmatched applicants in all categories could be particularly informative.

Because challenges associated with obtaining interviews can be exacerbated by the use of disadvantageous ranking behaviors, more research is needed to understand how U.S. seniors decide which programs to place on their ROLs. Ranking programs based on the perceived likelihood of matching rather than true preference suggests a fundamental misunderstanding of how the matching algorithm works. Moreover, failing to rank a mix of competitive and less competitive programs or a safety net program could reflect an overinflated sense of applicant confidence. An understanding of what drives the formation and utilization of ranking behaviors could guide discussions between U.S. seniors and their medical school advisors about the trade-off between narrowing their program preferences and being unmatched.

Lastly, input from program directors is needed to broaden the community’s understanding of the relationship among applicant behavior, characteristics, interview selection, ranking decisions, and Match outcome. Studies are needed that identify what program directors seek in selecting applicants for interview and the extent to which U.S. seniors possess attributes, apart from strong USMLE scores, that are deemed meaningful in determining an applicant’s goodness of fit. Such efforts, along with the availability of program-specific information about the characteristics and qualities sought by faculty in the applicants they select to interview and rank, could help applicants better leverage their candidacy and transform the Match experience for all.

References

1. Adashi EY, Gruppuso PA. Commentary: The unsustainable cost of undergraduate medical education: An overlooked element of U.S. health care reform. Acad Med. 2010;85:763765.
2. National Resident Matching Program. Results and data: 2015 Main Residency Match. http://www.nrmp.org/wp-content/uploads/2015/05/Main-Match-Results-and-Data-2015_final.pdf. Published May 1, 2015. Accessed October 6, 2016.
3. Sue GR, Narayan D. Generation Y and the integrated plastic surgery residency Match: A cross-sectional study of the 2011 Match outcomes. Plast Reconstr Surg Glob Open. 2013;1:e33. http://journals.lww.com/prsgo/Fulltext/2013/08000/Generation_Y_and_the_Integrated_Plastic_Surgery.7.aspx. doi: 10.1097/GOX.0b013e3182a47ed3. Accessed October 6, 2016.
4. Ortolon K. Losing the match game: Few residency slots leave more graduates unmatched. Tex Med. 2010;106:4345.
5. Gallegos A. New AAMC report forecasts “significant” physician shortage in the next decade. AAMC Reporter. https://www.aamc.org/newsroom/reporter/april2015/429680/physician-shortage.html. Published April 2015. Accessed October 6, 2016.
6. American Medical Association. Save GME.org. https://www.savegme.org. Accessed October 6, 2016.
7. National Resident Matching Program. First-year graduate medical education in the United States: 2002–2015. http://www.nrmp.org/wp-content/uploads/2015/12/First-Year-GME-in-the-US-2015.pdf. Published December 15, 2015. Accessed October 6, 2016.
8. National Resident Matching Program. Charting outcomes in the Match: Characteristics of applicants who matched to their preferred specialty in the 2014 Main Residency Match. http://www.nrmp.org/wp-content/uploads/2014/09/Charting-Outcomes-2014-Final.pdf. Published September 2014. Accessed October 6, 2016.
9. Loh AR, Joseph D, Keenan JD, Lietman TM, Naseri A. Predictors of matching in an ophthalmology residency program. Ophthalmology. 2013;120:865870.
10. Tadisina KK, Orra S, Bassiri Gharb B, Kwiecien G, Bernard S, Zins JE. Applying to integrated plastic surgery residency programs: Trends in the past 5 years of the Match. Plast Reconstr Surg. 2016;137:13441353.
11. Olazagasti J, Gorouhi F, Fazel N. A critical review of personal statements submitted by dermatology residency applicants. Dermatol Res Pract. 2014;2014:Article ID 984874. https://www.hindawi.com/journals/drp/2014/934874/. Accessed October 6, 2016.
12. Liang M, Curtin LS, Signer MM, Savoia MC. Understanding the interview and ranking behaviors of unmatched international medical students and graduates in the 2013 Main Residency Match. J Grad Med Educ. 2015;7:610616.
13. Sutton E, Richardson JD, Ziegler C, Bond J, Burke-Poole M, McMasters KM. Is USMLE Step 1 score a valid predictor of success in surgical residency? Am J Surg. 2014;208:10291034.
14. Green M, Jones P, Thomas JX Jr. Selection criteria for residency: Results of a national program directors survey. Acad Med. 2009;84:362367.
15. O’Donnell MJ, Obenshain SS, Erdmann JB. Background essential to the proper use of results of Step 1 and Step 2 of the USMLE. Acad Med. 1993;68:734739.
16. McGaghie WC, Cohen ER, Wayne DB. Are United States Medical Licensing Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med. 2011;86:4852.
17. National Resident Matching Program. Results of the 2012 NRMP Program Director Survey. http://www.nrmp.org/wp-content/uploads/2013/08/programresultsbyspecialty2012.pdf. Published August 2012. Accessed October 6, 2016.
18. National Resident Matching Program. Results of the 2014 NRMP Program Director Survey. http://www.nrmp.org/wp-content/uploads/2014/09/PD-Survey-Report-2014.pdf. Published September 15, 2014. Accessed October 6, 2016.
19. Prober CG, Kolars JC, First LR, Melnick DE. A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection. Acad Med. 2016;91:1215.
20. Brenner AM, Mathai S, Jain S, Mohl PC. Can we predict “problem residents”? Acad Med. 2010;85:11471151.
21. Puscas L, Esclamado R. Use of a secondary essay in the residency application process. JAMA Otolaryngol Head Neck Surg. 2015;141:591592.
22. Liang F, Rudnicki PA, Prince NH, Lipsitz S, May JW Jr, Guo L. An evaluation of plastic surgery resident selection factors. J Surg Educ. 2015;72:815.
23. National Resident Matching Program. ROL tips for success. http://www.nrmp.org/match-process/create-and-certify-rol-applicants/create-a-rol-applicants/tips-and-reminders/. Published September 15, 2013. Accessed October 6, 2016.
24. Weissbart SJ, Kim SJ, Feinn RS, Stock JA. Relationship between the number of residency applications and the yearly match rate: Time to start thinking about an application limit? J Grad Med Educ. 2015;7:8185.
Copyright © 2016 by the Association of American Medical Colleges