Secondary Logo

Journal Logo

Basic Research

What Factors Influence Applicants’ Rankings of Orthopaedic Surgery Residency Programs in the National Resident Matching Program?

Huntington, William P., MD1,a; Haines, Nikkole, MD1; Patt, Joshua C., MD, MPH1

Author Information
Clinical Orthopaedics and Related Research®: September 2014 - Volume 472 - Issue 9 - p 2859-2866
doi: 10.1007/s11999-014-3692-9
  • Free

Abstract

Introduction

Orthopaedics is a highly competitive career path for medical school applicants, with 1.50 applicants for every slot [15]. Studies have been published regarding residency program selection criteria for applicants, but which factors applicants regard as most important are under investigated [2, 3, 7, 13]. Research shows that the most important factors for programs include performance on orthopaedic rotations, class rank, United States Medical Licensing Exam (USMLE) Step 1 score, interview performance, and letters of recommendation [6]. Investigations in other medical specialties show that applicants rank residency programs based on many factors, including academic reputation, lifestyle, and geographic location [5, 8, 10, 13, 14]. However, the literature lacks information regarding the effect of these and other factors for orthopaedic surgery applicants. To our knowledge, only one study identified factors important to orthopaedic applicants [12]. Sanfilippo et al. [12] determined that early operative experience and location were the most important and program size and research were the least important factors.

One of the American Academy of Orthopaedic Surgeons’ (AAOS) strategic goals is to foster diversity in orthopaedics [1]. Program selection committees are a gateway to an orthopaedic career, yet they lack evidence-based information addressing the importance of specific factors that may lead an applicant to a particular residency program. The relative importance of these factors may differ between the traditional applicant and that of women and minorities pursuing a career in orthopaedics. Although women make up 51% of the US population and 49% of US medical school graduates, the 2008 AAOS census revealed that only 4.8% of responding orthopaedic surgeons were women [9]. These levels are expected to increase with women composing 14.5% of the applicant pool in the 2011 match, but this still represents a deficiency in the diversity of our workforce [4]. Identifying factors that draw women to orthopaedics and potential sex differences in the importance of these factors are an imperative endeavor.

Currently, program selection committees possess little insight and guidance regarding how to adjust their interviewing process to meet applicants’ needs and minimize gender and minority bias. A clearer understanding of these and other factors important to applicants will aid programs in highlighting key attributes that their institutions possess, allow feedback for improvement, and ultimately allow them to continue to attract the best and most qualified candidates. We primarily sought to identify the factors most important to residency applicants when selecting an orthopaedic residency program. Secondary objectives included identifying sex- and minority-specific differences in how these factors influence applicants. We also sought to identify the resources applicants use and value most to help make these life-changing decisions.

Materials and Methods

All 742 National Residency Matching Program (NRMP) applicants to Carolinas Medial Center Orthopaedic Surgery Residency Program during the 2013 cycle were contacted for study participation through email. The email described the academic purpose of the survey and contained a link to the survey web site (SurveyMonkey®, www.surveymonkey.com, Palo Alto, CA, USA). To avoid interfering with the match process, the survey was emailed to all applicants after medical students and residency programs had finalized their rank lists and the survey was closed on the day before the match results were revealed. Additionally, survey responses were anonymous. As an incentive for participation, three random participants were selected to receive a $50 iTunes® (Apple Inc, Cupertino, CA, USA) gift card. To preserve the anonymity of individual responses, a link was provided at the end of the main survey to a second, separate survey where respondents could provide their email address to be entered in the drawing for the gift cards. These surveys were not linked in a way that could identify respondents.

The survey began with a short demographic section where applicants were queried regarding their age, sex, race, marital status, geographic location, and participation and attitudes on away rotations (Table 1). Of the 742 students applying for orthopaedic residency at Carolinas Medical Center, 207 responded (28% response rate) to the survey. Of the respondents, 15.5% were female, which is slightly more than the 13% of the applicant pool that was female (94 of 742) this year. Thirty-six of 203 respondents were minority applicants (17.7%), and four respondents skipped this question. Two hundred two respondents completed away rotations (98%) (Table 1).

Table 1
Table 1:
Demographic variables

Factors that have been shown to influence an applicant's rank list order initially were selected after a review of the literature [5, 8, 10, 14, 16]. These factors then were modified after discussion with members of the interview committee and junior residents in our program who had participated in the NRMP match during the past 3 years. Thirty-seven factors eventually were identified and included in the questionnaire (Table 2). Respondents were asked to rate each item on a 5-point Likert scale from 1 (not important) to 5 (extremely important). Applicants also were asked to rank the top three factors in order of importance that influenced their rank list (Table 3). In addition, the authors categorized each of the 37 factors as having a low, medium, or high modifiability in an institution. Each author completed ranking of the factors. For each factor, the category was determined based on a majority (two of three authors in agreement). When all three authors disagreed, a discussion was held and a joint conclusion was drawn.

Table 2
Table 2:
Important factors in determining rank order lists
Table 2
Table 2:
Continued.
Table 3
Table 3:
Applicant factor percentages selected as top three choices

Nine potential sources of information regarding residency programs also were identified and included in the survey (Table 4). Applicants were asked again to rate the items on a 5-point Likert scale from 1 (not important) to 5 (extremely important). Applicants also were asked to rank what they felt were the top three most important factors residency selection committees used to rank applicants (Table 5). The survey concluded with a section addressing gender- and minority-specific questions and how these related to an applicant's decision to pursue a career in orthopaedics and in making his or her rank list (Table 6). Respondents were given opportunities to supply free-text responses throughout the survey.

Table 4
Table 4:
Important resources for determining rank order lists
Table 5
Table 5:
Applicant perception of residency selection committee's most important evaluating criteria
Table 6
Table 6:
Perceptions of gender and minority basis

Statistical data are presented as means to present a clear ordering of factors, and medians were used for statistical comparisons. Standard statistical measures were used to characterize respondent demographics and responses to each question when applicable. Comparisons of Likert scale ratings were analyzed using a Wilcoxon-Mann-Whitney test. For categorical data, chi-square (Fisher's exact tests when cell sizes were less than five) was used. Statistical significance was determined using a threshold of p less than 0.05 for all queries.

Results

Applicants identified the three most important factors as the perceived happiness/quality of life of current residents (mean score, 4.68; median score, 5; 95% CI, 5-5; resident camaraderie, mean, 4.62; median, 5; 95% CI, 5-5), and impression after an away rotation (mean, 4.46; median, 5; 95% CI, 5-5) (Table 2). The authors determined that programs have a moderate degree of influence on three of the five most important factors, but have a high degree of influence on only two of the top 20 most important factors to applicants. The most highly ranked factor that program directors have a high degree of influence on was the interview experience, which was the sixth most important factor. Men and women had similarities and differences in what each group felt were the most important factors (Table 3). Both groups felt that perceived happiness/quality of life of residents, resident camaraderie, and geographic location to be very important factors. Twenty-one percent of men favored their impressions after a medical student rotation at an institution versus 0% of women, a statistically significant finding. However, 22% women tended to weigh their personal interactions with residents significantly more than 7% of men when determining a rank list.

In general, women felt more confident than men in women's ability to perform the mental, physical, emotional, and educational tasks required of orthopaedic surgeons (p ≤ 0.001, Table 6). In addition, women more strongly agreed with the statement “the field of orthopaedics needs more female orthopaedists” compared with men (p < 0.001). Women valued the number of female orthopaedic residents and faculty when deciding on residency programs although not when deciding on which specialty to pursue. In accord, 68% of women eliminated residency programs from their options based on perceived sex biases versus less than 1% of men with the primary stated reason being that some programs were known to not take females. One male respondent noted that he eliminated programs with “a lot of female residents.” Interestingly, a few applicants identified eliminating programs based on their location in “the South” as a result of minority and sex biases.

According to our survey, women more commonly asked gender-related questions during their interviews (39.3% of women versus 0.67% of men, p < 0.001) and also were asked more gender-related questions (57.14% of women versus 5.33% of men, p < 0.001). Common questions asked by women revolved around the atmosphere for women at the program, the camaraderie between female residents, and the opportunities for women completing their residency at those programs. Men and women responded that they were asked about their marital status and family plans, but women also were asked about their physical ability to perform the job and their comfort entering a male-dominated field. Female applicants felt discriminated against by orthopaedic residents and attendings as a result of their gender more so than male applicants. Overall, 14.3 % of women felt discriminated against during their interviews versus 6.67% of men (p = 0.241). Of the men experiencing discrimination, one reason listed included his status as a foreign medical graduate.

Applicants primarily valued information received through direct interactions with programs when determining their rank lists. They put less weight into what they heard through word of mouth from fellow applicants, online blogs, and chat rooms when making their lists (Table 4). When queried on what applicants felt mattered most to the programs’ selection committees, more than 50% felt that USMLE scores, away rotation performances, interview performances, and letters of recommendation were most highly used (Table 5). Less than 20% of the applicants felt that the remaining factors were in the top three criteria that selection committees used when determining their rank lists. Interestingly, 47 respondents rotated at a specific program because they heard it was necessary to be invited to interview there (22.7%), and 43 respondents rotated at a specific program because they heard it was necessary to match at said program (20.8%).

Discussion

Although orthopaedic surgery continues to be one of the most competitive specialties for graduating physicians, with 1.50 applicants for every one spot as of 2013 [15], it is important for programs to understand what factors orthopaedic surgery applicants take into consideration when determining their rank lists. By better understanding these factors, orthopaedic surgery programs can attract the highest-quality graduates to their program and to the specialty in general. We aimed to identify factors that are important to orthopaedic surgery residency applicants; explore differences in factor importance for men, women, and minorities; and evaluate the relative importance of information sources used by the applicants.

Our survey has some potential limitations. All data were collected from applicants to one institution in one southeastern residency program. The population applying to this institution may not be representative of applicants applying to institutions in other geographic areas. Our response rate of 28% was low and we cannot assume that these opinions represent the opinions of all orthopaedic surgery applicants. Although this is better than the 11.2% response rate for the other published study in orthopaedics [12], a larger group of respondents would be needed to fully understand many of the complex dynamics that go into an applicant's rank list. Overall, the survey was sent to 742 applicants. This represents 71% of all applicants to orthopaedic surgery during 2013. Our response rate encompasses 20% of the total applicant pool, including US senior and International applicants [15]. Although the moderate response rate was a weakness of our study, the percentage of women respondents, one of the target groups for this investigation, was 15.5%. This mirrors the overall female applicant pool (13%) thus supporting the results. The survey also was done in a narrow time window between the time when rank lists were submitted and match results were revealed to minimize bias that postinterview contact with applicants may have on rank order lists and in which match results may change applicants’ attitudes on certain factors. Furthermore, although we attempted to create a comprehensive survey, there are other issues that may influence applicants. Additionally, no formal definitions were provided for the factors on the survey and individual interpretation of terms may have affected results. Like with any self-administered and anonymous survey, results are dependent on the accuracy of the self-reported data, which makes it vulnerable to error, because it cannot be externally validated.

This survey highlighted some interesting points. Categorizing these factors into interpersonal factors, reputation factors, and work factors, as done by Flynn et al. [8], provides a useful framework. Many of the factors identified as most important related to interpersonal factors with the top six factors fitting into this realm. Applicants first and foremost wish to be happy where they match and gauge this from their observations and interactions with the current residents at programs. Whether they would fit in with the residents and faculty based on their experiences during interviews and away rotations and the perceived relationships residents and faculties have with one another also play a prominent role in their decision-making. Only two of the top 10 factors were work-related issues—early operative experience, and case volumes and clinical variety. The majority of work-related factors such as the perceived workload, daily didactics, research opportunities/requirements, graduate medical education trainee benefits, employee benefits, and vacation time were in the bottom ½ of important factors. Program reputation factors such as successful placement of graduating residents and the national reputation of the program and its faculty were in the middle. Other factors that cannot be categorized to one of these domains are important to applicants, including geographic location, city characteristic, and proximity to family and friends. Our finding that interpersonal factors are more important to applicants than work-related and reputation-related factors are similar to the findings of applicants to radiology and internal medicine residency programs [8, 10]. To our knowledge, there has been only one similar study done in orthopaedic surgery to date [12]. This retrospective study was by Sanfilippo et al. in 2006 through queries of Postgraduate Years 1 and 2 residents and was limited by recall bias and possible changes in opinions resulting from matching at a particular institution and residency experiences. In addition, their response rate was only 11.2%. They found early operative experience to be the most important factor followed by location and program reputation. Interestingly, in our study, early operative experience was the eighth most important factor. Applicants in our survey placed much more emphasis on the interpersonal domain when selecting their top programs.

The degree of influence a program director has on each of the 37 factors also was categorized. The program director has a high degree of influence on eight factors, moderate degree of influence on 14, and low degree of influence on 15. These distinctions may be arbitrary, but it is important to consider because the program director is charged with recruiting new residents and establishing the collective attitude of the residency program. By understanding the level of control over each of these factors and which factors medical students consider important, program directors can respond accordingly to recruit the best applicants.

Although the women applying for orthopaedic residencies are confident in their abilities as shown by our survey, it remains to be seen whether all female medical students feel orthopaedics is a viable career option. “About 10-12% of applicants to orthopaedic residency programs are women, so we are selecting in proportion to the applicants. There just aren't enough women applicants” [11]. This statement held true for our program as well with approximately 12% of our applicant pool being women. One possible reason for the disparity between the percentage of female medical students and orthopaedic residents is that the perceived sex discrimination felt by the women who apply prevents other female students from considering an orthopaedic career. Other factors previously proposed as barriers include lack of exposure to musculoskeletal medicine during medical school and lack of female mentors. Although these barriers were not identified in our survey, this may be because the women ultimately applying for orthopaedic surgery residencies benefited from these during their training, whereas women who chose different career paths did not. Our study was not able to discriminate this difference. Alarmingly, there was a small number of free-text responses by male and female respondents relaying experiences of flagrant sex discrimination and of applicants using discriminatory statements in their responses. It is possible these responses are outliers, but even one act of discrimination by one individual has the potential to affect many applicants and programs and should be taken seriously to avoid misperceptions about the career potential, necessity for, and general desire for women and minorities in the field of orthopaedics.

Orthopaedic residency applicants have multiple sources of information to gain insight into orthopaedic residency programs as they make a rank list. Personal interactions with a program are of the highest importance, including experiences during an away rotation, talking with current residents, and the interview day experience. This is consistent with the findings of Pretorius and Hrung [10], who found that the interview day and talking with residents were the two most important sources of information for radiology applicants. Similar to their findings, a program's web site was more important than printed materials from a program. This is not unexpected as the new generation of surgeons turns to electronic media for information as it is readily accessible. Programs may find this interesting so that they may create comprehensive web pages.

Our study identified important factors considered by orthopaedic applicants, some that can be modified and some that cannot. Importantly, interpersonal factors play a large role in applicants’ decision-making processes. These findings should help identify potential strengths and deficiencies in residency programs’ appeal to applicants. Further research to identify factors important to all medical students when considering their career options would be beneficial for appealing to the top tier of applicants, including women and minority applicants. This study also supports the continued needs for increased diversity in the field of orthopaedics and for further initiatives to create a more diverse orthopaedic workforce.

Acknowledgments

We thank Rachel Seymour PhD for contributions to the statistical analyses in this study.

References

1. American Academy of Orthopaedic Surgeons. Diversity in Orthopaedics. Available at: http://www3.aaos.org/About/diversity/index.cfm. Accessed April 26, 2013.
2. Bernstein AD, Jazrawi LM, Elbeshbeshy B, Della Valle CJ, Zuckerman JD. Orthopaedic resident-selection criteria. J Bone Joint Surg Am. 2002;84:2090-2096.
3. Bernstein AD, Jazrawi LM, Elbeshbeshy B, Della Valle CJ, Zuckerman JD. An analysis of orthopaedic residency selection criteria. Bull Hosp Jt Dis. 2002-2003;61:49-57.
4. Department of Research and Scientific Affairs, American Academy of Orthopaedic Surgeons. 1998-2011 Resident Diversity Survey Report. March 9, 2012. Available at: http://www3.aaos.org/about/diversity/pdfs/resident_trend.pdf. Accessed April 26, 2013.
5. DeSantis M, Marco CA. Emergency medicine residency selection: factors influencing candidate decisions. Acad Emerg Med. 2005;12:559-561 10.1111/j.1553-2712.2005.tb00899.x.
6. Egol KA, Collins J, Zuckerman JD. Success in orthopaedic training: resident selection and predictors of quality performance. J Am Acad Orthop Surg. 2011;19:72-80.
7. Evarts CM. Resident selection: a key to the future of orthopaedics. Clin Orthop Relat Res. 2006;449:39-43.
8. Flynn TC, Gerrity MS, Berkowitz LR. What do applicants look for when selecting internal medicine residency programs? A comparison of rating scale and open-ended responses. J Gen Intern Med. 1993;8:249-254 10.1007/BF02600091.
9. Lewis VO, Scherl SA, O'Connor MI. Women in orthopaedics: way behind the numbers curve. J Bone Joint Surg Am. 2012;94:e30 10.2106/JBJS.J.01408.
10. Pretorius ES, Hrung J. Factors that affect National Resident Matching Program rankings of medical students applying for radiology residency. Acad Radiol. 2002;9:75-81 10.1016/S1076-6332(03)80298-2.
11. Proucznik MA. Where are the women orthopaedists? AAOS Now. 2008;2. Available at: http://www.aaos.org/news/aaosnow/feb08/cover2.asp. Accessed May 1, 2014.
12. Sanfilippo JA, Sharkey PF, Parvizi J. Criteria used by medical students to rank orthopaedic surgery residency programs. Am J Orthop (Belle Mead NJ). 2006;35:512-514.
13. Spitzer AB, Gage MJ, Looze CA, Walsh M, Zuckerman JD, Egol KA. Factors associated with successful performance in an orthopaedic surgery residency. J Bone Joint Surg Am. 2009;91:2750-2755 10.2106/JBJS.H.01243.
14. Stefanidis D, Miles WS, Greene FL. Factors influencing residency choice of general surgery applicants: how important is the availability of a skills curriculum? J Surg Educ. 2009;66:325-329 10.1016/j.jsurg.2009.06.004.
15. The Match National Residency Matching Program: Results and data: 2013 main residency match. Available at: http://www.nrmp.org/wp-content/uploads/2013/08/resultsanddata2013.pdf. Accessed July 9, 2013.
16. Yarris MS, Delorio NM, Lowe RA. Factors applicants value when selecting an emergency medicine residency. West J Emerg Med. 2009;10:159-1622729216.
© 2014 Lippincott Williams & Wilkins, Inc.