Introduction
Orthopaedic surgery is one of the most competitive specialties in the residency match1. In 2014, there were 994 applicants for 695 available positions, of which 77% were filled by US medical students and 28% were filled by international medical students2. For Match 2022, per the Association of American Medical Colleges (AAMC), there were 1,737 orthopaedic surgery applicants for 868 residency positions, with an average number of 87.07 applications per applicant3. Although the number of orthopaedic surgery residency positions has steadily increased over the years, the number of interested applicants has also increased, resulting in more applications per applicant and increased costs associated with the Electronic Residency Application Service (ERAS) application4,5. For Match 2022, an average number of 87.07 applications cost an applicant $1,936.56 for the ERAS application alone6. With interviews being held virtually for the past 2 interview cycles, expenses of travel and lodging have been replaced with the cost of ensuring high-speed internet, good camera and audio equipment, and a proper interview environment (i.e., child care during interview time). However, virtual interviews remain more affordable when compared with in-person interviews7. As in-person interviews resume, applicants will have to account for the aforementioned costs in addition to the ERAS application fees.
An initial step to increase transparency, and possibly decrease the number of applications per applicant, is to have complete and up-to-date information about residency programs on websites and databases commonly used by applicants to compare programs. There are multiple resources available for applicants to learn about residency programs. For orthopaedic surgery, program information is available through websites, social media, unsanctioned orthopaedic surgery spreadsheets on Reddit, virtual open houses, Doximity rankings, AAMC Fellowship and Electronic Residency Interactive Database (FRIEDA), and the American Orthopaedic Association (AOA) Orthopaedic Residency Information Network (ORIN)8–13.
During the coronavirus disease 2019 (COVID-19) pandemic, there was an increase in the use of social media (e.g., Instagram and Twitter) by orthopaedic surgery residency programs to provide applicants with additional information about the program9,14. In addition, applicants are able to weigh in on whether they prefer an in-person or virtual format, how this influences the residency application process, and what they hope to see in the future8. Understanding applicant preferences can further improve the application process, thereby decreasing the number of programs to which applicants apply.
The purpose of this study was to (1) determine which residency recruitment resources influence where applicants apply and how they rank programs, (2) identify what information would be most helpful for applicants to learn about programs, and (3) determine whether orthopaedic surgery applicants have preferential solutions to improve the application process.
Material and Methods
Medical students who applied to the senior author's orthopaedic surgery residency program during the 2021-2022 application cycle were invited through email to participate in this study. Invitations were sent out in March 2022 after interviews were finished and before match results were released. The anonymous survey was designed using the Qualtrics online survey platform.
The survey covered several domains related to demographics including age, race and ethnicity, sex, and personal background. Information regarding the type of medical school (allopathic, osteopathic, or international medical school) and location of the medical school (Northeast, South, Midwest, West, other) was also collected. Additional questions asked about recruitment resources such as information on residency websites, virtual open houses, in-person away rotations, and databases with information about orthopaedic surgery residency programs. The questions focused on how these tools influenced an applicant's decisions to apply and how to rank a program. The last question in the survey asked about how the recruitment process could be improved for applicants in the future (Supplemental Table 1).
Emails were sent to 900 orthopaedic surgery applicants. Reminder emails were sent 2 and 4 weeks after the initial invitation to encourage participation. Descriptive statistics were used to analyze the data, including evaluating the distribution of responses, means, and standard deviations for the questions.
Source of Funding
There was no source of funding for this study.
Results
Demographics of Survey Respondents
The survey was distributed to 900 applicants, 216 of whom completed it (response rate of 24%). Of the 216 survey respondents, 77 (35.6%) identified as female, 132 (61.1%) as male, 2 (1.2%) as nonbinary/nongender conforming, and 1 (0.6%) as transgender female. Most of the applicants (n = 171, 79.2%) were between the ages of 25 and 29 years. Complete demographics of survey respondents are summarized in Table I.
TABLE I -
Demographics of Survey Respondents
| Demographics |
|
| Total respondents, n |
216 |
| Sex, n (%) |
|
|  Female |
77 (35.6) |
|  Male |
132 (61.1) |
|  Transgender female |
1 (0.5) |
|  Transgender male |
0 (0.0) |
|  Nonbinary/nongender confirming |
2 (0.9) |
|  Prefer not to answer |
4 (1.9) |
| Race/ethnicity, n (%) |
|
|  White |
123 (56.9) |
|  Black, African American |
26 (12.0) |
|  Asian |
26 (12.0) |
|  Hispanic, Latino |
18 (8.3) |
|  American Indian |
2 (0.9) |
|  Native Hawaiian, Pacific Islander |
1 (0.5) |
|  Multiple races/ethnicities |
8 (3.7) |
|  Other |
3 (1.4) |
|  Prefer not to answer |
9 (4.2) |
| Age, n (%) |
|
|  19-24 |
8 (3.7) |
|  25-29 |
171 (79.2) |
|  30-34 |
31 (14.4) |
|  35-39 |
5 (2.3) |
|  40 |
1 (0.5) |
Of the survey respondents, 187 (86.6%) attended an allopathic medical school, 16 (7.4%) attended an international medical school, and 12 (5.6%) attended an osteopathic medical school. The greatest number of respondents was from medical schools located in the South (n = 89, 41.2%), followed by those who attended medical school in the Midwest (n = 51, 23.6%), Northeast (n = 34, 15.7%), outside the United States (n = 21, 9.7%), and the West Coast (n = 20, 9.3%).
Regarding an applicant's background, 24 (11.1%) were the first in their family to graduate college and 74 (34.3%) were the first ones in their family to attend graduate school. Twenty-nine (13.4%) respondents identified as first-generation immigrants, and 27 (12.5%) said English was not their first language. Forty-seven of 216 (21.8%) applicants had someone in their immediate family who was a physician (Table II).
TABLE II -
Personal Background Information of Survey Respondents
|
No. of Survey Respondents (n) |
Percentage of Survey Respondents (%) |
| First in family to graduate college |
24 |
11.1 |
| First in family to attend graduate school |
74 |
34.3 |
| First-generation immigrant |
29 |
13.4 |
| Recipient of government benefits (Medicaid/SNAP) |
19 |
8.8 |
| English was not my first language |
27 |
12.5 |
| Identify as having a disability |
7 |
3.2 |
| Someone in my immediate family is a physician |
47 |
21.8 |
| None apply to me |
62 |
28.7 |
Utilization of Resources for Applying to and Ranking Programs
The most important factor that students considered during the application process was the cutoff score for the US Medical Licensing Examination (USMLE) Step 2 or Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 2. One hundred five (48.6%) applicants ranked knowing the cutoff score as a 1 of 9 (1 = most important, 9 = least important), and 83 (38.4%) applicants listed knowing the median USMLE STEP 2 or COMLEX Level 2 score as a 2 of 9. Other factors that were identified as being important included knowing a program's research productivity, history of taking international medical graduates, and wellness opportunities for residents (Supplemental Table 2). Interestingly, 73 (33.8%) survey respondents listed knowing the number of female residents and attendings as a 7 of 9, and 92 (42.6%) applicants listed knowing the percentage of underrepresented in medicine residents and attendings in a program as an 8 of 9.
Regarding medical student experiences and information found on various resources, survey respondents selected in-person away rotations (n = 97, 44.9%) as being the most influential factor on deciding which orthopaedic surgery residency programs to apply to and how to rank programs. Fewer respondents found information obtained from FRIEDA (n = 8, 3.7%), the unsanctioned orthopaedic interview spreadsheet on Reddit (n = 10, 4.6%), AOA ORIN (n = 11, 5.1%), and a program's Doximity ranking (n = 13, 6.0%) to be a major influence on their decision to apply to or rank a program. Social media (n = 9, 4.2%), FRIEDA (n = 8, 3.7%), and ORIN (n = 11, 5.1%) did have more influence on where an applicant applied to than how an applicant ranked programs ([n = 6, 2.8%], [n = 3, 1.4%], and [n = 8, 3.7%], respectively). Residency program websites (n = 45, 20.8%) and in-person away rotations (n = 117, 54.2%) were more influential when ranking programs than when deciding on which programs to apply to ([n = 38, 17.6%] and [n = 97, 44.5%], respectively). Further breakdown of resource categories and survey responses are shown in Figures 1 and 2, respectively.
;)
Fig. 1: Degree of influence of the following sources on an applicant's decision to apply to an orthopaedic surgery residency program. FRIEDA = Fellowship and Electronic Residency Interactive Database, and ORIN = Orthopaedic Residency Information Network.
Fig. 2: Degree of influence of the following sources on an applicant's decision to rank an orthopaedic surgery residency program. FRIEDA = Fellowship and Electronic Residency Interactive Database, and ORIN = Orthopaedic Residency Information Network.
Residency Recruitment and Future Changes
Regarding orthopaedic surgery residency program recruitment, 152 survey respondents would like to see interview caps (70.4%), 128 would like to see an increase in program information available on FRIEDA and ORIN (59.3%), and 120 would like to see application caps (55.5%) (Table III). Additional changes that survey respondents proposed are listed in Supplemental Table 3.
TABLE III -
Changes Survey Respondents Would Want to See the Orthopaedic Surgery Residency Program Application Process
*
|
No. of Survey Respondents (n) |
Percentage of Survey Respondents (%) |
| Application caps |
120 |
55.6 |
| Interview caps |
152 |
70.4 |
| An increase in program information available on FRIEDA and ORIN |
128 |
59.3 |
| Monitored and sanctioned orthopaedic interview spreadsheet |
32 |
14.8 |
| Virtual and in-person interview options |
81 |
37.5 |
| In-person interviews only |
80 |
37.0 |
| Other changes |
42 |
19.4 |
*FRIEDA = Fellowship and Electronic Residency Interactive Database, and ORIN = Orthopaedic Residency Information Network.
Discussion
The COVID-19 pandemic challenged a lot of orthopaedic surgery residency programs to improve their online presence as a surrogate for in-person interaction with applicants. As programs are pushing for the return of in-person interviews and events, it is important to understand applicants' opinions on the resources they use to prepare for residency applications. Respondents in our study found in-person away rotations very influential in deciding where to apply and how to rank programs. In addition, we found that applicants would like the minimum USMLE Step 2 or COMLEX Level 2 cutoff score for individual residency programs to be more accessible. With the increasing competitiveness of the orthopaedic surgery residency application process, a majority of survey respondents would like to see interview caps, an increase in program information available on databases, and application caps.
In 2021, Tang et al. evaluated the online presence of various residency programs in the United States to determine where improvement in a program's virtual presence could be made8. The authors found that 152 of 156 orthopaedic surgery residency programs had websites; however, these websites on average had 18.3 of 30 desired content items that were found to be of interest to potential applicants, such as program director information, types of cases residents perform, resident workload, salary and benefits, and alumni placement. Boyd et al. evaluated changes that surgical subspecialties could enact because of the COVID-19 pandemic including a proposal for updating the content of websites to reflect the rotation schedule, operative experience, and information about the accomplishments and career paths of previous residents15. Despite the lack of certain information on program websites, Yong et al. found that during the 2018 to 2019 application cycle, 96% of orthopaedic surgery residency applicants still used program websites to learn more about the individual programs16.
Furthermore, a respondent in our study emphasized the need for a centralized system to access information about program-specific virtual meet and greets, video tours, and social media accounts. In 2021, Yong et al. noted that between July 2020 and November 2020, on orthopaedic surgery program websites, there was a 35.9% and 12.7% increase in the number of informative program videos and virtual tours, respectively17. These findings, along with the results of our study, show that program websites are actively used. This should encourage residency programs to revise their webpages and incorporate pertinent, up-to-date information to provide applicants with the opportunity to learn more about individual programs.
During the COVID-19 pandemic, the number of social media accounts created by orthopaedic surgery residency programs grew significantly18,19. Analysis of social media accounts, including Instagram, Facebook, and Twitter, showed a 300% increase between May 2019 and November 202018. Looking specifically at Instagram, as of November 4, 2021, approximately 66.1% of orthopaedic surgery residency programs were found to have an Instagram account18. Our survey found that 54.3% of applicants reported that the content they viewed on residency program social media pages made them more interested in the program. Checketts et al. also found that 51.2% of their survey respondents used social media platforms to learn about orthopaedic surgery residency programs before applying10.
In addition to increasing their social media presence, many programs created virtual away rotations, open houses, and interviews to adjust to the circumstances of the pandemic. A study by Seifi et al. surveyed US medical students and residents across all specialties about their opinion of virtual interviews19. The authors found that both groups favored in-person interviews over virtual interviews, but most participants believed virtual interviews should remain an option. Our study found approximately an equal number of medical students who favored solely in-person interviews (37.0%) and a combination of virtual and in-person interviews (37.5%). Having the option of performing virtual interviews could help minimize costs and limit the disruption to the applicant's medical school rotations20,21. However, virtual interviews do not allow applicants to see where the program is located or interact with the current residents, which may hinder their ability to determine whether a program is a good fit for them. In the coming years, residency programs can allow applicants to choose between an in-person and virtual interview or provide candidates with opportunities to acquire funding to cover application and travel costs.
Virtual social events, such as happy hours and open houses, may also provide applicants with more information about programs. One survey respondent in our study commented that more events, whether in-person or virtual, where applicants can meet and interact with residents and faculty would help increase exposure to the program. However, another survey respondent noted that attendance at these events should not be recorded to keep the application process fair for those who could not join an event. A survey of urology residency applicants demonstrated that applicants found virtual open houses to be beneficial, especially when the open house highlighted a program's strengths and weaknesses and provided the opportunity for a resident question-and-answer session22. Tawfik et al. surveyed applicants and program directors of orthopaedic surgery residency programs during the 2020-2021 application cycle and found that 64.0% of programs offered virtual happy hours or virtual conferences; however, these events did not significantly influence how medical students ranked programs23.
Several of our survey respondents were interested in learning about the history of a program accepting international medical school graduates (IMGs). As an IMG, securing a surgical residency position can be very challenging. Verma et al. found that from 2003 to 2015, radiation oncology had the fewest IMG matches (1.5%), followed by otolaryngology (1.9%) and orthopaedics (2.2%)24. By providing greater transparency on the number of IMG applicants, the number of IMG students interviewed, and the percentage of these applicants who match, residency programs can help IMG applicants make more informed decisions on which programs to apply to24,25.
As the number of applicants applying to orthopaedic surgery programs rises and the transition of USMLE Step 1 and COMLEX Level 1 to pass/fail, program directors may now place more emphasis on other factors, including USMLE Step 2 and COMLEX Level 2 scores26,27. However, a recent study by Bernstein et al. found that over 48% of programs in the FREIDA database failed to include USMLE Step requirements, and of the 141 programs participating in the ORIN database, most did not report the USMLE Step median or cutoff score28. Aside from examination scores, other factors such as performance during away rotations, personal knowledge of the applicant, clerkship grades, letters of recommendation from orthopaedic surgeons, Alpha Omega Alpha status, and research experience will gain significance26,27. Mun et al. found that 69.3% of orthopaedic surgery program directors surveyed are now advocating for a graded preclinical curriculum26. Some of our survey respondents also expressed support for greater emphasis on preclinical and clinical grades than board scores. In addition, of the program directors surveyed by Mun et al., 70.7% were advocating for caps on the number of residency programs an applicant can apply to26. The current lack of a cap on the number of applications has made it difficult for applicants to emphasize which programs they are most interested in, resulting in the introduction of preference signaling for the orthopaedic surgery residency match in 2023. In addition, several survey respondents have noted that knowing a program's view on reapplicants, influence of letters of intent, and view on postinterview contact would help them tailor their applications to individual programs. As programs and medical students prepare for Match 2023, residency programs can consider listing more information on their websites (e.g., AAMC FRIEDA and the AOA ORIN) to increase transparency about the application review process.
This study has some limitations. Because this study was a survey, it is limited by recall bias of respondents. In addition, the overall response rate was 24%, which is relatively low. The surveys were also only collected from applicants to 1 institution, which could have resulted in a biased response due to the applicants' desire to be accepted to the program distributing the survey. Furthermore, the response included many applicants from the South and Midwest regions, likely based on the residency program used to disseminate the survey, and thus, the results may not be generalizable to all the applicants across the United States.
Medical students applying to orthopaedic surgery residency programs would like to have more complete information on program websites, increased information on residency databases, and greater transparency of screening criteria. These changes would help applicants make more informed decisions on the number of programs to apply to, how to rank programs, and where to use signaling preference, a new addition to Match 2023 for orthopaedic surgery. Residency program directors may be able to apply these preferences to their individual programs to help facilitate a more equitable process for all students applying to orthopaedic surgery.
Appendix
Supporting material provided by the authors is posted with the online version of this article as a data supplement at jbjs.org (https://links.lww.com/JBJSOA/A473, https://links.lww.com/JBJSOA/A474, and https://links.lww.com/JBJSOA/A475). This content was not copyedited or verified by JBJS.
References
1. 10 Most Competitive Residencies (2022 Guide). TheMDJourney. 2022.
https://themdjourney.com/most-competitive-residencies/. Accessed June 20, 2022.
2. Charting Outcomes in the Match: Characteristics of Applicants Who Matched to Their Preferred Specialty in the 2014 Main Residency Match. Washington, DC: National Resident Matching Program; 2014.
https://www.nrmp.org/wp-content/uploads/2021/07/Charting-Outcomes-2014-Final.pdf. Accessed May 10, 2022.
3. Eras Statistics. Washington, DC: AAMC; n.d.
https://www.aamc.org/data-reports/interactive-data/eras-statistics-data. Accessed May 10, 2022.
4. Fogel HA, Finkler ES, Wu K, Schiff AP, Nystrom LM. The economic burden of orthopedic surgery residency interviews on applicants. Iowa Orthop J. 2016;36:26-30.
5. Camp CL, Sousa PL, Hanssen AD, Karam MD, Haidukewych GJ, Oakes DA, Turner NA. The cost of getting into orthopedic residency: analysis of applicant demographics, expenditures, and the value of away rotations. J Surg Educ. 2016;73(5):886-91.
6. Fees for ERAS Residency Applications. Washington, DC: AAMC; 2022.
https://students-residents.aamc.org/applying-residencies-eras/fees-eras-residency-applications. Accessed May 10, 2022.
7. Labiner HE, Anderson CE, Maloney Patel N. Virtual recruitment in surgical residency programs. Curr Surg Rep. 2021;9(11):25.
8. Tang OY, Ruddell JH, Hilliard RW, Schiffman FJ, Daniels AH. Improving the online presence of residency programs to ameliorate COVID-19's impact on residency applications. Postgrad Med. 2021;133(4):404-8.
9. Malyavko A, Kim Y, Harmon TG, Quan T, Gu A, Bernstein SA, Tabaie SA, Thakkar S. Utility of social media for recruitment by orthopaedic surgery residency programs. JB JS Open Access. 2021;6(3):e21.00076.
10. Checketts JX, Hunt T, Checketts BR, Scott JT, Johnson M, Boose M, Schwartz M, Chalkin B. Analysis of social media perceptions among orthopaedic surgery residency applicants and social media use by residency programs during the 2020 to 2021 cycle. JB JS Open Access. 2021;6(4):e21.00083.
11. Doximity Residency Navigator. Doximity. 2021.
https://www.doximity.com/residency/. Accessed January 12, 2022.
12. FREIDA Residency Program Database: Medical Fellowship Database: AMA. FREIDA.
https://freida.ama-assn.org/search/list. Accessed January 10, 2022.
13. Orthopaedic Residency Information Network.
http://orin.aoassn.org/. Accessed January 10, 2022.
14. LeDuc R, Lyons MM, Riopelle D, Wu K, Schiff A. Social media utilization trends in orthopaedic surgery residency programs during the COVID-19 pandemic. Iowa Orthop J. 2021;41(1):13-7.
15. Boyd CJ, Inglesby DC, Corey B, Greene BJ, Harrington MA, Johnson MD, King TW, Rais-Bahrami S, Tavana ML. Impact of COVID-19 on away rotations in surgical fields. J Surg Res. 2020;255:96-8.
16. Yong TM, Austin DC, Molloy IB, Torchia MT, Coe MP. Online information and mentorship: perspectives from orthopaedic surgery residency applicants. J Am Acad Orthop Surg. 2021;29(14):616-23.
17. El Shatanofy M, Brown L, Berger P, Gu A, Sharma AK, Campbell J, Tabaie S. Orthopedic surgery residency program website content and accessibility during the COVID-19 pandemic: observational study. JMIR Med Educ. 2021;7(3):e30821.
18. Holderread BM, Liu J, Wininger AE, Harris JD, Liberman SR. The effect of the COVID-19 pandemic on orthopaedic residency program social media utilization. JB JS Open Access. 2021;6(4):e21.00104.
19. Yong TM, Pappas MA, Ray GS, McManus TG, Coe MP. Analyzing the proliferation of social media use among orthopaedic surgery residency programs. JB JS Open Access. 2021;6(3):e21.00017.
20. Seifi A, Mirahmadizadeh A, Eslami V. Perception of medical students and residents about virtual interviews for residency applications in the United States. PLoS One. 2020;15(8):e0238239.
21. Joshi A, Bloom DA, Spencer A, Gaetke-Udager K, Cohan RH. Video interviewing: a review and recommendations for implementation in the era of COVID-19 and beyond. Acad Radiol. 2020;27(9):1316-22.
22. Jiang J, Key P, Deibert CM. Improving the residency program virtual open house experience: a survey of urology applicants. Urology. 2020;146:1-3.
23. Tawfik AM, Imbergamo C, Chen V, Filtes P, Butler A, Gatt C, Katt BM. Perspectives on the orthopaedic surgery residency application process during the COVID-19 pandemic. J Am Acad Orthop Surg Glob Res Rev. 2021;5(10):e21.00091.
24. Verma V, Shah C, Lautenschlaeger T, Lin C, Beriwal S, Zhen W, Mehta MP, Zietman AL. International medical graduates in radiation oncology: historical trends and comparison with other medical specialties. Int J Radiat Oncol Biol Phys. 2016;95(4):1102-6. doi:10.1016/j.ijrobp.2016.03.008
25. Zaidi Z, Dewan M, Norcini J. International medical graduates: promoting equity and belonging. Acad Med. 2020;95(12S Addressing Harmful Bias and Eliminating Discrimination in Health Professions Learning Environments):S82-7.
26. Mun F, Jeong S, Juliano PJ, Hennrikus WL. Perceptions of USMLE step 1 pass/fail score reporting among orthopedic surgery residency program directors. Orthopedics. 2022;45(1):e30-4.
27. Gu A, Farrar J, Fassihi SC, Stake S, Ramamurti P, Wei C, Wessel LE, Fufa DT, Rao RD. Effect of change in USMLE step 1 grading on orthopaedic surgery applicants: a survey of orthopaedic surgery residency program directors. J Am Acad Orthop Surg Glob Res Rev. 2021;5(5):e20.00216.
28. Bernstein SL, Wei C, Gu A, Fufa D, Levine WN. Evaluating databases with orthopaedic surgery residency program information. J Am Acad Orthop Surg. 2022;30(24):1177-83.
