The 2020 to 2021 orthopaedic surgery residency application cycle was unprecedented because of broad changes in response to the COVID-19 pandemic. This included disrupted medical student rotations, travel restrictions affecting interviews, and visiting student rotations.1-3 In response, the American Orthopaedic Association Council of Orthopaedic Residency Directors (CORD) issued guidelines for both applicants and residency programs regarding changes to application requirements, visiting student rotations, and interviews. These initiatives included a universal interview offer day (UIOD) that specified a synchronized date and time for programs to release interview offers (10:00 am EST on November 23, 2020), a departure from previous years in which interview invitations were offered with varying frequency over a span of months.4,5 The guidelines also stipulated a 48-hour response window for an interview offer by scheduling or declining an interview before programs could extend interviews to alternate candidates. Finally, programs were advised to not release more interview offers than available interview slots.
Most orthopaedic surgery residency programs agreed to adhere to the CORD guidelines, including the UIOD6—133 of the 208 programs (64%) participating in the National Resident Matching Program (NRMP) Match in 20217 and 132 of the 192 programs (69%) listed by the Electronic Residency Application Service.8 The novel UIOD among orthopaedic surgery residency programs for the 2020 to 2021 residency match cycle provides a unique opportunity for analysis. Thus, this study aims to determine (1) the extent to which residency programs adhered to the CORD guidelines during the 2021 application cycle, (2) the speed of interview scheduling after release, and (3) scheduling variance by the e-mail domain (edu versus com applicant e-mail domains) because of our hypothesis that edu e-mail domain delivery delays result in slower scheduling. This analysis aims to identify innovative methods for the UIOD to streamline and optimize selection and equity for all participants and stakeholders. These results may be useful for CORD, individual residency programs, and orthopaedic surgery residency applicants when considering how to approach future application cycles.
This study was reviewed by an Institutional Review Board (STUDY00022821) and determined to fall outside the regulatory definition of a human subject study, given that all data were coded and deidentified before analysis. Therefore, no written consent or review of the Institutional Review Board was required by applicants or programs. Thalamus (SJ MedConnect, dba ThalamusGME; https://thalamusgme.com/) is a commercially available cloud-based graduate medical education (GME) interview management platform and scheduling software used by residency programs and hosted on the Microsoft Azure/Structured Query Language (SQL) server.
We conducted a retrospective investigation using Thalamus data to estimate population-level parameters regarding the 2020 to 2021 orthopaedic surgery residency recruitment cycle. All analysis was conducted after the completion of the 2021 NRMP Match. Data were provided by Thalamus and are not publicly available, although it regularly collaborates with researchers in specialties across GME. The Thalamus database segments each institution by an institutional identifier and each program within each institution by the program and an American College of Graduate Medical Education identifier (or a similar number for non–American College of Graduate Medical Education-accredited programs).
Aggregate Program Level Data
Data were captured for residency programs on the aggregate level. Each of the 46 orthopaedic surgery residency programs using Thalamus for recruitment was deidentified at the onset of the study. Before deidentification, a program's commitment to the UIOD was categorized as follows based on the CORD UIOD guidelines6: (1) “participatory” (programs committed to release on the UIOD and time), (2) “alternative” (programs releasing interview offers on a date other than the UIOD), or (3) “nonparticipatory” (programs that did not commit to UIOD participation). Information on Doximity rankings of residency programs was collected through their online Residency Navigator tool.9
After deidentification, the following variables were collected and/or calculated for each program: (1) total number of invitations extended, (2) total number of interview positions offered, (3) total number of interviews scheduled, (4) total number of interviews completed, (5) time of first interview invitation release, (6) number of invitations released by calendar date, and (7) time elapsed between first interview invitation sent and for 80% of positions offered to be filled. An 80% threshold was chosen in assessing the rate of interview position filling because it accounted for the bulk of positions offered by a program without requiring that all positions fill.
To determine compliance with the UIOD guidelines, parameters were assessed and categorized for each program. Programs were defined as “strictly adherent” if interviews were released within a 6-minute window around the scheduled time of 10:00 am EST (9:57 am to 10:03 am EST on November 23, 2020) and “weakly adherent” if programs were released on the scheduled date but before or after the strictly adherent time window (any time before 9:57 am or after 10:03 am EST on November 23, 2020). This narrow window was selected because of the consistent messaging from CORD regarding the interview release time and date and the stated goals of the UIOD regarding transparency in communication to applicants with a consistent and equitable application process.10 Given the programmed functionality within Thalamus, we thought that a 6-minute window allowed reasonable time for a program to release interviews on time while being small enough to prevent an applicant from being able to book multiple interviews within the window. Programs were categorized as “nonadherent” if they released their initial interview invitations on any date other than November 23, 2020.
Aggregate Individual Level Data
Deidentified applicant data were analyzed to examine applicants using edu and com e-mail domains at the time of application submission to the Electronic Residency Application Service. The mean time from the first Thalamus interview invitation release to the first scheduled Thalamus interview was calculated for the two groups at the aggregate level. No other applicant information was observed or analyzed.
Analyses were conducted using descriptive statistics. Data were analyzed using R version 3.6.3 and ggplot2.11 Statistical tests included chi square and Student t-tests where appropriate.
In the 2020 to 2021 residency application cycle, 993 different applicants were invited to interview at one or more of 46 orthopaedic surgery residency programs using the Thalamus platform for interview management. This represents 77% of the 1,289 total orthopaedic surgery interviewees and 22.1% of the 208 orthopaedic surgery residency programs entering the 2020 to 2021 NRMP Match. The geographic region of each program was assigned based on US Census Bureau regional definitions (Figure 1). Programs were located in 24 states, with 19 Midwestern (41%), 11 Northeastern (24%), nine Southern (20%), and seven Western (15%) programs (Figure 1). Residency programs spanned a broad range of Doximity rank percentiles within orthopaedic surgery (Figure 2). A total of 3,143 interviews were scheduled through the Thalamus platform, 2,885 of which were completed (Table 1). An average of 12.5 interviews were completed per available residency position across programs.
Table 1 -
Residency Program and Interview Distribution by Region
||No. of Orthopaedic Residency Programs (% of Total)
||No. of Interviews Scheduled (% of Total)
Adherence to Universal Interview Offer Day Guidelines
Of the 46 orthopaedic surgery residency programs studied, 41 pledged to participate in the UIOD, two committed to alternative offer release days, and three were nonparticipatory. Nineteen of the 46 residency programs (41%) were strictly adherent to the UIOD release (Figure 3). Fourteen programs (30%) were weakly adherent, releasing interview offers on November 23, 2020, but outside the common release time. Of those programs, seven released interviews early and seven released interviews late, spanning a 9-hour period around the set UIOD time. Two programs each released two separate waves of interview offers on November 23, 2020, failing to adhere to the 48-hour window guideline. Thirteen programs (28%) were nonadherent and released interview offers on dates other than November 23, 2020. Among nonadherent programs, four released first interview offers before the UIOD and nine released after the UIOD, spanning October 24 to December 30, 2020.
How Quickly Were Interview Offers Filled?
Of the 33 programs that sent out interview offers on November 23, 2020, the mean (± SD) time to filling 80% of available interviews was 26 minutes (SD ± 14.1 minutes; 3 to 77 minutes) (Figure 4, A). For the 19 strictly adherent programs, the mean time to filling 80% of interview slots for each program was 29 minutes (SD ± 8.3 minutes; 13 to 49 minutes). No programs released more interview offers than available interview positions in the initial release, but many programs extended additional invitations in later waves to fill available interviews (Figure 4, B). Residency programs using Thalamus completed a mean of 64.2 interviews (SD ± 26.4).
Did Applicant E-mail Domain Affect Time to Scheduling?
Regarding the speed of interview scheduling, 728 of the 993 applicants (73.3%) received their initial interview invitation through Thalamus on the UIOD and responded within one hour. Of these applicants, the mean time from release to scheduling of the interview was 1.81 minutes longer in the edu e-mail domain group than in the com e-mail domain group (14.8 versus. 13.0 minutes, P < 0.05).
In this study, we assessed the adherence of interview offer release among orthopaedic residency programs to the novel UIOD guidelines implemented in the 2020 to 2021 residency application cycle. We found that most of the orthopaedic surgery programs (72%) in this cohort released interview offers on the UIOD, but only 41% were strictly adherent to the date and time. Within 30 minutes, over 80% of the available interviews in the sample were booked, and the fastest a program filled to the 80% capacity was 3 minutes. The mean time from interview release to scheduling of the first interview was 1.8 minutes slower for applicants with the edu e-mail domain compared with the com domain.
Within GME, there has been a call to improve transparency around the interview invitation and scheduling process and timing.4,12,13 The application process in previous cycles was a 1.5-year process, beginning with completing senior medical student rotations at home institution and then applying for and completing visiting medical student rotations at a minimum of two other institutions. Before the institution of UOID in orthopaedic surgery, candidates would typically submit applications to as many as 150 residency programs and then had to be constantly electronically available for a period of time spanning months to respond to interview invitations because they were sporadically and often unpredictably released. Given the very limited number of interview dates offered by each program and the restriction of travel logistics, minute delays in applicant response time could result in scheduling an interview for a less desirable date or loss of an interview slot entirely because of conflicts.4 The pressure on orthopaedic applicants is high because it remains one of the most competitive subspecialties, with an average of only three of four applicants matching into residency.14 The UIOD was instituted in part to increase predictability of interview release timing and overall transparency between residency programs and applicants.
The range of adherence of different programs within the UIOD highlights both a unique level of coordination across residency programs during the 2020 to 2021 Match cycle and demonstrates that a perfectly synchronized release is technically and operationally challenging. This is especially true because using a software platform such as Thalamus should increase the ease of adherence as it obviates the need to send individual e-mails and simply requires the triggered release of a prebuilt interview invitation list. Although not directly studied here, adherence may be lower among residency programs that did not use an interview management software to sync interview invites or scheduling. Potential barriers to adherence to specific time and date release guidelines include organizational planning, hospital e-mail and network policies, and Wi-Fi bandwidth. Although it is also possible that alternate release times were strategic on the part of some programs, the proportion of total interviews released on November 23 (Figure 4) suggests a good faith effort on the part of most of the programs studied.
These data also demonstrate the rapid speed at which interview slots filled. A common release time is akin to the “ticket-sale” problem, analogous to buying Super Bowl tickets or scheduling the general population for COVID-19 vaccines.15 By announcing exactly when interview offers will be released, applicants flock to their calendars several minutes before the release to manually refresh the page intermittently until the positions appear and then book interviews immediately. Releasing all interview offers simultaneously in this manner, therefore, runs the risk of creating possible issues of inequity related to the digital divide or applicants using surrogates or other technology to assist with interview scheduling, a common previous practice among residency applicants.16 Our data showed interviews filling within minutes from the time of interview offer release, implying that uninterrupted access to computer(s) or faster internet speeds would allow those with the financial means to access those resources more readily and thus book preferred interview dates more than those without access/resources. To mitigate this, one change introduced in the 2021 to 2022 application cycle was a delay of 48 hours between interview offer release and applicants being able to schedule the interview,17 allowing for all applicants to form a strategy for finalizing a desired interview schedule. However, possible inequity issues remain. Applicants able to use multiple devices or surrogates may have an advantage, especially if all scheduling becomes possible at a single time point.
One illustration of the challenges associated with simultaneous interview offer release is the difference in average time from release to booking seen between applicants using edu and com e-mail domains. We hypothesize that institutional informational technology policies including firewalls, VPNs, and spam filters used to protect hospital networks and patient information delay notification times compared with com e-mail domains. As some programs filled in 3 minutes, a mean difference of 1.8 minutes could easily be enough to disadvantage applicants using an edu e-mail domain. This study did not examine whether this altered the outcome of the Match for students in each group. However, the difference observed indicates one easily measurable, unforeseen source of inequity where others likely exist but are less easily measured. For this and other reasons, applicants should consider using an e-mail address with com domain as the main form of contact during the application cycle.
Although the data studied are only illustrative of the 2020 to 2021 Match cycle, it is important to interpret it within recent orthopaedic surgery residency application trends. The average number of applications submitted by prospective orthopaedic surgery residents has increased nearly twofold in recent years, from 46 in 2008 to 83 in 2018.14,18,19 Modeling studies in orthopaedic surgery and otolaryngology have demonstrated that to minimize unmatched spots and maximize applicant ability to indicate interest in a program, the optimal number of applications submitted per applicant is 50.19-21 The increasing applicant-to-residency position ratio and number of applications submitted per applicant are overwhelming both applicants and residency programs, diluting the weight of each application in signaling program interest. This skewed ratio of applicants to programs raises the question of different strategies that applicants, residency programs, and CORD might use to optimize the application process with the goal of matching applicants to programs in an equitable, efficient, and affordable process. The Coalition of Physician Accountability distributed recommendations for the 2021 to 2022 residency application cycle, based primarily on consensus agreements, with a call for data-driven solutions.22 Through data from residency application management platforms,23 we may better understand this process to the benefit of subsequent application cycles.
This study demonstrated that the UIOD generated rapid fill of interview offers, but maximum speed of scheduling was not the primary goal of the guidelines. A more equitable and optimal strategy might include inviting candidates through a standardized waved or phased approach. One recommendation for consideration is releasing interviews on two common dates, separated by a week with invitations for only half of the total interview slots released at a time. A two-wave structure would maintain the goal of increased transparency around interview release timing while providing applicants in the first wave an opportunity to pick an interview date in a less rushed manner, knowing that only 50% of the available positions will fill during the first week. This optimizes applicant choice and decreases the perceived urgency to schedule interviews while also serving as a potential mechanism for programs to signal preference to applicants.24 Programs could use phased interview offers to signal interest in applicants, and applicants would be able to make more informed decisions regarding which programs to prioritize in their interview schedule when conflicts inevitably arise. This may decrease the scheduling of undesired interviews or promote earlier cancellations. The 48-hour delay introduced in the 2021 to 2022 Match cycle may help alleviate the pressure to schedule an interview immediately on receiving the invitation and allow applicants to instead make scheduling decisions based on the totality of interview invitations received.
Limitations of this study include that only 22% of the orthopaedic surgery residency programs and 77% of the applicants are included in the data set. As a sensitivity analysis, for each state, we compared the proportion of all programs with the proportion of programs in this study. We found that the two distributions are very similar, with normally distributed errors with mean zero. One idiosyncrasy was that Ohio programs are over-represented. We recognize that this is not a truly randomly drawn sample and that the over-representation of one state could have an unforeseen effect. However, given the breadth of the sample and the singular outlier, we think that this analysis remains reliable and valuable. Real-time, granular data for a broad subset of applicants hold advantages over recall surveys done after the application cycle is completed, which forms the basis for a large portion of the residency application literature.25 Although programs using interview management software could be systematically different from those who do not, we think these data hold notable value due for the 2021 to 2022 application cycle and beyond, especially given the potential for continued use of video interviewing.26
Alternative definitions of adherence to the UIOD guidelines could be constructed. The 6-minute window was chosen because of its technical feasibility within the interview management system and to be consistent with the stated goals of transparency and equity. If a 15-minute window was chosen instead, then an additional four programs would be considered strictly adherent; this includes three programs that sent invites 9 to 13 minutes early and one program that released interviews 10 minutes late. There was also one program each that sent interviews 40 and 60 minutes early. This sensitivity analysis does not meaningfully change our results. However, it does suggest that a small number of programs either incidentally or purposefully sent interview invites far enough in advance of the stated release time to allow for interview scheduling.
Additional research on the UIOD structure would benefit decisions regarding future application cycles. For example, are there trends regarding programs that are nonadherent to the UIOD? Does the UIOD structure change outcomes compared with the pre-2020, more variable interview offer release schedule? Medical students often only go through the application cycle one time. This makes it challenging to compare experiences from year to year, especially regarding the level of stress induced by the interviews being released over the course of several months compared with the UIOD and given the exceptional circumstances surrounding the 2020 to 2021 application cycle.
The 2020 to 2021 cycle included unprecedented, synchronized changes to the orthopaedic residency application process. We found that residency program adherence to a universal interview release was low in the 2020 to 2021 application cycle, with only 41% of the programs studied (19 of 46) strictly adhering to the UIOD date and time and 71% of the programs (33 of 46) adhering to the interview release date. The bulk of available interview slots filled within 30 minutes. By analyzing interview scheduling data directly, we propose a phased approach to universal residency interview offers. Prospective applicants should consider using a com domain e-mail address to interface with residency programs, given the average relative delay from invitation release to scheduling for students using edu e-mail domains. Using an evidence-based approach with the help of targeted data sources, we may be able to improve our understanding of the orthopaedic surgery match in ways that assist applicants and programs in future application cycles.
References printed in bold type are those published within the past 5 years.
1. Aiyer AA, Granger CJ, McCormick KL, et al.: The impact of COVID-19 on the orthopaedic surgery residency application process. J Am Acad Orthop Surg 2020;28:e633-e641.
2. Boyd CJ, Inglesby DC, Corey B, et al.: Impact of COVID-19 on away rotations in surgical fields. J Surg Res 2020;255:96-98.
3. Rajesh A, Asaad M: Alternative strategies for evaluating general surgery residency applicants and an interview limit for MATCH 2021: An impending necessity. Ann Surg 2021;273:109-111.
4. Ramkumar PN, Navarro SM, Chughtai M, Haeberle HS, Taylor SA, Mont MA: The orthopaedic surgery residency application process: An analysis of the applicant experience. J Am Acad Orthop Surg 2018;26:537-544.
5. 2020 ACGME-Accredited Orthopaedic Residency Interviews & Virtual Experiences. American Orthopaedic Association. Available at: https://www.aoassn.org/interviews-and-visits/
. Accessed March 11, 2021.
6. Universal Interview Offer Day: American Orthopaedic Association. Available at: https://www.aoassn.org/programs/
. Accessed March 27, 2021.
7. The National Resident Matching Program: Advance Data Tables: 2021 Main Residency Match. Available at: https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2021/03/Advance-Data-Tables-2021_Final.pdf
. Accessed April 25, 2021.
8. ERAS 2021 Participating Specialties & Programs. Available at: https://services.aamc.org/eras/erasstats/par/display.cfm?NAV_ROW=PAR&SPEC_CD=260
. Accessed April 25, 2021.
11. Wickham H: Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016. Available at: https://ggplot2.tidyverse.org
. Accessed April 19, 2021.
12. Hariton E, Bortoletto P, Ayogu N: Residency interviews in the 21st century. J Grad Med Educ 2016;8:322-324.
13. Gruppuso PA, Adashi EY: Residency placement fever: Is it time for a reevaluation?. Acad Med J Assoc Am Med Coll 2017;92:923-926.
14. Trikha R, Keswani A, Ishmael CR, Greig D, Kelley BV, Bernthal NM: Current trends in orthopaedic surgery residency applications and match rates. J Bone Joint Surg Am 2020;102:e24.
15. Goralnick E, Kaufmann C, Gawande AA: Mass-vaccination sites—an essential innovation to curb the covid-19 pandemic. N Engl J Med 2021;18:e67.
16. Lai J, Widmar NO: Revisiting the digital divide in the COVID‐19 era. Appl Econ Perspect Pol. 2020;Oct 12:10.1002/aepp.13104.
17. AOA/CORD 2021-2022 Application Cycle and Universal Interview Offer Day Updated Recommendations for Orthoapedic Surgery Residency Programs. Available at: https://2wq9z720przc30yf331grssd-wpengine.netdna-ssl.com/wp-content/uploads/2021/09/AOACORD-Guidelines-21-22-App-Season-and-UOD.pdf
. Accessed February 20, 2022.
18. Chen AF, Secrist ES, Scannell BP, Patt JC: Matching in orthopaedic surgery. J Am Acad Orthop Surg 2020;28:135-144.
19. Li NY, Gruppuso PA, Kalagara S, Eltorai AEM, DePasse JM, Daniels AH: Critical assessment of the contemporary orthopaedic surgery residency application process. J Bone Joint Surg Am 2019;101:e114.
20. Nagarkar PA, Janis JE: Fixing the “match”: How to play the game. J Grad Med Educ 2012;4:142-147.
21. Whipple ME, Law AB, Bly RA: A computer simulation model to analyze the application process for competitive residency programs. J Grad Med Educ 2019;11:30-35.
22. The Coalition for Physician Accountability's Work Group on Medical Students Moving Across Institutions For Updated Recommendations on Away Rotations for Medical Education Institutions of LCME®-Accredited, U.S. Osteopathic, and Non-U.S. Medical School Applicants. Available at: https://www.aamc.org/media/52326/download
. Accessed April 18, 2021.
23. Love ER, Dexter F, Reminick JI, Sanford JA, Karan S: Interview data highlight importance of “same-state” on anesthesiology residency match. Anesth Analg 2021;132:223-230.
24. Bernstein J: Not the last word: Predicting chaos in the residency match. Clin Orthop Relat Res 2021;479:452-457.
25. Gundle KR, Mickelson DT, Cherones A, Black J, Hanel DP: Rapid web-based platform for assessment of orthopedic surgery patient care milestones: A 2-year validation. J Surg Educ 2017;74:1116-1123.
26. Association of American Medical Colleges: Medical School Admission Requirements (MSAR) Report for Applicant and Advisors: Interview Procedures, 2022. Available at: https://students-residents.aamc.org/media/7051/download
. Accessed August 11, 2021.