Orthopaedic Surgery Residency Program Adherence to Universal Interview Offer Day Guidelines: A Retrospective Analysis

Abstract

Introduction: 

The 2020 to 2021 application cycle was marked by structural changes due to the COVID-19 pandemic. Adaptations included the American Orthopaedic Association Council of Residency Program Directors recommendations for a universal interview offer day (UIOD), synchronizing applicant interview offer release. As a novel process within orthopaedics, there are limited data on the execution and effect of a UIOD. Study goals include determining (1) residency program adherence to the Council of Residency Program Directors UIOD guidelines during the 2020 to 2021 cycle, (2) how quickly programs filled available interviews, and (3) any difference in time from release to booking between applicants using edu and com e-mail domains.

Methods: 

Orthopaedic residency programs and applicants using the Thalamus interview management software platform during the 2020 to 2021 residency application cycle were analyzed, representing 46 residency programs and 993 interviewees (22% and 77% of the national total, respectively).

Results: 

Of the programs included in this study, 19 (41%) were strictly adherent to the UIOD and time, 14 (30%) were weakly adherent and sent out offers outside of the assigned time, and 13 (28%) were nonadherent and sent out offers on dates other than the UIOD. The average time to fill to 80% capacity was 26 ± 14 minutes (range 3 to 77 minutes) for the 33 programs that released on the UIOD. Applicants with edu e-mail domains scheduled their first interview an average of 1.8 minutes after those with com e-mail domains (14.8 versus 13.0 minutes, P < 0.05).

Conclusion: 

Despite more than 60% of the residency programs committing to participate, less than half of the programs that initially agreed to participate were strictly adherent to guidelines during the first UIOD in orthopaedic surgery. Although additional research is needed to analyze the unique, fully virtual 2020 to 2021 recruitment season, a phased or waved approach to the UIOD may improve the process for all stakeholders in future cycles.

Level of Evidence: 

Level V

The 2020 to 2021 orthopaedic surgery residency application cycle was unprecedented because of broad changes in response to the COVID-19 pandemic. This included disrupted medical student rotations, travel restrictions affecting interviews, and visiting student rotations.^1–3 In response, the American Orthopaedic Association Council of Orthopaedic Residency Directors (CORD) issued guidelines for both applicants and residency programs regarding changes to application requirements, visiting student rotations, and interviews. These initiatives included a universal interview offer day (UIOD) that specified a synchronized date and time for programs to release interview offers (10:00 am EST on November 23, 2020), a departure from previous years in which interview invitations were offered with varying frequency over a span of months.^4,5 The guidelines also stipulated a 48-hour response window for an interview offer by scheduling or declining an interview before programs could extend interviews to alternate candidates. Finally, programs were advised to not release more interview offers than available interview slots.

Most orthopaedic surgery residency programs agreed to adhere to the CORD guidelines, including the UIOD⁶—133 of the 208 programs (64%) participating in the National Resident Matching Program (NRMP) Match in 2021⁷ and 132 of the 192 programs (69%) listed by the Electronic Residency Application Service.⁸ The novel UIOD among orthopaedic surgery residency programs for the 2020 to 2021 residency match cycle provides a unique opportunity for analysis. Thus, this study aims to determine (1) the extent to which residency programs adhered to the CORD guidelines during the 2021 application cycle, (2) the speed of interview scheduling after release, and (3) scheduling variance by the e-mail domain (edu versus com applicant e-mail domains) because of our hypothesis that edu e-mail domain delivery delays result in slower scheduling. This analysis aims to identify innovative methods for the UIOD to streamline and optimize selection and equity for all participants and stakeholders. These results may be useful for CORD, individual residency programs, and orthopaedic surgery residency applicants when considering how to approach future application cycles.

Methods

This study was reviewed by an Institutional Review Board (STUDY00022821) and determined to fall outside the regulatory definition of a human subject study, given that all data were coded and deidentified before analysis. Therefore, no written consent or review of the Institutional Review Board was required by applicants or programs. Thalamus (SJ MedConnect, dba ThalamusGME; https://thalamusgme.com/) is a commercially available cloud-based graduate medical education (GME) interview management platform and scheduling software used by residency programs and hosted on the Microsoft Azure/Structured Query Language (SQL) server.

We conducted a retrospective investigation using Thalamus data to estimate population-level parameters regarding the 2020 to 2021 orthopaedic surgery residency recruitment cycle. All analysis was conducted after the completion of the 2021 NRMP Match. Data were provided by Thalamus and are not publicly available, although it regularly collaborates with researchers in specialties across GME. The Thalamus database segments each institution by an institutional identifier and each program within each institution by the program and an American College of Graduate Medical Education identifier (or a similar number for non–American College of Graduate Medical Education-accredited programs).

Aggregate Program Level Data

Data were captured for residency programs on the aggregate level. Each of the 46 orthopaedic surgery residency programs using Thalamus for recruitment was deidentified at the onset of the study. Before deidentification, a program's commitment to the UIOD was categorized as follows based on the CORD UIOD guidelines⁶: (1) “participatory” (programs committed to release on the UIOD and time), (2) “alternative” (programs releasing interview offers on a date other than the UIOD), or (3) “nonparticipatory” (programs that did not commit to UIOD participation). Information on Doximity rankings of residency programs was collected through their online Residency Navigator tool.⁹

After deidentification, the following variables were collected and/or calculated for each program: (1) total number of invitations extended, (2) total number of interview positions offered, (3) total number of interviews scheduled, (4) total number of interviews completed, (5) time of first interview invitation release, (6) number of invitations released by calendar date, and (7) time elapsed between first interview invitation sent and for 80% of positions offered to be filled. An 80% threshold was chosen in assessing the rate of interview position filling because it accounted for the bulk of positions offered by a program without requiring that all positions fill.

To determine compliance with the UIOD guidelines, parameters were assessed and categorized for each program. Programs were defined as “strictly adherent” if interviews were released within a 6-minute window around the scheduled time of 10:00 am EST (9:57 am to 10:03 am EST on November 23, 2020) and “weakly adherent” if programs were released on the scheduled date but before or after the strictly adherent time window (any time before 9:57 am or after 10:03 am EST on November 23, 2020). This narrow window was selected because of the consistent messaging from CORD regarding the interview release time and date and the stated goals of the UIOD regarding transparency in communication to applicants with a consistent and equitable application process.¹⁰ Given the programmed functionality within Thalamus, we thought that a 6-minute window allowed reasonable time for a program to release interviews on time while being small enough to prevent an applicant from being able to book multiple interviews within the window. Programs were categorized as “nonadherent” if they released their initial interview invitations on any date other than November 23, 2020.

Aggregate Individual Level Data

Deidentified applicant data were analyzed to examine applicants using edu and com e-mail domains at the time of application submission to the Electronic Residency Application Service. The mean time from the first Thalamus interview invitation release to the first scheduled Thalamus interview was calculated for the two groups at the aggregate level. No other applicant information was observed or analyzed.

Data Analyses

Analyses were conducted using descriptive statistics. Data were analyzed using R version 3.6.3 and ggplot2.¹¹ Statistical tests included chi square and Student t-tests where appropriate.

Results

Study Population

In the 2020 to 2021 residency application cycle, 993 different applicants were invited to interview at one or more of 46 orthopaedic surgery residency programs using the Thalamus platform for interview management. This represents 77% of the 1,289 total orthopaedic surgery interviewees and 22.1% of the 208 orthopaedic surgery residency programs entering the 2020 to 2021 NRMP Match. The geographic region of each program was assigned based on US Census Bureau regional definitions (Figure 1). Programs were located in 24 states, with 19 Midwestern (41%), 11 Northeastern (24%), nine Southern (20%), and seven Western (15%) programs (Figure 1). Residency programs spanned a broad range of Doximity rank percentiles within orthopaedic surgery (Figure 2). A total of 3,143 interviews were scheduled through the Thalamus platform, 2,885 of which were completed (Table 1). An average of 12.5 interviews were completed per available residency position across programs.

Figure 1: Diagram showing US Census Bureau regions. Map of the states included in each region (Northeast, South, Midwest, and West) as defined from US Census Bureau regional definitions.

Figure 2: Graph showing a histogram of Doximity reputation rankings of residency programs by percentile. Distribution of rank percentiles of included orthopaedic surgery residency programs (n = 46) as calculated by 1- (program rank/total programs ranked in Doximity), by decile.

Table 1 - Residency Program and Interview Distribution by Region

Region	No. of Orthopaedic Residency Programs (% of Total)	No. of Interviews Scheduled (% of Total)
Northeast	11 (24)	847 (27)
South	9 (20)	605 (19)
Midwest	19 (41)	1,300 (41)
West	7 (15)	391 (12)
Total	46	3,143

Adherence to Universal Interview Offer Day Guidelines

Of the 46 orthopaedic surgery residency programs studied, 41 pledged to participate in the UIOD, two committed to alternative offer release days, and three were nonparticipatory. Nineteen of the 46 residency programs (41%) were strictly adherent to the UIOD release (Figure 3). Fourteen programs (30%) were weakly adherent, releasing interview offers on November 23, 2020, but outside the common release time. Of those programs, seven released interviews early and seven released interviews late, spanning a 9-hour period around the set UIOD time. Two programs each released two separate waves of interview offers on November 23, 2020, failing to adhere to the 48-hour window guideline. Thirteen programs (28%) were nonadherent and released interview offers on dates other than November 23, 2020. Among nonadherent programs, four released first interview offers before the UIOD and nine released after the UIOD, spanning October 24 to December 30, 2020.

Figure 3: Graph showing adherence to universal interview offer day (UIOD) guidelines by initial commitment. Distribution of residency programs (n = 46) by adherence to the UIOD guidelines defined as follows: (1) strongly adherent programs release interviews within a 6-minute window around 10:00 am EST on the common release date, (2) weakly adherent programs released interviews on the common date but after the strictly adherent time window (any time after 10:03 am EST on November 23), and (3) nonadherent programs released initial interview invitations on any day other than November 23, 2020. Program commitment to UIOD defined by the list published online by the Council for Orthopaedic Residency Directors (CORD).

How Quickly Were Interview Offers Filled?

Of the 33 programs that sent out interview offers on November 23, 2020, the mean (± SD) time to filling 80% of available interviews was 26 minutes (SD ± 14.1 minutes; 3 to 77 minutes) (Figure 4, A). For the 19 strictly adherent programs, the mean time to filling 80% of interview slots for each program was 29 minutes (SD ± 8.3 minutes; 13 to 49 minutes). No programs released more interview offers than available interview positions in the initial release, but many programs extended additional invitations in later waves to fill available interviews (Figure 4, B). Residency programs using Thalamus completed a mean of 64.2 interviews (SD ± 26.4).

Figure 4: A–B, Graphs showing interview activity during the 2020 to 2021 residency application cycle. A, Interview activity by day over the 2020 to 2021 residency application cycle. Proportion of total interview slots (%) offered by orthopaedic surgery residency programs using Thalamus software (n = 46) with activity on each calendar day of the 2020 to 2021 application cycle, where black represents interview invitations extended to applicants, dark gray represents interviews subsequently scheduled by applicants, and light gray represents completed interviews. B, Cumulative interview activity over the 2020 to 2021 residency application cycle. Cumulative proportion of total interview slots (%) offered by orthopaedic surgery residency programs (n = 46) by calendar date over the 2020 to 2021 application cycle, where black represents interview invitations extended to applicants, dark gray represents interviews subsequently scheduled by applicants, and light gray represents completed interviews. The black line surpasses 100% to account for programs extending additional invitations beyond the number of available slots as previously invited applicants canceled.

Did Applicant E-mail Domain Affect Time to Scheduling?

Regarding the speed of interview scheduling, 728 of the 993 applicants (73.3%) received their initial interview invitation through Thalamus on the UIOD and responded within one hour. Of these applicants, the mean time from release to scheduling of the interview was 1.81 minutes longer in the edu e-mail domain group than in the com e-mail domain group (14.8 versus. 13.0 minutes, P < 0.05).

Discussion

In this study, we assessed the adherence of interview offer release among orthopaedic residency programs to the novel UIOD guidelines implemented in the 2020 to 2021 residency application cycle. We found that most of the orthopaedic surgery programs (72%) in this cohort released interview offers on the UIOD, but only 41% were strictly adherent to the date and time. Within 30 minutes, over 80% of the available interviews in the sample were booked, and the fastest a program filled to the 80% capacity was 3 minutes. The mean time from interview release to scheduling of the first interview was 1.8 minutes slower for applicants with the edu e-mail domain compared with the com domain.

Within GME, there has been a call to improve transparency around the interview invitation and scheduling process and timing.^4,12,13 The application process in previous cycles was a 1.5-year process, beginning with completing senior medical student rotations at home institution and then applying for and completing visiting medical student rotations at a minimum of two other institutions. Before the institution of UOID in orthopaedic surgery, candidates would typically submit applications to as many as 150 residency programs and then had to be constantly electronically available for a period of time spanning months to respond to interview invitations because they were sporadically and often unpredictably released. Given the very limited number of interview dates offered by each program and the restriction of travel logistics, minute delays in applicant response time could result in scheduling an interview for a less desirable date or loss of an interview slot entirely because of conflicts.⁴ The pressure on orthopaedic applicants is high because it remains one of the most competitive subspecialties, with an average of only three of four applicants matching into residency.¹⁴ The UIOD was instituted in part to increase predictability of interview release timing and overall transparency between residency programs and applicants.

The range of adherence of different programs within the UIOD highlights both a unique level of coordination across residency programs during the 2020 to 2021 Match cycle and demonstrates that a perfectly synchronized release is technically and operationally challenging. This is especially true because using a software platform such as Thalamus should increase the ease of adherence as it obviates the need to send individual e-mails and simply requires the triggered release of a prebuilt interview invitation list. Although not directly studied here, adherence may be lower among residency programs that did not use an interview management software to sync interview invites or scheduling. Potential barriers to adherence to specific time and date release guidelines include organizational planning, hospital e-mail and network policies, and Wi-Fi bandwidth. Although it is also possible that alternate release times were strategic on the part of some programs, the proportion of total interviews released on November 23 (Figure 4) suggests a good faith effort on the part of most of the programs studied.

These data also demonstrate the rapid speed at which interview slots filled. A common release time is akin to the “ticket-sale” problem, analogous to buying Super Bowl tickets or scheduling the general population for COVID-19 vaccines.¹⁵ By announcing exactly when interview offers will be released, applicants flock to their calendars several minutes before the release to manually refresh the page intermittently until the positions appear and then book interviews immediately. Releasing all interview offers simultaneously in this manner, therefore, runs the risk of creating possible issues of inequity related to the digital divide or applicants using surrogates or other technology to assist with interview scheduling, a common previous practice among residency applicants.¹⁶ Our data showed interviews filling within minutes from the time of interview offer release, implying that uninterrupted access to computer(s) or faster internet speeds would allow those with the financial means to access those resources more readily and thus book preferred interview dates more than those without access/resources. To mitigate this, one change introduced in the 2021 to 2022 application cycle was a delay of 48 hours between interview offer release and applicants being able to schedule the interview,¹⁷ allowing for all applicants to form a strategy for finalizing a desired interview schedule. However, possible inequity issues remain. Applicants able to use multiple devices or surrogates may have an advantage, especially if all scheduling becomes possible at a single time point.

One illustration of the challenges associated with simultaneous interview offer release is the difference in average time from release to booking seen between applicants using edu and com e-mail domains. We hypothesize that institutional informational technology policies including firewalls, VPNs, and spam filters used to protect hospital networks and patient information delay notification times compared with com e-mail domains. As some programs filled in 3 minutes, a mean difference of 1.8 minutes could easily be enough to disadvantage applicants using an edu e-mail domain. This study did not examine whether this altered the outcome of the Match for students in each group. However, the difference observed indicates one easily measurable, unforeseen source of inequity where others likely exist but are less easily measured. For this and other reasons, applicants should consider using an e-mail address with com domain as the main form of contact during the application cycle.

Although the data studied are only illustrative of the 2020 to 2021 Match cycle, it is important to interpret it within recent orthopaedic surgery residency application trends. The average number of applications submitted by prospective orthopaedic surgery residents has increased nearly twofold in recent years, from 46 in 2008 to 83 in 2018.^14,18,19 Modeling studies in orthopaedic surgery and otolaryngology have demonstrated that to minimize unmatched spots and maximize applicant ability to indicate interest in a program, the optimal number of applications submitted per applicant is 50.^19–21 The increasing applicant-to-residency position ratio and number of applications submitted per applicant are overwhelming both applicants and residency programs, diluting the weight of each application in signaling program interest. This skewed ratio of applicants to programs raises the question of different strategies that applicants, residency programs, and CORD might use to optimize the application process with the goal of matching applicants to programs in an equitable, efficient, and affordable process. The Coalition of Physician Accountability distributed recommendations for the 2021 to 2022 residency application cycle, based primarily on consensus agreements, with a call for data-driven solutions.²² Through data from residency application management platforms,²³ we may better understand this process to the benefit of subsequent application cycles.

This study demonstrated that the UIOD generated rapid fill of interview offers, but maximum speed of scheduling was not the primary goal of the guidelines. A more equitable and optimal strategy might include inviting candidates through a standardized waved or phased approach. One recommendation for consideration is releasing interviews on two common dates, separated by a week with invitations for only half of the total interview slots released at a time. A two-wave structure would maintain the goal of increased transparency around interview release timing while providing applicants in the first wave an opportunity to pick an interview date in a less rushed manner, knowing that only 50% of the available positions will fill during the first week. This optimizes applicant choice and decreases the perceived urgency to schedule interviews while also serving as a potential mechanism for programs to signal preference to applicants.²⁴ Programs could use phased interview offers to signal interest in applicants, and applicants would be able to make more informed decisions regarding which programs to prioritize in their interview schedule when conflicts inevitably arise. This may decrease the scheduling of undesired interviews or promote earlier cancellations. The 48-hour delay introduced in the 2021 to 2022 Match cycle may help alleviate the pressure to schedule an interview immediately on receiving the invitation and allow applicants to instead make scheduling decisions based on the totality of interview invitations received.

Limitations of this study include that only 22% of the orthopaedic surgery residency programs and 77% of the applicants are included in the data set. As a sensitivity analysis, for each state, we compared the proportion of all programs with the proportion of programs in this study. We found that the two distributions are very similar, with normally distributed errors with mean zero. One idiosyncrasy was that Ohio programs are over-represented. We recognize that this is not a truly randomly drawn sample and that the over-representation of one state could have an unforeseen effect. However, given the breadth of the sample and the singular outlier, we think that this analysis remains reliable and valuable. Real-time, granular data for a broad subset of applicants hold advantages over recall surveys done after the application cycle is completed, which forms the basis for a large portion of the residency application literature.²⁵ Although programs using interview management software could be systematically different from those who do not, we think these data hold notable value due for the 2021 to 2022 application cycle and beyond, especially given the potential for continued use of video interviewing.²⁶

Alternative definitions of adherence to the UIOD guidelines could be constructed. The 6-minute window was chosen because of its technical feasibility within the interview management system and to be consistent with the stated goals of transparency and equity. If a 15-minute window was chosen instead, then an additional four programs would be considered strictly adherent; this includes three programs that sent invites 9 to 13 minutes early and one program that released interviews 10 minutes late. There was also one program each that sent interviews 40 and 60 minutes early. This sensitivity analysis does not meaningfully change our results. However, it does suggest that a small number of programs either incidentally or purposefully sent interview invites far enough in advance of the stated release time to allow for interview scheduling.

Additional research on the UIOD structure would benefit decisions regarding future application cycles. For example, are there trends regarding programs that are nonadherent to the UIOD? Does the UIOD structure change outcomes compared with the pre-2020, more variable interview offer release schedule? Medical students often only go through the application cycle one time. This makes it challenging to compare experiences from year to year, especially regarding the level of stress induced by the interviews being released over the course of several months compared with the UIOD and given the exceptional circumstances surrounding the 2020 to 2021 application cycle.

Conclusion

The 2020 to 2021 cycle included unprecedented, synchronized changes to the orthopaedic residency application process. We found that residency program adherence to a universal interview release was low in the 2020 to 2021 application cycle, with only 41% of the programs studied (19 of 46) strictly adhering to the UIOD date and time and 71% of the programs (33 of 46) adhering to the interview release date. The bulk of available interview slots filled within 30 minutes. By analyzing interview scheduling data directly, we propose a phased approach to universal residency interview offers. Prospective applicants should consider using a com domain e-mail address to interface with residency programs, given the average relative delay from invitation release to scheduling for students using edu e-mail domains. Using an evidence-based approach with the help of targeted data sources, we may be able to improve our understanding of the orthopaedic surgery match in ways that assist applicants and programs in future application cycles.