Introduction
In selection of orthopaedic surgery residents, the interview is a critical component for the applicant and the training program1,2. Previous studies attempted to identify predictors of success both in residency and in practice at the time of residency application and interview/ranking process3-6. Some of these previously identified factors include US Medical Licensing Examination (USMLE) scores correlating with in-training scores and surgical skills and medical school honors grades correlating with higher overall performance, interpersonal skills, knowledge, and surgical skills3-6. Poor-performing residents were notably difficult to predict4. However, there is a dearth of research on the applicant interview and ranking process. Anecdotal experience suggests significant variability in faculty and resident evaluations of applicants.
As of January 2022, the USMLE converted the Step 1 examination to pass/fail scoring, eliminating a key historical objective measure for students applying to orthopaedic surgery residency7,8. Because of the coronavirus disease 2019 (COVID-19) pandemic, residency programs conducted interviews virtually during the 2020-2022 application cycles and subinternships were canceled or limited, despite record application numbers9. Many medical schools have also changed the information they report to residency programs such as using pass-fail grades, elimination of class rank, and variable use of Alpha Omega Alpha (AOA) designation because of concern of ethnic and socioeconomic biases10,11. These developments offer significant challenges and changes to the residency candidate evaluation process and may place more weight on the interview. Contemporizing the interview process to minimize bias becomes more paramount given these changes5.
Therefore, this study sought to characterize the residency applicant interview and ranking process at a large urban academic university setting. We hypothesized that large variability exists among faculty interviewers' evaluations, and key applicant academic data were factors associated with final ranking.
Materials and Methods
Study Design
Per STROBE guidelines12, data were collected from orthopaedic surgery residency interviews at our institution from December 2020 to December 2021. Applicants consented to the use of their Electronic Residency Application Service (ERAS) application data for residency program research purposes. This study was also given exemption status by the institutional review board (IRB Control #21E.454). Included applicants were at least 18 years and deidentified. Applicants with inadequate applicant interview data were excluded.
Institutional Interview Process
Screening of ERAS applications was conducted by the program's postgraduate year 5 orthopaedic residents, the program director (PD), and senior faculty. This process identifies approximately 100 qualified applicants to interview of approximately 1,000 applicants each year. This initial review of applicants was not analyzed within this study. Applicants who rotated as subinterns were given a consensus grade by residents during a separate review session. On interview days, each applicant had 3 interviews with faculty and chief residents, as well as a group interview with the PD and chairman. Interviews were conducted virtually for the first time at our institution in 2020 and 2021 due to the COVID-19 pandemic. Each interviewer graded the applicant on an 8-point scale from A1 to D and ranked all of the applicants they interviewed. Individual applicant grades and ranks were converted to a 100-point scale and averaged to a “score.” An additional score was generated for each applicant by a committee meeting of all faculty/chief resident interviewers of a given applicant after interviews occurred and individual interviewer scores were generated. Faculty and PD/chairman interview scores were averaged to a final combined score used as a guide in generating a final rank list (Fig. 1).
;)
Fig. 1: Diagram summarizing the annual orthopaedic resident interview and scoring process at our institution. PD = program director.
Data Abstraction
Abstracted demographic data included sex, underrepresented minority (URM) status, years since finishing their undergraduate education, geographical distances of applicant hometown and medical school from our program, other degrees (i.e., PhD, MBA), prior match history, couples match, international medical graduate (IMG) status, notable attribute(s), and subinternship participation. Notable attributes were subjectively recorded and consisted of unique attributes or life experiences, that is, college athletics and Eagle Scout. The following academic data were collected: USMLE Step 1 and 2 scores, educational breaks (i.e., before/during medical school), extended research (i.e., >1 year spent in a research role), postbaccalaureate (postbac), major, home student, and US News and World Report undergraduate and medical school rank. Interview scores were evaluated also including faculty/chief resident score, PD/chairman score, consensus resident scores for rotators, and combined mean scores.
Statistical Methods
Our departmental biostatistician performed all statistics. Advanced statistical analyses included bivariate Poisson regression and multivariate stepwise Poisson regression to identify final rank number predictors. For power analysis, this was a pilot study of natural history and sample size was one of convenience. Significance was determined at p value <0.05. All statistical analyses were performed using R Studio (version 4.1.2).
Results
Cohort
Ninety-nine applicants were interviewed in the 2020-2021 cycle and 102 applicants were interviewed in 2021 to 2022 for a total of 201 applicants. Six applicants from the 2020-2021 cycle were excluded from the study because of interviews conducted separately from the standard protocol, providing 195 included applicants for the study.
The application cycle cohorts were statistically similar. Differences from 2020 to 2021 included greater medical school class rank (73.5 ± 21.7 in 2021, 54.2 ± 23.6 percentile in 2020, p = 0.002); fewer applicants with notable attributes (22.5% vs. 72.0%, p < 0.001); and improved interview performance for mean interviewer scores, PD/chairman scores, and combined scores (p < 0.001). There were 20% more applications received between years and an increase in interviewing faculty (16 in 2020, 31 in 2021).
Interviewer Inter-Rater Reliability
There was no true agreement between interviewers for both grade and rank in both application years with intraclass coefficient value ranges between 0.0 and 0.2.
Predictors of Applicant being Ranked vs. Unranked
Applicant demographic, academic, and interview variables were compared among those ultimately ranked or not ranked. For applicant demographics (Table I), the only statistically significant variable was applicants with previous unsuccessful match to orthopaedic surgery were less likely to be ranked (p = 0.023). For applicant academics and interview scores (Table II), USMLE Step 1 (p = 0.005) and Step 2 (p = 0.004) scores, number of textbook chapters published (p = 0.033), educational breaks (p = 0.024), class rank percentiles (p = 0.002), and all interviewer score types (p < 0.001) were statistically significantly greater in ranked applicants.
TABLE I -
Applicant Demographics
*
| Variable |
Combined (n = 195) |
Not Ranked (n = 31) |
Ranked (n = 164) |
p Value |
| Prior match, n (%) |
8 (4.1) |
4 (12.9) |
4 (2.44) |
0.023†
|
| Male sex, n (%) |
147 (75.4) |
26 (83.9) |
121 (73.8) |
0.333 |
| URM, n (%) |
49 (25.1) |
9 (30.0) |
40 (25.0) |
0.729 |
| Mean years since undergrad, mean ± SD (range) |
5.75 ± 2.07 (0-18) |
6.26 ± 3.61 |
5.65 ± 1.62 |
0.362 |
| Mean home distance (km), mean ± SD (range) |
946.6 ± 1,088.2 (0-4,052.3) |
1,099.2 ± 1,308.4 |
1,432.32 ± 2,018.1 |
0.246 |
| Mean medical school distance (km), mean ± SD (range) |
836.7 ± 1,306.6 (0-9,150.6) |
1,247.2 ± 1,947.3 |
939.9 ± 1,693.0 |
0.415 |
| Other degrees, n (%) |
36 (18.5) |
6 (19.4) |
30 (18.3) |
1.000 |
| Couples match, n (%) |
11 (5.6) |
2 (6.45) |
9 (5.49) |
0.688 |
| IMG, n (%) |
5 (2.6) |
1 (3.23) |
4 (2.44) |
0.583 |
| Subinternship participation, n (%) |
28 (14.4) |
0 (0.00) |
28 (31.1) |
N/A |
*IMG = international medical graduate, N/A = not available, and URM = underrepresented minority.
†Statistically significant.
TABLE II -
Applicant Academics and Interview Scores
*
| Variable |
Combined (n = 195) |
Not Ranked (n = 31) |
Ranked (n = 164) |
p Value |
| USMLEs |
| Step 1 score |
248.3 ± 12.1 (205-274) |
240 (17.6) |
250 (10.1) |
0.005†
|
| Step 2 score |
256.7 ± 12.4 (217-280) |
249 (13.9) |
258 (11.5) |
0.004†
|
| Step 2 available |
148/195 (75.9) |
26 (83.9) |
122 (74.4) |
0.367 |
| Research experience |
| Chapters |
0.3 ± 1.1 (0-8) |
0.10 (0.40) |
0.35 (1.18) |
0.033†
|
| Submissions |
6.9 ± 8.0 (0-44) |
5.67 (6.20) |
7.17 (8.29) |
0.280 |
| Publications |
7.9 ± 10.0 (0-69) |
5.90 (8.75) |
8.33 (10.2) |
0.175 |
| Posters |
9.5 ± 10.8 (0-55) |
8.13 (9.88) |
9.71 (11.0) |
0.426 |
| Podiums |
5.8 ± 9.1 (0-44) |
4.55 (7.78) |
6.09 (9.37) |
0.335 |
| Extended research |
48/195 (24.6) |
5 (16.1) |
43 (26.2) |
0.333 |
| Medical school background |
| Education break |
146/195 (74.9) |
18 (58.1) |
128 (79.0) |
0.024†
|
| Class rank percentile |
75.1 ± 19.9 (10-99) |
54.2 (23.6) |
79.4 (16.0) |
0.002†
|
| AOA |
57/195 (29.2) |
5 (16.1) |
52 (31.7) |
0.183 |
| Home student |
34/195 (17.4) |
9 (29.0) |
25 (15.2) |
0.110 |
| Med school US News & World Report research rank |
39.5 ± 24.8 (1-93) |
43.9 (25.2) |
38.8 (24.7) |
0.339 |
| Med school US News & World Report primary care rank |
53.8 ± 25.3 (2-93) |
52.5 (26.5) |
54.0 (25.1) |
0.788 |
| Other background |
| Science major |
172/195 (23) |
27 (87.1) |
145 (88.4) |
0.767 |
| Notable attribute |
90/195 (46.2) |
12 (38.7) |
78 (47.6) |
0.478 |
| Postbac |
15/195 (7.7) |
3 (9.68) |
12 (7.32) |
0.712 |
| Undergrad US News & World Report rank |
59.7 ± 96.9 (1-1,068) |
104 (190) |
51.1 (62.8) |
0.135 |
| Interview performance |
| Mean interviewer score |
45.9 ± 17.7 (11.7-91.0) |
65.8 (14.0) |
43.6 (16.6) |
<0.001†
|
| Mean PD/chairman score |
47.2 ± 25.9 (10.3-100) |
79.2 (21.3) |
43.5 (23.8) |
<0.001†
|
| Combined score |
46.4 ± 19.6 (11.6-95.5) |
72.5 (14.1) |
43.4 (17.9) |
<0.001†
|
*AOA = Alpha Omega Alpha, PD = program director, and USMLE = United States Medical Licensing Examination.
†Statistically significant.
Predictors of Applicant Final Rank
Bivariate correlations between all continuous variables and final rank number were performed. All interviewer scores showed statistically significant correlation (Table III). Multivariate Poisson regression analyses showed the relative risk of a variable on final rank number (Table IV). Applicants with other degrees, couples match, AOA designation, educational breaks, notable attributes, URM status, and greater years since graduating college had higher match likelihood (lower rank). Conversely, male sex, IMG, prior match history, undergraduate science majors, extended research, and home students were associated with lower match likelihood (higher rank). Applicants with extended research experience had statistically greater number of research publications (7.0 ± 5.8) and submissions (10.0 ± 7.7) vs. those without (4.7 ± 4.7, 3.9 ± 4.3; p = 0.021, p < 0.001, respectively). Applicants with extended research otherwise did not differ in class rank, USMLE Step 1 and 2 scores, AOA designation, research posters/podiums, or undergraduate and medical school US News & World Report ranking.
TABLE III -
Interview Score Correlation Results with Final Rank Number
*
| Interview score |
Correlation |
p Value |
| Mean interviewer score (2020) |
0.780 |
<0.001 |
| Mean interviewer score (2021) |
0.757 |
<0.001 |
| Mean PD score (2020) |
0.460 |
<0.001 |
| Mean PD score (2021) |
0.780 |
<0.001 |
| Mean combined score (2020) |
0.715 |
<0.001 |
| Mean combined score (2021) |
0.977 |
<0.001 |
| Mean resident score (2021) |
0.863 |
<0.001 |
| Mean committee score (2021) |
0.860 |
<0.001 |
TABLE IV -
Multivariate Regression Identifying Final Rank Predictors
*
| Variable |
Relative Risk |
95% Confidence Interval |
| Factors associated with lower final rank (greater match likelihood) |
| Any other degrees |
0.73 |
0.68-0.79 |
| MBA degree |
0.68 |
0.60-0.79 |
| Couples match |
0.48 |
0.41-0.55 |
| AOA |
0.68 |
0.65-0.72 |
| Education break |
0.87 |
0.82-0.92 |
| Postbaccalaureate |
0.78 |
0.71-0.86 |
| Notable attribute |
0.86 |
0.85-0.95 |
| Research chapters |
0.97 |
0.95-0.99 |
| URM |
0.87 |
0.82-0.92 |
| Greater years since undergraduate |
0.95 |
0.93-0.96 |
| Factors associated with higher final rank (lesser match likelihood) |
| Male sex |
1.31 |
1.24-1.39 |
| IMG |
1.54 |
1.36-1.74 |
| Prior match history |
1.60 |
1.41-1.80 |
| Science major |
1.45 |
1.34-1.59 |
| Extended research experience |
1.27 |
1.20-1.33 |
| Home student |
1.22 |
1.15-1.30 |
| Factors showing no association with final rank (not predictive of match likelihood) |
| Medical school distance |
1.00 |
0.99-1.0 |
| Home distance |
1.00 |
1.0 |
| Mean interview score |
1.03 |
1.02-1.03 |
| Mean PD/chair score |
1.02 |
1.01-1.03 |
| Combined interview score |
1.03 |
1.02-1.03 |
| Undergraduate US News & World Report rank |
1.00 |
1.0 |
| Medical school US News & World Report rank |
1.00 |
1.0 |
| USMLE Step 1 score |
0.99 |
0.99-1.0 |
| USMLE Step 2 score |
0.99 |
0.99-1.0 |
| USMLE Step 2 available |
0.98 |
0.93-1.04 |
| Submissions |
0.99 |
0.99 |
| Publications |
0.99 |
0.99 |
| Class rank percentile |
0.98 |
0.98-0.99 |
| Posters |
0.99 |
0.98-0.99 |
| Podiums |
0.99 |
0.98-0.99 |
| PhD degree |
0.92 |
0.77-1.07 |
| MPH degree |
0.94 |
0.80-1.10 |
*AOA = Alpha Omega Alpha, IMG = international medical graduate, PD = program director, URM = underrepresented minority, USMLE = United States Medical Licensing Examination.
Discussion
Overall, this study presents residency interview analyses over 2 years at our large academic institution, when pivotal changes faced the orthopaedic surgery resident selection process nationwide. Our hypothesis that significant variability existed in faculty interview evaluations was affirmed. Our hypothesis on which specific factors were related to final rank was partially affirmed. Orthopaedic surgery is one of the most competitive medical specialties, with 100% of 875 positions filled in 2022, leaving 852 applicants (49.3%) unmatched13. The mean number of applications submitted per applicant to programs increased by 72%, from 48 in 2006 to 83 in 2017. Similarly, mean total applications received per program have increased 46.4%, from 457 in 2010 to 669 in 2017. As a result, estimated in-depth reviews (subjectively defined as thorough review of all application components in a survey of PDs) of applicants decreased to only 45% of applications in 201614. Some medical schools have switched to pass/fail grades and discarded class rank and honors societies. National changes have occurred including the transition of USMLE Step 1 scores to pass/fail and the loss of most in-person interactions during the COVID-19 pandemic.
In our study, being ranked was more based on academic factors such as USMLE Step 1 and 2 scores and class rank. Whether the USMLE Step 2 score was available did not have bearing on rank likelihood. Demographics were equivalent between ranked and unranked applicants, except those with prior match participation, which could reflect reasons for the original failure to match. Studies have highlighted the lower match likelihood facing repeat applicants when compared with first-time applicants15,16.
Greater class rank signifies an applicant's academic competitiveness relative to their own medical school class. The nature of educational breaks (i.e., years spent out of medical training before or during medical school) was variable but ranged from nonmedical former careers to military service. This time away may give applicants a more enriched background and life experience applicable to residency training or patient care. When compared with national averages of matched applicants in orthopaedic surgery, our institution's ranked applicants had higher USMLE Step 1 and 2 scores, research experiences, AOA designation, PhD or other degrees, and notable attributes including work/volunteer experiences13. Interestingly, once reaching the interview, factors traditionally believed to be as important, including research publications and subinternships, did not relate to rank likelihood. Notably, the overall number of rotating students was decreased from previous years due to COVID-19 restrictions; therefore, this result may be limited because of the current period while COVID-19 restrictions are in place.
Interestingly, our results showed poor interviewer inter-reliability and no true agreement, yet interview scores still heavily correlated with final rank. This highlights a potential for bias and an important area for attention. Legato et al. similarly reported that the interview correlated with final rank list order and that inter-rater reliability among faculty members was low for the applicant scoring process2. In our study, resident and committee scores showed higher correlation with final rank, suggesting possible greater weight to these scores (relative to PD/chairman score). Resident scores may reflect “program fit” from unique prolonged applicant interaction both clinically and socially. Postinterview committee consensus scores may facilitate greater discussion and less biased reflection of applicants, vs. a single interviewer's first impression.
Final rank number in our study appeared based on applicant characteristics including multiple degrees, educational breaks, and notable attributes. Other studies have attempted to ascertain what applicant factors are associated with successful residents. Fryer et al. found that USMLE scores were predictive of in-training scores only, and raw objective performance scores were more predictive of later performance than faculty-biased scores. The authors also concluded that poor-performing residents were notably difficult to predict4. The extensive review by Egol and Zuckerman on resident selection reported that high USMLE Step 1 scores correlated with higher Orthopaedic In-Training Examination scores and surgical skills during residency while honors grades correlated with overall performance, interpersonal and surgical skills, and knowledge3. In a study of Canadian orthopaedic residency programs, work ethics, interpersonal qualities, orthopaedic experience, and enthusiasm correlated most with final rank order. They also found that agreement across programs was poor5.
Having other degrees, couples matching, AOA designation, taking an educational break, notable attributes, publishing textbook chapters, and greater years since graduating college all increased match likelihood in our study and may serve as markers of applicant maturity. Interestingly, objective measures traditionally believed to be very important for match likelihood, such as geographical origin, interview performance, class rank, USMLE Step 1 and 2 scores, and research accolades, had no bearing on final rank/match likelihood. IMG applicants were rare overall but associated with decreased match likelihood, consistent with other literature3. More historically common applicant factors, such as male sex, extended research, home institution, and undergraduate science major, were associated with a lower match likelihood, reflecting an effort to diversify the residency program.
Along these lines, our study showed that URM and female applicants were associated with greater match likelihood. While women constitute approximately 49% of the US workforce, a recent American Academy of Orthopaedic Surgeons (AAOS) article reports that only 6.5% of practicing orthopaedic surgeons and 14.8% of candidate AAOS members (resident/fellow trainees) are women17. Minorities constitute over 30% of the US population, yet represent only 6% of US physicians. Numbers are among the lowest in orthopaedic surgery, but have increased between 0.5% and 1.4% over the past decade18. Increased diversity in the field is imperative toward improving patient satisfaction and addressing healthcare inequalities. Female and URM applicants made up 25% of our respective applicant pool and were ranked with greater match likelihood. However, there is still improvement to be made, and successfully matching these applicants may require focused recruitment effort and faculty/resident representation. Target pipeline curricula such as summer internships or musculoskeletal rotations in medical school (or earlier) have shown promise in increasing numbers of diverse applicants18.
USMLE scores, AOA designation, research, and volunteer experiences have been cited as other reasons for the disparity19-21. Eliminating these application factors may come from noble intent but could have negative consequences. In a survey of 78 orthopaedic surgery PDs, 59% responded that USMLE Step 2 Clinical Knowledge score will be most important after Step 1 transitions to pass/fail, and 90% will encourage applicants to include it in their application8. Pass/fail grading could merely increase applicant stress surrounding USMLE Step 2, with fewer opportunities to demonstrate academic performance. Students from schools with lesser known or nonexistent orthopaedic departments may face greater challenges, if programs weigh remaining application factors more heavily, such as recommendation letters and school reputation. Perhaps limiting the number of resident applications overall, as other medical specialties have done, could increase programs' ability to more closely review all applicants14. Future studies must examine the “preference signal” system initiated for the 2022-2023 orthopaedic surgery residency application cycle by the American Orthopaedic Association Council of Orthopaedic Residency Directors22.
Other potential solutions for the interview process include increasing the number of interviewers for a given applicant or incorporating personality tests to account for bias1. Many business corporations have used questionnaires identifying personal characteristics important for success, such as ambition, humility, and curiosity, during the hiring process7. Motor tasks have not been shown to correlate with later resident performance or operative ability, although they do improve over time when measured throughout training23-25. Athletic or musical talents correlate poorly with operative ability, despite historically favorable bias8.
This study also represents the first virtual interviews conducted at our institution due to the COVID-19 pandemic. A recent survey of applicants and PDs found that virtual experiences did not adequately replicate the social factors that applicants found most important when ranking a program, such as perception of resident happiness/camaraderie26. In another survey, PDs reported less ability to determine applicant fit; clinical, social, and surgical skills; and genuine interest while greater importance was placed on research. Most candidates (81%) and PDs (79%) preferred in-person interviews, despite cost savings for both27. Future studies at our institution must analyze the effect of virtual interview processes.
Strengths of this study include a large series of consecutive data over 2 years at a strong orthopaedic academic center, whereas limitations are that the data presented may be institution-specific. Another potential limitation of this study is selection bias because only applicants invited to interview were analyzed. The initial screening of applications to invite for interview was not assessed but should be further examined in future studies.
In conclusion, there was significant variability and poor reliability at our institution among faculty interviewers' applicant ratings. Being ranked at all was based more on academic record and interview performance while final rank number seemed based on applicant qualities. We hope this study can help optimize the resident interview and selection process to be more objective, reliable, and diverse while facilitating discussion among orthopaedic residency programs on current and future challenges.
Source of Funding
There was no funding source for this study.
Note: The authors acknowledge orthopaedic surgery residency program coordinator Kelly Quici. There was no funding source for this study.
References
1. Quintero AJ, Segal LS, King TS, Black KP. The personal interview: assessing the potential for personality similarity to bias the selection of orthopaedic residents. Acad Med. 2009;84(10):1364-72.
2. Legato JM, Fuller DA, Kirbos C, Pollard M, Tase D, Kim TW, Miller LS. Matching into an orthopedic residency: which application components correlate with final rank list order? J Surg Educ. 2019;76(2):585-90.
3. Egol KA, Collins J, Zuckerman JD. Success in orthopaedic training: resident selection and predictors of quality performance. J Am Acad Orthop Surg. 2011;19(2):72-80.
4. Fryer JP, Corcoran N, George B, Wang E, Darosa D. Does resident ranking during recruitment accurately predict subsequent performance as a surgical resident? J Surg Educ. 2012;69(6):724-30.
5. Gilbart MK, Cusimano MD, Regehr G. Evaluating surgical resident selection procedures. Am J Surg. 2001;181(3):221-5.
6. Raman T, Alrabaa RG, Sood A, Maloof P, Benevenia J, Berberian W. Does residency selection criteria predict performance in orthopaedic surgery residency? Clin Orthop Relat Res. 2016;474(4):908-14.
7. Porter SE, Graves M. Resident selection beyond the United States Medical Licensing Examination. J Am Acad Orthop Surg. 2017;25(6):411-5.
8. Cohn MR, Bigach SD, Bernstein DN, Arguello AM, Patt JC, Ponce BA, Beal MD, Kogan M, Dyer GSM; Collaborative Orthopaedic Educational Research Group. Resident selection in the wake of United States Medical Licensing Examination Step 1 transition to pass/fail scoring. J Am Acad Orthop Surg. 2020;28(21):865-73.
9. Mikhail D, Margolin EJ, Sfakianos J, Clifton M, Sorenson M, Thavaseelan S, Haleblian G, Kavoussi L, Badalato GM, Richstone L. Changing the status quo: developing a virtual sub-internship in the era of COVID-19. J Surg Educ. 2021;78(5):1544-55.
10. Gordon M. A medical school tradition comes under fire for racism [published Public Health]. Shots. 2018.
https://www.npr.org/sections/health-shots/2018/09/05/643298219/a-medical-school-tradition-comes-under-fire-for-racism. Accessed July 20, 2022.
11. Smith TM. How UCSF is working to overcome bias in clerkship grading. Medical School Diversity. 2021.
https://www.ama-assn.org/education/medical-school-diversity/how-ucsf-working-overcome-bias-clerkship-grading. Accessed July 20, 2022.
12. Cuschieri S. The STROBE guidelines. Saudi J Anaesth. 2019;13(suppl 1):S31-4.
13. National Resident Matching Program. Charting Outcomes in the Match: Senior Students of U.S. MD Medical Schools. Washington, DC: NRPM; 2022.
14. Li NY, Gruppuso PA, Kalagara S, Eltorai AEM, DePasse JM, Daniels AH. Critical assessment of the contemporary orthopaedic surgery residency application process. J Bone Joint Surg Am. 2019;101(21):e114.
15. Amin NH, Jakoi AM, Cerynik DL, Kumar NS, Johanson N. How should unmatched orthopaedic surgery applicants proceed? Clin Orthop Relat Res. 2013;471(2):672-9.
16. Kheir MM, Tan TL, Rondon AJ, Chen AF. The fate of unmatched orthopaedic applicants: risk factors and outcomes. JB JS Open Access. 2020;5(2):e20.00043.
17. DeMaio M. Making the case (again) for gender equity. AAOS Now. 2019.
https://www.aaos.org/aaosnow/2019/jun/youraaos/youraaos05/. Accessed July 21, 2022.
18. Jiménez-Almonte JH, Jensen AR, Ghodasra JH, Chan WW. Minority representation among orthopaedic surgery residents. AAOS Now. 2017.
https://www.aaos.org/aaosnow/2017/jul/clinical/clinical06/. Accessed July 21, 2022.
19. Stanton T. Study probes reasons for underrepresentation of minorities in orthopaedic residencies. AAOS Now. 2019.
https://www.aaos.org/aaosnow/2019/nov/research/research03/. Accessed July 21, 2022.
20. Adelani MA, Harrington MA, Montgomery CO. The distribution of underrepresented minorities in U.S. orthopaedic surgery residency programs. J Bone Joint Surg Am. 2019;101(18):e96.
21. Poon S, Nellans K, Rothman A, Crabb RAL, Wendolowski SF, Kiridly D, Gecelter R, Gorroochurn P, Chahine NO. Underrepresented minority applicants are competitive for orthopaedic surgery residency programs, but enter residency at lower rates. J Am Acad Orthop Surg. 2019;27(21):e957-e968.
22. Guthrie ST. Orthopaedic surgery residents will be able to identify their preferences for programs at the time of application. AAOS Now. 2022.
https://www.aaos.org/aaosnow/2022/aug/residency/residency02. Accessed November 20, 2022.
23. Williams JF, Watson SL, Baker DK, Ponce BA, McGwin G, Gilbert SR, Khoury JG. Psychomotor testing for orthopedic residency applicants: a pilot study. J Surg Educ. 2017;74(5):820-7.
24. Anderson DD, Long S, Thomas GW, Putnam MD, Bechtold JE, Karam MD. Objective structured assessments of technical skills (OSATS) does not assess the quality of the surgical result effectively. Clin Orthop Relat Res. 2016;474(4):874-81.
25. Lopez G, Wright R, Martin D, Jung J, Bracey D, Gupta R. A cost-effective junior resident training and assessment simulator for orthopaedic surgical skills via fundamentals of orthopaedic surgery: AAOS exhibit selection. J Bone Joint Surg Am. 2015;97(8):659-66.
26. Tawfik AM, Imbergamo C, Chen V, Filtes P, Butler A, Gatt C, Katt BM. Perspectives on the orthopaedic surgery residency application process during the COVID-19 pandemic. J Am Acad Orthop Surg Glob Res Rev. 2021;5(10):e21.00091.
27. Brueggeman DA, Via GG, Froehle AW, Krishnamurthy AB. Virtual interviews in the era of COVID-19: expectations and perceptions of orthopaedic surgery residency candidates and program directors. JB JS Open Access. 2021;6(3):e21.00034.
