Diversity in medicine has been shown to improve communication and quality of care for patients as well as the educational environment for residents in training [35, 38, 40, 46]. Orthopaedic surgery is one of the most competitive specialties, with yearly increases in the number of applicants per residency position  and increasingly competitive academic metrics [30, 31]. Despite the competitiveness of the specialty, orthopaedic surgery continues to lag behind other surgical subspecialties in terms of gender and racial/ethnic diversity in its residency composition [8, 29]. Across specialties, residency programs have reported using academic criteria such as United States Medical Licensing Examination (USMLE) scores, class rank, Alpha Omega Alpha membership, and meaningful involvement in research and extracurricular activities as important selection criteria for residency candidates [3, 17, 18, 25]. There is some evidence that using these criteria as residency interview screening tools may maximize training program outcomes such as first-time board certification success [12, 34].
Recent studies, however, suggest that the use of standardized measures may differentially affect underrepresented minorities [11, 14, 21]. Specifically, USMLE scores and Alpha Omega Alpha status have been shown to differ among applicant subsets, including race and gender [5, 7, 37, 43]. Within orthopaedic surgery, prior studies have found that small differences exist in the academic metrics of residency applicants and admitted residents when comparing groups based on racial ethnicity but less so based on gender [30, 31]. Some evidence suggests that these trends have not changed over time [4, 29], and it remains unknown whether differences in application metrics by race/ethnicity sufficiently explain the low admission of underrepresented minorities into residency programs.
We therefore sought to determine (1) the relative weight of academic variables for admission into orthopaedic residency, and (2) whether race and gender are independently associated with admission into an orthopaedic residency.
Materials and Methods
Orthopaedic surgery residency application data from 2005 to 2014 submitted through the Electronic Residency Application System (ERAS) was obtained from the Association of American Medical Colleges (AAMC); the National Board of Medical Examiners (NBME) provided de-identified USMLE Step 1 and Step 2 Clinical Knowledge scores for these applicants. Information on whether an orthopaedic surgery residency applicant was admitted into an orthopaedic surgery residency program was also provided by AAMC through the Graduate Medical Education (GME) Track Resident Survey dataset (part of the National GME Census), which is completed annually by residency program directors. The ERAS, NBME, and GME Track data were linked and de-identified by AAMC.
Data included information about first-time applicants (n = 12,093) for orthopaedic surgery residency positions in the United States. Only an applicants’ first year of entry into the application process was included in this analysis. Because of lower overall admission rates (8%-10%) and heterogeneous educational/training experience, non-US citizen applicants who graduated from non-US medical schools (8% [1008 of 12,093]), international medical school graduates (11% [1341 of 12,093]), doctors of osteopathic medicine (6% [776 of 12,093]), and those whose medical school type was not indicated (0.02% [2 of 12,093]) were excluded from the analysis. The remaining first-time applicants from allopathic US medical schools (n = 8966) and those who subsequently were admitted into an orthopaedic surgery residency during the 10-year study period (n = 6218) were analyzed (Fig. 1). Admitted orthopaedic surgery residents are defined as those who entered into orthopaedic surgery residency through the Match, Post-Match Supplemental Offer and Acceptance Program (SOAP), or some other agreement before or after the Match. The admission rate (percentage of applicants who were admitted in a residency program/total applicants) varied by race groups (Table 1).
Table 1. -
Applicants and accepted residents by race and gender
|Applied n = 6370
73% (4647 of 6370)
|Applied n = 1057
68% (724 of 1057)
|Applied n = 513
46% (238 of 513)
|Applied n = 231
48% (110 of 231)
|Applied n = 795
63% (499 of 795)
|Applied n = 8966
69% (6199 of 8966)
||73% (4002 of 5478)
||68% (617 of 906)
||50% (195 of 389)
||46% (94 of 203)
||64% (439 of 684)
||70% (5331 of 7669)
||72% (645 of 892)
||71% (107 of 151)
||37% (43 of 115)
||57% (16 of 28)
||54% (60 of 111)
||67% (868 of 1297)
We used logistic regression analyses to investigate the association between admission outcomes and applicant metrics. Independent categorical variables included Alpha Omega Alpha status, race, and gender. Independent continuous variables included USMLE Step 1 and Step 2 Clinical Knowledge scores, self-reported volunteer experience, research experience, work experience, and number of publications. Race and gender were categorized based on information provided by the applicants. Race was categorized as white, Asian, Black, Hispanic or Latino, or other, which consisted of applicants who self-identified as American Indian or Alaskan Native, Hawaiian or other Pacific Islander, other, or unknown. For the race analysis, the comparator group was white. For the gender analysis, the comparator group was men, and for the Alpha Omega Alpha status, the comparator group was students who were not selected for Alpha Omega Alpha. The outcome variable was whether or not the applicant was admitted into any US orthopaedic surgery residency program.
For our analysis, we performed hierarchical logistic regression analyses. The first model focused on academic metrics only and included the following variables: Alpha Omega Alpha status, Step 1 score, Step 2 Clinical Knowledge score, volunteer experience count, research experience count, publication count, and work experience count. The purpose of this academic metrics model is to examine the relative weight of these academic metrics to admission into an orthopaedic surgery residency. The second model added gender and race to the academic variables to determine whether the intrinsic factors of race and gender are independently associated with admissions and to investigate the potential of race and gender explaining any of the variance in admission above and beyond those included in the academic metrics model.
The Nagelkerke R2 was computed to compare the model’s prediction to the actual admission of each applicant of each model. A Nagelkerke R2 in the range of 25% represents good predictive fit based on statistical definitions . To determine how well the models simulated the actual admissions data, we computed the receiver operating characteristics (ROC) including the area under curve (AUC), which measures the model’s ability to simulate which applicants were admitted or not admitted, with an AUC equal to 1.0 representing a perfect simulation. In each model, the odds ratio and the corresponding confidence interval was computed to quantify the strength of the association between each variable and admission. Variables with p less than 0.05 were considered significant. To allow for relative comparison of the weight of factors (which are measured using different scales) in the model, we also calculated standardized β coefficients for continuous factors.
To further assess the association of race and gender, we used the results from the model with race and gender in addition to academic metrics to calculate the probability of admission for an average hypothetical applicant from differing race or gender groups. Admission probability was computed with input variables set to the mean values from all applicants (Step 1: 233, Step 2: 235, research experience: 3, publication count: 4, volunteer experience: 6, work experience: 3). Gender (men, women), race (white, Asian, Black, Hispanic, other), and Alpha Omega Alpha status (yes, no) were varied, and the resulting admission probability was computed for each combination.
Relative Weight of Academic Metrics on Residency Admission
Results from the model evaluating academic metrics only demonstrate that Alpha Omega Alpha status (OR 2.12 [95% CI 1.80 to 2.50]; p < 0.001) was associated with residency admission. Of the continuous variables evaluated, each unit increase in the USMLE Step 1 score (OR 1.04 [95% CI 1.03 to 1.04]; p < 0.001), the USMLE Step 2 Clinical Knowledge score (OR 1.01 [95% CI 1.01 to 1.02]; p < 0.001), publication count (OR 1.04 [95% CI 1.03 to 1.05]; p < 0.001), and volunteer experience (OR 1.03 [95% CI 1.01 to 1.04]; p < 0.001) were all associated with increased odds of orthopaedic residency admissions (Table 2). Standardized β demonstrate that Alpha Omega Alpha (β = 0.75) and Step1 Clinical Knowledge (β = 0.67) had comparable contributions to the model, while Step 2 Clinical Knowledge (β = 0.24), publication count (β = 0.25), and volunteer experience (β = 0.10) had smaller relative weight. Work experience (p = 0.17) and research experience (p = 0.12) were not associated with residency admissions. This model exhibited a good fit (Nagelkerke R2 = 24.5%) and exhibited a good estimate of admissions based on the ROC curve, with an AUC of 0.755 (Fig. 2A).
Table 2. -
Model results with only academic variables included
||OR (95% CI)
|Step 1 score
|Step 2 Clinical Knowledge score
For the dichotomous variables, the comparison group was white for race, men for gender, and No AOA for AOA status.
The Nagelkerke R2 of the model compares the model prediction to the actual admission of each applicant, with the range of 25% representing a good fit based on statistical definitions. The Nagelkerke R2 of this model = 24.5%; β = standardized beta coefficient (allows for relative comparison of each variable’s impact on the model’s outcome); AOA = Alpha Omega Alpha.
Association of Gender and Race with Admission into Residency
The second model with gender and race in addition to the academic variables demonstrated that Asian (OR 0.78 [95% CI 0.67 to 0.092]; p = 0.002), Black (OR 0.63 [95% CI 0.51 to 0.77]; p < 0.001), Hispanic (OR 0.48 [95% CI 0.36 to 0.65]; p < 0.001), or other race groups (OR 0.65 [95% CI 0.55 to 0.77]; p < 0.001) had lower odds of admission into orthopaedic surgery residency compared with white applicants (Table 3). Gender (p = 0.76) was not associated with admission into residency. The goodness of fit of this model (Nagelkerke R2 = 25.3%) was similar to that of the academics only model, which also exhibited a good estimate of admissions with an AUC of 0.759 (Fig. 2B). Alpha Omega Alpha status (OR 2.067 [95% CI 1.75 to 2.44]; p < 0.001) was associated with orthopaedic residency admission. For the continuous variables evaluated, Step 1 scores (OR 1.037 per unit increase [95% CI 1.03 to 1.04]; p < 0.001), Step 2 Clinical Knowledge scores (OR 1.011 per unit increase [95% CI 1.01 to 1.01]; p < 0.001), publication count (OR 1.04 per publication [95% CI 1.03 to 1.05]; p < 0.001), and volunteer experience count (OR 1.03 per experience [95% CI 1.01 to 1.04]; p < 0.001) continued to be associated with admissions into a residency program. Results also show that research experience (OR 1.04 per research experience [95% CI 1.00 to 1.07]; p < 0.03) but not work experience (p = 0.10) was associated with residency admission. Although the magnitude of standardized betas changed slightly in this model, the relative weights of the academic factors in this model was similar as in the first model where only academic variables were analyzed.
Table 3. -
Model with race and gender included with academic variables
||OR (95% CI)
|Step 1 score
|Step 2 CK score
For the dichotomous variables, the comparison group was white for race, men for gender, and No AOA for AOA status.
The Nagelkerke R2 of the model compares the model prediction to the actual admission of each applicant, with the range of 25% representing a good fit based on statistical definitions. The Nagelkerke R2 of this model = 25.3%; β = standardized beta coefficient (allows for relative comparison of each variable’s impact on the model’s outcome); AOA = Alpha Omega Alpha; CK = Clinical Knowledge.
Although diversity in medicine has been shown to be beneficial to patient care and the learning environment [35, 38, 40, 46], orthopaedic surgery continues to lag behind other specialties in terms of diversity . Academic metrics commonly used as selection criterion for the resident selection process exhibit differences based on race/ethnicity of the applicants, which may differentially affect underrepresented minorities and prolong the lack of diversity trends in orthopaedic surgery [5, 31, 43, 44]. This study sought to model the extent to which modifiable applicant academic metrics and intrinsic factors including race and gender are associated with admission into an orthopaedic residency program. We found that ethnicity/race, but not gender, was associated with admission into orthopaedic residency, even when accounting for academic metrics. Alpha Omega Alpha and Step 1 scores had the greatest relative weight on admissions, while the relative weight of Step 2 Clinical Knowledge scores, publication count, and volunteer experience was smaller. Adding gender and race did not substantially alter the fit quality of the model. Work experience was not associated with admission into an orthopaedic surgery residency.
This study has several limitations. To maintain the anonymity of the medical students, an individual’s medical school information was not included in the data set, limiting the ability to perform multilevel models of school ranking. This also limited our ability to track whether a student did not achieve Alpha Omega Alpha because their school does not have an Alpha Omega Alpha chapter. We postulate that this limitation may inflate the importance of the Alpha Omega Alpha results slightly, since only a small percentage of medical schools (∼15%) do not have an Alpha Omega Alpha chapter. Although excluding graduates from non-US medical schools may have reduced the overall number of racial/ethnic minorities in the applicant pool, their admission rate is highly skewed, with only 8% to 10% of non-US applicants having been admitted into an orthopaedic residency over the course of the 10-year study. Excluding non-US students from the data more closely reflects the admission profile of US orthopaedic residency programs and is not expected to have an appreciable effect on the conclusions of this study. Although the data used in this study is the largest and most comprehensive data set made available by the data generating organizations for orthopaedics to date, changes in the number of applicants over time for periods after the study timeframe (2005-2014) are unknown. However, there have been no important changes to the residency selection process over the span of the study period, suggesting that these findings likely are relevant to current and future residency admissions. Recently announced changes to the Step 1 exam, which is expected to become pass/fail after 2022, will change the criteria by which students can be evaluated for orthopaedic surgery resident admissions, making our analysis of USMLE Step 1 scores a historical one. Nevertheless, the factor was maintained in the model for completeness of the variables used in admission decisions for the time period of this study. Since prior studies have shown racial/ethnic differences exist in Step 1 scores for orthopaedic applicants and USMLE Step 1 in general [31, 37], the change in test score reporting may alter the current study’s association between race and admissions. Finally, this is an observational study and therefore, the model can only show associations and not causations of the variables studied with admission into orthopaedic surgery residency.
Relative Weight of Academic Metrics on Residency Admission
Students who earned Alpha Omega Alpha status had greater odds of admission into an orthopaedic residency, even when accounting for race, gender, and other academic metrics. Although racial disparities in the selection process for Alpha Omega Alpha have been described [5, 43], Alpha Omega Alpha membership is also associated with higher evaluation scores of residents by faculty and higher American Board of Orthopaedic Surgery pass rates [3, 17]. Step 1 score had a higher relative weight on residency admission compared with Step 2 Clinical Knowledge. Although standardized examinations are often used by residency directors as a consistent method to evaluate applicants for both interview selection as well as final ranking [13, 24, 42], Step 1 and Step 2 Clinical Knowledge were not primarily designed to predict success in residency [33, 34]. Indeed, there are unexplained demographic differences in USMLE scores based on gender and race [7, 37]. Prior studies have also shown that within the orthopaedic surgery applicant and admitted resident pool, Step 1 and Step 2 Clinical Knowledge scores vary more by race than by gender [30, 31]. The scores are commonly used nonetheless because they serve as convenient tools to stratify the large pool of applicants applying to the small number of available residency spots. Applicants are expected to participate in volunteer and research experiences. Both research and volunteer experience had comparable relative weights in the model. Publication count had a higher weight than research experience, which is to be expected given that coauthorship on publications is a key productivity metric of academic research.
Association of Gender and Race with Admission into Residency
We found an association of race but not gender on residency admissions, even when other metrics were held constant in the model. All race groups modeled in this study had lower odds of entering orthopaedic residency compared with white applicants. Recent data show that minority students applying to residency, including orthopaedic surgery, tend to have lower average Step 1 and Step 2 Clinical Knowledge scores than white applicants [7, 18, 31-33, 37]. However, we found that minority students have lower odds of admission into an orthopaedic surgery residency than white applicants, independent of application metrics. Our findings extend the understanding of the field beyond the fact that applications from underrepresented minority students exhibit lower metrics (such as USMLE Step 1 and Step 2 Clinical Knowledge scores or Alpha Omega Alpha) and thus end up with lower likelihoods of admission. Rather, the current study finds that applicants from minority races have lower odds of entering orthopaedic residency compared with white applicants of similar quantitative application metrics. Interestingly, gender was not associated with residency admissions, which is consistent with recent findings that women and men applicants to orthopaedic residency programs have similar academic metrics . Low representation of medical student applicants who are women in the orthopaedic surgery applicant pool likely is the major contributor to the underrepresentation of women in orthopaedic surgery residency programs [29, 30].
The finding that race/ethnicity but not gender is associated with an applicant’s odds of admission may point to the existence of structural inequalities in admissions practices. This observational analysis does not define causation and clearly the disparities are complex and multifactorial. Nevertheless, below, we provide suggestions for those interested in changing the training landscape, as not all programs have equal access to resources and recognizing that the suggested tools do not fully address the structural deficiencies/inequalities in the system. The tools are categorized into two themes: application process and environment.
It is well established that there are inherent biases in the medical student evaluation process [2, 5, 9, 16, 20, 37]. It is important for residency selection committee members to recognize and familiarize themselves with the biases in these evaluation tools and adjust their definition of academic achievement before evaluating potential applicants. In addition, implicit bias and social stereotypes have been shown to be barriers to inclusive recruitment and evaluation [9, 10, 28]. Bias training is necessary as an ongoing activity for selection committee members and current residents who interface with medical student candidates because there is some evidence that certain characteristics such as age and years of experience may be associated with interview scores . Careful review and redefining the makeup of a residency selection committee to improve racial and gender diversity may also impact the disparity in evaluation between racial groups [6, 41, 45]. In the absence of a diverse faculty, engaging residency alumni, faculty peers from other departments, or institutional diversity and equity program leaders may be necessary . Other application features such as the use of blinded applications, where by the means to identify the applicant such as photo and race are removed from the application material, may be helpful in eliminating implicit bias . Additional blinding of medical school name, scores on exams or clerkships, or gender/ethnicity-related text in the letters may be achieved through computerized algorithms to minimize aspects of unconscious bias during the interview selection process. To address the potential for overreliance on academic variables with known racial disparities [5, 37], removal of these metrics from an interviewer’s candidate profile before the interview may provide a clean slate for candidates when evaluated postinterview [21, 23, 44]. Additionally, a holistic review of applications including eliminating the use of cutoffs for stratifying applications will provide a more personalized process and an opportunity to emphasize characteristics important to the department, such as teamwork, leadership, or research activity [1, 15].
Cultivating an environment of inclusion is important for recruitment and support of underrepresented minorities. Recruitment and retention of underrepresented minorities at all levels of the academic institution, not just at the residency level, will strengthen the peer support and mentorship network for potential students from underrepresented groups [26, 27]. Orthopaedic surgery leaders in certain universities are also well positioned to team up with peers in other specialties (including general surgery or emergency medicine) who have published plans for fostering and addressing racial bias in their departments and their plan to evaluate the success of the programs implemented [9, 10]. In addition to an inclusive mission statement, the participation by diverse individuals among institutional leadership and featured images and stories on public-facing materials (such as a website) may positively contribute to the learning environment and recruitment of minorities [9, 14, 22]. Other additional programming to enhance sponsorship and mentorship of minority medical students may also increase interest and attract more applicants, and ultimately improve admissions to orthopaedic surgery. In closing, this study is not meant to discourage underrepresented minorities from applying to orthopaedics, rather is a call to action for the orthopaedic residency leadership to strengthen ongoing effort into recruiting and retaining qualified students.
We thank the Association of American Medical Colleges and National Board of Medical Examiners for providing the dataset used in this study.
1. Aibana O, Swails JL, Flores RJ, Love L. Bridging the gap: holistic review to increase diversity in graduate medical education. Acad Med. 2019;94:1137-1141.
2. Berg K, Blatt B, Lopreiato J, et al. Standardized patient assessment of medical student empathy: ethnicity and gender effects in a multi-institutional study. Acad Med.
3. Bernstein AD, Jazrawi LM, Elbeshbeshy B, Della Valle CJ, Zuckerman JD. Orthopaedic resident-selection criteria. J Bone Joint Surg Am. 2002;84:2090-2096.
4. Blakemore LC, Hall JM, Biermann JS. Women in surgical residency training programs. J Bone Joint Surg Am. 2003;85:2477-2480.
5. Boatright D, Ross D, O'Connor P, Moore E, Nunez-Smith M. Racial disparities in medical student membership in the Alpha Omega Alpha Honor Society. JAMA Intern Med. 2017;177:659-665.
6. Capers Q IV, Clinchot D, McDougle L, Greenwald AG. Implicit racial bias in medical school admissions. Acad Med. 2017;92:365-369.
7. Dawson B, Iwamoto CK, Ross LP, Nungester RJ, Swanson DB, Volle RL. Performance on the National Board of Medical Examiners. Part I examination by men and women of different race and ethnicity. JAMA. 1994;272:674-679.
8. Day CS, Lage DE, Ahn CS. Diversity based on race, ethnicity, and sex between academic orthopaedic surgery and other specialties: a comparative study. J Bone Joint Surg Am. 2010;92:2328-2335.
9. Diaz T, Navarro JR, Chen EH. An institutional approach to fostering inclusion and addressing racial bias: implications for diversity in academic medicine. Teach Learn Med. 2020;32:110-116.
10. Dossett LA, Mulholland MW, Newman EA. Building high-performing teams in academic surgery: the opportunities and challenges of inclusive recruitment strategies. Acad Med. 2019;94:1142-1145.
11. Edmond MB, Deschenes JL, Eckler M, Wenzel RP. Racial bias in using USMLE step 1 scores to grant internal medicine residency interviews. Acad Med. 2001;76:1253-1256.
12. Egol KA, Collins J, Zuckerman JD. Success in orthopaedic training: resident selection and predictors of quality performance. J Am Acad Orthop Surg. 2011;19:72-80.
13. Green M, Jones P, Thomas JX Jr. Selection criteria for residency: results of a national program directors survey. Acad Med. 2009;84:362-367.
14. Hadinger MA. Underrepresented minorities in medical school admissions: a qualitative study. Teach Learn Med. 2017;29:31-41.
15. Harrison LE. Using holistic review to form a diverse interview pool for selection to medical school. Proc (Bayl Univ Med Cent). 2019;32:218-221.
16. Hauer KE, Boscardin C, Gesundheit N, Nevins A, Srinivasan M, Fernandez A. Impact of student ethnicity and patient-centredness on communication skills performance. Med Educ. 2010;44:653-661.
17. Herndon JH, Allan BJ, Dyer G, Jawa A, Zurakowski D. Predictors of success on the American Board of Orthopaedic Surgery examination. Clin Orthop Relat Res. 2009;467:2436-2445.
18. Karnes JM, Mayerson JL, Scharschmidt TJ. Is orthopedics more competitive today than when my attending matched? An analysis of National Resident Matching Program data for orthopedic PGY1 applicants from 1984 to 2011. J Surg Educ. 2014;71:530-542.
19. Lausen B, Krolak-Schwerdt S, Böhmer M. Data Science, Learning by Latent Structures and Knowledge Discovery. Springer-Verlag Berlin; 2015.
20. Lee KB, Vaishnavi SN, Lau SKM, Andriole DA, Jeffe DB. Cultural competency in medical education: demographic differences associated with medical student communication styles and clinical clerkship feedback. J Natl Med Assoc. 2009;101:116-126.
21. McDonald TC, Drake LC, Replogle WH, Graves ML, Brooks JT. Barriers to increasing diversity in orthopaedics: the residency program perspective. JB JS Open Access. 2020;5:e0007-e0007.
22. Metz AM. Racial and ethnic underrepresentation in medicine: lessons from the past and a vision of the future. Teach Learn Med. 2013;25:S33-S38.
23. Miles WS, Shaw V, Risucci D. The role of blinded interviews in the assessment of surgical residency candidates. Am J Surg. 2001;182:143-146.
24. National Resident Match Program. Results of the 2016 NRMP Program Director Survey. National Resident Match Program. Available at: http://www.nrmp.org/wp-content/uploads/2016/09/NRMP-2016-Program-Director-Survey.pdf
. Accessed April 22, 2019.
25. National Resident Match Program. Charting Outcomes in the Match for U.S. Allopathic Seniors 2016. National Resident Matching Program. Available at: http://www.nrmp.org/wp-content/uploads/2016/09/Charting-Outcomes-US-Allopathic-Seniors-2016.pdf
. Accessed May 10, 2019.
26. Odom KL, Roberts LM, Johnson RL, Cooper LA. Exploring obstacles to and opportunities for professional success among ethnic minority medical students. Acad Med. 2007;82:146-153.
27. Orom H, Semalulu T, Underwood W 3rd. The social and learning environments experienced by underrepresented minority medical students: a narrative review. Acad Med. 2013;88:1765-1777.
28. Oyler J, Thompson K, Arora VM, Krishnan JA, Woodruff J. Faculty characteristics affect interview scores during residency recruitment. Am J Med. 2015;128:545-550.
29. Poon S, Kiridly D, Mutawakkil M, et al. Current trends in sex, race, and ethnic diversity in orthopaedic surgery sesidency. J Am Acad Orthop Surg. 2019;27:e725-e733.
30. Poon S, Nellans K, Crabb RAL, et al. Academic metrics do not explain the underrepresentation of women in orthopaedic training programs. J Bone Joint Surg Am. 2019;101:e32.
31. Poon S, Nellans K, Rothman A, et al. Underrepresented minority applicants are competitive for orthopaedic surgery residency programs, but enter residency at lower rates. J Am Acad Orthop Surg. 2019;27:e957-e968.
32. Porter SE, Graves M. Resident selection beyond the United States Medical Licensing Examination. J Am Acad Orthop Surg. 2017;25:411-415.
33. Prober CG, Kolars JC, First LR, Melnick DE. A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection. Acad Med. 2016;91:12-15.
34. Raman T, Alrabaa RG, Sood A, Maloof P, Benevenia J, Berberian W. Does residency selection criteria predict performance in orthopaedic surgery residency? Clin Orthop Relat Res. 2016;474:908-914.
35. Ramirez RN, Franklin CC. Racial diversity in orthopedic surgery. Orthop Clin North Am. 2019;50:337-344.
36. Reid IN. Gender-correlated systematics in HST proposal selection. Publications of the Astronomical Society of the Pacific. 2014;126:923-934.
37. Rubright JD, Jodoin M, Barone MA. Examining demographics, prior academic performance, and United States Medical Licensing Examination Scores. Acad Med. 2019;94:364-370.
38. Saha S, Guiton G, Wimmers PF, Wilkerson L. Student body racial and ethnic composition and diversity-related outcomes in US medical schools. JAMA. 2008;300:1135-1145.
39. Schrock JB, Kraeutler MJ, Dayton MR, McCarty EC. A comparison of matched and unmatched orthopaedic surgery residency applicants from 2006 to 2014: data from the National Resident Matching Program. J Bone Joint Surg Am. 2017;99:e1.
40. Shen MJ, Peterson EB, Costas-Muñiz R, et al. The effects of race and racial concordance on patient-physician communication: a systematic review of the literature. J Racial Ethn Health Disparities. 2018;5:117-140.
41. Sherman MD, Ricco J, Nelson SC, Nezhad SJ, Prasad S. Implicit bias training in a residency program: aiming for enduring effects. Fam Med. 2019;51:677-681.
42. Wagoner NE, Suriano JR, Stoner JA. Factors used by program directors to select residents. J Med Educ. 1986;61:10-21.
43. Wijesekera TP, Kim M, Moore EZ, Sorenson O, Ross DA. All other things being equal: exploring racial and gender disparities in medical school honor society induction. Acad Med. 2019;94:562-569.
44. Williams M, Kim EJ, Pappas K, et al. The impact of United States Medical Licensing Exam (USMLE) step 1 cutoff scores on recruitment of underrepresented minorities in medicine: a retrospective cross-sectional study. Health Sci Rep. 2020;3:e2161.
45. Zeidan AJ, Khatri UG, Aysola J, et al. Implicit bias education and emergency medicine training: step one? Awareness. AEM Educ Train
46. Zuckerman JD. What's important: diversity in orthopaedic surgery. J Bone Joint Surg Am. 2018;100:1351-1352