How Medical School Applicant Race, Ethnicity, and Socioeconomic Status Relate to Multiple Mini-Interview–Based Admissions Outcomes

Findings From One Medical School

Abstract

Purpose 

To examine associations of medical school applicant underrepresented minority (URM) status and socioeconomic status (SES) with Multiple Mini-Interview (MMI) invitation and performance and acceptance recommendation.

Method 

The authors conducted a correlational study of applicants submitting secondary applications to the University of California, Davis, School of Medicine, 2011–2013. URM applicants were black, Southeast Asian, Native American, Pacific Islander, and/or Hispanic. SES from eight application variables was modeled (0–1 score, higher score = lower SES). Regression analyses examined associations of URM status and SES with MMI invitation (yes/no), MMI score (mean of 10 station ratings, range 0–3), and admission committee recommendation (accept versus not), adjusting for age, sex, and academic performance.

Results 

Of 7,964 secondary-application applicants, 19.7% were URM and 15.1% self-designated disadvantaged; 1,420 (17.8%) participated in the MMI and were evaluated for acceptance. URM status was not associated with MMI invitation (OR 1.14; 95% CI 0.98 to 1.33), MMI score (0.00-point difference, CI −0.08 to 0.08), or acceptance recommendation (OR 1.08; CI 0.69 to 1.68). Lower SES applicants were more likely to be invited to an MMI (OR 5.95; CI 4.76 to 7.44) and recommended for acceptance (OR 3.28; CI 1.79 to 6.00), but had lower MMI scores (−0.12 points, CI −0.23 to −0.01).

Conclusions 

MMI-based admissions did not disfavor URM applicants. Lower SES applicants had lower MMI scores but were more likely to be invited to an MMI and recommended for acceptance. Multischool collaborations should examine how MMI-based admissions affect URM and lower SES applicants.

The Multiple Mini-Interview (MMI) is replacing traditional interviews at medical schools.^1–5 In the MMI, trained raters evaluate applicants in a series of brief, timed, structured stations, and station ratings are pooled to yield a summary score. Stations are designed to assess skills that are difficult to ascertain from medical school applications, such as interpersonal communication, teamwork, ability to handle stress, problem solving, and integrity/ethics. The MMI is well accepted by applicants, reasonably reliable, and predictive of medical school and subsequent performance.^1–3

Little studied is how underrepresented racial/ethnic minority (URM) and lower socioeconomic status (SES) applicants may be affected by adoption of the MMI. This is a key issue given that U.S. medical schools admit disproportionately few URM and lower SES individuals.^6–8 URM admissions declined at many schools in the wake of legislation restricting consideration of race and ethnicity in acceptance decisions,^9,10 a trend that may accelerate following a Supreme Court decision upholding the constitutionality of such restrictions.¹¹ Concurrently, medical education and health policy groups call for a more diverse physician workforce.^12–16 In such a climate, it is necessary to evaluate how evolving admissions trends like MMI adoption may influence the diversity of medical school classes.

A long-recognized problem with traditional nonstructured interviews is vulnerability to interviewer biases triggered by various applicant characteristics.^17–22 Implicit (i.e., unconscious) biases disfavoring racial/ethnic minority and lower SES persons are common in U.S. society,²³ including among physicians.²⁴ The effects of bias during interviews can be reduced by increasing structure (removing ambiguity and, therefore, the tendency to rely on stereotype-driven judgments) and pooling evaluations from multiple raters (potentially diluting or offsetting individual biases).^20,25–27 In being structured and incorporating multiple raters’ perspectives, the MMI may thus be less susceptible to implicit bias effects than traditional medical school interviews.

Only three studies to our knowledge have explored the associations of medical school applicants’ racial/ethnic minority status or SES with MMI performance. In one study, involving six Canadian schools, aboriginal status was negatively correlated with MMI scores, whereas family income level was not significantly associated with MMI performance.²⁸ The analyses included few aboriginal participants (< 3%) and did not explore other race/ethnicity categories. Further, a robust indicator of SES would consider factors beyond income, such as parental education.²⁹ A single-school U.S. study using a limited dichotomous (yes/no) single-item self-report indicator of disadvantaged status found no evidence of an association between disadvantaged status and MMI performance.³⁰ Similarly, a single-school United Kingdom study, using a geographic area-based measure of deprivation, found no association between applicants’ deprivation score and MMI performance.³¹ However, ecological measures such as geographic area deprivation scores have significant limitations.³²

To our knowledge, no studies have examined whether applicants’ race/ethnicity influences acceptance following MMI participation, or whether race/ethnicity or SES influences the likelihood of being invited to an MMI. Such outcomes are likely to be more strongly influenced by parochial concerns (e.g., institutional mission focus) than the MMI process,^12,33 which is relatively similar across schools.^34,35 Nonetheless, it is important to consider MMI invitation and acceptance decisions to provide context for evaluating the MMI-based admissions process.

We examined the associations of applicants’ URM status and SES with MMI invitation, MMI performance, and post-MMI acceptance recommendation among applicants to the University of California, Davis (UCD), School of Medicine (SOM) in Sacramento, California, over three admission cycles (2011–2013), adjusting for other applicant demographic characteristics and postsecondary academic performance.

Method

We employed data collected as part of the routine admissions processes during the 2011, 2012, and 2013 application cycles. The admissions office provided relevant application data in an electronic spreadsheet with personal identifiers removed. The study was conducted from April 18, 2014, through the end of August 2014. The UCD institutional review board reviewed the protocol and determined it was exempt.

Application, screening, and MMI invitation and scheduling

Applicants initially applied to the UCD SOM via the American Medical College Application Service (AMCAS). Following initial screening based on cumulative grade point average (GPA) and Medical College Admission Test (MCAT) scores, admissions committee members reviewed all applications and invited a subset of applicants to submit a secondary application. Faculty evaluated secondary applications for invitation to an MMI based on cumulative GPA and MCAT scores, personal statements, extracurricular activities, recommendation letters, and other characteristics that could contribute to fulfilling the educational and service missions of the school. Invited applicants self-scheduled their MMI sessions via an online portal.

MMI process and scoring

The MMI consisted of 10 individual 10-minute stations. At each station, applicants had 2 minutes to read a brief set of instructions, and 8 minutes to address the assigned tasks on entering the room. Nine stations assessed skills in the following domains: integrity/ethics, professionalism, interpersonal communication, diversity/cultural awareness, teamwork, ability to handle stress, and problem solving. An additional station asked applicants to explain their choice to pursue a career in medicine. Most stations were adapted from content developed at McMaster University and marketed by ProFitHR.³⁴

A single trained rater, blinded to participants’ AMCAS application information, attended each station. In some stations, raters interacted directly with applicants. At others, raters observed applicant interactions with actors or other applicants. There were 216 different raters during the study period; the mean number of MMI stations that each evaluated was 104 (standard deviation [SD] 61.9; range 8–276). Women made up 61% of raters. Rater professional backgrounds were as follows: physicians, 31%; medical students, 15%; other clinicians (e.g., nurses), 11%; basic science faculty, 6%; patients, 2%; and various nonclinician leaders (e.g., deans), professionals (e.g., lawyers), and high-level administrative staff (e.g., curriculum manager), 35%. The range of rater backgrounds reflected the conviction that diverse perspectives are helpful in selecting future physicians who will be able to work effectively with people from all walks of life. Mandatory rater training included a one-hour course reviewing the admissions process, rater roles and duties, and the need to avoid pursuing protected class issues (e.g., race/ethnicity, gender).³⁶

At each station, raters scored overall applicant performance using an anchored four-point scale: 0, < 25th percentile performance (relative to other applicants); 1, 25th–50th percentile; 2, 51st–75th percentile; or 3, > 75th percentile. Raters were instructed to consider both the applicant’s communication abilities and the content (e.g., comprehensiveness) of their statements in assigning ratings. The total MMI score was the mean of each applicant’s individual station scores. Scale internal consistency (Cronbach alpha = 0.67) was comparable to that observed in other MMI studies.^2,18,37–41

Acceptance recommendation

The admissions committee met weekly during the admission cycle to review each participant’s MMI performance, AMCAS application, and secondary application. Subsequently, the committee made one of the following recommendations: reject, low waitlist, high waitlist, or offer acceptance. For the current analyses, we dichotomized the recommendation (offer acceptance versus not).

URM status

We determined URM status (URM [black, Southeast Asian, Native American, or Pacific Islander race and/or Hispanic ethnicity] versus not [all other responses]) from self-reported race/ethnicity information in the AMCAS application. These groups remain underrepresented in the medical profession relative to the general population.⁴²

Socioeconomic disadvantage

We developed a composite measure of SES using self-reported information in the AMCAS application, screening candidate indicators for inclusion using logistic regression analyses and based on their contribution to predicting applicants’ self-designated disadvantaged status. The following predictors (yes/no items except where indicated) were significant and maximized the area under the receiver operating characteristic curve (0.95): fee assistance received for medical school application (yes/no); childhood spent in an underserved area; family recipients of family assistance program; income level category of applicant’s family (< $25,000; $25,000 to < $50,000; $50,000 to < $75,000; or > $75,000); applicant contributed to family income; any financial-need-based scholarship(s) in paying for postsecondary education; percentage of postsecondary education costs contributed by the family; and parents’ highest level of educational attainment (< high school, high school graduate, some college, or college graduate). The model yielded a predicted probability of being self-designated disadvantaged, ranging continuously from 0 to 1.0 (higher predicted probability = lower SES). We employed the score, which correlated 0.91 with a factor-analytic-derived score, as the study measure of applicant SES. The SES score was preferable to alternatives²⁹ because the continuous scale acknowledges that socioeconomic disadvantage is not a binary characteristic and reduces misclassification (false positives and negatives).

Other characteristics

The admissions office also provided information from AMCAS regarding applicant age, sex, cumulative postsecondary GPA, and total MCAT score.

Data analysis

We analyzed the data using Stata version 13.1 (Stata Corporation Inc., College Station, Texas). We modeled MMI invitation (yes/no) and medical school acceptance recommendation (accept versus not) using logistic regression (a separate regression for each dependent variable). We modeled total MMI score using linear regression. All models included the following characteristics: age category (< 22 [reference], 22, 23, or > 24 years); female gender (yes/no); URM status (versus not); cumulative GPA category (< 3.4, 3.4–3.6, > 3.6–3.8, or > 3.8 [reference]); total MCAT score category (19–26, 27–30, 31–32, 33–34, or > 34 [reference]); SES (0–1.0 continuous score), and application year (2011, 2012, or 2013). The acceptance recommendation model additionally included the total MMI score (0–3).

Results

During the three study application cycles, 15,844 people applied, and 8,933 (56.4%) were invited to submit secondary applications. Of the invitees, 7,964 (89.2%) submitted secondary applications, and 1,575 (19.8%) were invited to an MMI. Of the MMI invitees, 1,420 (90.2%) attended an MMI.

Applicant characteristics

Table 1 summarizes the characteristics of screened applicants. Compared with those not invited to an MMI, those who participated in an MMI were older and more likely to be female, from an URM group, disadvantaged (based on both self-designation and the continuous SES score), and had higher GPAs and MCAT scores.

Table 1: Characteristics of 7,964 Applicants to the University of California, Davis, School of Medicine Submitting Secondary Applications, by MMI Participation Status, From a Study of Applicant Race and Ethnicity, Socioeconomic Status, and MMI-Based Admissions Outcomes, 2011–2013

MMI invitation

Although adjusted URM status was not associated with MMI invitation, lower SES was associated with receiving an MMI invitation, as were older age, female gender, and higher GPA and MCAT score (Table 2).

Table 2: Results of a Logistic Regression Model Examining Associations of Characteristics of 7,964 Applicants to the University of California, Davis, School of Medicine Submitting a Secondary Application With MMI Invitation, From a Study of Applicant Race and Ethnicity, Socioeconomic Status, and MMI-Based Admissions Outcomes, 2011–2013

MMI score

Mean MMI score was 1.30 (SD 0.62). In adjusted analyses, URM status was not associated with MMI score, but lower SES was associated with lower MMI scores (Table 3). Older age, female gender, and lower GPA were also associated with MMI score (Table 3).

Table 3: Results of a Linear Regression Model Examining Associations of Characteristics of 1,420 Applicants to the University of California, Davis, School of Medicine Completing an MMI With Total MMI Score, From a Study of Applicant Race and Ethnicity, Socioeconomic Status, and MMI-Based Admissions Outcomes, 2011–2013

Acceptance recommendation

Of the 1,420 MMI participants, 334 (23.5%) were recommended for acceptance. Lower SES was associated with being recommended for acceptance, whereas URM status was not (Table 4). Of other factors examined, mean MMI score was the most strongly associated with being recommended for acceptance. Older age, female gender, and higher GPA and MCAT scores were also associated with being recommended for acceptance (Table 4).

Table 4: Results of a Logistic Regression Model Examining Associations of Characteristics of 1,420 Applicants to the University of California, Davis, School of Medicine Completing an MMI With Admissions Committee Recommendation to Accept, From a Study of Applicant Race and Ethnicity, Socioeconomic Status, and MMI-Based Admissions Outcomes, 2011–2013

Discussion

In a diverse sample of applicants over three admission cycles at UCD, superior MMI performance was strongly linked with being recommended for acceptance. Further, URM applicants were no less likely than non-URM applicants to receive an MMI invitation, performed similarly on the MMI, and were just as likely to be recommended for acceptance. In the only prior study to exploring this issue, MMI performance at six Canadian medical schools was worse for the < 3% of applicants with self-reported aboriginal status, whereas income was not significantly associated with MMI scores.²⁸ Our findings provide some reassurance that adoption of the MMI-based admissions process at U.S. medical schools need not adversely affect admission prospects for URM applicants.

The similar MMI scores for URM and non-URM participants support the notion that structured interview processes that incorporate the perspectives of multiple evaluators like the MMI may be less vulnerable to the effects of individual evaluator implicit biases.^20,25–27 Although we did not measure rater implicit biases regarding racial/ethnic minorities, such biases have been documented to be pervasive in U.S. society, including among physicians and other professionals,^23,24 and can affect the outcomes of employment interviews in various fields including medicine.^17,19–22 Thus, it is likely that implicit biases were present among our raters; however, they did not exert a significant net influence, given that mean MMI scores did not differ between URM and non-URM applicants. Because lack of URMs in medicine is a widely acknowledged problem,^{6,7,13,33,42–44} it is possible that biases against URM applicants were offset by ratings biased in favor of URM applicants, made by raters seeking to address limited racial/ethnic diversity in the physician workforce.

Our findings regarding MMI performance may have the broadest applicability, given the relatively high standardization of the MMI across institutions.^34,35 By contrast, MMI invitation and acceptance decisions are shaped more by parochial concerns such as institutional mission and local workforce needs.^12,33 In this context, our finding that lower SES applicants had worse adjusted MMI performance may be cause for concern. Although three prior studies reported no association of applicant SES with MMI performance, all relied on less robust measures of SES. Nonetheless, the decrement in MMI performance with decreasing SES in our study was small: The MMI score (scale of 0–3 points) declined by a mean of 0.12 points across the 0–1 range of the SES score. Further, the lower MMI scores among lower SES applicants were more than offset by their greater likelihood of being invited to an MMI and recommended for acceptance. These findings may reflect the ongoing shift from a purely metric-based applicant review process toward the more holistic process advocated by the Association of American Medical Colleges.^12,15

Although the reasons for the lower MMI performance among lower SES applicants are unclear, poorer postsecondary academic preparation and performance are unlikely explanations because we adjusted for GPA and MCAT score. Lower SES applicants may have fewer life experiences bolstering skills assessed by the MMI. Similar reasoning has been suggested to explain the lower MCAT scores among such applicants.⁴⁵ Although less affluent applicants are more likely to report paid employment during postsecondary education, their financial circumstances may require taking jobs that do not require MMI-type preemployment screening. Lack of prior experience with MMI-type screening may be a disadvantage in the medical school MMI because prior experience with a particular interview format is associated with better future performance with that format.⁴⁶ Lower-level jobs also may not facilitate the higher-level communication, critical thinking, and problem-solving skills the MMI assesses, and the time required for such jobs may limit participation in pursuits that build such skills (e.g., scholarly presentations, volunteer clinic work).

Rater implicit bias could also help to explain the lower MMI scores among lower SES applicants. Raters were not provided any information about participants, and only one MMI station afforded applicants the opportunity to describe their backgrounds. Nonetheless, subtle information apparent to raters could have led to implicitly biased ratings, possibly triggered by the generally relatively large social distance between lower SES applicants and the typical rater.^47,48 Most of our MMI raters were well educated and relatively affluent. Prior work indicates that applicant factors such as use of language unfamiliar to the typical rater could trigger a biased low rating.^20,21 Applicants’ verbal skills have been shown to determine immediate interviewer impressions and, in turn, final appraisals.⁴⁹ The issue of SES-based physician workforce disparities has received less attention than race/ethnicity-based disparities.⁶ Thus, it is less likely that raters consciously biased their evaluations in favor of lower SES applicants to address SES-based physician workforce disparities. If our findings are replicated, it would suggest the need to consider rater training to minimize the influence of SES-based biases.

We found that older applicants and women performed better than their younger and male counterparts both in the MMI (consistent with prior studies^28,37) and throughout the admissions process. Older applicants are more likely to have had life experiences requiring effective communication. Women, more than men, have been noted to communicate in ways that quickly build rapport in novel social situations such as the MMI. Previously underrepresented in medicine, women now constitute well over half of all U.S. medical students.⁵⁰ Given the increasing adoption of the MMI and strong influence of MMI performance on medical school acceptance, our findings suggest the potential for a disparity to develop disfavoring male applicants, and warrant monitoring.

Limitations

Our study was correlational, limiting causal inferences. Our data were derived from applicants at a single school. How the findings generalize to other schools is unclear. We lacked sociodemographic information regarding MMI raters and admissions committee personnel, precluding examination of how such characteristics may have influenced the study outcomes. We employed a novel composite measure of SES, albeit one with theoretical advantages over other indicators.²⁹ Because schools vary widely in institutional mission,³³ applicant pools, admissions personnel, and other attributes, multi-institution studies are needed to better gauge the impact of MMI-based admissions processes on URM and socioeconomically disadvantaged applicants. Studies ideally should account for the characteristics of admissions personnel and applicants, and examine the relative utility of different SES indicators. It is also unknown whether URM or lower SES applicants would have fared differently in one-on-one interview-based admissions processes. Ideally, randomized trials would compare traditional interview processes with MMI-based processes, examining their impact on the prospects of URM and lower SES applicants.

Conclusion

In conclusion, in analyses of data from one California medical school, an MMI-based admissions process did not disfavor racial/ethnic minority groups underrepresented in the physician workforce. Applicants from lower SES backgrounds, also underrepresented in medicine, had lower MMI scores but were more likely to receive an MMI invitation and be recommended for acceptance. Multischool collaborations are needed to further evaluate the impact of MMI-based medical school admissions processes on URM and lower SES applicants.