The Multiple Mini-Interview (MMI) is replacing traditional interviews at medical schools.1–5 In the MMI, trained raters evaluate applicants in a series of brief, timed, structured stations, and station ratings are pooled to yield a summary score. Stations are designed to assess skills that are difficult to ascertain from medical school applications, such as interpersonal communication, teamwork, ability to handle stress, problem solving, and integrity/ethics. The MMI is well accepted by applicants, reasonably reliable, and predictive of medical school and subsequent performance.1–3
Little studied is how underrepresented racial/ethnic minority (URM) and lower socioeconomic status (SES) applicants may be affected by adoption of the MMI. This is a key issue given that U.S. medical schools admit disproportionately few URM and lower SES individuals.6–8 URM admissions declined at many schools in the wake of legislation restricting consideration of race and ethnicity in acceptance decisions,9,10 a trend that may accelerate following a Supreme Court decision upholding the constitutionality of such restrictions.11 Concurrently, medical education and health policy groups call for a more diverse physician workforce.12–16 In such a climate, it is necessary to evaluate how evolving admissions trends like MMI adoption may influence the diversity of medical school classes.
A long-recognized problem with traditional nonstructured interviews is vulnerability to interviewer biases triggered by various applicant characteristics.17–22 Implicit (i.e., unconscious) biases disfavoring racial/ethnic minority and lower SES persons are common in U.S. society,23 including among physicians.24 The effects of bias during interviews can be reduced by increasing structure (removing ambiguity and, therefore, the tendency to rely on stereotype-driven judgments) and pooling evaluations from multiple raters (potentially diluting or offsetting individual biases).20,25–27 In being structured and incorporating multiple raters’ perspectives, the MMI may thus be less susceptible to implicit bias effects than traditional medical school interviews.
Only three studies to our knowledge have explored the associations of medical school applicants’ racial/ethnic minority status or SES with MMI performance. In one study, involving six Canadian schools, aboriginal status was negatively correlated with MMI scores, whereas family income level was not significantly associated with MMI performance.28 The analyses included few aboriginal participants (< 3%) and did not explore other race/ethnicity categories. Further, a robust indicator of SES would consider factors beyond income, such as parental education.29 A single-school U.S. study using a limited dichotomous (yes/no) single-item self-report indicator of disadvantaged status found no evidence of an association between disadvantaged status and MMI performance.30 Similarly, a single-school United Kingdom study, using a geographic area-based measure of deprivation, found no association between applicants’ deprivation score and MMI performance.31 However, ecological measures such as geographic area deprivation scores have significant limitations.32
To our knowledge, no studies have examined whether applicants’ race/ethnicity influences acceptance following MMI participation, or whether race/ethnicity or SES influences the likelihood of being invited to an MMI. Such outcomes are likely to be more strongly influenced by parochial concerns (e.g., institutional mission focus) than the MMI process,12,33 which is relatively similar across schools.34,35 Nonetheless, it is important to consider MMI invitation and acceptance decisions to provide context for evaluating the MMI-based admissions process.
We examined the associations of applicants’ URM status and SES with MMI invitation, MMI performance, and post-MMI acceptance recommendation among applicants to the University of California, Davis (UCD), School of Medicine (SOM) in Sacramento, California, over three admission cycles (2011–2013), adjusting for other applicant demographic characteristics and postsecondary academic performance.
We employed data collected as part of the routine admissions processes during the 2011, 2012, and 2013 application cycles. The admissions office provided relevant application data in an electronic spreadsheet with personal identifiers removed. The study was conducted from April 18, 2014, through the end of August 2014. The UCD institutional review board reviewed the protocol and determined it was exempt.
Application, screening, and MMI invitation and scheduling
Applicants initially applied to the UCD SOM via the American Medical College Application Service (AMCAS). Following initial screening based on cumulative grade point average (GPA) and Medical College Admission Test (MCAT) scores, admissions committee members reviewed all applications and invited a subset of applicants to submit a secondary application. Faculty evaluated secondary applications for invitation to an MMI based on cumulative GPA and MCAT scores, personal statements, extracurricular activities, recommendation letters, and other characteristics that could contribute to fulfilling the educational and service missions of the school. Invited applicants self-scheduled their MMI sessions via an online portal.
MMI process and scoring
The MMI consisted of 10 individual 10-minute stations. At each station, applicants had 2 minutes to read a brief set of instructions, and 8 minutes to address the assigned tasks on entering the room. Nine stations assessed skills in the following domains: integrity/ethics, professionalism, interpersonal communication, diversity/cultural awareness, teamwork, ability to handle stress, and problem solving. An additional station asked applicants to explain their choice to pursue a career in medicine. Most stations were adapted from content developed at McMaster University and marketed by ProFitHR.34
A single trained rater, blinded to participants’ AMCAS application information, attended each station. In some stations, raters interacted directly with applicants. At others, raters observed applicant interactions with actors or other applicants. There were 216 different raters during the study period; the mean number of MMI stations that each evaluated was 104 (standard deviation [SD] 61.9; range 8–276). Women made up 61% of raters. Rater professional backgrounds were as follows: physicians, 31%; medical students, 15%; other clinicians (e.g., nurses), 11%; basic science faculty, 6%; patients, 2%; and various nonclinician leaders (e.g., deans), professionals (e.g., lawyers), and high-level administrative staff (e.g., curriculum manager), 35%. The range of rater backgrounds reflected the conviction that diverse perspectives are helpful in selecting future physicians who will be able to work effectively with people from all walks of life. Mandatory rater training included a one-hour course reviewing the admissions process, rater roles and duties, and the need to avoid pursuing protected class issues (e.g., race/ethnicity, gender).36
At each station, raters scored overall applicant performance using an anchored four-point scale: 0, < 25th percentile performance (relative to other applicants); 1, 25th–50th percentile; 2, 51st–75th percentile; or 3, > 75th percentile. Raters were instructed to consider both the applicant’s communication abilities and the content (e.g., comprehensiveness) of their statements in assigning ratings. The total MMI score was the mean of each applicant’s individual station scores. Scale internal consistency (Cronbach alpha = 0.67) was comparable to that observed in other MMI studies.2,18,37–41
The admissions committee met weekly during the admission cycle to review each participant’s MMI performance, AMCAS application, and secondary application. Subsequently, the committee made one of the following recommendations: reject, low waitlist, high waitlist, or offer acceptance. For the current analyses, we dichotomized the recommendation (offer acceptance versus not).
We determined URM status (URM [black, Southeast Asian, Native American, or Pacific Islander race and/or Hispanic ethnicity] versus not [all other responses]) from self-reported race/ethnicity information in the AMCAS application. These groups remain underrepresented in the medical profession relative to the general population.42
We developed a composite measure of SES using self-reported information in the AMCAS application, screening candidate indicators for inclusion using logistic regression analyses and based on their contribution to predicting applicants’ self-designated disadvantaged status. The following predictors (yes/no items except where indicated) were significant and maximized the area under the receiver operating characteristic curve (0.95): fee assistance received for medical school application (yes/no); childhood spent in an underserved area; family recipients of family assistance program; income level category of applicant’s family (< $25,000; $25,000 to < $50,000; $50,000 to < $75,000; or > $75,000); applicant contributed to family income; any financial-need-based scholarship(s) in paying for postsecondary education; percentage of postsecondary education costs contributed by the family; and parents’ highest level of educational attainment (< high school, high school graduate, some college, or college graduate). The model yielded a predicted probability of being self-designated disadvantaged, ranging continuously from 0 to 1.0 (higher predicted probability = lower SES). We employed the score, which correlated 0.91 with a factor-analytic-derived score, as the study measure of applicant SES. The SES score was preferable to alternatives29 because the continuous scale acknowledges that socioeconomic disadvantage is not a binary characteristic and reduces misclassification (false positives and negatives).
The admissions office also provided information from AMCAS regarding applicant age, sex, cumulative postsecondary GPA, and total MCAT score.
We analyzed the data using Stata version 13.1 (Stata Corporation Inc., College Station, Texas). We modeled MMI invitation (yes/no) and medical school acceptance recommendation (accept versus not) using logistic regression (a separate regression for each dependent variable). We modeled total MMI score using linear regression. All models included the following characteristics: age category (< 22 [reference], 22, 23, or > 24 years); female gender (yes/no); URM status (versus not); cumulative GPA category (< 3.4, 3.4–3.6, > 3.6–3.8, or > 3.8 [reference]); total MCAT score category (19–26, 27–30, 31–32, 33–34, or > 34 [reference]); SES (0–1.0 continuous score), and application year (2011, 2012, or 2013). The acceptance recommendation model additionally included the total MMI score (0–3).
During the three study application cycles, 15,844 people applied, and 8,933 (56.4%) were invited to submit secondary applications. Of the invitees, 7,964 (89.2%) submitted secondary applications, and 1,575 (19.8%) were invited to an MMI. Of the MMI invitees, 1,420 (90.2%) attended an MMI.
Table 1 summarizes the characteristics of screened applicants. Compared with those not invited to an MMI, those who participated in an MMI were older and more likely to be female, from an URM group, disadvantaged (based on both self-designation and the continuous SES score), and had higher GPAs and MCAT scores.
Although adjusted URM status was not associated with MMI invitation, lower SES was associated with receiving an MMI invitation, as were older age, female gender, and higher GPA and MCAT score (Table 2).
Mean MMI score was 1.30 (SD 0.62). In adjusted analyses, URM status was not associated with MMI score, but lower SES was associated with lower MMI scores (Table 3). Older age, female gender, and lower GPA were also associated with MMI score (Table 3).
Of the 1,420 MMI participants, 334 (23.5%) were recommended for acceptance. Lower SES was associated with being recommended for acceptance, whereas URM status was not (Table 4). Of other factors examined, mean MMI score was the most strongly associated with being recommended for acceptance. Older age, female gender, and higher GPA and MCAT scores were also associated with being recommended for acceptance (Table 4).
In a diverse sample of applicants over three admission cycles at UCD, superior MMI performance was strongly linked with being recommended for acceptance. Further, URM applicants were no less likely than non-URM applicants to receive an MMI invitation, performed similarly on the MMI, and were just as likely to be recommended for acceptance. In the only prior study to exploring this issue, MMI performance at six Canadian medical schools was worse for the < 3% of applicants with self-reported aboriginal status, whereas income was not significantly associated with MMI scores.28 Our findings provide some reassurance that adoption of the MMI-based admissions process at U.S. medical schools need not adversely affect admission prospects for URM applicants.
The similar MMI scores for URM and non-URM participants support the notion that structured interview processes that incorporate the perspectives of multiple evaluators like the MMI may be less vulnerable to the effects of individual evaluator implicit biases.20,25–27 Although we did not measure rater implicit biases regarding racial/ethnic minorities, such biases have been documented to be pervasive in U.S. society, including among physicians and other professionals,23,24 and can affect the outcomes of employment interviews in various fields including medicine.17,19–22 Thus, it is likely that implicit biases were present among our raters; however, they did not exert a significant net influence, given that mean MMI scores did not differ between URM and non-URM applicants. Because lack of URMs in medicine is a widely acknowledged problem,6,7,13,33,42–44 it is possible that biases against URM applicants were offset by ratings biased in favor of URM applicants, made by raters seeking to address limited racial/ethnic diversity in the physician workforce.
Our findings regarding MMI performance may have the broadest applicability, given the relatively high standardization of the MMI across institutions.34,35 By contrast, MMI invitation and acceptance decisions are shaped more by parochial concerns such as institutional mission and local workforce needs.12,33 In this context, our finding that lower SES applicants had worse adjusted MMI performance may be cause for concern. Although three prior studies reported no association of applicant SES with MMI performance, all relied on less robust measures of SES. Nonetheless, the decrement in MMI performance with decreasing SES in our study was small: The MMI score (scale of 0–3 points) declined by a mean of 0.12 points across the 0–1 range of the SES score. Further, the lower MMI scores among lower SES applicants were more than offset by their greater likelihood of being invited to an MMI and recommended for acceptance. These findings may reflect the ongoing shift from a purely metric-based applicant review process toward the more holistic process advocated by the Association of American Medical Colleges.12,15
Although the reasons for the lower MMI performance among lower SES applicants are unclear, poorer postsecondary academic preparation and performance are unlikely explanations because we adjusted for GPA and MCAT score. Lower SES applicants may have fewer life experiences bolstering skills assessed by the MMI. Similar reasoning has been suggested to explain the lower MCAT scores among such applicants.45 Although less affluent applicants are more likely to report paid employment during postsecondary education, their financial circumstances may require taking jobs that do not require MMI-type preemployment screening. Lack of prior experience with MMI-type screening may be a disadvantage in the medical school MMI because prior experience with a particular interview format is associated with better future performance with that format.46 Lower-level jobs also may not facilitate the higher-level communication, critical thinking, and problem-solving skills the MMI assesses, and the time required for such jobs may limit participation in pursuits that build such skills (e.g., scholarly presentations, volunteer clinic work).
Rater implicit bias could also help to explain the lower MMI scores among lower SES applicants. Raters were not provided any information about participants, and only one MMI station afforded applicants the opportunity to describe their backgrounds. Nonetheless, subtle information apparent to raters could have led to implicitly biased ratings, possibly triggered by the generally relatively large social distance between lower SES applicants and the typical rater.47,48 Most of our MMI raters were well educated and relatively affluent. Prior work indicates that applicant factors such as use of language unfamiliar to the typical rater could trigger a biased low rating.20,21 Applicants’ verbal skills have been shown to determine immediate interviewer impressions and, in turn, final appraisals.49 The issue of SES-based physician workforce disparities has received less attention than race/ethnicity-based disparities.6 Thus, it is less likely that raters consciously biased their evaluations in favor of lower SES applicants to address SES-based physician workforce disparities. If our findings are replicated, it would suggest the need to consider rater training to minimize the influence of SES-based biases.
We found that older applicants and women performed better than their younger and male counterparts both in the MMI (consistent with prior studies28,37) and throughout the admissions process. Older applicants are more likely to have had life experiences requiring effective communication. Women, more than men, have been noted to communicate in ways that quickly build rapport in novel social situations such as the MMI. Previously underrepresented in medicine, women now constitute well over half of all U.S. medical students.50 Given the increasing adoption of the MMI and strong influence of MMI performance on medical school acceptance, our findings suggest the potential for a disparity to develop disfavoring male applicants, and warrant monitoring.
Our study was correlational, limiting causal inferences. Our data were derived from applicants at a single school. How the findings generalize to other schools is unclear. We lacked sociodemographic information regarding MMI raters and admissions committee personnel, precluding examination of how such characteristics may have influenced the study outcomes. We employed a novel composite measure of SES, albeit one with theoretical advantages over other indicators.29 Because schools vary widely in institutional mission,33 applicant pools, admissions personnel, and other attributes, multi-institution studies are needed to better gauge the impact of MMI-based admissions processes on URM and socioeconomically disadvantaged applicants. Studies ideally should account for the characteristics of admissions personnel and applicants, and examine the relative utility of different SES indicators. It is also unknown whether URM or lower SES applicants would have fared differently in one-on-one interview-based admissions processes. Ideally, randomized trials would compare traditional interview processes with MMI-based processes, examining their impact on the prospects of URM and lower SES applicants.
In conclusion, in analyses of data from one California medical school, an MMI-based admissions process did not disfavor racial/ethnic minority groups underrepresented in the physician workforce. Applicants from lower SES backgrounds, also underrepresented in medicine, had lower MMI scores but were more likely to receive an MMI invitation and be recommended for acceptance. Multischool collaborations are needed to further evaluate the impact of MMI-based medical school admissions processes on URM and lower SES applicants.
1. Eva KW, Reiter HI, Rosenfeld J, Norman GR. The ability of the multiple mini-interview to predict preclerkship performance in medical school. Acad Med. 2004;79(10 suppl):S40–S42
2. Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, Norman GR. Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ. 2009;43:767–775
3. Reiter HI, Eva KW, Rosenfeld J, Norman GR. Multiple mini-interviews predict clerkship and licensing examination performance. Med Educ. 2007;41:378–384
4. Griffin B, Wilson I. Associations between the big five personality factors and multiple mini-interviews. Adv Health Sci Educ Theory Pract. 2012;17:377–388
6. Magnus SA, Mick SS. Medical schools, affirmative action, and the neglected role of social class. Am J Public Health. 2000;90:1197–1201
7. Saha S, Shipman SA. Race-neutral versus race-conscious workforce policy to improve access to care. Health Aff (Millwood). 2008;27:234–245
8. Association of American Medical Colleges. Minorities in Medical Education: Facts and Figures 2005. 2005 Washington, DC Association of American Medical Colleges
13. Saha S, Guiton G, Wimmers PF, Wilkerson L. Student body racial and ethnic composition and diversity-related outcomes in US medical schools. JAMA. 2008;300:1135–1145
14. Walker KO, Moreno G, Grumbach K. The association among specialty, race, ethnicity, and practice location among California physicians in diverse specialties. J Natl Med Assoc. 2012;104:46–52
16. Committee on Institutional and Policy-Level Strategies for Increasing the Diversity of the U.S. Health Care Workforce, Board on Health Sciences Policy. In the Nation’s Compelling Interest: Ensuring Diversity in the Health Care Workforce. 2004 Washington, DC National Academies Press
17. Salvatori P. Reliability and validity of admissions tools used to select students for the health professions. Adv Health Sci Educ Theory Pract. 2001;6:159–175
18. Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: The multiple mini-interview. Med Educ. 2004;38:314–326
19. Fiske ST, Markus HR Facing Social Class: How Societal Rank Influences Interaction. 2012 New York, NY Russell Sage Foundation
20. Macan T. The employment interview: A review of current studies and directions for future research. Hum Resour Manage R. 2009;19:203–218
21. Purkiss SLS, Perrewé PL, Gillespie TL, Mayes BT, Ferris GR. Implicit sources of bias in employment interview judgments and decisions. Organ Behav Hum Dec. 2006;101:152–167
22. Cuddy AJ, Fiske ST, Kwan VS, et al. Stereotype content model across cultures: Towards universal similarities and some differences. Br J Soc Psychol. 2009;48(pt 1):1–33
23. Greenwald AG, Poehlman TA, Uhlmann EL, Banaji MR. Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. J Pers Soc Psychol. 2009;97:17–41
24. Chapman EN, Kaatz A, Carnes M. Physicians and implicit bias: How doctors may unwittingly perpetuate health care disparities. J Gen Intern Med. 2013;28:1504–1510
25. Kutcher EJ, Bragger JD. Selection interviews of overweight job applicants: Can structure reduce the bias? J Appl Soc Psychol. 2004;34:1993–2022
26. Brecher E, Bragger J, Kutcher E. The structured interview: Reducing biases toward job applicants with physical disabilities. Employ Responsib Rights J. 2006;18:155–170
27. Levashina J, Hartwell CJ, Morgeson FP, Campion MA. The structured employment interview: Narrative and quantitative review of the research literature. Pers Psychol. 2014;67:241–293
28. Reiter HI, Lockyer J, Ziola B, Courneya CA, Eva KCanadian Multiple Mini-Interview Research Alliance (CaMMIRA). . Should efforts in favor of medical student diversity be focused during admissions or farther upstream? Acad Med. 2012;87:443–448
29. Grbic D, Jones DJ, Case ST Effective Practices for Using the AAMC Socioeconomic Status Indicators in Medical School Admissions. 2013 Washington, DC Association of American Medical Colleges
30. Uijtdehaage S, Doyle L, Parker N. Enhancing the reliability of the multiple mini-interview for selecting prospective health care leaders. Acad Med. 2011;86:1032–1039
31. Taylor CA, Green KE, Spruce A. Evaluation of the effect of socio-economic status on performance in a Multiple Mini Interview for admission to medical school [published online July 2, 2014]. Med Teach. 2015;37:59–63
32. Morgenstern H. Uses of ecologic analysis in epidemiologic research. Am J Public Health. 1982;72:1336–1344
33. Mullan F, Chen C, Petterson S, Kolsky G, Spagnola M. The social mission of medical education: Ranking the schools. Ann Intern Med. 2010;152:804–811
35. Pau A, Jeevaratnam K, Chen YS, Fall AA, Khoo C, Nadarajah VD. The Multiple Mini-Interview (MMI) for student selection in health professions training—a systematic review. Med Teach. 2013;35:1027–1041
37. Jerant A, Griffin E, Rainwater J, et al. Does applicant personality influence multiple mini-interview performance and medical school acceptance offers? Acad Med. 2012;87:1250–1259
38. Dowell J, Lynch B, Till H, Kumwenda B, Husbands A. The multiple mini-interview in the U.K. context: 3 years of experience at Dundee. Med Teach. 2012;34:297–304
39. Harris S, Owen C. Discerning quality: Using the multiple mini-interview in student selection for the Australian National University Medical School. Med Educ. 2007;41:234–241
40. Till H, Myford C, Dowell J. Improving student selection using multiple mini-interviews with multifaceted Rasch modeling. Acad Med. 2013;88:216–223
41. Sebok SS, Luu K, Klinger DA. Psychometric properties of the multiple mini-interview used for medical admissions: Findings from generalizability and Rasch analyses. Adv Health Sci Educ Theory Pract. 2014;19:71–84
43. Ko M, Heslin KC, Edelstein RA, Grumbach K. The role of medical education in reducing health care disparities: The first ten years of the UCLA/Drew medical education program. J Gen Intern Med. 2007;22:625–631
44. Whitla DK, Orfield G, Silen W, Teperow C, Howard C, Reede J. Educational benefits of diversity in medical school: A survey of students. Acad Med. 2003;78:460–466
45. Davis D, Dorsey JK, Franks RD, Sackett PR, Searcy CA, Zhao X. Do racial and ethnic group differences in performance on the MCAT exam reflect test bias? Acad Med. 2013;88:593–602
46. Huffcutt AI, Van Iddekinge CH, Roth PH. Understanding applicant behavior in employment interviews: A theoretical model of interviewee performance. Hum Resour Manage R. 2011;21:353–367
47. Dohrenwend BS, Colombotos J, Dohrenwend BP. Social distance and interviewer effects. Milbank Q. 1969;41:213–226
48. Howard JL, Ferris GR. The employment interview context: Social and situational influences on interviewer decisions. J Appl Soc Psychol. 1996;26:112–136
49. Barrick MR, Dustin SL, Giluk TL, Stewart GL, Shaffer JA, Swider BW. Candidate characteristics driving initial impressions during rapport building: Implications for employment interview validity. J Occup Organ Psychol. 2012;85:330–352