Interviews are an integral part of the residency selection process. Programs use a variety of interview methods, including unstructured traditional interviews (TIs), a one-on-one format where the interviewer directs the content and tone of the encounter. However, previous studies of the TI method have shown poor interrater reliability, perceived gender and cultural bias by applicants, and lack of value in predicting future success.1–3
In response, researchers at McMasters University developed the multiple mini-interview (MMI).4 The MMI consists of multiple interview stations, each assigned a unique, validated case scenario designed to assess particular character traits. Applicants first read and then discuss the scenario with an interviewer for less than 10 minutes. Rather than judge specific content, the interviewer is trained to evaluate critical thinking skills. The theory behind the MMI is that, by standardizing interview content and increasing the total number of interviews, one can both decrease perceived bias and increase interrater reliability.
Initial trials of the MMI involving Canadian medical school applicants demonstrated increased interviewer agreement compared with the TI.4 In addition, surveys showed a decreased perception of gender and cultural bias as well as applicant preference for the MMI process.5 Subsequent studies involving Canadian,5,6 UK,7,8 and U.S. medical school applicants9 have validated the acceptability of the MMI. As a result, the MMI process has replaced TIs in 14 of 17 Canadian medical schools and is increasingly used in U.S. medical schools.10
Studies involving Canadian residency programs in pediatrics, internal medicine, obstetrics–gynecology,11 family medicine,12 otolaryngology,13 and physiatry14 demonstrate an overall preference for MMI over TI styles. However, all studies have involved Canadian or international graduates, many with prior MMI experience; there have been no studies evaluating perceptions of the MMI among U.S. residency applicants, whose expectations of the interview process may differ.
Recognizing the limitations of our TI process in resident selection, emergency medicine faculty members at Alameda Health System–Highland Hospital (AHS) elected to integrate the MMI with the standard TI during the 2011–2012 interview season. Given results from prior studies, we hypothesized that U.S. emergency medicine residency (EMR) applicants would favor the MMI over the TI. A postinterview questionnaire was created to assess applicant perceptions of the MMI format.
AHS has been the site of a four-year emergency medicine training program for over 30 years. The institution is a trauma referral center and county emergency department with an annual volume of over 90,000 visits.
Historically, the EMR interview process consisted of three TIs—defined as 20-minute, one-on-one discussions with a faculty member or senior resident. The interviewer had access to the applicant’s file prior to the interview, and questions focused on applicants’ past experiences and reasons for choosing emergency medicine; however, specific content was left to the discretion of the interviewer. Immediately following the interview, applicants were ranked on a 1–10 scale (1 = unsuitable, 3 = less suitable, 5 = satisfactory, 7 = above average, 10 = outstanding); evaluators were encouraged to use the full scale; however, there was no standardized evaluation method.
During the 2011 retreat, AHS emergency medicine faculty members elected to reduce the number of TIs to two and to integrate a four-station MMI for EMR applicants.
Applicants to the AHS EMR program during the 2011–2012 interview season were notified via e-mail of the new MMI process when initially offered an invitation to interview. Included in the original e-mail was information regarding the MMI, including prior publications and instructions, as well as examples of scenarios (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A238). Participants of the study included those applicants who chose to complete an optional, anonymous postinterview survey evaluating their perceptions of the MMI and TI process.
The MMI curriculum was purchased from Advanced Psychometrics for Transition (APT) Incorporated. Scenarios used during the interview season were selected by faculty and encompassed the following values: ethical judgment, responsibility, conflict resolution, self-awareness, and communication. Emergency medicine faculty, residents, nurses, and administrators volunteered to serve as interviewers. All MMI interviewers completed a standard one-hour interviewer training session provided by APT prior to participation. The training session was a prerecorded video, which was administered during small-group sessions prior to the start of the interview season.
One week prior to the interview day, 4 MMI scenarios were selected from a bank of 120 total scenarios at the discretion of the medical student clerkship director (A.S.). During the interview day, each EMR applicant randomly completed at total of 4 MMI scenarios (along with 2 TIs). Applicants were given two minutes to read each MMI scenario. After two minutes, applicants entered the interview room and had eight minutes to discuss the scenario with the interviewer.
The interviewer was given the MMI scenario prior to the interview day, along with optional follow-up questions to help applicants progress through scenarios. Interviewers were free to ask questions; however, in accordance with APT training, they were encouraged to allow the applicant to express their thoughts fully. Immediately after the interview, the applicant was assigned a score on the same 10-point scale used in the TIs (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/A238).
After completion of interviews, we sent an online link to an anonymous electronic questionnaire to all applicants (www.Surveymonkey.com). The questionnaire was derived using survey questions from prior studies involving satisfaction with the MMI.5,9,12,15 We obtained consent electronically by agreeing to participate in the survey; no incentives were offered for participation. We asked applicants to rate their perceptions of the MMI and TIs using 23 multiple-choice, single-answer questions presented using a five-point Likert scale (definitely disagree, disagree, neither agree nor disagree, agree, definitely agree). Topics included perceptions of gender and cultural bias, enjoyment and preference of interview, stress, and portrayal of self. In addition, 3 multiple-choice questions assessed how applicants perceived and ranked AHS before and after their interview experience. We included a general comments free-text optional section at the end of the survey.
To reduce bias, we sent the survey after applicants and residency programs had submitted their rank lists, but prior to the release of National Resident Matching Program (NRMP) results. The survey was anonymous, collected no demographic data, and stated that responses could not affect any applicant’s position on the program’s rank order list.
We calculated median and interquartile ranges for all multiple-choice survey questions. To aid in comparison with prior MMI studies, mean and standard deviation of multiple-choice questions were also included. We collapsed the five-point Likert scale anchored by “1–definitely disagree” and “5–definitely agree” into three categories: 1 to 2 (disagree), 3 (neutral), and 4 to 5 (agree). Analysis of the collapsed data was conducted using simple descriptive statistics. We chose McNemar chi-square test to evaluate differences in nonparametric, paired questions. All analyses were done using R statistical software version 2.14.0 (R Core Team, Vienna, Austria).
Participants’ comments were coded as either primarily positive toward the MMI, primarily negative toward the MMI, or neither positive nor negative by three authors (W.S., H.H., A.S.). Kappa scores were calculated to verify coder agreement. We then analyzed comments using a modified thematic network analysis. First, comments were grouped according to recurring words and phrases. Grouped comments were then analyzed and coded into general recurring themes that were present in five or more participant comments. We then reread comments to count the frequency of each theme.
The AHS committee for the protection of human subjects reviewed the protocol and granted the study exempt status.
Of 150 applicants interviewed, 110 completed the survey (73% response rate). Ninety-nine (90%) respondents had never experienced the MMI, whereas 8 (7%) had one prior experience and 4 (3%) had two or more prior MMI interviews.
Applicants reported that they had received adequate advanced information about the MMI process (96, or 87%). Applicants felt the MMI process was free from gender bias (95, or 86%) compared with the TI (80, or 73%, McNemar chi-square = 8.52, P = .004). In comparison, cultural bias was not statistically different between groups (MMI = 77, or 70%; TI = 71, or 65%; McNemar chi-square = 1.04, P = .31) (see Table 1).
Applicants perceived the TI as more enjoyable than the MMI process (98, or 89% compared with 48, or 44%, McNemar chi-square = 28.66, P < .01). Additionally, respondents reported that they preferred the TI to the MMI process (66, or 60% compared with 9, or 10%, McNemar chi-square = 40.81, P < .01).
When asked the ideal future interview method, 57 respondents (52%) indicated that they would prefer a mix of MMI and TI formats compared with only the TI (51, or 46%) or only the MMI (2, or 2%). Whereas 91 (83%) respondents felt the MMI process did not alter their ranking of our program, 3 applicants (3%) stated that they ranked AHS higher (positively/closer to their first choice) because of the MMI and 16 applicants (14%) stated they ranked AHS lower (negatively/closer to their last choice) because of the MMI process.
Of the 110 applicants who completed the survey, 60 applicants (55%) wrote in the optional general comments section. Overall, 17 (28%) of those comments were primarily positive, 29 (49%) were primarily negative, and 14 (23%) comments contained both positive and negative statements regarding the MMI process (unweighted kappa = 0.71). Commonly expressed positive themes included the ability of applicants to demonstrate their unique strengths using the MMI (10, or 18%) and feeling the MMI was an innovative interview style (7, or 13%). Commonly expressed negative themes included the inability of the applicant to form a meaningful connection with the interviewer (14, or 25%) and the inability to learn more about the interviewer, hospital, or residency program (13, or 24%) (see Table 2).
During the 2011–2012 interview season, applicants to the AHS EMR program were interviewed by both TI and MMI techniques. Contrary to previously published studies, our respondents reported the MMI as less enjoyable and preferred the TI to the MMI process.
We hypothesize many reasons why applicants indicated a preference for the TI. First, lack of familiarity with the MMI process may have contributed to negative perceptions. With 89% of applicants experiencing the MMI for the first time, the new and unfamiliar process during the interview day may have biased participants against the MMI. Unfortunately, the subgroup of 12 participants with prior MMI experience was too small to perform analysis.
Applicants also indicated feeling that it was difficult to portray themselves accurately during the MMI. In contrast to a TI, which involves discussion of achievements and experiences, the MMI focuses on the applicant’s ability to demonstrate character traits by discussing a preselected scenario. Whereas the applicant may have opportunities to refine their TI technique, the MMI requires improvised responses, which may lead applicants to question how they portrayed themselves and negatively rate the MMI. Unfortunately, our study did not directly compare applicants’ perceptions of self-portrayal and their interview score or ranking.
Although it is possible that lack of familiarity and perceptions of self affected applicants’ rankings, our analysis of comments revealed that many applicants disliked the MMI because they felt unable to form a personal connection with the interviewer.
Previous research suggests that personal connections are important to U.S. residency applicants. A 2004 study by Nuthalapaty and colleagues16 surveyed U.S. medical school graduates and found that the top three factors influencing residency selection were perceived resident satisfaction (98%), how the program cared for its trainees (98%), and how well the applicant thought she or he would fit with the program (97%). Similarly, DeIorio and colleagues17 noted that EMR applicants felt the personality of the program, which included perceived happiness of residents, faculty–resident relationships, and shared values, was the most important part of the interview day. Together, these studies suggest that U.S. residency applicants value personal connection during the interview day.
Analysis of participants’ comments showed that many applicants felt unable to form a meaningful connection with the MMI interviewer. The difficulty of forming personal connections highlights a difference in the goals of the MMI and the applicants’ interview experience. The MMI was created to provide an unbiased interview that increases interrater agreement. It accomplishes this by standardizing interactions, thus decreasing the variability associated with an interviewer’s background, experiences, and values.18 However, eliminating interviewers’ backgrounds and experiences may make the interview feel more rigid, with less opportunity for personal connection.
Beyond a lack of personal connection, some participants commented that they were unable to even learn about the interviewer, hospital, or residency program. Nearly half of comments expressed concern that the MMI format hindered participants’ ability to form meaningful connections or learn about the program. Even with ample opportunities to interact outside of interviews (applicant mixer, conference, interview day lunch), many respondents still felt the MMI decreased their ability to determine their fit with the program.
In summary, although lack of experience may have contributed to decreased preference for the MMI, analysis of comments indicated that many felt unable to form a personal connection during the interview, which negatively affected their perceptions of the MMI. For some participants, this lack of connection negatively affected the perceptions and subsequent ranking of the residency program.
Although overall participants preferred the TI, it is important to note that many applicants enjoyed the MMI format. Participants commented that the MMI allowed them to display unique qualities and strengths that do not normally surface during TIs, and that they enjoyed the innovation and the challenge of the MMI. Further, consistent with prior studies, applicants perceived the MMI as free from gender bias when compared with the TI. Finally, when asked their ideal future interview method, over half of respondents indicated that they would prefer a mix of MMI and TI formats, suggesting that some participants felt the MMI offered positive qualities and should have some continued role in the interview process.
Although many participants enjoyed the MMI, significantly more preferred the TI format. We believe this is due to multiple factors, including lack of familiarity with the MMI, inability to form a personal connection with the interviewer, and difficulty perceiving fit with the program. However, it remains unclear how much each of these individual factors contributed to our overall findings. It is possible that, by simply increasing familiarity, applicants may prefer the MMI. We recommend that future studies involve analysis of applicant perceptions of the MMI—including participants with prior MMI experience—to assess acceptability of the MMI as an interview method among U.S. residency applicants.
Many previous MMI studies requested voluntary enrollment of students who were applying to their respective medical training programs. We felt that having applicants volunteer for a study during their interview day was unfair, and that the only equitable way to use the MMI among our applicants was to revise the interview process for all applicants.
Additionally, the decision to time our survey during the period when no change to the rank list was possible (by applicant or program) and before publication of NRMP results was an attempt to collect unbiased, unfiltered responses. Our relatively high response rate suggests that our findings are more likely to represent of the entire group and are perhaps less subject to selection bias.
Our study is constrained by several limitations. First, the study only involved one EMR program in the United States during one interview year. Although we are not aware of particular bias involving our program, we use caution generalizing our results. In addition, although we had a high applicant response rate to our survey, participation was voluntary, which may have preferentially selected extreme viewpoints. Finally, given our preference to incorporate both the MMI and TI, we did not use enough MMI stations to reach significant levels of interrater reliability. Therefore, we cannot comment on rates of interviewer agreement in the MMI as compared with the TI.
In conclusion, applicants to the EMR program at AHS preferred TIs to MMIs, in part because of the lack of personal connection inherent in the MMI format. Although the MMI has been shown to increase interrater reliability, it may come at the cost of interpersonal interactions. As a result, we propose further investigation into the perceptions of the MMI versus other interview techniques for U.S. EMR applicants.
Acknowledgments: The authors would like to thank the applicant class of 2011–2012 for their participation in the study.
1. Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM. Assessing personal qualities in medical school admissions. Acad Med. 2003;78:313–321
2. Morris JG. The value and role of the interview in the student admissions process: A review. Med Teach. 2001;21:473–481
3. Edwards JC, Johnson EK, Molidor JB. The interview in the admission process. Acad Med. 1990;65:167–177
4. Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: The multiple mini-interview. Med Educ. 2004;38:314–326
5. Razack S, Faremo S, Drolet F, Snell L, Wiseman J, Pickering J. Multiple mini-interviews versus traditional interviews: Stakeholder acceptability comparison. Med Educ. 2009;43:993–1000
6. Brownell K, Lockyer J, Collin T, Lemay JF. Introduction of the multiple mini interview into the admissions process at the University of Calgary: Acceptability and feasibility. Med Teach. 2007;29:394–396
7. Dowell J, Lynch B, Till H, et al. The multiple mini-interview in the UK context: 3 years of experience at Dundee. Med Teach. 2012;34:297–304
8. O’Brien A, Harvey J, Shannon M, Lewis K, Valencia O. A comparison of multiple mini-interviews and structured interviews in a UK setting. Med Teach. 2011;33:397–402
9. Uijtdehaage S, Doyle L, Parker N. Enhancing the reliability of the multiple mini-interview for selecting prospective health care leaders. Acad Med. 2011;86:1032–1039
10. Eva KW, Reiter HI, Rosenfeld J, Trinh K, Wood TJ, Norman GR. Association between a medical school admission process using the multiple mini-interview and national licensing examination scores. JAMA. 2012;308:2233–2240
11. Dore KL, Kreuger S, Ladhani M, et al. The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions. Acad Med. 2010;85(10 suppl):S60–S63
12. Hofmeister M, Lockyer J, Crutcher R. The acceptability of the multiple mini interview for resident selection. Fam Med. 2008;40:734–740
13. Campagna-Vaillancourt M, Manoukian J, Razack S, Nguyen LH. Acceptability and reliability of multiple mini interviews for admission to otolaryngology residency. Laryngoscope. 2014;124:91–96
14. Finlayson HC, Townson AF. Resident selection for a physical medicine and rehabilitation program. Am J Phys Med Rehabil. 2011;90:330–335
15. Humphrey S, Dowson S, Wall D, Diwakar V, Goodyear HM. Multiple mini-interviews: Opinions of candidates and interviewers. Med Educ. 2008;42:207–213
16. Nuthalapaty FS, Jackson JR, Owen J. The influence of quality-of-life, academic, and workplace factors on residency program selection. Acad Med. 2004;79:417–425
17. Deiorio NM, Yarris LM, Gaines SA. Emergency medicine residency applicant views on the interview day process. Acad Emerg Med. 2009;16(suppl 2):S67–S70
18. Harasym PH, Woloschuk W, Mandin H, Brundin-Mather R. Reliability and validity of interviewers’ judgments of medical school candidates. Acad Med. 1996;71(1 suppl):S40–S42