Journal Logo

Feature Article

Maximizing Effectiveness of the Holistic Admission Process

Implementing the Multiple Mini Interview Model

Wagner, Robin DNP, RN, CNS, CHSE; Maddox, Krista R. EdD; Glazer, Greer PhD, RN, CNP, FAAN; Hittle, Beverly M. MSN, RN

Author Information
doi: 10.1097/NNE.0000000000000702
  • Free

Abstract

Producing diverse nurse leaders begins with admitting applicants to nursing programs that possess qualities and characteristics (eg, critical thinking, communication skills, ethical integrity) necessary to be successful in the program and to become leaders. These qualities and characteristics cannot be measured by quantitative metrics alone, such as grade point average (GPA) and test scores. Rather, nursing programs should consider a wide variety of metrics including applicants' experiences and attributes, in addition to academic measures. Considering a balance of these metrics is known as holistic admission.1

Planning and instituting a holistic admissions process can be a daunting project, yet several health care disciplines have accomplished it with success.2,3 In the United States, 93% of dental schools and 91% of medical schools report using a holistic admissions process, but only 47% of undergraduate nursing schools report the same.4 First outlined by the Association of American Medical Colleges5 and later adopted by the American Association of Colleges of Nursing,6 the core principles to holistic admissions are designed to place balance on an applicant's experience, attributes, and academics metrics. Evaluating applicants' life experiences can be readily achieved via an essay, and academic metrics can be assessed via traditional measurements such as GPA and standardized testing. Capturing applicant attributes, which align with those desired in the nursing workforce, are more elusive. Therefore, a holistic admission process is preferable over customary practices because it allows nursing programs to critically analyze each applicant's experiences, motivation, and character that will make them nurse leaders.

The Problem: Pros and Cons of Interviews

Interviews (including nonstructured, 1-on-1, and panel interviews) have been implemented in admission processes as a method for measuring applicant noncognitive attributes.7 In a study of allied health programs (n = 228), 57% of the schools reported using interviews in the admission process.8 The most sought-after skills gleaned from the interviews included “communication skills, motivation, readiness for the profession, service, and problem solving.”8 An integrative review of admission processes7 found similar characteristics reported by schools as a motivator for using interviews. Schools that have incorporated interviews into the admission process have reported positive results including increased diversity and maintaining or exceeding GPA and pass rates on the NCLEX-RN.9-11

Despite reported successes, interviews are not without limitations. Interviewer biases such as the halo/horn effect, explicit bias, implicit bias, and assessment of extraneous characteristics have resulted in poor reliability and validity.7,12,13 Resulting inconsistencies and reports of poor previous experiences with admission interviews were one of the barriers to holistic admissions identified by nursing school deans in the United States.14 In a longitudinal study to evaluate the use of interview scores in a baccalaureate nursing program admission process, Hendricks and Krothe15 report their interview process (first traditional and then group interviews) did not contribute to increased diversity (ethnic or gender) or retention. The results prompted a decision to stop using interviews in the admission process.

Multiple Mini Interviews as a Solution

Because traditional interview methods have poor reliability and validity, there is a need for an improved method.16 The Multiple Mini Interview (MMI) model is an emerging option. With stronger psychometrics than those found in traditional interview methods, MMI model addresses many of the concerns for interviewer bias.16,17 The MMI model is an admission interview format, developed in 2002 at McMasters University, based on the Objective Structured Clinical Examination process.18 Instead of a traditional panel interview, applicants participate in a circuit interview format rotating through each of 6 to 8 stations with 8 to 10 minutes per station with a different rater at each station. Each station provides a validated prompt to specifically target an attribute of interest. At each station, the applicant is allotted 2 minutes to read the provided prompt and then upon entering the room has 6 to 8 minutes to respond to the prompt. At the end of the circuit, raters' scores are compiled to give a composite interview score for each applicant. Discipline-specific knowledge is not required to respond to interview prompts; rather, raters assess the critical thought process and how the applicant frames his/her response.

Based on the MMI model's advantages over traditional interview methods, various professional schools have adopted MMI models as part of the holistic admission process. Evidence supports that MMI models successfully predict medical student clinical performance, program advancement, and licensure examination scores.18-20 Researchers report similar success with MMI models in dental school admissions.21,22 Nursing schools in the United Kingdom using MMI models have reported a decrease in racial and gender biases and success in capturing desired noncognitive attributes deemed successful in the nursing field.23,24 Additionally, qualitative responses from students and interviewers present a positive experience with MMI models admissions processes.25,26

The MMI model is seen in the existing literature as a convincing alternative to the traditional style of interview when the objective of the interview is identifying noncognitive constructs of interest. Ross et al27 caution, however, that any change in process can produce unintended consequences. Potential consequences identified include personality type, gender bias, station sequence/difficulty, and increased cost17,27-29 but were viewed as controllable based on the structure of the MMI model process.17 Pau et al17 suggest that although the MMI model is generally accepted as reliable, more longitudinal research is needed to support the validity of the MMI. With the international success of MMI models and because the advantages outweigh the disadvantages, the University of Cincinnati College of Nursing (CON) decided to adopt an MMI model into the existing BSN program holistic admission procedure.

The intent for incorporating an MMI model was 2-fold: (1) to select applicants whose life experiences, motivation, and character are aligned with the core values, vision, and mission of the college and (2) to improve the diversity of the student body through enhanced holistic admission practices. Nursing programs that strive to achieve their vision and mission should ask the critical question of whether the students they admit will allow them to do so. We have found some students with high GPAs and grades had difficulty communicating effectively with faculty, fellow students, and other health care team members. A holistic admission process allows for a critical examination of applicants' experiences, attributes, and characteristics to determine if they are aligned with the program, which will further develop students, enabling them to be successful in the program and become nurse leaders. Additionally, improving diversity of the nursing student body will prepare nurses who reflect the diversity of those they serve to affect change in health outcome equity.3,30 “Educating students in environments that value diversity and inclusion produces graduates better prepared to practice in underserved communities and whose understanding of the cultural needs of patients improves patient satisfaction and trust.”3(p5)

Foundation to the efforts of incorporating an interview process was the development of an admission mission statement. The goal of this statement was to provide guidance in identifying applicants who possess attributes aligned with the CON values, vision, and mission: “The University of Cincinnati CON is committed to admitting diverse students who will become nurse leaders in the transformation of health care. The admission process values intellect, life experiences, motivation, and character, which exemplify the core values of the CON. Candidates will be selected using broad-based admission criteria to create a diverse student body consistent with the University of Cincinnati CON mission with an aim to cultivate collegiality by promoting innovation and engagement in our communities.”

In an effort to realize the admission mission statement, the MMI model was selected as a reliable way to address both aims. In this article, we outline our steps for incorporating the MMI model into our existing holistic admissions model. Although the planning and implementation process are unique to our CON, there are many issues other sites would need to anticipate, such as who should lead the project, how to involve faculty, how to schedule interviews, other logistical considerations, and potential costs. Additionally, many nursing programs do not have their own admission process, instead relying on the central college or university admissions office.8 In this scenario, implementing the MMI model will require coordination with the central office.

Approach for Incorporating MMI Model Into Holistic Admissions Process

Increasing diversity at the CON was part of a multistep collaborative project within the university's Academic Health Center, led by the dean of the CON and a professor in the College of Medicine.31 The project managers, assistant dean of Student Affairs and chair of the Admission Task Force, assessed the current admission process and determined the process employed holistic admission techniques but was lacking an intentional alignment of candidate attributes with the CON values, vision, and mission. Core values of the CON could have been better realized by using more than the applicants' written word. Therefore, an interview process was desired. After investigating MMI models at other universities, this format was chosen for achieving the admission goals.

Planning

It was decided to first focus on adding the MMI model in admissions to the most competitive program, the BSN program. A BSN Admission Task Force, made up of 2 cochairs and 6 undergraduate program faculty, was created to develop the admission rubric and plan and implement the MMI model process for admission to the BSN program. Prior to planning, the task force led the creation of the admission mission statement as a pathway toward the type of student and nursing graduate the CON desired. Even with the admission mission statement as a guide, the necessary attributes to be captured by the MMI model needed to be defined. The task force sought feedback from the college's faculty, clinical partners, and community stakeholders to compile a list of characteristics that best fit the program mission and core values.

Partnering With Central Admissions

Undergraduate admissions are handled by a central university office, with central office staff reviewing and making admissions decisions based on guidelines and criteria provided by the CON. Incorporating an interview process required a partnership and logistical planning to effectively implement this process change to ensure congruency with the CON admissions' mission and overall needs of the central process. Case in point, the central admissions office at the same time had adopted a new customer relation management system for managing all recruitment and admissions communication and activities. Understanding how the central system would handle admissions decisions, interview offers, reporting of interview scores, and reporting of nursing applicant status had to be determined. The interview scores presented a challenge, as CON staff did not have administrative access to the central office system, yet the MMI model scores needed to be reported for the final admission decisions. To work out these details, central admissions and the CON staff conducted weekly collaborative planning meetings over several months to ensure a seamless admissions process.

Recruiting and Training Raters

Perhaps one of the most significant challenges of the MMI model initiative was recruiting and securing raters. Six to 10 raters were needed for each interview session, with multiple sessions planned to complete the almost 250 interviews. Raters were recruited using a variety of electronic messaging and social media posts to raise interest of faculty, alumni, clinical partners, community stakeholders, and others. As no specialized nursing knowledge is required to become a rater, anyone identified as a potential recipient of nursing care is eligible to serve. To prepare raters, a 90-minute online or in-person training session, developed by the team, was required for each rater to complete prior to engaging in the interviews.

Training incorporated not only an introduction to the MMI model and training on how to conduct an interview, but also included detailed information on the insidious nature of implicit bias and how to forestall potential (or avoidable) variations in reliability. The nature of this model of interviewing allows for different raters to assess the same attribute in various stages of the interview cycle. As such, interrater reliability is a challenge. The rater training attempts to address this by requiring raters to watch a simulated interview video. They rate the interview and then watch a video of a group of raters discussing appropriate ratings for the interview. Through this video they are provided guidance on specifically what they should be assessing in the interviews in an attempt to increase the interrater reliability. Additionally, at the end of the interview season, raters are provided feedback of their overall rating as compared with the aggregate ratings of all raters. This allows raters to determine how closely they align with other raters and allows them to correct any issues of misalignment in subsequent rating experiences.

Implementation: Logistics and Cost

The MMI model is complex because of the fact that it involves many interviewees and raters interacting at the same time. Appropriate preparation for the inherent complexity ensures logistics do not detract from the overall impact of the MMI model. Therefore, to ensure a seamless experience for all, it was necessary to fine tune the logistics of the interview day, which primarily occurred during non–high school hours, including weekends. Specifically, one needs to consider the location and number of interview stations, as well as the setup of each station to provide efficient transition of interviewees through the process. Equally important was the setup of the individual interview stations to provide necessary guidance to the raters. Implicit bias information was also included in the rater packet at each interview station as a reminder. To guarantee efficiency and integrity of the process, it was necessary to have someone provide oversight of the interviews. This individual had the responsibility to welcome interviewees, set interviewees' expectations, ensure smooth transition between stations, and keep time during the interviews.

Depending on the needs and abilities of the institution, interview prompts can be developed internally or purchased from an existing MMI vendor. One resource that cannot be overlooked to implement a successful MMI model is the faculty and staff time needed. It is necessary to have dedicated, committed faculty and staff for whom this project is an expectation of their role to ensure the integrity and necessary oversight of the work. One should also consider additional costs of the MMI model experience such as those related to rater training, creating the physical space, materials used for interviews, and hosting both interviewees and raters. While the resources necessary to implement a holistic admission process that includes an interview can vary from one institution to another, it is helpful to manage the potential resource barriers by incorporating the necessary work into existing roles, such as recruitment/admission staff and admission committees. Additionally, the work will vary depending on the institution's available technology platforms to handle tasks related to scheduling, analyzing, and managing interviews.

Outcomes of MMI Model Implementation

Success of MMI model implementation was based on which applicants would have been admitted based solely on quantitative metrics alone versus the applicants accepted based on the holistic admission process, including the MMI model. Using the characteristics for what makes a great nurse, based on a prior stakeholder survey, the CON was able to choose questions directly reflecting these attributes. In the overall aggregate score, applicants with the highest interview scores were perceived as being most aligned with the college vision, mission, and core values.

When comparing the students accepted for admission under the MMI model versus those who would have been accepted based on quantitative metrics alone, it was determined there was a 72% (n = 247) overlap. Therefore, 28% (n = 96) of the applicants offered admission under the holistic process that included the MMI model would not have been offered admission prior to the addition of the MMI model interview to the admission process. In evaluation of our second objective to improve the diversity of the student body, admission under the MMI model increased the number of students identified as nonwhite, male, first-generation college student, or as coming from out of state as compared with the same applicants who would have been admitted on quantitative metrics alone (Table).

Table
Table:
Applicant Admission Comparison Based on Determining Admission Metrics

A postevent evaluation was sent to applicants (n = 221) and raters (n = 56). For the responding applicants (n = 125), there was an overall positive response to the process. A majority (89%) believed the MMI model experience allowed an opportunity to demonstrate good fit to the CON program beyond what an application process allowed. Similarly, a majority (83%) agreed the process was a fair assessment of personal characteristics and abilities. When asked about whether the process was free from gender, cultural, socioeconomic, or other biases, a majority (94%) agreed. Some qualitative responses included “I am thankful that you look beyond grades and scores. I felt that these interviews gave me a great chance to show who I really am outside of grades.” The rater questionnaire was more directed at the confidence in rater training, as well as efficiency and logistics on the day of the interviews. When responding to effectiveness of rater training (n = 65), 65% of raters reported adequate, 32% moderate, and only 3% under preparation.

Overall, the short-term results associated with implementing the MMI model admissions process were positive. Our objectives were achieved, particularly when looking at the congruency of applicant noncognitive characteristics with the CON desired attributes. This highlights the ability of the MMI model process to delve deeper than quantitative metrics alone, allowing admission to the applicants most aligned with the CON's core values, vision, and mission. Encouragingly, 96 applicants were invited to enroll that would otherwise have not been offered admission without the addition of the MMI model to the existing holistic admission process. This reinforces literature findings that the MMI model provides the opportunity to consider applicants who have barriers achieving the high quantitative metrics, but still possess characteristics to be a successful professional nurse representative of the populations for which they will care.19,20 We believe the MMI model was a positive addition to our existing holistic admission process and is complementary to our other strategies for increasing student diversity.

Some key aspects of the planning and implementation are believed to have contributed to the success of the CON's initial BSN MMI model implementation admission experience. First, creating an admission mission statement created a roadmap for what the CON values, helping to identify what type of applicant should be admitted. Second, identifying desired “great nurse” characteristics allowed for aligning applicant attributes with the mission statement and admission process. Third, collaborating with the central admissions department was a lengthy yet vital process. This partnership between central admissions and the CON allowed MMI model interviews to be seamlessly integrated into the admission process without prolonging the decision time. Finally, a dedicated staff and faculty, committed to the vision of holistic admissions, were essential to this project.

Lessons Learned and Future Plans

Moving forward, there are anticipated steps for sustaining and improving this holistic admissions process. The greatest challenge will be to continue to secure an adequate number of raters to conduct the interviews. Gaining buy-in to incorporate MMI model as part of the admission process versus commitment to carrying out the interviews is important. The rater resource is the single most taxing resource of implementing an MMI model process and requires a significant commitment from the faculty, staff, and other stakeholders. Not only do faculty and staff need to agree to the incorporation, but they also need to serve as raters.

Conclusion

The course for planning and implementing an MMI model method into a holistic admission process is a multiyear commitment, requiring strong leadership and internal and external collaboration. It requires intentional forethought into the type of student that matches the nursing program's vision, mission, and values. Despite the potential challenges of the MMI model, it is an achievable process. When one considers the implications of this new diverse nursing student population on the improved health of their patients and the community, the rewards for the hard work and dedication required for MMI model implementation become evident.

References

1. Dewitty VP. What is holistic admissions review, and why does it matter? J Nurs Educ. 2018;57(4):195–196.
2. Grabowski CJ. Impact of holistic review on student interview pool diversity. Adv Health Sci Educ Theory Pract. 2018;23(3):487–498.
3. Holistic admission in the health professions: findings from a national survey. Urban Universities for Health website. Available at http://urbanuniversitiesforhealth.org/media/documents/Holistic_Admissions_in_the_Health_Professions.pdf. Published 2014. Accessed December 5, 2018.
4. Wells A, Brunson D, Sinkford JC, Valachovic RW. Working with dental school admissions committees to enroll a more diverse student body. J Dent Educ. 2011;75(5):685–695.
5. Roadmap to excellence: key concepts for evaluating the impact of medical school holistic admissions. Association of American Medical Colleges website. Available at https://members.aamc.org/eweb/upload/Holistic%20Review%202013.pdf. Published 2013. Accessed December 5, 2018.
6. Holistic review: a quick primer. American Association of Colleges of Nursing website. Available at https://www.aacnnursing.org/Diversity-Inclusion/Holistic-Admissions. Published 2016. Accessed December 5, 2018.
7. Salvatori P. Reliability and validity of admissions tools used to select students for the health professions. Adv Health Sci Educ Theory Pract. 2001;6:159–175.
8. Glazer G, Startsman LF, Bankston K, Michaels J, Danek JC, Fair M. How many schools adopt interviews during the student admission process across the health professions in the United States of America? J Educ Eval Health Prof. 2016;13:12.
9. Rosenberg L, Perraud S, Willis L. The value of admission interviews in selecting accelerated second-degree baccalaureate nursing students. J Nurs Educ. 2007;46(9):413–416.
10. Scott LD, Zerwic J. Holistic review in admissions: a strategy to diversify the nursing workforce. Nurs Outlook. 2015;63(4):488–495.
11. Trice LB, Foster PH. Improving nursing school diversity through use of a group admission interview. AORN J. 2008;87(3):522–532.
12. Kreiter CD, Yin P, Solow C, Brennan RL. Investigating the reliability of the medical school admissions interview. Adv Health Sci Educ Theory Pract. 2004;9:147–159.
13. Patrick LE, Altmaier EM, Kuperman S, Ugolini K. A structured interview for medical school admission, phase 1: initial procedures and results. Acad Med. 2001;76(1):66–71.
14. Glazer G, Clark A, Bankston K, Danek J, Fair M, Michaels J. Holistic admissions in nursing: we can do this. J Prof Nurs. 2016;32(4):306–313.
15. Hendricks SM, Krothe JS. Outcomes and lessons learned regarding the use of interviewing for baccalaureate nursing school admission. J Prof Nurs. 2014;30(5):392–398.
16. Patterson F, Knight A, Dowell J, Nicholson S, Cousans F, Cleland J. How effective are selection methods in medical education? A systematic review. Med Educ. 2016;50(1):36–60.
17. Pau A, Jeevaratnam K, Chen YS, Fall AA, Khoo C, Nadarajah VD. The Multiple Mini-Interview (MMI) for student selection in health professions training—a systematic review. Med Teach. 2013;35(12):1027–1041.
18. Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: the Multiple Mini-Interview. Med Educ. 2004;38(3):314–326.
19. Eva KW, Reiter HI, Rosenfeld J, Trinh K, Wood TJ, Norman GR. Association between a medical school admission process using the Multiple Mini-Interview and national licensing examination scores. JAMA. 2012;308(21):2233–2240.
20. Reiter HI, Eva KW, Rosenfeld J, Norman GR. Multiple Mini-Interviews predict clerkship and licensing examination performance. Med Educ. 2007;41:378–384.
21. Alaki SM, Yamany IA, Shinawi LA, Hassan MH, Tekian A. Can Multiple Mini-Interviews predict academic performance of dental students? A two-year follow-up. J Dent Educ. 2016;80(11):1376–1383.
22. Foley JI, Hijazi K. The admissions process in a graduate-entry dental school: can we predict academic performance? Br Dent J. 2013;214(2):E4.
23. Gale J, Ooms A, Grant R, Paget K, Marks-Maran D. Student nurse selection and predictability of academic success: the Multiple Mini Interview project. Nurse Educ Today. 2016;40:123–127.
24. Perkins A, Burton L, Dray B, Elcock K. Evaluation of a Multiple-Mini-Interview protocol used as a selection tool for entry to an undergraduate nursing programme. Nurse Educ Today. 2013;33:465–469.
25. Kumar K, Roberts C, Rothnie I, du Fresne C, Walton M. Experiences of the Multiple Mini-Interview: a qualitative analysis. Med Educ. 2009;43(4):360–367.
26. McAndrew R, Ellis J. An evaluation of the Multiple Mini-Interview as a selection tool for dental students. Br Dent J. 2012;212(7):331–335.
27. Ross M, Walker I, Cooke L, et al. Are female applicants rated higher than males on the Multiple Mini-Interview? Findings From the University of Calgary? Acad Med. 2017;92(6):841–846.
28. Jerant A, Griffin E, Rainwater J, et al. Does applicant personality influence Multiple Mini-Interview performance and medical school acceptance offers? Acad Med. 2012;87(9):1250–1259.
29. Kim KJ, Kwon BS. Does the sequence of rotations in Multiple Mini Interview stations influence the candidates performance? Med Educ Online. 2018;23(1):1485433.
30. Phillips JM, Malone B. Increasing racial/ethnic diversity in nursing to reduce health disparities and achieve health equity. Public Health Rep. 2014;129(suppl 2):45–50.
31. Glazer G, Tobias B, Mentzel T. Increasing healthcare workforce diversity: urban universities as catalysts for change. J Prof Nurs. 2018;34(4):239–244.
Keywords:

admissions; diversity; holistic admissions; interview; nursing education

Copyright © 2020 Wolters Kluwer Health, Inc. All rights reserved