Medical school admissions have the difficult role of acting as the door to the profession. Less than half of all applicants are offered admission to a single medical school.1 However, once matriculated, academic attrition is exceedingly rare (< 2%).2 The medical school selection process is high stakes, is expensive, and has a significant power imbalance between most students and schools.3,4 The selection process could be improved by increasing the efficiency of admission mechanisms. An accurate model about which students are likely to enroll in a specific institution would improve the efficiency of medical school admissions. This increased efficiency can be seen most notably in financial benefits to both the applicant and the institution and in class composition control. Model creation can be guided by a series of important questions that reflect these benefits.
First, how do we maximize the return on financial investment per student by minimizing the costs of the selection process? The interview process is costly to medical school applicants and institutions, requiring significant time and expense. Recruitment in the form of “second look” visits creates further costs. Additionally, there is a set amount of financial aid and scholarship money offered as an incentive for a student to choose a school.
Second, how do we meet our institutional goals and values through intentional class composition? Underenrollment (not accepting adequate numbers of students) requires the admission of students from the waitlist instead of more aggressively recruiting students who may be more qualified. Additionally, some schools provide financial incentives to defer enrollment when there is overenrollment. These issues are even more pronounced where an institution is interested in meeting goals of diversity or in-state representation while maintaining selectivity. Admissions yield is defined as the number of enrolled students divided by the number of offered positions. A miscalculation can result in an undesired balance of students in a class. In the case of a public institution, if the yield of out-of-state students decreases, this can cause significant losses in tuition-based revenue. Similarly, a decrease in in-state student yield may result in negative political consequences. In balancing numerous enrollment goals simultaneously, having an accurate, real-time measurement tool of class composition is highly desirable.
Higher education institutions have addressed similar issues through the study of student enrollment decision making and the development of enrollment management (EM) techniques.5 EM is based on consideration of theories of student choice6 and bounded rational choice making.7 This combination of theories posits that students choose an institution on the basis of maximizing personal utility within a limited search behavior.8 The resultant student enrollment behavior is thus subject to both external and internal factors that can be observed.9 EM operationalizes these theories by incorporating multiple aspects of higher education processes into the admissions decision-making process. Through synergy in previously disparate areas, institutions can use EM to adapt their programs to their goals and needs.10 For example, EM has also been used to increase recruitment of women and underrepresented minority students to doctoral fields.11 Given the commitment to diversity of the medical profession,12 efforts to improve the fairness of the admissions processes,13 and the inability of novel admissions processes to counteract limitations in the pool of diverse candidates,14 EM should be considered as an adjunct to enhance diversity efforts in medical schools. Efficiencies in financial aid offer disbursements can result in more aid being available to promote diversity and equity goals, thereby increasing educational access.15 In EM, marketing, recruitment, and financial aid are included under a single administrative program because their separate processes did not allow for this fusion of purpose.16
EM techniques are pervasive in higher education across Carnegie Classification institutional types17; this includes tuition-dependent schools, Research I institutions, and graduate programs.18–20 Although the majority of the research and applications of EM have been in higher education, it is reasonable to consider its applicability in medical education. Even though some factors found to be important in undergraduate enrollment may not hold for medical school admissions (e.g., distance of the institution from the home of the applicant21), other factors such as demographic characteristics, scholarships, and in-state tuition are important in explaining students’ choice of schools. Extension to professional education is the next logical step for EM.
The purpose of this study was to investigate the application of EM techniques to the admissions process at the University of Michigan Medical School (U-M). We examined its usefulness in constructing a yearly class that included high academic achievers, an appropriate number of in-state students, and diversity along racial/ethnicity, personal life experiences, and socioeconomics dimensions. Our hypothesis was that preexisting data at U-M could be used to create a predictive model for decision makers.
To estimate the enrollment propensities of admitted students, we conducted retrospective data analysis of a single medical school’s admissions and enrollment process. Data included variables available at U-M and information submitted by students and contained in the American Medical College Application Service school report. The U-M institutional review board found the study to be IRB exempt. We used STATA 12 statistical software (StataCorp LP, College Station, Texas) for statistical analysis. The institutional database contained all applicants from 2006 to 2014, resulting in 41,796 applications with suitable data for analysis. During this time period, 3,453 applications were accepted (8.2%). For individuals who applied in multiple years, we included only the year they were admitted. As admittance was required for possible enrollment, all others were not considered.
EM is grounded in a student-choice theoretical framework.6 On the basis of this theory, the model should include individual factors such as competitiveness of a student’s application and also institutional factors (e.g., financial aid offer).22 For our analyses, independent categorical variables included gender, underrepresented minority in medical (URM) status as defined by the Association of American Medical Colleges,23 in-state residency, and institutional financial aid offered. The independent continuous variables were undergraduate grade point average (GPA), Medical College Admission Test (MCAT) average score, and the admissions committee score (ACS). The ACS is a composite of scores, references, previous experiences, and interview recommendations and is used to stratify applicants for admission. Applicants’ demographic information for these variables is presented in Table 1. We chose these factors on the basis of student-choice theory, prior EM research, and knowledge of medical school admissions.24 Other factors suggested by theory, such as issues of climate and student fit, were not available for inclusion in our models. Collection of these data and inclusion in future models is planned.
Binomial regression model
We considered multiple statistical models, and our preferred (based on model fit) specifications for the binomial and multinomial outcomes are discussed below. Assumptions for logistic regression were considered, and the data and our approach did not violate any of these requirements. The outcome variable for the binomial regression was, conditional on being accepted, whether an admitted student enrolled at U-M or not. The covariates included in the model were gender, URM status, undergraduate GPA, MCAT average, applicant residency status, and financial aid offer. The objective was to examine how important the covariates included were in explaining enrollment behavior and to provide an overall estimate of each student’s probability of enrollment to be used to aid decision making. This model consisted of all 3,453 accepted applicants.
Multinomial regression model
We used records from 2010 to 2014 for the multinomial regression because these included applicants’ specific institutional enrollment choices. Data prior to 2010 did not have specific institutional choices available for students who did not enroll at U-M. Data from these four years were for 1,349 individuals. The multinomial outcome variable included enrollment at U-M versus three alternative institution groupings. The first group consisted of two highly selective institutions that students often choose to enroll in instead of at U-M. The second group comprised 12 institutions that directly compete for students with U-M but do not enroll our accepted applicants at as high a rate as the previous group. The third category included all other institutions. We chose these categories on the basis of perceived peer institutions. We considered an additional category, in-state competitors to U-M, but it did not contain enough applicants enrolling elsewhere in the previous four years to be viable. Once again, the covariates included in the model were gender, URM status, undergraduate GPA, MCAT average, applicant residency status, and financial aid offer.
After model generation, an analytic “dashboard” was produced using Microsoft Visual Basic and Excel for practical application. The analytic dashboard was designed to allow the easy input of potential student factors and provide a predicted enrollment to a decision maker. In using the dashboard, the decision maker could vary the institutional financial aid offer and observe how enrollment behavior was expected to change in response.
Binomial regression model
In the binomial regression model all the covariates were statistically significant (P < .05) except two financial aid offer levels ($20,000–$30,000 and “named scholarships”) (Table 2). In this model, the regression constant was equivalent to a male, majority student, from out-of-state and with no financial aid offer. Applicants who were female and self-identified URM were less likely to enroll at U-M. Admissions with higher undergraduate GPAs, MCAT average scores, and ACSs were also less likely to enroll at U-M. Financial aid offers had differential effects on enrollment probabilities. For example, compared with students who received no aid, students who received a full scholarship had 1.53-fold higher odds of enrolling. Conversely, admissions who received $30,000 to $46,000 in aid had odds of enrolling that were actually smaller than those of their peers who received no aid.
Multinomial regression model
Estimation of the multinomial logistic regression allowed us to examine which competitors were most successful recruiting applicants away from U-M. This regression resulted in a statistically significant model (P < .001), and within each category of the outcome there were statistically significant covariates (P < .05) (Table 3). Being a member of a URM was associated with an increase in the odds of enrolling at the two most competitive peer groups: “two most competitive” (relative risk ratio [RRR] 2.28) and “direct competitors” (RRR 2.13). An increase in an applicant’s undergraduate GPA was associated with increased enrollment at the two most competitive peers of the U-M (RRR 4.38). Similarly, an increase in the applicant’s MCAT average score was associated with an increased likelihood of enrollment at another institution. For example, an increase from 11 to 12 in the MCAT average, with all other factors held constant, was associated with an increase in the likelihood of enrollment from 29.8% to 50.8% at the “two most competitive” (RRR 2.43), an increase from 54.0% to 69.4% in the likelihood of enrollment at a “direct competitor” (RRR 1.94), and an increase from 29.7% to 39.2% in the likelihood of enrollment at any other institution not included above (RRR 1.52). Accepted applicants who were considered in-state had a decreased likelihood of enrolling at an institution other than U-M: “two most competitive” (RRR 0.21), “direct competitors” (RRR 0.16), “all others” (RRR 0.13).
The amount of financial aid provided by U-M had a nonlinear association with enrollment. Small financial aid offers (less than $10,000; $10,000–$20,000) were associated with increased enrollment at U-M. Students were 0.16 times as likely to enroll at a “direct competitor” and 0.26 times as likely to enroll at an institution in the “other” group when offered less than $10,000 in institutional aid. Similar results were found for offers of $10,000 to $20,000: “direct competitors” (RRR 0.39) and “all others” (RRR 0.17). Larger institutional financial aid offers ($30,000–$46,000), but not full tuition offers, were associated with an increased enrollment at one of the “two most competitive” (RRR 2.76). A “full ride” (complete tuition support regardless of state residency) was associated with a decreased likelihood of enrolling at the “all other” schools (RRR 0.21).
We created an interactive decision-maker dashboard for both educative purposes and to demonstrate how adjustments in applicant characteristics and recruitment in the form of financial aid offers could affect the likelihood of enrollment in a dynamic manner. Although the financial aid offer is the primary method by which an institution can change the enrollment probability, the dashboard allows for manipulation of all variables to allow for consideration of both real and hypothetical applicants. Downloadable examples of this tool are available.25 Two static examples of the output of this program are provided in Figures 1 and 2.
EM has been shown to be useful across higher education and is therefore likely to be translatable to the medical school context. We studied the applicability of a higher educational model of informed decision making and manage ment in a new context. Using an EM approach provides insight into the medical selection process, and the benefits for such an approach are likely generalizable between medical schools. Our specific models were tailored to meet the needs of our recruitment and admissions office. The selection of specific competing institutions is not meant to be immediately generalizable to other schools but, in fact, is chosen to show how a given medical school could easily change the possible competing institutions to refine EM techniques to their specific needs.
Examination of the individual covariates in the model can provide insights into unexpected barriers to enrollment. First, URM enrollment is lower at U-M than at comparison institutions in both regression models. A reconsideration by U-M of the recruitment and enrollment planning may be necessary for URM applicants. A possible intervention could include increasing the number of offers of admission while assuming that the rate of enrollment remains constant in this applicant group. However, this is not an option at U-M given state law prohibiting race-based affirmative action. A second intervention could include a targeted financial aid offer approach, based on the results regarding both high and low amount offers. This modified financial aid approach would be applicable to all admitted students regardless of URM status. Finally, the lower enrollment rate of URM students is likely related to increased competition for these applicants. A lower rate secondary to competition could be tested through comparison with data from other medical schools.
EM results such as these can also prompt a reevaluation of climate and culture at an institution. U-M has an explicit commitment to the creation of a diverse class along multiple domains including racial/ethnic variation, socioeconomic variation, different intellectual interests, and experiential diversity. Our findings demonstrate that URM students are more likely to choose another institution than U-M. This could be due to signif icant competition from peer institutions, meaning the difference between admit tance and enrollment rates is similar in other medical schools. Alternatively, this could be a result of perceived issues of institutional culture here. Previous higher education studies have used similar approaches to demonstrate to stakeholders interested in promoting diversity that local institutional issues of class makeup may be less due to admissions decisions (i.e., who is admitted) and more to other factors such as the above (i.e., why aren’t they attending). Surveys distributed to accept ed students who elected to enroll elsewhere can help answer this question.
In the general context of all financial aid offers, the increased enrollment of students receiving small amounts of institutional aid from U-M likely represents a more generous offer from U-M versus other schools. Given the low amount of these offers and the assumption that cost is a discriminatory factor for students, it is likely that these students received little/no other local financial support from equally desirable options. Conversely, for applicants receiving high amounts of aid, but not full tuition support, the decreased likelihood of enrollment likely represents an aid offer inferior to competing peer institutions. Analysis of enrollment based on financial aid offers helps decision makers maximize the likelihood of attracting an admitted student. Our findings suggest differential effects in terms of the provision of financial aid. Use of EM models can predict the aid sensitivities of different groups of students and lead to more efficient and effective targeting of these scarce resources. For example, analysts can group students into categories based on their predicted probability of enrolling and simulate how changes in their aid awards affect these probabilities. Targeted aid to the groups for whom there is the largest change in enrollment probabilities can be offered. These groups can include specific target applicants of the institution, such as URM students, or targeted aid can be levied to fulfill other institutional goals. This approach has been successful in undergraduate EM and may have some utility for medical schools.26 Similar to students receiving high aid offers, the decreased likelihood of enrollment associated with increasing MCAT average is indicative of high-scoring students being in high demand. As might be expected with a public institution that has lower tuition for state residents, there is a strong association between in-state status and decision to enroll at U-M. Given the reputation of U-M as a selective school, in-state tuition offers a decreased cost of attendance compared with peer institutions out of state.
The model can also be used to predict the size and composition of an incoming class. Using EM analytics in this manner can allow a single institution the ability to “course correct” if they are concerned that they are not going to hit an admissions goal (e.g., size, selectivity, class diversity). Even adoption of the simple enrollment dashboard described here (Figure 1) would allow for a data-driven approach to class creation.27 Use of EM analytics allows for real-time adjustment in the students being selected to interview, recruitment of students off the waitlist, and changes in financial aid offers given. Whereas the content of medical education may be different from that of an undergraduate program, the class composition needs are similar. Successful implementation at other types of institutions bodes well for adaptation to medical education. Similarly, although EM techniques for prospective diversity recruitment are subject to the same laws as admissions in general, we found no instance where the use of EM techniques was in itself a separate legal issue. As such, EM must not be used to fulfill specific student quotas28 but, rather, as a means of understanding potential future student population characteristics. Whereas this study looked retrospectively at a single institution over eight years, future studies from U-M will look to validate the model prospectively with an ongoing partnership with the Admissions Office. Similarly, generalizability to other institutions, especially ones that are dissimilar in selectivity, institutional goals, or resources, may be limited given a single site; however, this weakness has not been observed in studies of EM in higher education.
We would advocate the expansion of EM to other schools to demonstrate both the usefulness of this technique and the specific differences that EM can identify between schools. Whereas this model fits our current data, EM is not a static intervention but is instead an ongoing process. Therefore, even as the model above shows good predictive ability, further study of practical validation through actual implementation is necessary in the future. Once introduced, EM requires continuous refinement and adjustment based on the observed behavior of both the applicants and institutions.
In conclusion, EM techniques can provide an evidence-based adjunct in admissions decisions and class composition creation. Dissemination to additional institutions and national scale data analysis could provide insight into the efficiency and nature of student selection to medical school.
1. Association of American Medical Colleges. MCAT Scores and GPAs for Applicants and Matriculants to U.S. Medical Schools, 2002–2013. 2013.Washington, DC: Association of American Medical Colleges.
2. Garrison G, Mikesell C, Mathew D. Medical school graduation and attrition rates. AAMC Analysis in Brief. April 2007;7:12.
5. Coomes MD. The historical roots of enrollment management. New Dir Stud Serv. 2000;89:518Spring.
6. Paulsen MB. College Choice: Understanding Student Enrollment Behavior. 1990. Washington, DC: George Washington University, School of Education and Human Development; ASHE-ERIC higher education report no. 6.
7. Jones BD. Bounded rationality. Annu Rev Polit Sci. 1999;2:297321.
8. Simon HA. McGuire CB, Radner R. Theories of bounded rationality. In: Decision and Organization. 1972:Vol 1. Amsterdam, Netherlands: North-Holland Publishing Company; 161176.
9. Kinzie J, Palmer M, Hayek J, Hossler D, Jacob SA, Cummings H. Fifty years of college choice: Social, political, and institutional influences on the decision-making process. Lumina. September 2004. http://files.eric.ed.gov/fulltext/ED484237.pdf
. Accessed February 14, 2016.
10. Bontrager B. Enrollment management: An introduction to concepts & structures. Coll Univ J. 2004;79:1116.
11. Perna LW. Understanding the decision to enroll in graduate school: Sex and racial/ethnic group differences. J Higher Educ. 2004;75:487527.
12. Kirch DG. Culture and the courage to change. Paper presented at: Association of American Medical Colleges Annual Meeting; November 4, 2007; Washington, DC.
13. Sebok SS, Luu K, Klinger DA. Psychometric properties of the multiple mini-interview used for medical admissions: Findings from generalizability and Rasch analyses. Adv Health Sci Educ Theory Pract. 2014;19:7184.
14. Reiter HI, Lockyer J, Ziola B, Courneya CA, Eva K; Canadian Multiple Mini-Interview Research Alliance (CaMMIRA). Should efforts in favor of medical student diversity be focused during admissions or farther upstream? Acad Med. 2012;87:443448.
15. Hossler D. The role of financial aid in enrollment management. New Dir Stud Serv. 2000;89:7790Spring.
16. Hossler D. Effective admissions recruitment. New Dir Higher Educ. 1999;108:1530Winter.
17. Shulman LS. The Carnegie Classification of Institutions of Higher Education. 2001.Menlo Park, Calif: Carnegie Publication.
18. Binder SF, Aldrich-Langen C. Fiscal and human resources to support enrollment management. New Dir Stud Serv. 1995;71:5572Fall.
20. Campbell CD, Smith J. Crossing the GEM frontier: Graduate admissions professionals’ participation in enrollment management. Coll Univ. 2014;89:311.
21. Leppel K. Logit estimation of a gravity model of the college enrollment decision. Res Higher Educ. 1993;34:387398.
22. DesJardins SL, Ahlburg DA, McCall BP. An integrated model of application, admission, enrollment, and financial aid. J Higher Educ. 2006;77:381429.
24. Edwards JC, Elam CL, Wagoner NE. An admission model for medical schools. Acad Med. 2001;76:12071212.
26. DesJardins SL. Assessing the effects of changing institutional aid policy. Res Higher Educ. 2001;42:653678.
27. Glover RH. Designing a decision-support system for enrollment management. Res Higher Educ. 1986;24:1534.
Copyright © 2016 by the Association of American Medical Colleges
28. DeVille K, Kopelman LM. Diversity, trust, and patient care: Affirmative action in medical education 25 years after Bakke. J Med Philos. 2003;28:489516.