And who, only studying
birdtracks in the sand,
could think those little forks
had decamped on the wind?
In 2015, the Association of American Medical Colleges (AAMC) introduced a new version of the Medical College Admission Test (MCAT) to assess knowledge and skills in natural, behavioral, and social sciences.2 The new test was intended to align with the adoption of competency-based curricula in medical schools and was devised to place greater weight on scientific reasoning and analysis rather than memorization of facts. The new MCAT exam is divided into 4 sections: chemical and physical foundations of biological systems; critical analysis and reasoning skills; biological and biochemical foundations of living systems; and psychological, social, and biological foundations of behavior.
The total number of questions increased in the new MCAT exam, and the scoring system was changed to create a broader, more normal (“bell-shaped”) distribution of scores.2 The top score on the old test was 45 points, while the top score on the new test is 528 points. A new MCAT score report was developed and provides scores for each of the 4 sections along with an overall score. Percentiles and “confidence” bands are shown on the report, and scores are centered such that half of test takers’ total scores are below 500 and half are above. The rollout of the new MCAT exam was advanced carefully, with recommendations to medical school admissions committees on how to evaluate candidates with data derived from old, new, or both MCAT exams.2
The new MCAT exam has been viewed favorably for encompassing domains of working knowledge, such as social determinants of human health and quantitative methods, that are increasingly important for physicians-in-training and future leaders and innovators in medicine.3–6 And, with its expanded content and emphasis on skills, the new MCAT exam is seen as a better instrument in support of holistic review by medical school admissions committees.3,6
The AAMC has encouraged admissions committees to consider test takers with new MCAT scores at the “top of the curve” rather than at the top of the scale.2 Such guidance is recognized for its importance in stemming a real or perceived practice of considering only top-scoring applicants, which advantages majority-identifying candidates and candidates with access to greater resources.4,7–9 In other words, the AAMC has suggested that admissions committees give full consideration to individuals across a wide range of subtest scores and with overall scores both above and below the 500 mark.2 This inclusive approach has been supported by education leaders seeking to strengthen representation in medicine.7–10
As we in academic medicine together endeavor to create a highly capable, representative, and effective physician workforce to serve the needs of the public, we should reflect on the proper role of the MCAT exam in medical education. The overvaluation, overreliance on, or misuse of test results when assessing candidates may bring unintended and undesired consequences, principally related to diversity and inclusion,9–12 an observation that has arisen elsewhere in higher education.13,14 Thinking about the utility of the MCAT exam, new or old, leads to consideration of optimal and fair approaches to selecting candidates whose academic factors and values align well with a school’s mission commitments.7,10 Such reflection encourages careful review of the attributes15 that define a capable medical student who will go on to become a physician who is effective and lives up to the expectations of the profession.16 The related issue of how these individual decisions at individual medical schools form, by accretion, the physician workforce of the future also deserves our attention.
What We Are Learning About the New MCAT Exam and Its Utility
In this issue of Academic Medicine, we present a number of peer-reviewed empirical reports17–20 and conceptual pieces7,8,21–24 that document the real-world performance of the new MCAT exam and explore important questions related to the role of tests in the selection of medical students. Violato et al20 present initial empirical evidence of the validity of the new MCAT exam, including incremental validity of the subtest entitled Psychological, Social, and Behavioral Foundations of Behavior. The authors highlight the need for additional study of the psychometric properties of the exam, given its “high-stakes” use in medical school admissions. Meanwhile, the AM Last Page features an infographic with advice for prospective test takers on how best to prepare for the MCAT exam.22
Articles in this issue of the journal8,17,19 present findings on the preparation strategies of new MCAT examinees from different backgrounds, patterns of scores, and key early training outcomes. The analyses performed by Busche et al19 show that multisite data on grade point averages and new MCAT scores, taken together, correlate well with student performance on summative examinations in the first year of medical school. In this study, students with a wide range of scores on the new MCAT exam successfully progressed to their second year of medical school. Moreover, rates of progression did not vary by the student’s gender, race/ethnicity, or by the highest educational achievement of the student’s parent(s). These data may offer reassurance to those who fear that accepting students with a broader range of MCAT scores may compromise intellectual rigor in medicine.
Salutary educational trajectories of medical students with scores in the middle third of the scoring range of the new MCAT exam (i.e., near-“top of the curve” scorers) were also documented by Terregino et al.8 The authors were also able to show that applicants in the middle third of the new MCAT scoring range had more diverse characteristics (e.g., having no parent with a bachelor’s degree, being underrepresented in medicine based on race/ethnicity, having grown up in a rural or medically underserved area) than did applicants in the top third. Medical schools that have accepted more applicants with midrange scores clearly have attained more diverse matriculating classes.
Do test takers from different groups perform differently on the new MCAT exam? In a study published in this issue, Girotti et al17 revealed a small gap between the average total scores of men (503.1 average) and scores of women (500.2 average) who were test takers on new MCAT in 2017. The authors found larger differences in average total MCAT scores when comparing test takers from higher-resource schools (502.1 average) to those from lower-resource schools (492.3 average). Comparisons based on race/ethnicity varied, with the largest gaps between test takers who identified as white (503.3 average) and test takers who identified as belonging to groups underrepresented in medicine, for example, black or African American (494.2 average) or Hispanic, Latino, or Spanish (496.7 average). As noted by the authors,
these mean differences in scores do not tell the whole story about the performance of examinees from different backgrounds … [as there] were examinees with low, middle, and high MCAT total scores in each [racial/ethnic] group.
Nevertheless, individuals who identify as belonging to groups underrepresented in medicine and applicants from lower-resource schools face distinct challenges when applying to medical school. These challenges may be especially salient and impactful when medical school admissions committees adopt strict “cutoff” scores. Girotti et al’s findings again highlight the importance of considering applicants across a wide range of total MCAT scores and remaining mindful of the various factors, independent of “readiness” for medical training, that may contribute to lower scores. The peer-reviewed empirical reports17–20 and conceptual pieces7,8,21–24 in this issue indicate the need for further research on this topic.
Translating What We are Learning—The Importance of Holistic Review
Reassuringly, the first few years of experience with the new MCAT exam suggest that it is a reliable test that appears to correlate with meaningful outcomes in medical training.19,20 For this reason, total scores and subtest scores on the new MCAT exam can be reasonably expected to continue to play a part in the holistic evaluation of applicants.10,24,25 Additionally, excluding MCAT data may disadvantage applicants whose grade point averages do not reflect their true academic capability and potential. Holistic review in medical school admissions is defined as
a flexible, individualized way of assessing an applicant’s capabilities by which balanced consideration is given to experiences, attributes, and academic metrics … and, when considered in combination, how the individual might contribute value as a medical student and future physician.26(p2)
Holistic review encompasses the traits of an ideal physician, including intellectual ability, commitment to service, cultural sensitivity, empathy, capacity for growth, emotional resilience, strength of character, interpersonal skills, and curiosity and engagement, as reflected in the academic record, personal history, essay, letters of recommendation, interview, and life experiences documented in the candidate’s application materials.16 Scores on the new MCAT exam are but one element of this rich portfolio.
Admissions committees that embrace best practices in holistic review and choose not to establish arbitrary “cutoff” test scores at the high end of the scoring range are required to complete a great deal of work. In 2019, U.S. medical schools received a total of 896,819 applications from prospective medical students.27 One medical school received 15,415 applications for its entering class of 189 individuals. Fifteen medical schools received more than 10,000 applications each. Despite the cost to applicants and the administrative burden to schools, this trend is likely to continue. The Princeton Review advises premedical students located in competitive states such as California to apply to 20 schools or more.28U.S. News & World Report suggests that students apply to no more than 50 schools.29 With application materials from so many candidates, it is natural that admissions committees might look for benchmarks to help lessen the burden of comprehensive review. The volume of applications, a dramatic problem of “scale,” causes admissions committees to overly rely on test scores. Further, MCAT scores can be a large factor in the ranking of medical schools by certain entities, such as U.S. News & World Report, which, in turn, may lead admissions committees to overvalue MCAT scores.30
But the true cost of non-holistic, strictly by-the-numbers practices is very dear, and it is not one that will be paid by those of us who already have a place in the field of medicine. Applicants and patients carry the burden. To be clear, psychometric evaluation of early data regarding the new MCAT exam does not suggest that the test itself is biased, but clearly educational opportunities, financial realities, and societal supports before the taking of a high-stakes examination such as an admissions test for medical school, law school, graduate school, or dentistry school are not equitable.7,13,23,31,32 Other fields are also coming to terms with this observation; for example, graduate programs at Brown, Princeton, and Yale are no longer requiring applicants to submit Graduate Record Examination scores, a movement referred to as “GRExit.”14,33–35
Admissions committees that focus solely on highest-achieving test takers inevitably advance an unfair system and will unfortunately miss the opportunity to help build a more inclusive and representative future physician workforce. More positively, admissions committees that carefully evaluate test takers with a wide range of scores on the new MCAT exam, based on data and analysis such as that presented in this issue of Academic Medicine, should be quite delighted with their students’ success and should be well positioned as they seek to select medical students from more diverse backgrounds. Moreover, individual schools that are more inclusive in the students they accept will, step by step, create a more multifaceted workforce that brings wider life experiences and greater creativity and awareness to the important issues affecting our world, such as health disparities.10
As illuminated in the beautiful poem by U.S. Poet Laureate Kay Ryan,1 one will never extrapolate the exquisite qualities of a bird in flight by looking only at its footprints in the sand. We must continue to approach the process of selecting our medical students with a sense of humility and the desire to always do better. MCAT scores are signals that should be appreciated for what they are: birdtracks in the sand.
MCAT scores reflect test takers’ cognitive strengths, yet, like grade point averages, letters of recommendation,36,37 and many other features of an applicants’ portfolio,38 scores can be influenced, sometimes unfairly, by context and circumstance.11–13 A test, even a very good one, is still just a test—a tool that has arisen in a certain context and possesses inherent properties, intended uses, and intrinsic limitations.
Authentic and richly informed holistic review, cognizant of the contingent dimensions of application materials, can do much to create a more complete view of the applicant. This more complete view, which includes data derived from the new, more comprehensive knowledge- and skills-oriented MCAT exam, along with the academic record, can certainly help in identifying applicants who fit well with a medical school’s curriculum and mission. The new MCAT exam can help with assessment of intellectual ability, while assessment of other ideal physician traits, such as commitment to service, cultural sensitivity, empathy, capacity for growth, emotional resilience, strength of character, interpersonal skills, and curiosity and engagement, requires much more depth. Holistic review in medical school admissions, undertaken in this thorough manner, will help individuals from all backgrounds—individuals who would do well in caring for patients and fulfilling key roles as leaders and innovators in society—fulfill their potential. And that, Dear Reader, will help ensure a more robust physician workforce of the future.
1. Ryan K. Paired things. In: Flamingo Watching. 1994.Providence, RI: Copper Beech Press.
2. Association of American Medical Colleges. The New Score Scales for the 2015 MCAT Exam: An Overview of What Admissions Officers Need to Know. 2014.Washington, DC: AAMC.
3. Kirch DG, Mitchell K, Ast C. The new 2015 MCAT: Testing competencies. JAMA. 2013;310:2243–2244.
4. Cohen JJ. Will changes in the MCAT and USMLE ensure that future physicians have what it takes? JAMA. 2013;310:2253–2254.
5. Sorrel AL. Wanted: Well-rounded physicians. Tex Med. 2015;111:24–31.
6. Roberts LW, Louie AK, Goldsmith M, et al. Elevating the behavioral and social sciences in premedical training: MCAT2015. Acad Psychiatry. 2015;39:127–131.
7. Lucey CR, Saguil A. The consequences of structural racism on MCAT scores and medical school admissions: The past is prologue. Acad Med. 2020;95:351–356.
8. Terregino CA, Saguil A, Price-Johnson T, Anachebe NF, Goodell K. The diversity and success of medical school applicants with scores in the middle third of the MCAT score scale. Acad Med. 2020;95:344–350.
9. Heller CA, Rúa SH, Mazumdar M, Moon JE, Bardes C, Gotto AM Jr.. Diversity efforts, admissions, and national rankings: Can we align priorities? Teach Learn Med. 2014;26:304–311.
10. Capers Q, McDougle L, Clinchot DM. Strategies for achieving diversity through medical school admissions. J Health Care Poor Underserved. 2018;29:9–18.
11. Kelly ME, O’Flynn S, Mulligan J, Murphey AW. A systematic review of stakeholder views of selection methods for medical school admission. BMC Medical Educ. 2018;18:139.
12. Ross DA. The Match: Magic versus the machine. J Graduate Medical Educ. 2019;11:274–276.
13. Gilmore H. Standardized testing and race: A reply to Professor Subotnik. Seattle J Social Justice. 2014;13:1–56.
14. Sealy L, Saunders C, Blume J, Chalkley R. The GRE over the entire range of scores lacks predictive ability for PhD outcomes in the biomedical sciences. PLoS One. 2019;14:e0201634.
15. Cohen JJ. Our compact with tomorrow’s doctors. Acad Med. 2002;77:475–480.
16. Witzburg RA, Sondheimer HM. Holistic review—Shaping the medical profession one applicant at a time. N Engl J Med. 2013;368:1565–1567.
17. Girotti JA, Chanatry JA, Clinchot DM, et al. Investigating group differences in examinees’ preparation for and performance on the new MCAT exam. Acad Med. 2020;95:365–374.
18. Lam JTH, Hanson MD, Martimianakis MAT. Exploring the socialization experiences of medical students from social science and humanities backgrounds. Acad Med. 2020;95:401–410.
19. Busche K, Elks ML, Hanson JT, et al. The validity of scores from the new MCAT exam in predicting student performance: Results from a multisite study. Acad Med. 2020;95:387–395.
20. Violato C, Gauer JL, Violato EM, Patel D. A study of the validity of the new MCAT exam. Acad Med. 2020;95:396–400.
21. Dewan MJ, Norcini JJ. We must graduate physicians, not doctors. Acad Med. 2020;95:336–339.
22. Swan Sein A, Cuffney F, Clinchot D. How to help students strategically prepare for the MCAT exam and learn foundational knowledge needed for medical school. Acad Med. 2020;95:484.
23. Poole KG Jr, Jordan BL, Bostwick JM. Mission drift: Are medical school admissions committees missing the mark on diversity? Acad Med. 2020;95:357–360.
24. Schwartzstein RM. Leveraging the medical school admissions process to foster a smart, humanistic, and diverse physician workforce. Acad Med. 2020;95:333–335.
25. Monroe A, Quinn E, Samuelson W, Dunleavy DM, Dowd KW. An overview of the medical school admission process and use of applicant data in decision making: What has changed since the 1980s? Acad Med. 2013;88:672–681.
26. Association of American Medical Colleges. AAMC Holistic Review Project. https://www.aamc.org/download/358700/data/hrp2-pager.pdf
. Published 2013. Accessed December 2, 2019.
27. Association of American Medical Colleges. Table A-1: U.S. medical school applications and matriculants by school, state of legal residence, and sex, 2019-2020. https://www.aamc.org/system/files/2019-11/2019_FACTS_Table_A-1.pdf
. Accessed December 2, 2019.
28. The Princeton Review. How many med schools should you apply to? https://www.princetonreview.com/med-school-advice/how-many-med-schools-should-you-apply-to
. Accessed December 2, 2019.
29. Marinelli R. How many medical schools should you apply to? U.S. News & World Report. https://www.usnews.com/education/blogs/medical-school-admissions-doctor/articles/2019-06-25/how-many-medical-schools-should-you-apply-to
. Published June 25,2019. Accessed December 2, 2019.
30. McGaghie WC. America’s best medical schools: A renewed critique of the U.S. News & World Report rankings. Acad Med. 2019;94:1264–1266.
31. Freeman BK, Landry A, Trevino R, Grande D, Shea JA. Understanding the leaky pipeline: Perceived barriers to pursuing a career in medicine or dentistry among underrepresented-in-medicine undergraduate students. Acad Med. 2016;91:987–993.
32. Davis D, Dorsey JK, Franks RD, Sackett PR, Searcy CA, Zhao X. Do racial and ethnic group differences in performance on the MCAT exam reflect test bias? Acad Med. 2013;88:593–602.
33. Spear S. Brown eliminates GRE test requirement for 24 doctoral programs. News from Brown. https://www.brown.edu/news/2019-10-03/gre
. Published October 3,2019. Accessed December 2, 2019.
34. Aronson E. Princeton eliminates GRE test requirements for 14 graduate programs. Princeton University. https://www.princeton.edu/news/2019/09/17/princeton-eliminates-gre-test-requirement-14-graduate-programs
. Published September 17,2019. Accessed December 2, 2019.
35. A note on removing the GRE from graduate admissions. Department of History. https://history.yale.edu/news/note-removing-gre-graduate-admissions
. Published May 23,2019. Accessed December 2, 2019.
36. Grimm LJ, Redmond RA, Campbell JC, Rosette AS. Gender and racial bias in radiology residency letters of recommendation. J Am Coll Radiol. 2020;17:64–71.
37. Saudek K, Saudek D, Treat R, Bartz P, Weigert R, Weisgerber M. Dear program director: Deciphering letters of recommendation. J Grad Med Educ. 2018;10:261–266.
38. White BA, Sadoski M, Thomas S, Shabahang M. Is the evaluation of the personal statement a reliable component of the general surgery residency application? J Surg Educ. 2012;69:340–343.