Schwartzstein, Richard M. MD; Rosenfeld, Gary C. PhD; Hilborn, Robert PhD; Oyewole, Saundra Herndon PhD; Mitchell, Karen PhD
Much has changed in both academic medicine and health care since the current version of the Medical College Admission Test (MCAT) was introduced in 1991. Scientific knowledge has exploded. Patients live longer and have more complicated medical problems. The ways in which physicians interact with their patients have altered radically. The populations of the United States and Canada are more diverse, with better health outcomes in the aggregate; however, health outcomes are more disparate when examined by population group.
These changes have important implications for medical education and the future physician workforce. They affect medical school curricula and teaching methods as well as criteria for admission. Medical school admission committees today are seeking applicants who have strong natural science backgrounds, solid foundations for learning about the social and behavioral determinants of health, strong critical thinking and problem-solving skills, and appreciation for the cultural differences that patients bring with them into the examining room.
In standardized testing, periodic reviews of examinations are considered a best practice, especially in fields with rapidly changing knowledge and practice patterns, like medicine.1 In this commentary, we discuss the development of the blueprint for the new MCAT exam that will be launched in 2015. We describe the deliberations that drove the redesign during the fifth comprehensive review of the MCAT exam (MR5). We focus more on the tensions and trade-offs that characterized the review and shaped the new exam rather than on the supporting data. Nonetheless, it is important to note that the blueprint for the MCAT2015 exam rests on a broad evidence base. It reflects advice from expert panels and recommendations from national reports about the competencies2 that entering medical students need to be prepared to learn and to succeed in medical school. It takes into account feedback on the current and proposed tests by participants at more than 90 outreach events and data from more than 2,700 surveys completed by faculty and administrators from U.S. and Canadian undergraduate institutions and medical schools. It was also informed by analyses of test takers’ work on the current exam and by course-taking data from medical school applicants. The qualitative and quantitative data that we and our colleagues on the MR5 Committee gathered, and the ways in which we used these data in developing the new exam’s blueprint, are described in Appendixes 1 and 2.
These datasets and our conversations as a committee reflected the many objectives and varying priorities of the MCAT testing program’s different stakeholder groups, including prospective examinees, medical school administrators and faculty, prehealth advisors and baccalaureate faculty, and members of disciplinary societies and higher education organizations. Many of these groups view the MCAT exam as a powerful lever for change in medical and prehealth education. Our work made it clear, however, that these groups’ numerous and differing goals could not be met without constructing an exam to test many more things than is reasonable.
For example, whereas some stakeholders called for the addition of biochemistry, cellular/molecular biology, and genetics concepts,2 others worried about the increased undergraduate course-taking burden that testing these concepts would imply. Some called for assessing foundational knowledge in the social and behavioral sciences,3 whereas others were concerned about undergraduate institutions’ capacity to increase enrollments in social and behavioral sciences courses to accommodate prehealth students. Many stakeholders recognized the difficulty of using a standardized test to increase diversity in medical school admissions, but, even so, they hoped that the new exam would support efforts to increase racial, ethnic, socioeconomic, and other types of diversity.
The tensions created by these and other competing interests made the MCAT2015 design process complex. We had to make compromises in order to balance the limitations of a single exam with the need to address the goals and concerns of a diverse student population and collegiate system. Throughout the process, we had to address the trade-off between content coverage and exam duration as each concept included had to be subject to a minimum number of questions to assess applicants’ capabilities reliably. Our goal was to design an exam that
* responds to the recommendations of the Scientific Foundation for Future Physicians (SFFP) Committee2 regarding the natural sciences competencies that are needed by entering medical students;
* tests competencies that provide a solid foundation for learning in medical school about the human and social aspects of medicine3;
* asks examinees to demonstrate that they can use their knowledge of the natural sciences and the social and behavioral sciences to solve problems that call for scientific reasoning and research skills;
* communicates the expectation that students who are preparing for medical school should read broadly in the humanities and social sciences; and
* balances the assessment of a broad range of competencies in the natural sciences, competencies in the social and behavioral sciences, and critical analysis and reasoning skills.
Below, we discuss the compromises we made and the decision rules we followed as the MR5 Committee worked to design the MCAT2015 exam to meet each of these five goals.
Goal 1: Test Natural Sciences Competencies
In 2009, the SFFP Committee2 released a report describing the natural sciences competencies with which students should enter medical school and those with which they should graduate. The committee highlighted the increasingly rapid rate at which new knowledge revises our understanding of the natural sciences fundamental to medicine and argued that future physicians must be scientifically inquisitive and possess the knowledge, skills, and habits of mind that will enable them to integrate new scientific discoveries into their work. The committee suggested that focusing on competencies, rather than traditional course requirements, would allow baccalaureate faculty to develop innovative interdisciplinary and integrative courses to help students build strong foundations in the natural sciences. Many of the SFFP Committee’s recommendations mirrored the recommendations of other recent seminal reports in natural sciences education.4–6
The SFFP recommendations, together with other recommendations from the literature and feedback from MCAT stakeholders at outreach events, helped shape our surveys of medical school and baccalaureate faculty concerning content to include on the new exam. We collected more than 2,700 completed surveys.7 Medical school faculty rated the importance of candidate natural sciences concepts to entering students’ success in medical school. Baccalaureate faculty described the levels at which these concepts are taught in their institutions’ introductory core course sequences in biology, organic and general chemistry, physics, biochemistry, and cellular/molecular biology. In addition, we analyzed course-taking data for applicants to U.S. medical schools, which we drew from the Association of American Medical College’s medical school applicant database.8 (Data were not centrally available for applicants to Canadian medical schools.)
When we examined the concepts to which medical school faculty members gave high importance ratings, our results suggested a larger testing domain in the natural sciences than could reasonably be covered on the new exam. We determined that incorporating all of the highly rated concepts would stretch the new exam’s limits and duration.
In addition, expanding the testing domain to include all of the highly rated concepts in biochemistry and cellular/molecular biology would increase the number of premedical courses needed to prepare for the exam. Although the SFFP Committee2 encouraged the development of integrative college-level courses that would include many of these competencies, baccalaureate faculty participating in MCAT outreach events highlighted the difficulty of teaching in this integrated model, particularly at smaller or relatively underresourced schools. They also pointed to the need for textbooks written explicitly to support an integrated approach.
Because 80% of MD-granting medical schools in the United States and Canada require or recommend a biochemistry course for admission,9 and two-thirds of applicants to U.S. medical schools already take biochemistry,8 we decided to include biochemistry concepts as they are taught in first-semester biochemistry courses. In contrast, few medical schools require or recommend cellular/molecular biology courses,9 and few applicants take them,8 so we decided to limit the testing of such concepts to those taught in introductory biology courses. (Many introductory biology courses now devote a significant amount of time to concepts in cellular/molecular biology.)
The MCAT2015 exam will include two natural sciences sections: the Biological and Biochemical Foundations of Living Systems and the Chemical and Physical Foundations of Biological Systems. Together, these sections will test five foundational concepts10:
1. Biomolecules have unique properties that determine how they contribute to the structure and function of cells and how they participate in the processes necessary to maintain life.
2. Highly organized assemblies of molecules, cells, and organs interact to carry out the functions of living organisms.
3. Complex systems of tissues and organs sense the internal and external environments of multicellular organisms and, through integrated functioning, maintain a stable internal environment within an ever-changing external environment.
4. Complex living organisms transport materials, sense their environment, process signals, and respond to changes using processes understood in terms of physical principles.
5. The principles that govern chemical interactions and reactions form the basis for a broader understanding of the molecular dynamics of living systems.
Goal 2: Test Foundations for Learning About the Human and Social Aspects of Medicine
In 2011, the Behavioral and Social Science Foundations for Future Physicians (BSSFFP) Committee released a report on preparing medical students and doctors to deal with the human and social issues of medicine.11 Building on the recommendations of the Institute of Medicine’s (IOM’s) Improving Medical Education report,12 the BSSFFP Committee described the critical roles that behavioral and sociocultural factors play in health and illness and the ways in which they interact with biological factors to influence health outcomes. The committee emphasized the importance of behavioral- and social-science-derived competencies to the effective practice of medicine. It recommended that students enter medical school with the foundational knowledge necessary to learn about the behavioral and sociocultural determinants of health, concepts that are receiving increasing attention in medical school curricula.
Input from stakeholders at MCAT outreach events reinforced these recommendations, as did data from our surveys of admission and academic affairs officers, who rated disciplinary knowledge in psychology, sociology, cultural studies, and public health as important or somewhat important to entering medical students’ success.13,14
Just as in the natural sciences, a larger testing domain was suggested than was manageable. Including concepts from psychology, sociology, population health, anthropology, cultural studies, economics, geography, cognitive science, and other disciplines3 would have resulted in an unacceptably long test. In addition, it would have been difficult to build test forms for the MCAT2015 exam that are equivalent to each other in content coverage and difficulty. Importantly, applicants would have had difficulty preparing for an exam that covered such a wide range of competencies.
We ultimately decided to add a section focused on the concepts taught in introductory psychology and sociology courses. These concepts are linked to several of the IOM’s “high priority” areas in the behavioral and social sciences,12 as well as areas discussed in the BSSFFP report.11 This decision was bolstered by admission requirements and applicant course-taking data: Half of the MD-granting medical schools in the United States and Canada require or recommend a course in the social or behavioral sciences for admission,9 whereas about two-thirds of applicants to U.S. medical schools take introductory psychology and one-third take introductory sociology.8 Many undergraduate institutions require or make available introductory psychology and sociology courses as part of their core curricula, which might ease some of the burden of preparing for the new test section.
The new Psychological, Social, and Biological Foundations of Behavior section of the MCAT2015 exam will be organized around five foundational concepts10:
1. Biological, psychological, and sociocultural factors influence the ways that individuals perceive, think about, and react to the world.
2. Biological, psychological, and sociocultural factors influence behavior and behavior change.
3. Biological, psychological, and sociocultural factors influence how we think about ourselves and others.
4. Social and cultural differences influence well-being.
5. Social stratification affects access to resources and well-being.
Goal 3: Test Scientific Inquiry and Research Skills
Like the SFFP and BSSFFP Committees, the MR5 Committee recognized the importance of scientific inquiry and research skills to medical students’ and physicians’ success. Medical school faculty want to be assured that students can use their knowledge about the natural sciences and the social and behavioral sciences to solve problems that call for scientific reasoning. Using data from our survey of medical school faculty and the recommendations of the SFFP and BSSFFP Committees and others,2,4–7,11–14 we identified the scientific inquiry, research, and statistics concepts and skills that are important to test for prospective medical students.
We designed a conceptual framework for testing these in the two natural sciences sections and in the Psychological, Social, and Biological Foundations of Behavior section. The MCAT2015 exam will ask examinees to demonstrate10
* knowledge of scientific concepts and principles by showing their understanding of scientific principles and by identifying the relationships between closely related concepts;
* scientific reasoning and problem solving by reasoning with scientific principles, theories, and models and by analyzing and evaluating scientific explanations and predictions;
* reasoning about the design and execution of research by demonstrating their understanding of important components of scientific research and by reasoning about ethical issues in research; and
* data-based and statistical reasoning by interpreting patterns in data presented in tables, figures, and graphs and by reasoning about data and drawing conclusions from them.
The research methods and statistics concepts that medical school faculty rated as important to success in medical school included basic probability, measures of central tendency, measures of variability, confidence intervals, statistical significance levels, graphical presentation of data, research ethics, hypothesis formulation, independent and dependent variables, hypothesis testing, and reporting research results.7 To demonstrate the third and fourth skills in the conceptual framework above, examinees will need a basic understanding of these concepts, which are discussed in many introductory psychology and sociology courses. Further, baccalaureate faculty survey data confirmed their importance to undergraduate students’ work in introductory biology, chemistry, physics, and biochemistry courses.7 Therefore, examinees should not need additional targeted course work in research methods or statistics to prepare for the new exam.
Goal 4: Communicate the Need to Read Broadly in the Humanities and Social Sciences
At the beginning of our deliberations, we reviewed data on the predictive validity of the Verbal Reasoning (VR) section of the current MCAT exam. These data showed that VR scores predict a range of academic outcomes in medical school, including grades, performance on licensure examinations, and the numbers of years required to complete undergraduate medical education.15–18 Data from the admission and academic affairs officer surveys,13,14 input from colleagues on the Holistic Review Project Advisory Committee,19 and feedback from stakeholders at outreach events also suggested that testing analysis and reasoning skills on the MCAT exam was of continuing importance.
In addition, we reviewed predictive validity data for the Writing Sample (WS) section of the current exam as well as survey data about admission officers’ use of WS scores.13,20,21 Validity data showed that WS scores add little to the prediction of medical student outcomes once applicants’ VR scores and undergraduate grades are considered. Survey data showed that many admission officers only use WS scores in making decisions about a small percentage of their applicants (primarily those applicants whose application essays and interviews suggest communication difficulties and those with modest VR scores). Given these data, we decided not to include the WS section on the new exam.
Our outreach and survey data highlighted ethics, philosophy, cultural studies, and population health as important new disciplines to include. With the addition of biochemistry, psychology, and sociology concepts, however, we were concerned about the impact that testing more subjects would have on examinees. For example, the time and tuition that might be needed to prepare for the exam could disadvantage, in particular, examinees from lower socioeconomic backgrounds. Similarly, examinees from “alternative backgrounds” (e.g., individuals who decide later in life to pursue a medical career or begin their undergraduate work in community colleges) might have difficulty completing the course work necessary to perform well.
We therefore decided to test examinees’ critical analysis and reasoning skills using passages from a wide range of humanities and social science disciplines but without requiring specific knowledge of particular disciplines. The new Critical Analysis and Reasoning Skills section will replace the VR section. It will test examinees’ skill at comprehending and analyzing what they read, drawing inferences from text, and applying the arguments and ideas that the passages describe to new situations. It will address stakeholders’ priorities by including passages from the fields of ethics, philosophy, cultural studies, and population health. We expect that our drawing attention to these disciplines will encourage examinees to familiarize themselves with the issues and arguments these disciplines raise and to read broadly in preparation for medical school.
Additional details about this and each of the other sections described in this commentary are available in the Preview Guide for the MCAT2015 Exam.10 That resource also provides sample questions.
Goal 5: Balance Testing in the Natural, Social, and Behavioral Sciences and Critical Analysis and Reasoning
We believe that the MR5 Committee’s recommendations for the MCAT2015 exam preserve what works about the current exam, eliminate what does not, and enrich the exam by adding concepts that will help ensure that future medical students are adequately prepared in key competencies in the natural sciences and the social and behavioral sciences. They also recognize the importance of critical analysis and reasoning skills. At the same time, our recommendations are mindful of the need for a diverse physician workforce and of the practical challenges of adapting baccalaureate curricula and applicants’ course taking to meet these new expectations.
Recommendations for Admission Committees
In the MR5 Committee’s deliberations, we and our colleagues recognized many paths to achieving the competencies that the MCAT2015 exam will test. We are heartened by the baccalaureate faculty who are developing multidisciplinary, competency-based courses that will make it possible for prospective medical students to efficiently prepare for the new MCAT exam and medical school. Examples of such courses include Integrated Introduction to the Life Sciences: Chemistry, Molecular Biology, and Cell Biology at Harvard University22; the Transformation in Medical Education Initiative at the University of Texas23; and the Howard Hughes Medical Institute’s National Experiment in Undergraduate Science Education at Purdue University, the University of Maryland at Baltimore County, the University of Maryland at College Park, and the University of Miami.24
It is important, however, to recognize that some colleges and universities are not as well positioned to examine their curricula and refine them. It would be unrealistic to expect that faculty everywhere have the resources needed to rethink what and how they teach. Indeed, more than 300 undergraduate institutions in the United States do not have prehealth advisors on staff. Further, some state schools and minority-serving institutions have been hard hit by the downturn in the U.S. economy.
Better communication between undergraduate colleges and medical schools about the content and focus of existing courses and their alignment with the competencies is important. In addition, we encourage medical school admission committees to continue to be flexible as they review applicants’ portfolios. They should carefully review the transcripts of applicants who have not completed all of the standard prerequisite courses or have taken integrated courses that have unusual titles. They should consider applicants who have taken alternate paths to touchstone courses such as biochemistry and/or have credits from community colleges, online courses, and other learning alternatives. We acknowledge that such flexibility in reviewing the credentials of applicants who have followed different academic paths will be difficult. This flexibility is, however, an additional, critical aspect of the holistic processes that admission committees already use to review applicants’ academic, personal, and experiential credentials in ways that reflect their institutions’ educational, scientific, clinical, and service-oriented goals.
The production of doctors is a complex process that incorporates experiences and course work in college and medical school, as well as training in residency programs. To the degree that the MCAT exam influences the college experiences of applicants and is predictive of their success in medical school, the exam may play a positive role in shaping the future physician workforce.
In the 21st century, physicians have to be “Renaissance” people with expertise in areas as different as psychology, biology, statistics, sociology, economics, culture, and communication skills. With the explosion of medical knowledge and the technologies that make information easily available, physicians must be able to find and evaluate information and to think critically when applying that information to solve their patients’ problems. We believe that the MCAT2015 exam’s broader focus, emphasis on analytical reasoning, and adoption of scientific competencies will help foster the development of such physicians.
MR5 Committee. Chair and vice chair: Steven G. Gabbe, MD, Ohio State University Wexner Medical Center; Ronald D. Franks, MD, University of South Alabama. Committee members: Lisa T. Alty, PhD, Washington and Lee University; Dwight Davis, MD, Pennsylvania State University College of Medicine; J. Kevin Dorsey, MD, PhD, SIU School of Medicine; Michael J. Friedlander, PhD, Virginia Tech Carilion Research Institute; Robert Hilborn, PhD, American Association of Physics Teachers; Barry A. Hong, PhD, MDiv, Washington University in St. Louis School of Medicine; Richard Lewis, PhD, Pomona College; Maria F. Lima, PhD, Meharry Medical College; Catherine R. Lucey, MD, University of California, San Francisco, School of Medicine; Alicia Monroe, MD, University of South Florida College of Medicine; Saundra Herndon Oyewole, PhD, Trinity Washington University; Erin A. Quinn, PhD, MEd, Community Memorial Health System and Keck School of Medicine, University of Southern California; Richard K. Riegelman, MD, PhD, George Washington University School of Public Health and Health Services; Gary C. Rosenfeld, PhD, University of Texas Medical School at Houston; Wayne M. Samuelson, MD, University of Utah School of Medicine; Richard M. Schwartzstein, MD, Harvard Medical School and Beth Israel Deaconess Medical Center; Maureen Shandling, MD, Mount Sinai Hospital and Faculty of Medicine, University of Toronto; Catherine (Katie) Spina, MD/PhD candidate, Boston University Medical School; Ricci Sylla, MD, Kaiser Permanente Santa Clara.
Acknowledgments: The authors and committee chairs would like to thank their colleagues on the MR5 Committee for their dedication and their spirit of collaboration. Many of them are authors of articles that appear in this issue of Academic Medicine. The others contributed in vital ways to the ideas these articles describe. The authors and chairs thank them for their work on these articles and, more important, for their hard work in shaping the new MCAT exam that will play a part in selecting tomorrow’s physicians. The authors and chairs would also like to thank Paul Sackett and James Pellegrino for their expert advice during the review. Additionally, they want to thank two anonymous reviewers for their suggestions, which greatly improved the manuscript.
Other disclosures: None.
Ethical approval: Not applicable.
1. American Educational Research Association; American Psychological Association; National Council on Measurement in Education. Standards for Educational and Psychological Testing. 19992nd ed Washington, DC American Educational Research Association
2. Association of American Medical Colleges; Howard Hughes Medical Institute. Scientific Foundations for Future Physicians. 2009 Washington, DC Association of American Medical Colleges
3. Kaplan RM, Satterfield JM, Kington RS. Building a better physician—The case for the new MCAT. N Engl J Med. 2012;366:1265–1268
4. American Association for the Advancement of Science. Vision and Change in Undergraduate Biology Education—A Call to Action. 2009 Washington, DC American Association for the Advancement of Science
5. National Research Council. A Framework for K–12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. 2011 Washington, DC National Academies Press
6. College Board. Science, College Board Standards for College Success. 2009 New York, NY College Board
7. Association of American Medical Colleges. MR5 Content Review Study of Natural Sciences for the Medical College Admission Test. 2011 Washington, DC Association of American Medical Colleges
8. Association of American Medical Colleges. AAMC Data Warehouse, Medical School Applicant Database. 2009 Washington, DC Association of American Medical Colleges ; November
9. Association of American Medical Colleges. Medical School Admissions Requirements, 2009–2010. 2009 Washington, DC Association of American Medical Colleges
11. Association of American Medical Colleges. Behavioral and Social Science Foundations for Future Physicians. 2011 Washington, DC Association of American Medical Colleges
12. Institute of Medicine. Improving Medical Education: Enhancing the Behavioral and Social Science Content of Medical School Curricula. 2004 Washington, DC National Academies Press
13. Association of American Medical Colleges. Study of Medical School Admissions Policies and Practices [unpublished report]. 2008 Washington, DC Association of American Medical Colleges
14. Association of American Medical Colleges. Study of Academic Affairs Officers’ Judgments About the Attributes Required for Student Success in Medical School [unpublished report]. 2009 Washington, DC Association of American Medical Colleges
15. Violato C, Donnon T. Does the Medical College Admission Test predict clinical reasoning skills? A longitudinal study employing the Medical Council of Canada clinical reasoning examination. Acad Med. 2005;80(10 suppl):S14–S16
16. Donnen TP, Elizabeth O, Violato C. The predictive validity of the MCAT for medical school performance and medical board licensing examinations: A meta-analysis of the published research. Acad Med. 2007;82:100–106
17. Callahan CA, Hojat M, Veloski J, Erdmann JB, Gonnella JS. The predictive validity of three versions of the MCAT in relation to performance in medical school, residency, and licensing examinations: A longitudinal study of 36 classes of Jefferson Medical College. Acad Med. 2010;85:980–987
18. Julian ER. Validity of the Medical College Admission Test for predicting medical school performance. Acad Med. 2005;80:910–917
19. Addams AN, Bletzinger RB, Sondheimer HM, White SE, Johnson LM Roadmap to Diversity: Integrating Holistic Review Practices Into Medical School Admission Processes. 2010 Washington, DC Association of American Medical Colleges
20. Association of American Medical Colleges. AAMC Data Warehouse, Medical College Admission Test (MCAT) Examinee Database. 2009 Washington, DC Association of American Medical Colleges ; October
21. Association of American Medical Colleges. AAMC Data Warehouse, National Board of Medical Examiners (NBME) Database. October 2009 Washington, DC Association of American Medical Colleges