Secondary Logo

Journal Logo

The Current Use of United States Medical Licensing Examination Step 1 Scores: Holistic Admissions and Student Well-Being Are in the Balance

Moynahan, Kevin F., MD

doi: 10.1097/ACM.0000000000002101
Invited Commentaries
Free
SDC

United States Medical Licensing Examination Step 1 scores increasingly are being used by graduate medical education programs to “screen out” applicants to invite for an interview; medical students with scores below a certain (usually unpublished) point are not considered for an interview. Conversely, many students are extended an interview invitation based solely on their Step 1 scores, prior to the release of their Medical Student Performance Evaluation. In this Invited Commentary, the author discusses why this practice has gained popularity as well as the unintended effects it has had in multiple domains—holistic undergraduate medical education admissions practices, student well-being, and medical school curricula. Possible solutions are considered, including a pass/fail reporting structure, a national basic science curriculum, and better alignment between undergraduate and graduate medical education admissions processes and values. Through collaborative work between the Association of American Medical Colleges, the National Board of Medical Examiners, the Accreditation Council for Graduate Medical Education, and medical educators, an alternative, more holistic standardized metric by which to compare students’ applications should be developed.

K.F. Moynahan is deputy dean for education and professor of medicine, University of Arizona College of Medicine–Tucson, Tucson, Arizona.

Funding/Support: None reported.

Other disclosures: None reported.

Ethical approval: Reported as not applicable.

Correspondence should be addressed to Kevin F. Moynahan, University of Arizona College of Medicine–Tucson, 1501 N. Campbell Ave., Tucson, AZ 85724-5017; telephone: (520) 626-6505; e-mail: moynahan@email.arizona.edu.

“My life is over,” she told me tearfully. This statement was not from one of my patients but, rather, from one of my medical students. In fact, it was from one of my most promising students, one who effortlessly combined the traits she will need to be an excellent physician, including intelligence, empathy, excellent clinical skills, and outstanding communication skills. The fact that her statement was in reference to a United States Medical Licensing Examination (USMLE) Step 1 score that was passing but below her expectation will not surprise anyone who works closely with medical students in today’s environment. The same scenario plays out for multiple students every year at every medical school accredited by the Liaison Committee on Medical Education in the United States. How did we get here, what are the consequences, and what are some possible solutions?

A call for reevaluating the purpose and use of USMLE Step 1 scores in medical education is hardly new.1 By now, it is widely accepted that, as it is a threshold exam used to ensure that students have the appropriate basic science knowledge before progressing, a passing Step 1 score has little to no predictive value in terms of how good of a physician a student will become.2 Nonetheless, Step 1 scores are used by many residency programs in the United States to “screen out” and select applicants for interviews. It is not hard to understand why this practice occurs. Much has been written about the increasing numbers of graduate medical education (GME) applications per student (an increase of 50% between 2005 and 2015),3 perceived increased competition for residency positions given the increasing number of medical students and relatively static number of GME positions, increasing prevalence of pass/fail grading in basic science curricula, and the heterogeneity and limited utility of the Medical Student Performance Evaluation (MSPE) to GME program directors in their selection of students to interview and accept. These factors are coupled with the pragmatic and understandable desire of GME program directors to select students using a known metric (e.g., past performance on standardized multiple-choice exams) with the expectation that students who obtain high scores on the Step 1 exam will have little difficulty passing their specialty board certifying exam.

What are the unintended consequences of using Step 1 scores in a way that was not intended and for which the test was not designed? Why are medical school leaders and students so concerned? Perhaps the most important reason is the incompatibility of the current use of Step 1 scores with the holistic admissions processes championed by the Association of American Medical Colleges (AAMC) and embraced by many medical schools in the United States. The rationale for holistic admissions is sound and multifaceted, and it includes efforts to diversify our physician workforce. As noted by the AAMC, “With the nation’s population growing and becoming increasingly diverse, it is crucial that the physician workforce reflect the changing demographics of the country to mitigate racial, ethnic, and socioeconomic health disparities.”4 Holistic admissions policies have been successful at starting to diversify the medical student body at many medical schools. Many students chosen through a holistic admissions process come from underrepresented populations in medicine or minority communities; evidence suggests that such students may not perform as well on standardized tests compared with their nonminority peers and thus may be more likely to be denied an interview.5 Of course, students chosen through a holistic admissions process are not selected for their test-taking ability; past a certain threshold of Medical College Admission Test (MCAT) score and grade point average, characteristics such as distance traveled, life experiences, resiliency, communication skills, and desire to practice in underserved communities take precedence. However, these same students who are carefully admitted to medical school through these holistic processes soon learn that their future will be heavily influenced by a single score on a standardized multiple-choice test; it is the ultimate “bait and switch.”

In addition to undermining the holistic admissions process, the current use of Step 1 scores has other problematic unintended consequences, including significant adverse effects on students’ well-being and mental health. The large percentage of medical students (and unfortunately, all health care providers) who suffer from depression, anxiety, and burnout is well documented.6 In my ample experience, mental health disorders are significantly amplified and more likely to occur in the months preceding sitting for the Step 1 exam as students prepare for a test that will significantly impact their future in medicine, including their specialty choice and where they will be accepted for GME training.

Furthermore, medical school curricula are unduly influenced by the Step 1 exam. Although most medical educators are aware of the content of the Step 1 exam from an outline provided by the National Board of Medical Examiners (NBME) and, ironically, from commercial resources used by students, attempts to design a curriculum that is centered around what a school’s faculty believe are the most important concepts rather than what is thought to be on the Step 1 exam are undermined. Currently, students largely dismiss as irrelevant content in the curriculum that they believe is not included on the Step 1 exam due to their focused attention on this exam. Despite this flawed logic, it is easy to understand why they feel this way. Students are required to learn and memorize vast, even overwhelming, amounts of information and are wary of any content that they perceive as not included on the Step 1 exam or as irrelevant to their preparation for it.

At many medical schools, the curriculum presented in the months prior to the Step 1 exam might as well not exist; students put almost all of their effort into studying for the Step 1 exam. They routinely tell us that they feel there are two separate curricula they have to negotiate—their school’s curriculum and a separate one for the Step 1 exam. The never-ending stream of Step 1 study aids and programs that our students spend increasing amounts of money on are a testament to this second curriculum. This industry seems to be growing at a rapid pace, given the solicitations my school receives. Some students, despite increasingly high tuition costs and individual debt, completely eschew portions of their school’s curriculum in favor of an outsourced commercial curriculum focused on the Step 1 exam.

Of course, it is not enough to voice angst over how Step 1 scores are used; creative solutions need to be considered, vetted, and implemented. What might some of these solutions be? Some have advocated a pass/fail reporting structure. Certainly this change would be consistent with how the exam was designed and with how other licensing exams are interpreted, such as the USMLE Step 3 exam and specialty board exams. As a physician, one is either board certified or not; the actual score is not relevant to one’s employability. Some, including those who are at odds with the current use of Step 1 scores,7 have disagreed with implementing a pass/fail reporting structure, stating that Step 1 scores are important for individual and program improvement. Ideally, scores on Step 1 would be available to the examinee as they are for all other certifying exams but not meaningful to residency program directors, assuming a passing score is attained. A hybrid approach might entail reporting scores and detailed test metrics to students but not to program directors until after the Match.

A change in how the success of GME programs is measured may help; an innovative method to judge the success of GME programs has been advocated. In such a system, a program would not be judged by the board pass rates of its residents but, rather, by the improvement they show on their board scores over board score projections based on their NBME exam scores. A similar approach may be used for undergraduate medical education (UME) as well, wherein students’ Step 1 scores are interpreted in light of their MCAT scores and their relative improvement.

However, I believe that any approach that does not use actual Step 1 scores will not gain acceptance until GME program directors are given an alternative, more holistic standardized metric by which to compare students’ applications. This challenge needs to be explored by medical schools, GME programs, the AAMC, and the Accreditation Council for Graduate Medical Education (ACGME). Equivalent reviews of numerous applications have been done at a number of medical schools, which receive thousands of applications for few slots; recognizing that resources are required, why can this process not be used in GME programs? A holistic tool that compares students in a way that allows GME program directors to choose applicants who are a good fit for the mission of their program will be challenging to construct and validate. A first step, currently being championed by the AAMC, is to standardize the MSPE in a way that makes it more useful to program directors. However, given the variations in medical school curricula and methods used to rank students, as well as a certain level of mistrust among GME program directors regarding the MSPE, it is unlikely that they will accept a standardized MSPE as a substitute for Step 1 scores.

Institutions with both medical schools and GME programs have an opportunity to align their UME and GME admissions processes. Making these changes at a local level is much more feasible in the short term, will allow for data to be collected about the process and success of the initiative, potentially attract students to stay within the state for residency and practice (a key outcome for many states with physician shortages), and reduce the “bait-and-switch” perception that many students currently have. At my institution, the University of Arizona College of Medicine–Tucson, we have begun this process by aligning the desired attributes of applicants between our UME and GME admissions processes, and we are hopeful that this change will allow us to both reduce anxiety among our students and recruit our own students to our GME programs. Another approach is to offer admission to combined UME/GME programs. Such programs are currently being explored by a number of medical schools as part of the Consortium of Accelerated Medical Pathway Programs.8 Five of the schools in this pilot program offer admitted students preliminary acceptance to a residency program at their own institution, pending successful completion of graduation requirements. These programs combine decreased training time and educational debt with potentially less stress over GME placement.

Other short-term solutions include creating a national basic science curriculum and/or question bank that can be used by all schools to ensure that they are addressing the necessary content and using questions that are similar to and of sufficient rigor to enhance students’ success when taking the Step 1 exam. However, the real elephant in the room is how we test students, as well as physicians, for their certification and recertification exams. In an era when portable electronic resources allow students and physicians to look up evidence-based knowledge in real time, why do we continue to test and reward the memorization of knowledge rather than the ability to access, assess, and apply appropriate knowledge based on the characteristics and presentation of the patient? This concern is especially true for the Step 1 exam given the sheer volume of facts that need to be memorized and the lack of opportunity students have had to apply this knowledge to real patients in a meaningful or repetitive fashion at the time of the examination. Thus, high Step 1 scores are generally achieved by students who are able to successfully memorize large amounts of information. Changing how we test medical professionals, including students, will be challenging, but it is necessary as medical knowledge and evidence continue to increase exponentially and our ability to rapidly access this information to make point-of-care decisions continues to be enhanced.

I do not minimize the importance of the knowledge that students need to acquire to pass the USMLE Step 1 exam. However, it is past time for medical schools to take back control of our students and curricula from the “Step 1 madness” that has infected our schools. To do so, we will need help and resources directed toward this issue from the AAMC, creative thinking, and a willingness of stakeholders—including the NBME, the ACGME, and GME program directors—to work collaboratively to find a satisfactory, equitable, and enduring solution. Our students and our patients deserve no less.

Acknowledgments: The author thanks Summers G. Kalishman for her review of and suggestions about this Invited Commentary.

Back to Top | Article Outline

References

1. McMahon GT, Tallia AF. Perspective: Anticipating the challenges of reforming the United States Medical Licensing Examination. Acad Med. 2010;85:453–456.
2. McGaghie WC, Cohen ER, Wayne DB. Are United States Medical Licensing Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med. 2011;86:48–52.
3. Gruppuso PA, Adashi EY. Residency placement fever: Is it time for a reevaluation? Acad Med. 2017;92:923–926.
4. Association of American Medical Colleges. Policy Priorities to Improve the Nation’s Health. From America’s Medical Schools and Teaching Hospitals. 2016. Washington, DC: Association of American Medical Colleges; https://www.aamc.org/advocacy/472826/policy-priorities-improve-nations-health.html. Accessed December 6, 2017.
5. Edmond MB, Deschenes JL, Eckler M, Wenzel RP. Racial bias in using USMLE Step 1 scores to grant internal medicine residency interviews. Acad Med. 2001;76:1253–1256.
6. Rotenstein LS, Ramos MA, Torre M, et al. Prevalence of depression, depressive symptoms, and suicidal ideation among medical students: A systematic review and meta-analysis. JAMA. 2016;316:2214–2236.
7. Prober CG, Kolars JC, First LR, Melnick DE. A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection. Acad Med. 2016;91:12–15.
8. Cangiarella J, Fancher T, Jones B, et al. Three-year MD programs: Perspectives from the Consortium of Accelerated Medical Pathway Programs (CAMPP). Acad Med. 2017;92:483–490.
© 2018 by the Association of American Medical Colleges