Secondary Logo

Journal Logo

Why Not Wait? Eight Institutions Share Their Experiences Moving United States Medical Licensing Examination Step 1 After Core Clinical Clerkships

Daniel, Michelle MD, MHPE; Fleming, Amy MD, MHPE; Grochowski, Colleen O’Conner PhD; Harnik, Vicky PhD; Klimstra, Sibel MD; Morrison, Gail MD; Pock, Arnyce MD; Schwartz, Michael L. PhD; Santen, Sally MD, PhD

doi: 10.1097/ACM.0000000000001714
Perspectives
Free

The majority of medical students complete the United States Medical Licensing Examination Step 1 after their foundational sciences; however, there are compelling reasons to examine this practice. This article provides the perspectives of eight MD-granting medical schools that have moved Step 1 after the core clerkships, describing their rationale, logistics of the change, outcomes, and lessons learned. The primary reasons these institutions cite for moving Step 1 after clerkships are to foster more enduring and integrated basic science learning connected to clinical care and to better prepare students for the increasingly clinical focus of Step 1. Each school provides key features of the preclerkship and clinical curricula and details concerning taking Steps 1 and 2, to allow other schools contemplating change to understand the landscape. Most schools report an increase in aggregate Step 1 scores after the change. Despite early positive outcomes, there may be unintended consequences to later scheduling of Step 1, including relatively late student reevaluations of their career choice if Step 1 scores are not competitive in the specialty area of their choice. The score increases should be interpreted with caution: These schools may not be representative with regard to mean Step 1 scores and failure rates. Other aspects of curricular transformation and rising national Step 1 scores confound the data. Although the optimal timing of Step 1 has yet to be determined, this article summarizes the perspectives of eight schools that changed Step 1 timing, filling a gap in the literature on this important topic.

M. Daniel is assistant dean of curriculum and assistant professor, Departments of Emergency Medicine and Learning Health Sciences, University of Michigan Medical School, Ann Arbor, Michigan.

A. Fleming is associate dean for student affairs and associate professor of pediatrics, Vanderbilt University School of Medicine, Nashville, Tennessee.

C.O. Grochowski is associate dean for curricular affairs, Duke University School of Medicine, Durham, North Carolina.

V. Harnik is associate dean for curriculum and assistant professor, Department of Cell Biology, New York University School of Medicine, New York, New York.

S. Klimstra is Ehrenkranz Associate Dean of Academic Affairs and professor of clinical psychiatry, Weill Cornell Medical College, New York, New York.

G. Morrison is senior vice dean for education and professor of medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania.

A. Pock is associate dean for curriculum and associate professor of medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland.

M.L. Schwartz is associate dean for curriculum and associate professor, Department of Neuroscience, Yale University School of Medicine, New Haven, Connecticut.

S. Santen is assistant dean for educational research and quality improvement and professor of emergency medicine and learning health sciences, University of Michigan Medical School, Ann Arbor, Michigan.

Funding/Support: None reported.

Other disclosures: The University of Michigan School of Medicine, Vanderbilt School of Medicine, and New York University School of Medicine have Accelerating Change in Medical Education grants from the American Medical Association.

Ethical approval: Reported as not applicable.

Disclaimer: The views expressed are those of the authors and do not reflect the official policy or position of their universities, the Department of Defense, the United States Air Force, or the United States Government.

Previous presentations: Learn Serve Lead, the Association of American Medical Colleges Annual Meeting, Seattle, Washington; November 11–15, 2016.

Correspondence should be addressed to Michelle Daniel, University of Michigan Medical School, 6123 Taubman Health Sciences Library, 1135 Catherine St., SPC 5726, Ann Arbor, MI 48109; telephone: (401) 525-0251; e-mail: micdan@med.umich.edu.

Historically, students take the United States Medical Licensing Examination (USMLE) Step 1 before clerkships; however, there are compelling reasons to examine this practice. According to data published by the Association of American Medical Colleges in 2015–2016, 130/142 of accredited medical schools have their students take Step 1 of the USMLE during their first or second year of medical school.1 This typically occurs following completion of the basic science curricula, after a dedicated study period of variable duration. Passing Step 1 is often a requirement for advancement to clinical clerkships.2 A number of schools are undergoing curricular revisions that shorten the traditional two-year preclerkship curricula to 18 months or even 1 year,3 and an increasing number are altering or have altered the timing of Step 1 to follow completion of the core clerkships. Other schools are still contemplating where to best place Step 1 to optimize student learning and performance.

Back to Top | Article Outline

Context

Research on the timing of Step 2 Clinical Knowledge (CK) suggests that student performance declines over time after completion of the core clerkships.4 A similar decline in scores might be predicted for Step 1 as the distance from completing the foundational sciences increases. Ling et al5 found a 6.4% decrease in overall performance on USMLE Step 1 basic science items administered in unscored sections of USMLE Step 2 CK. The magnitude of the decline varied by discipline, with the steepest occurring in biochemistry (17.5%). This and other studies by the National Board of Medical Examiners (NBME) examining basic science knowledge retention6,7 raise the concern that students may not perform as well on Step 1 questions after clerkships.

Other studies suggest that Step 1 performance could improve with clinical exposure. Petrusa et al8 investigated the performance of medical students from years 1, 3, and 4 on sample Step 1 questions. Their results demonstrated that students with more clinical experience performed better, suggesting that moving Step 1 after clerkships might raise scores. Unfortunately, the study was limited by a small sample size and the use of practice questions rather than the actual USMLE exam.

At this time, consensus on the impact of moving Step 1 after the core clinical years on student outcomes is lacking. In the absence of empirical data, schools are looking to peer institutions for guidance to aid them in decision making and advocating to their administrations. Numerous queries on the optimal timing of Step 1 are appearing on medical education listservs,9 and there is a need for a collated resource of institutions that have made the change. This article offers the perspective of eight medical schools: Duke University School of Medicine, Perelman School of Medicine at University of Pennsylvania, New York University School of Medicine, Uniformed Services University of the Health Sciences, Vanderbilt University School of Medicine, Weill Cornell Medical College, Yale University School of Medicine, and University of Michigan Medical School. These schools have either realigned or are in the process of moving Step 1 to after completion of the core clinical clerkships. Here we discuss our rationale, the logistics, outcomes, and lessons learned, to begin to address the gap in the literature on this important topic.

Back to Top | Article Outline

Rationale

The primary reasons our institutions cite for moving Step 1 after the core clerkships are to help improve retention of foundational science, to promote more integrated basic science learning in clinical contexts, and to better prepare students for the increasingly clinical focus of Step 1.

At our institutions, we see a concern amongst faculty that students prepare intensively for Step 1, pass the exam, and promptly forget their basic science. Research in medical education shows that students retain two-thirds to three-fourths of knowledge at one year, and slightly below 50% at two years in the absence of deliberate retention and retrieval practices.10 We believe that students cannot afford to forget significant portions of their scientific foundations. We want their foundational knowledge to be enduring, to enable students to delve into the biochemical and molecular basis of disease when working with patients. As part of curricular reform, our institutions have all moved to more integrated, organ-system-based approaches and employed deliberate strategies to ensure that students revisit the basic sciences in the clinical setting. We believe that placement of Step 1 after the core clerkships adds to these instructional efforts by tapping into student motivation to review basic science content. Assessment is one of the most powerful motivators of student learning.11 Thus, the timing of the exam has the potential to influence study behaviors by making the material more “relevant.”

Over time, Step 1 has become increasingly clinically focused, with longer and more complicated vignette-style question stems. We believe it makes pedagogical sense that having clinical experience prior to taking the examination may improve scores. Students at our institutions have anecdotally reported that taking the exam after the core clerkships has allowed them to read and analyze the clinical vignettes quickly and with greater comprehension. Students presumably use pattern recognition and can “think fast,” rather than using slower, more analytical thinking during the exam.12 Additionally, students have commented that practice with the NBME clinical subject “shelf” exams facilitates more effective and efficient progress through the clinical stem questions on Step 1.

Back to Top | Article Outline

Logistics

There are a number of logistical issues associated with changing the timing of Step 1 that span the curricular continuum. For peer institutions to best learn from our institutional examples, they must be able to determine whether our practices are applicable to their curricula. Table 1 outlines the logistical details surrounding Step 1 at each of our eight schools, detailing timing and curricular features of import. Appendix 1 offers the logistical details of additional schools making this change, and there are likely others. Duke University and the University of Pennsylvania have the longest experience with placing Step 1 after the core clinical clerkships, with 24 years and 19 years, respectively. The other schools made the change more recently as part of their curriculum transformations.

Table 1

Table 1

Based on our collective experiences to date, we have identified key facets of the preclerkship curricula to consider when determining the optimal timing of Step 1, including the duration of the basic science curriculum, the type of curriculum, and grading policies. The length of the preclerkship curriculum in our eight schools ranges from 45 to 60 weeks, excluding vacations. All but one school (Duke) report having single-pass curricula, organized around organ systems, where scientific disciplines are interwoven and the basic and clinical sciences are integrated. Duke has a two-pass curriculum of normal systems followed by abnormal systems.

Preclerkship student assessment at most institutions is pass/fail, with only a few reporting discriminatory grading. Some institutions assist students in Step 1 preparation by offering either the NBME subject exams or customized exams spaced throughout the preclerkship years. Two institutions offer the Comprehensive Basic Science Examination (CBSE) as a summative experience similar to Step 1 to help students consolidate knowledge before entering clerkships. These preclerkship curricular features are detailed for each institution in Table 1.

We have also identified important features of the clinical curricula to consider, including the total duration of the core clinical clerkships, grading policies, and whether or not schools use the NBME clinical subject examinations. These core clinical curricular features are also detailed in Table 1. Notably, in addition to traditional clerkship didactics, our schools all use a variety of pedagogical strategies for deliberate science integration in the clinical setting, including linkage of specific clerkships with basic science disciplines (i.e., surgery and anatomy), special days dedicated to deep science dives, weeklong intersessions, four-week selectives, and the deliberate encouragement of self-directed, patient-based scientific inquiry. Several institutions have invested in question banks, and two schools use learning platforms to help provide the testing effect of spaced repetition of basic science content during the clinical years. We highlight these curricular features because we believe they help foster retention and retrieval of basic science knowledge, and drive the integration of basic and clinical science learning, complementing the change in Step 1 timing.

Of the eight institutions represented, six mandate students to take Step 1 after the core clinical clerkships, and two provide flexible options (i.e., students may take Step 1 immediately following the basic science curricula or after clerkships). The time provided to prepare for Step 1 varies from four to eight weeks, and averages six. Of note, Duke, which has the longest experience administering Step 1 after the clerkships, offers the shortest guaranteed study period, though students may negotiate with their research mentor for additional time if needed. Institutions generally offer some flexibility as to when students take Step 2 CK, but most require it by December of the fourth year. Some institutions allow students to take Step 2 CK before Step 1. In practice, few students have exercised this option, though some students do take the exam soon after Step 1, and many take it earlier than ever before. The majority completed Step 2 CK prior to the submission of their data to the Electronic Residency Application Service in mid-September. Most schools require passing both Step 1 and Step 2 as a prerequisite for graduation. Although some schools have expressed concerns about stacking USMLE Step 1 and Step 2 CK close together, this has not, in our experience, been a significant issue. Some students have found it helpful to have the option to take both exams in rapid succession or to spread them out.

Back to Top | Article Outline

Outcomes and Lessons Learned

Arguably, the most desirable outcomes of moving Step 1 after the core clerkships are students’ deeper mastery and retention of the foundational sciences and integration of basic science and clinical learning, but direct measures of these outcomes are elusive. One measurable objective outcome is performance on Step 1.

Although we do not teach to the USMLE exams, we recognize the importance of Step 1 scores for our students, and we want to ensure noninferiority or improvement in performance with the change in exam timing. At the institutional level, our schools have reported higher mean Step 1 scores after changing the timing of Step 1 to post clerkship. Duke has consistently high Step 1 scores, but they do not have comparison data from before the change, which occurred more than 20 years ago. The University of Pennsylvania overhauled its curriculum in 1996, and their students have consistently demonstrated mean scores 20 points above their pretransformation average. New York University has 4 years of data, showing an average 7- to 12-point improvement in mean scores, with the smallest change occurring the first year after the transformation. Notably, they report that their MD/PhD candidates, who may serve as an internal control group, take the exam directly after the basic sciences. This group, while scoring above the national average, does not score as highly as the students who take it later. The Uniformed Services University of the Health Sciences has 3 years of data demonstrating improved Step 1 scores. This was manifest by an 11-point improvement the first year after the change, a 13-point improvement in the second year, followed by a slight (1- to 2-point) return toward baseline in the third year, but still with aggregate scores remaining higher than the historical baseline. Vanderbilt has 1 year of data. Acknowledging that scores fluctuate from year to year, they compared the single year (postclerkship Step 1 mean) against the average performance over the prior 4 years of data. The mean score improved by 9 points. They reported “cautious optimism” with this delta, suggesting that students may have been nervous about the change, and hence more motivated in their studies. At Weill Cornell, students may choose to take Step 1 either before or after the core clerkships. For the first 2 years of the new curriculum, approximately 80% opted to take the exam during the “traditional” time immediately following the basic science curriculum. Scores are still pending for students who opted to take the exam after clerkships. Yale and the University of Michigan do not yet have their first year of data.

There are other unintended outcomes to moving Step 1 or offering a choice of timing. For example, a lesson learned from Weill Cornell’s experience was that allowing students autonomy concerning when to take the exam resulted in a notable increase in anxiety. Students worried that they were making a choice in the absence of clear data that might significantly impact their performance and possibly career plans. Students tended to fall back on tradition, choosing to take the exam early. Duke also offers flexibility in the timing of the exam, but their longer experience with taking the exam after the core clerkships leads ~99% of students to take the exam late. At Vanderbilt, students were concerned that they might decide on a particular specialty, then need to reconsider their career choices relatively late if their Step 1 score was not competitive, and thus their candidacy for certain residency positions was not as strong as they might have otherwise anticipated. Although the scores may be somewhat predictable, based on prior performance (i.e., basic science class cumulative scores, CBSE and NBME shelf exam scores), the fact remains that a small number of students may be confronted with a real limitation.

Back to Top | Article Outline

Discussion and Next Steps

The optimal timing of Step 1 has yet to be determined, but an increasing number of medical schools are starting to deviate from tradition, and early outcomes from our institutions are promising. As new schools develop, and older institutions undergo curriculum transformation, they might carefully consider where to place Step 1 to align best with their pedagogical aims, rather than simply following the status quo. If major goals for learning include fostering retention of basic science knowledge and the integration of basic and clinical science, then placement of Step 1 after the core clerkships may help drive this process by tapping into student motivation.

A few institutions (i.e., Duke and the University of Pennsylvania) moved the timing of Step 1 many years ago, but most institutions have implemented this change relatively recently. One might question why there is this seemingly sudden trend to move the timing of Step 1 as represented by our eight schools and others (Appendix 1). Perhaps schools now feel they have a “morsel of permission” to follow others, or perhaps the centennial Flexner Report is pushing schools to change. In Educating Physicians: A Call for Reform of Medical School and Residency, Cooke et al13 emphasize the need to strengthen the connections between formal and experiential knowledge, by providing more clinical exposure earlier in medical school, and more opportunities for deep science learning later in training. This notable work has prompted rethinking of many major structural and architectural aspects of medical curricula, of which Step 1 timing is one.

In their plea to reassess the role of USMLE Step 1 scores on residency selection, Prober et al14 noted: “Because students recognize the high stakes of USMLE, they prioritize learning what they believe to be important for the test during their preclerkship courses. They are emotionally stressed about perceived disconnects between what they need to learn for the test and what they need to know to care for their patients and prepare for lifelong learning.” Repositioning Step 1 may help students connect the basic sciences to patient care and shift the focus in the preclerkship years away from memorization of facts, toward developing habits of learning that will prompt them to ask deep scientific questions when confronted with complex patient problems. Of course, a change in Step 1 timing alone is not sufficient to overcome a curriculum in which these connections are not modeled and reinforced in other ways. This is why we firmly believe that other structural changes should accompany the change in Step 1 timing, though a discussion of such changes is outside the scope of this article.

For institutions concerned about student performance on Step 1, our collective experience suggests noninferiority, if not slight improvement, when Step 1 is placed after clerkships.

Our current outcomes data are aggregate and not amenable to statistical analysis. Thus, they should be interpreted with caution. To date, we have not looked at the effect of the change in Step 1 timing on residency match rates, but this is another avenue for future exploration.

It seems logical that a year of clinical training may improve student performance on clinically based vignettes. The modest rise in scores may be further influenced by students’ total time on task. Although most institutions that change the timing of Step 1 also shorten the preclerkship curriculum, students generally spend between 6 and 12 additional months learning medicine before taking the exam.

One concern amongst educators about delaying Step 1 is that students may lose a needed opportunity to consolidate basic science knowledge before clerkships and may not be as prepared for clerkships, potentially leading to poorer scores on NBME clinical subject exams, which many schools use as a measure of clerkship performance. Specifically, there is a concern for the effect on struggling students. The changes in aggregate scores that we report say nothing about low-performing students compared with high-performing students. In the future, we plan to study both class-based and individual student metrics to further characterize these potentially important effects.

Other factors complicate the interpretation of the improved Step 1 scores: There has been a slow but steady increase in national Step 1 averages.15 From 2009 to 2015, mean scores increased from 221 to 229, with incremental changes of 0 to 3 points per year. Taking this into account, the deltas we report are likely smaller but still appear to show increases slightly above overall trends. Another factor that influences interpretation of these outcomes is the measurement error of the USMLE. This provides an index of the imprecision for the examination, making these differences of unclear significance.16

Our observations have other limitations. Our institutions are not representative of all schools in terms of geographic representation, rankings, or baseline USMLE scores. The majority of these eight schools’ mean USMLE scores before the change were above the national average, and failure rates were in the range of 0% to 4%. We do not know how a change in Step 1 timing would affect schools with significantly lower mean Step 1 scores and higher failure rates. Although our experiences may not be generalizable, we have attempted to provide as much information as possible about our curricular structures to make our experiences transferable. Finally, we should note that the outcomes on USMLE Step 1 scores reported in association with changing Step 1 timing are potentially confounded, as we are unable to separate the effects of changing the timing of Step 1 from the effects of other curricular transformations simultaneously occurring at our institutions.

Back to Top | Article Outline

Concluding Remarks

This article summarizes our eight institutions’ experience with moving the timing of Step 1 after the core clerkships, filling an important gap in the literature. Early outcomes are promising, and as such, the practice warrants further investigation.

Acknowledgments: The authors wish to thank Kim Lomis, associate dean of undergraduate medical education, Vanderbilt University School of Medicine; Peter Marzuk, associate dean of curricular affairs, Weill Cornell Medical College; Mel Rosenfeld, senior associate dean for medical education, New York University School of Medicine; Judy Shea, associate dean for evaluation and assessment, Perelman School of Medicine at the University of Pennsylvania; Jennifer Christner, dean, School of Medicine, Baylor College of Medicine; Jonathan Amiel, associate dean for curricular affairs, Columbia University College of Physicians and Surgeons; Richard Iuli, medical education specialist, Stony Brook School of Medicine; Peter Ziemkowski, associate dean for student affairs, Western Michigan University Homer Stryker MD School of Medicine; Karin Esposito, associate dean for curriculum and medical education, Florida International University Herbert Wertheim College of Medicine; Todd Cassese, director, Clinical Arts and Science Course, Frank H. Netter MD School of Medicine at Quinnipiac University; Edward Krupat, director, Center for Evaluation, Harvard Medical School; Beth Nelson, associate dean for undergraduate medical education, University of Texas at Austin Dell Medical School; and Owen Thompson and Genevieve Allen, medical students, University of Michigan School of Medicine, for their contributions to this article.

Back to Top | Article Outline

References

1. Association of American Medical Colleges. Time in the curriculum in which medical schools require students to take the United States Medical Licensing Examinations (USMLE): USMLE Step 1. https://www.aamc.org/initiatives/cir/406430/10c.html. Accessed February 27, 2017.
2. Association of American Medical Colleges. Number of medical schools requiring the United States Medical Licensing Examination (USMLE) for advancement/promotion. https://www.aamc.org/initiatives/cir/406442/10b.html. Accessed February 27, 2017.
3. Association of American Medical Colleges. 2016;3(1). https://www.aamc.org/download/464758/data/ciic03-1jan2016.pdf. Accessed February 27, 2017.
4. Pohl CA, Robeson MR, Veloski J. USMLE Step 2 performance and test administration date in the fourth year of medical school. Acad Med. 2004;79(10 suppl):S49–S51.
5. Ling Y, Swanson DB, Holtzman K, Bucak SD. Retention of basic science information by senior medical students. Acad Med. 2008;83(10 suppl):S82–S85.
6. Kennedy WB, Kelley PR Jr, Saffran M. Use of NBME examinations to assess retention of basic science knowledge. J Med Educ. 1981;56:167–173.
7. Swanson DB, Case SM, Luecht RM, Dillon GF. Retention of basic science information by fourth-year medical students. Acad Med. 1996;71(10 suppl):S80–S82.
8. Petrusa ER, Reilly CG, Lee LS. Later is better: Projected USMLE performance during medical school. Teach Learn Med. 1995;7:163–167.
9. Timing of Step 1 Exam. Dr-ED listserv discussions. January 6–7, 2016, and March 5, 2014. http://list.msu.edu/archives/dr-ed.html. Accessed March 2, 2017.
10. Custers EJ. Long-term retention of basic science knowledge: A review study. Adv Health Sci Educ Theory Pract. 2010;15:109–128.
11. Mann KV. Motivation in medical education: How theory can inform our practice. Acad Med. 1999;74:237–239.
12. Kahneman D. Thinking, Fast and Slow. 2011.New York, NY: Farrar, Straus and Giroux.
13. Cooke M, Irby DM, O’Brien BC. Educating Physicians: A Call for Reform of Medical School and Residency. 2010.San Francisco, CA: John Wiley & Sons.
14. Prober CG, Kolars JC, First LR, Melnick DE. A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection. Acad Med. 2016;91:12–15.
15. National Board of Medical Examiners (NBME). NBME self-service portal. USMLE Step 1 performance summary score reports. https://nsp.nbme.org [user name and password required]. Accessed February 27, 2017.
16. United States Medical Licensing Examination. USMLE score interpretation guidelines. http://www.usmle.org/pdfs/transcripts/USMLE_Step_Examination_Score_Interpretation_Guidelines.pdf. Accessed July 27, 2016 [no longer available].
Appendix 1

Appendix 1

© 2017 by the Association of American Medical Colleges