Student Performances on Step 1 and Step 2 of the United States Medical Licensing Examination Following Implementation of a Problem-based Learning Curriculum

Abstract

PURPOSE 

To examine students' performances on Step 1 and Step 2 of the United States Medical Licensing Examination (USMLE) following the implementation of a problem-based learning curriculum.

METHOD 

Performances on Step 1 of the USMLE for four classes at the University of Missouri-Columbia School of Medicine that completed a new problem-based learning curriculum (1997, 1998, 1999, and 2000) were compared with those of the last two classes to learn in the traditional curriculum (1995 and 1996). Performances on Step 2 of the USMLE for the classes of 1997, 1998, and 1999 were also compared with those of the classes of 1995 and 1996. The authors analyzed matriculation data (GPAs and MCAT scores) for all six classes. They compared all data with those of U.S. and Canadian first-time USMLE takers.

RESULTS 

The mean scores were higher on USMLE Step 1 for classes in the problem-based learning curriculum than for classes in the traditional curriculum. The mean scores for Step 2 were above the national mean for classes in the revised curriculum and below the national mean for classes in the traditional curriculum. The admission profiles of these classes were essentially the same before and after the change in curriculum.

CONCLUSIONS 

Major PBL revisions of the curriculum did not compromise the performances of medical students on the licensing examinations; in fact, they may have contributed to higher scores.

In recent years, problem-based learning (PBL) has generated considerable interest in the medical education community. Two meta-analyses of PBL,^1,2 published in 1993, identified several advantages of this approach to learning, but concluded that students exposed to PBL curricula generally did not perform as well as did students in traditional curricula on standardized examinations that measure basic science knowledge. Also in 1993, Mennin and colleagues³ similarly reported that medical students who completed the PBL track at the University of New Mexico, when compared with their peers in the conventional track of the curriculum, had lower mean scores on Part I of the National Board of Medical Examiners examination (which examined knowledge of the basic sciences). Those same students, however, scored on a par with their peers on Part II (which measured knowledge of clinical sciences during medical school) and scored better than did their peers on Part III (which measured clinical decision making and patient management during residency).

Until 1993, first- and second-year medical students at the University of Missouri-Columbia (UMC) School of Medicine learned the basic sciences in discipline-based, departmentally conducted courses that relied on lectures and laboratory experiences. In 1988, a member of a visiting Liaison Committee on Medical Education accreditation team had described that curriculum as perfectly preserved from the 1960s. That description stimulated the dean of the school of medicine to begin looking for a new curriculum.

A faculty committee, after collecting information about innovative strategies for teaching and learning in medical education, recommended that any new curriculum (1) incorporate PBL as a major component during the first two years, (2) expose students to clinical experiences beginning in their first year, (3) evaluate students' ability to solve problems as well as their knowledge base, (4) emphasize primary care, (5) review the basic sciences in the fourth year, and (6) be a responsibility shared between the dean's office and the faculty.

Following those recommendations, a second faculty committee developed a new curricular framework, and the medical school faculty voted by two to one to approve it. The major curricular changes would be made to the first two years; smaller adjustments would be made to the third and fourth years. UMC implemented the new first- and second-year basic science curricula in 1993 and 1994, respectively. Revisions of the third- and fourth-year curricula were approved and implemented in 1995 and 1996, respectively.

The restructured curriculum features two components: Basic Science/Problem-based Learning and Introduction to Patient Care (IPC) (which presents basic clinical knowledge and skills). Each of the first two years consists of four nine-week blocks. During eight weeks of each block, students work in groups of eight with a faculty facilitator (tutor) to study the primary basic science. Each week, the students discuss and analyze a new clinical case addressing issues in anatomy, biochemistry, physiology, pharmacology, pathology, epidemiology, behavioral sciences, and other basic sciences of medicine. These PBL activities are supplemented by lectures and some laboratory-based instruction. All evaluation activities take place during the ninth week. During the first year, students spend eight to ten hours a week in the small-group PBL setting and not more than ten hours in traditional lectures and laboratory work. During the second year, students spend approximately eight hours in PBL sessions and not more than ten hours in traditional learning methods.

The IPC component of the curriculum includes two to four hours per week in small groups and not more than two to three hours a week in lectures, demonstrations, or other large-group activities. IPC addresses a variety of subjects and skills, including interviewing and history taking, physical examination skills, clinical epidemiology, behavioral sciences, and diagnostic tests. During the first year, IPC also includes a clinical experience in which each student spends one half-day every other week with faculty or community practitioners. In the third year, students have an eight-week family medicine clerkship, which was created by reducing the internal medicine and surgery clerkships from 12 weeks to eight weeks. The time was returned to internal medicine and surgery in the fourth year as required advanced clinical selectives. An eight-week Advanced Bio-medical Science selective is required in the fourth year and can be fulfilled by original research or a scholarly review of the basic sciences in the context of a clinical problem.

Medical students at the UMC are required to pass Steps 1 and 2 of the United States Medical Licensing Examination (USMLE) (the successors to the NBME Parts I and II) prior to graduation. The curriculum does not provide formal preparation for the USMLE examination, and the timing of the curricular blocks does not accommodate enrollment in commercial preparatory courses. This article reports the performances on Step 1 and Step 2 for the first four classes of students who completed the new curriculum.

METHOD

To assess the effects of the new curriculum on the knowledge of the basic sciences and clinical medicine, we compared the performances on Step 1 of the USMLE for the first four classes that completed the new curriculum (classes of 1997, 1998, 1999, and 2000) with those of the last two classes to complete the traditional curriculum (classes of 1995 and 1996). We also compared the performances on Step 2 for the first three classes in the new curriculum (classes of 1997, 1998, and 1999) with those of the last two classes to complete the traditional curriculum. It should be noted that the USMLE began with the class of 1995; thus only two classes could be used as comparison groups.

We compared mean scores, the percentages of students passing, and the percentages of students scoring in the top 90th, 95th, and 99th percentiles of all students taking the examinations for the first time. The Student t test was used to assess differences between means, and chi square was used to assess differences between proportions.

RESULTS

Table 1 shows the mean scores for Step 1 for the six graduating classes. Prior to the implementation of the new curriculum, students at UMC generally had mean Step 1 scores that were at or slightly lower than the national means. The mean Step 1 scores for the classes of 1995 and 1996 were two and six points below the national means. The difference was statistically significant for the class of 1996. In contrast, the mean scores for the four classes that completed the new curriculum were improved from just one point below the national mean to as much as eight points above. The differences were statistically significant for the classes of 1999 and 2000.

Table 1: Performances on Step 1 of the United States Medical Licensing Examination for Six Classes of the University of Missouri-Columbia School of Medicine (UMC) and the Corresponding US/Canada First-time Takers

Table 2 shows the mean scores on Step 2 for the five classes in that comparison. Those classes learning under the traditional curriculum (1995 and 1996) had scores that were below the national mean. For the class of 1996, the difference was statistically significant. The mean scores for the new-curriculum classes of 1997, 1998, and 1999 were above the national mean; the differences for 1998 and 1999 were statistically significant.

Table 2: Performances on Step 2 of the United States Medical Licensing Examination for Six Classes of the University of Missouri-Columbia School of Medicine (UMC) and the Corresponding US/Canada First-time Takers

Table 3 shows the percentages of students in each class who scored in the 90th, 95th, and 99th percentiles on Steps 1 and 2. Students who completed the new curriculum were much more likely than were students who completed the traditional curriculum to achieve high scores. One third of the classes of 1999 and 2000 scored in the top 10% nationally on Step 1.

Table 3: Percentages of University of Missouri-Columbia Medical Students Scoring in the 90th, 95th, and 99th Percentiles of the USMLE Steps 1 and 2, by Class

The improved performances on the USMLE are unlikely to reflect changes in the characteristics of students at the time of matriculation to medical school. Students entering the UMC School of Medicine have generally had grade-point averages (GPAs) slightly above the national average and Medical College Admission Test (MCAT) scores slightly below the national mean (Table 4).

Table 4: Grade Point Averages (GPAs) and Medical College Admission Test (MCAT) Scores for University of Missouri-Columbia School of Medicine and U.S. Medical School Matriculants, by Class

DISCUSSION

Previous studies have found that students exposed to a PBL curriculum scored lower on standardized tests of basic science knowledge and higher on tests of clinical knowledge than students who completed traditional curricula.^1,2,3 At UMC, students in the second, third, and fourth classes to complete the new PBL curriculum performed substantially better on both Step 1 (basic science) and Step 2 (clinical science) of the USMLE than did previous classes at the school. While these students had MCAT scores that were slightly below the national average, their USMLE scores exceeded the national means. It is unlikely that the students' level of knowledge at the onset of their medical education accounts for their superior performances on the licensing exam. Our data indicate that the use of a PBL curriculum as a major educational method during the first two years of medical school does not compromise performances on standardized tests of basic science knowledge and clinical knowledge.

Previously published studies of the effects of PBL on basic science knowledge involved students who completed National Board of Medical Examiner tests in the 1980s. More recently, the examinations have shifted emphasis from measuring factual knowledge to assessing the application of knowledge.⁴ Step 1 of the USMLE increasingly frames questions as clinical vignettes; students in a PBL curriculum may be at an advantage with such questions. Thus, the better performances of PBL-trained students on these standardized exams may partially reflect changes in the format and content of the test.

Other variables that must be considered as potentially influencing the USMLE performances are the motivation of the students in a new curriculum and faculty expectations. The dean's office has downplayed competition among the classes; however, each graduating class has wanted to surpass the USMLE mean score of the previous class.

Performance on Step 1 of the USMLE is generally considered to be an important measure of factual knowledge. However, the evaluation of students should not be limited to standardized tests that use multiple-choice questions. Valid assessment of a curriculum and accurate evaluation of students require a variety of strategies that measure appropriate attitudes, skills, and knowledge.

Blake and Parkison evaluated the new curriculum at UMC by interviewing clinical faculty members and asking them to compare students who had completed the new curriculum with students who had completed the traditional curriculum.⁵ The majority of faculty rated students who completed the first two years of the new curriculum superior in knowledge of pathophysiology and disease processes, ability to obtain an appropriate history, and clinical reasoning and problem solving.

Curriculum evaluation is a complex and challenging task. Performance on the USMLE is an important component, but is not sufficient to assure excellence of the education process. The early experiences with the PBL curriculum at the University of Missouri-Columbia provide reassurance that the use of PBL as a major educational method does not compromise the attainment of basic science and clinical knowledge.