Secondary Logo

Journal Logo

Relationship Between Performance on the NBME Comprehensive Basic Sciences Self-Assessment and USMLE Step 1 for U.S. and Canadian Medical School Students

Morrison, Carol A.; Ross, Linette P.; Fogle, Thomas; Butler, Aggie; Miller, Judith; Dillon, Gerard F.

Section Editor(s): Hess, Brian PhD; Hawkins, Richard MD

doi: 10.1097/ACM.0b013e3181ed3f5c
Predictive Validity and High-Stakes Exams
Free
SDC

Background This study examined the relationship between performance on the National Board of Medical Examiners Comprehensive Basic Science Self-Assessment (CBSSA) and performance on United States Medical Licensing Examination Step 1.

Method The study included 12,224 U.S. and Canadian medical school students who took CBSSA prior to their first Step 1 attempt. Linear and logistic regression analyses investigated the relationship between CBSSA performance and performance on Step 1, and how that relationship was related to interval between exams.

Results CBSSA scores explained 67% of the variation in first Step 1 scores as the sole predictor variable and 69% of the variation when time between CBSSA attempt and first Step 1 attempt was also included as a predictor. Logistic regression results showed that examinees with low scores on CBSSA were at higher risk of failing their first Step 1 attempt.

Conclusions Results suggest that CBSSA can provide students with a realistic self-assessment of their readiness to take Step 1.

Correspondence: Carol A. Morrison, PhD, 3750 Market Street, Philadelphia, PA 19104; e-mail: cmorrison@nbme.org.

Step 1 of the United States Medical Licensing Examination (USMLE) is the first in the sequence of four examinations that all students and graduates of MD-granting U.S. medical schools must pass in order to be licensed to practice medicine in the United States. Typically taken at the end of the second year of medical school, Step 1 is an eight-hour, approximately 350-item, multiple-choice, computer-administered examination that assesses whether students can apply important basic science concepts, with emphasis on principles and mechanisms underlying health, disease, and modes of therapy.1 Although Step 1 is designed to be part of the licensing process, it also has important secondary uses for U.S. students that make the pressure to obtain a passing score on the first attempt extremely high. Two common secondary uses are the requirement of a passing score for promotion to the third year at many medical schools and the importance of a high score for acceptance into competitive residency training programs.

Given the high stakes associated with Step 1, it has been reported that students can typically spend several weeks preparing for the examination. In 2003, the National Board of Medical Examiners introduced a series of Web-based self-assessments to provide a tool for both U.S./Canadian and international medical students and graduates to assess their readiness for USMLE using examinations that were similar to the USMLE examinations in terms of content, item types, and pacing. The Comprehensive Basic Science Self-Assessment (CBSSA) is a four-hour, 200-item, Web-administered, multiple-choice examination that includes content that is typically covered during basic science courses in medical school. The content of the CBSSA items resembles the content of the items on Step 1. Participants may elect to take the CBSSA in either a standard-paced format that matches the pacing on Step 1 or a self-paced format that allows additional time. Immediate feedback is provided at the end of the self-assessment in the form of a performance profile and a score interpretation guide that enables participants to “translate” their CBSSA score to an approximate score on Step 1.2 Many medical schools encourage students to take CBSSA prior to taking Step 1 to assess whether they are at risk of failing, to alleviate test anxiety, and to become familiar with the content and format of Step 1.

Only one study to date has examined the relationship between performance on the CBSSA and Step 1. This study was conducted shortly after the CBSSA was introduced and was based on a fairly small sample of participants who took the CBSSA under either self-paced or standard-paced conditions. In addition, some of the participants took CBSSA for free during a field test. The standard-paced CBSSA scores for the group who paid for the self-assessment explained the greatest amount of variation in Step 1 scores (R2 = 0.69); the relationship was weakest for the self-paced group who took the self-assessment for free (R2 = 0.49).3

The current study extends previous research on the relationship between CBSSA scores and Step 1 scores for U.S. and Canadian medical school students based on a much larger and more recent sample who all took CBSSA under standard-paced conditions for a fee. The CBSSA was designed to help students prepare for Step 1, and it is important to confirm that its relationship with Step 1 continues to be robust as use of CBSSA has increased each year. For this study, it was hypothesized that the strength of the relationship between CBSSA and Step 1 would be similar to that reported in an earlier study because of the continued similarities between the measures. Further, it was hypothesized that there would be a relationship between Step 1 performance and the time between CBSSA and Step 1 attempts, with less time between the last CBSSA attempt and first Step 1 attempt associated with higher first Step 1 performance. Students often take CBSSA several times as they prepare for Step 1, and the theory behind the second hypothesis was that the level of basic science knowledge would be similar if the two exams were taken in close proximity in terms of time. We also examined whether the number of CBSSA attempts was related to Step 1 scores, expecting that scores would improve with practice. These hypotheses are consistent with a distributed or spaced practice strategy with a relatively brief retention interval.4 With a distributed practice strategy, practice time is divided across multiple sessions rather than being concentrated into one session. This strategy would be consistent with students taking the CBSSA multiple times to practice with test material similar to Step 1 and receive feedback about their readiness to sit for Step 1, with their last CBSSA attempt occurring just before their first Step 1 attempt.

Back to Top | Article Outline

Method

The data set included U.S. and Canadian medical school students who took an English version of CBSSA under standard-paced conditions between December 2, 2006, and December 28, 2008, and whose first Step 1 attempt was on or before November 17, 2009. Participants gave consent for their deidentified data to be used for research purposes when registering for each examination. CBSSA participants were eliminated from the analyses if their pattern of performance on CBSSA suggested that they did not take the self-assessment seriously as indicated by a short test duration (<55% of total test time used), large percentage of blank responses (>5%), or very low CBSSA score (scored >3 standard deviations below mean), or if they did not complete CBSSA. Many participants who were dropped from the analyses were part of pilot projects and did not take CBSSA for the purpose of self-assessment in preparation for Step 1. Approximately 7% of CBSSA takers were dropped. The final sample included 12,224 students who took CBSSA at least once prior to their first Step 1 attempt and who had scores for both examinations. The most recent CBSSA attempt prior to the first Step 1 attempt was used for the analyses. Students often took more than one form of the CBSSA prior to their first Step 1 attempt; the most recent attempt was used in this study because it was closest in time to the first Step 1 attempt, and it was hypothesized that students were likely to be motivated to perform as well as possible to gauge their readiness for Step 1. The analyses used first Step 1 attempt because students from U.S. and Canadian medical schools are under considerable pressure to pass Step 1 on their first attempt in order to be competitive in the residency selection process. For this population, the relationship between CBSSA performance and first Step 1 performance seems to be of the most interest, although additional analyses indicated that the relationship between CBSSA performance and last Step 1 performance was essentially the same.

In addition to descriptive analyses, a multiple regression analysis was performed to investigate the relationship between CBSSA scale scores (range between 200 and 800) and Step 1 three-digit scores (range between 1 and 300). Both sets of scores used in the analysis were equated scores and were comparable across different forms of the respective examinations. A stepwise procedure was used to investigate whether adding time between most recent CBSSA completion date and first Step 1 attempt date to the equation explained more variation in Step 1 scores than using CBSSA scale score as the only predictor variable. A logistic regression analysis predicting first Step 1 pass/fail outcome from CBSSA scale score was also performed.

Back to Top | Article Outline

Results

Table 1 provides descriptive information on the performance of the study participants on their most recent CBSSA attempt prior to their first Step 1 attempt and on their first Step 1 attempt. Comparative information is provided for the first Step 1 attempt for all U.S. and Canadian medical school examinees who tested during the same time frame. As shown in Table 1, the study sample was more able than the population of U.S. and Canadian examinees who took their first Step 1 during the same time frame. It is possible that students with a higher level of motivation, who might otherwise be expected to perform better on Step 1, chose to use CBSSA to help prepare.

Table 1

Table 1

In general, students used CBSSA to help them prepare for Step 1 right up to the Step 1 administration date, and they often took CBSSA multiple times. The median number of days between completion of the most recent CBSSA attempt and the first Step 1 attempt was eight days, and the mode was seven days. The number of CBSSA attempts prior to the first Step 1 attempt ranged from one to eight; 53% had one attempt, 29% had two attempts, 12% had three attempts, and 5% had four attempts. In general, performance on CBSSA improved with multiple attempts, and students with only one CBSSA attempt scored significantly lower on their first Step 1 attempt than students with three to six attempts (P < .05). The means for the numbers of attempts are 224 (SD = 21), 225 (SD = 21), 227 (SD = 21), and 230 (SD = 21) for one through four attempts, respectively.

A stepwise multiple regression analysis was conducted with first Step 1 three-digit score as the dependent variable and most recent CBSSA scale score prior to first Step 1 attempt and time in days between most recent CBSSA attempt and first Step 1 attempt as predictor variables. Both predictors were statistically significant in the equation (P < .01), and the increase in the proportion of variance explained by the model was statistically significant when time between most recent CBSSA attempt and first Step 1 attempt was added to the equation as a predictor variable (R2 = 0.67 versus 0.69, respectively). In general, Step 1 performance was better when the last CBSSA attempt was closer in time to the Step 1 attempt. The top portion of Figure 1 graphically depicts the relationship between CBSSA scores and Step 1 scores based on the study sample, with the regression line included. The unusual shape of the distribution at the lower end of the CBSSA scale is due to the reported scores being truncated at 200. A follow-up analysis using an underlying continuous score scale found that a relatively small percentage (<2%) of examinees performed at a level that would be considered below a reported score of 200 and that the R2 value was not impacted by the truncation.

Figure 1

Figure 1

A logistic regression analysis was also conducted predicting Step 1 first attempt pass/fail outcome from most recent CBSSA scale score prior to the first Step 1 attempt. As expected on the basis of the multiple regression results, higher CBSSA scores were associated with a greater probability of passing Step 1 on the first attempt (OR = 1.024, CI = 1.022-1.025). The strength of the relationship is most easily interpreted by examining the probability of passing the first Step 1 attempt associated with various CBSSA scores. For example, students with CBSSA scores of 230 had a predicted probability of 0.63 of passing Step 1 on the first attempt; students with CBSSA scores of 370 had a predicted probability of 0.98 of passing Step 1 on the first attempt. The bottom portion of Figure 1 shows the predicted probability of passing Step 1 on the first attempt for a given CBSSA score.

Back to Top | Article Outline

Discussion

The results of this study are consistent with the results of previous research that found that CBSSA performance is strongly related to Step 1 performance when CBSSA is taken under the standard-paced condition, with 67% to 69% of the variation in Step 1 scores being explained by CBSSA scores in both studies.3 In addition, the current study confirms that the proximity in time of the most recent CBSSA attempt to the first Step 1 attempt also matters, with higher Step 1 scores being associated with fewer days between the most recent CBSSA attempt and first Step 1 attempt. Further, students with multiple CBSSA attempts perform better than students with only one attempt. Motivation and other personal circumstances may help explain these findings. A logistic regression analysis showed a similar strong relationship with Step 1 pass/fail outcomes; examinees with low CBSSA scores were at greater risk for failing their first Step 1 attempt. These findings should be especially useful for faculty members who counsel students about their readiness to sit for Step 1. Faculty at schools that purchase CBSSA vouchers for their students have access to each student's performance feedback. This feedback may be used to assess the student's readiness to take Step 1 and to provide additional feedback to the student about his or her relative strengths and weaknesses in various content areas.

The strength of the relationship between performance on CBSSA and Step 1 for U.S. and Canadian students is not surprising because CBSSA has been designed to cover important basic science content that is also taught in U.S. medical schools and the overlap with the basic science content covered on Step 1 is considerable. Additionally, CBSSA contains similar item types to those that appear on Step 1, and the standard-paced CBSSA mirrors the pacing of Step 1. For these reasons, CBSSA provides students with a valuable practice tool and a realistic self-assessment of their readiness to take Step 1. Of course, the strength of the relationship between these two measures and, therefore, the usefulness of CBSSA in preparing for Step 1, may vary by individual. Motivational factors, the conditions under which CBSSA is taken, and other relevant circumstances should be considered by individual participants when interpreting results. These results are consistent with a distributed practice strategy with a relatively short retention interval. Additional research is needed to assess the relationship between performance on CBSSA and Step 1 for international medical school students and graduates. It is not unreasonable to expect that there will be factors unique to this population that may impact the results.

Back to Top | Article Outline

Funding/Support:

None.

Back to Top | Article Outline

Other disclosures:

None.

Back to Top | Article Outline

Ethical approval:

Participants gave consent for their de-identified data to be used for research purposes when registered for each examination.

Back to Top | Article Outline

References

1Federation of State Medical Boards; National Board of Medical Examiners. Bulletin of Information for the United States Medical Licensure Examination. Philadelphia, Pa: National Board of Medical Examiners; 2009.
2National Board of Medical Examiners. Web-Based Self-Assessment Services. Philadelphia, Pa: National Board of Medical Examiners. 2009.
3Sawhill A, Butler A, Ripkey D, et al. Using the NBME self-assessments to project performance on USMLE Step 1 and Step 2: Impact of test administration issues. Acad Med. 2004; 79(10 suppl):S55–S57.
4Rohrer D, Taylor K. The effects of overlearning and distributed practice on the retention of mathematics knowledge. Appl Cogn Psychol. 2006;20:1209–1224.
© 2010 Association of American Medical Colleges