Edmond, Michael B. MD, MPH; Deschenes, Jennifer L. MS; Eckler, Maia MS; Wenzel, Richard P. MD, MSc
Over the last century, standardized examinations have played an important role in medical education and physicians' advancement. Because studies have shown that African-American students do not perform as well on standardized tests as do white students, the prominent role of standardized examinations in the physicians' licensing process has become problematic.
Colleges and universities currently rely on standardized test scores to better compare students from high schools of varying quality. For the Scholastic Aptitude Test, significant differences between the scores of white and African-American students have been documented, and the racial gap has widened over the past ten years.1
Documented differences in the Medical College Admissions Test (MCAT) performances of African-American and white students have also presented challenges for medical school admission committees. A 1998 study examined two cohorts stratified by race.2 In the first cohort of 1,109 MCAT examinees, the mean component score for white students was 9.24, while the mean score for African-American students was 7.50. In the second cohort, all first-time takers of the United States Medical Licensing Examination (USMLE) Step 1 in 1994, the mean MCAT component score was 9.67 for white students and it was 7.97 for African-American students.
Studies have found that residency program directors view USMLE performance (Steps 1 and 2) as a valuable criterion in the selection process to obtain a standardized comparison of the medical knowledge among applicants from various medical schools and as a predictor of success on future licensing and certifying examinations.3 While some program directors simply use USMLE scores to verify that an applicant has passed the licensing exams, other program directors from more competitive programs will frequently use these scores to rank the desirability of candidates.4 Substantial differences in test performances across racial groups on the National Board of Medical Examiners' (NBME) examinations, the USMLE's predecessor, were previously reported.5 In 1988, the overall mean score on NBME Part I was 480; however, African-American students had mean scores that were 100–120 points lower than those for white students. Only limited data on USMLE Step 1 performances by racial groups have been reported. However, analysis of first-time takers (n = 11,279) of the USMLE Step 1 in June 1994 revealed that the mean score for white students was 210, while the mean score for African-American students was 187.2
Since the USMLE Step 1 scores of African-American medical students have been shown to be lower than those of their white counterparts, we hypothesized that there would be a negative impact on African-American students in gaining access to residency interviews and, thereby, on their selection by some residency programs.
We performed our study in the Department of Internal Medicine at the Virginia Commonwealth University School of Medicine. Of the department's 134 house officers, 133 are graduates of 48 Liaison Committee on Medical Education-accredited medical schools and one is a graduate of an osteopathic school. The median USMLE Step 1 score of entering interns is 220. The study was performed after the interviewing process was completed and the residency program's rank list was submitted to the National Residency Matching Program for the March 2000 Match.
The study's design consisted of a survey and a cohort analysis. Program directors of U.S. internal medicine residency programs were surveyed to determine the prevalence of the use of the USMLE Step 1 score when selecting applicants to interview. The questionnaire, a postcard, was mailed to each residency program listed in the membership listing of the Association of Program Directors in Internal Medicine. The questionnaire asked whether (1) the program was university-based or community-based, (2) the USMLE Step 1 score was used in the decision to grant an interview, and (3) the program had a minimum Step 1 score requirement to grant an interview.
In the cohort analysis, we constructed a database of USMLE Step 1 scores from the Electronic Residency Application Service (ERAS) database of applications from U.S., Canadian, and osteopathic medical students to our residency program for internship beginning July 2000. The 744 applicants represented 18% of the U.S. medical students matching in categorical, primary care, and preliminary internal medicine positions in the 2000 match.6 The applicant pool we analyzed contained students from 106 (85%) of the 125 U.S. medical schools.
Race is not reported in the ERAS application form. Therefore, race was assessed by review of the applicant's photograph. Each photograph was reviewed by two independent reviewers (without knowledge of the applicant's USMLE score) to determine whether the applicant was African American. Applicants determined to be any other race were pooled as non—African American. When disagreement occurred between the two reviewers, the applicant's personal statement was reviewed to ascertain whether the student had disclosed his or her racial status. If the applicant had not disclosed race, the record was eliminated from the database. Applicants without a USMLE Step 1 score were also excluded. Rejection rates were then calculated for each five-point increment from a hypothetical cutoff score of <180 to a cutoff score of <215.
We compared the proportion of rejected applicants by race at each hypothetical cutoff score using the chisquare test. Alpha was set at .05 and all tests of significance were two-tailed.
Program Directors' Survey
We received responses from 259 (69%) of the 377 residency programs we surveyed. Of the respondents, 122 classified their programs as being university-based and 137, as being community-based. Overall, 92% of the programs used the USMLE Step 1 score in the decision to grant an interview. Sixty percent used a minimum score requirement. There was no significant difference in the use of the scores between the university-based and community-based programs.
Applicant Cohort Analysis
Thirty-six of the 744 reviewed applications were eliminated because they reported no USMLE Step 1 score. Of these, 22 (61%) were those of osteopathic students. An additional 22 applicants were eliminated from the cohort due to absence of a photo. For the remaining 686 applicants, the two independent reviewers disagreed on the race of 18 individuals (3.1%). Five of the 18 had disclosed their racial status, leaving undetermined 13, who were eliminated from the analysis. The final database consisted of 626 non—African-American and 47 African-American applicants.
The mean USMLE Step 1 score for the African-American students in the applicant cohort was 200, while the mean score for the non—African-American students was 216. (The mean score of the 13 students excluded from the study because their race could not be determined was 206.) The proportion of applicants below each incremental hypothetical rejection score was significantly higher for African-American applicants (p < .05 at each level). Depending on the threshold score used, an African-American applicant was three to six times less likely to be offered an interview (see Table 1). As an example, if a residency program set the minimum score to grant an interview at 200, 53% of the African-American students would be rejected, whereas only 20% of the non—African-American students would be rejected (the odds ratio for rejection of an African-American applicant was 4.5).
Internal medicine residency programs are the largest of the postgraduate medical education training programs. Because many of these programs receive large numbers of applications, easily obtainable and efficient screening tools are desired to select applicants for interviews. From the ERAS application, those data that are commonly used in the interview decision are the USMLE Step 1 scores and medical school grades. Because dean's letters take significantly longer to review, they are often not reviewed until after the applicant has been invited and has accepted an interview. We previously reported the difficulties in using dean's letters in determining the quality of applicants.7 Our present study confirms that the vast majority of residency programs use the USMLE Step 1 score in the decision to offer an interview, and nearly two thirds of programs responding to our survey set a minimum score for invitations to interviews.
Because our cohort analysis was performed in only one residency program, the data may not be generalizable to other programs. However, the large size of the residency program and the large pool of applicants from representative medical schools across the country minimize this risk. The African-American students applying to our residency program represented 29 medical schools in 19 states. While the number of African Americans in our applicant pool was relatively small (n = 47, or 7%), it is important to note that only approximately 1,200 African-American students graduated each year from U.S. medical schools in 1997, 1998, and 1999.8 Assuming that the proportion of African-American students matching in internal medicine programs is similar to that of the entire graduating cohort in the United States (approximately 30%),8 we can estimate that the number of African-American students applying to our program represents approximately 13% of all African-American students applying to internal medicine programs that year. We could find only one published mean USMLE Step 1 score for African-American students in the literature.2 It is important to note, however, that the score reported in that study, 187, was lower than the mean USMLE Step 1 score of our African-American applicants, which was 200. This suggests that the African-American students in our applicant pool were at least as competitive academically as was the African-American cohort generally.
It is also possible that misclassification bias may have affected the results, since racial identification cannot always be accurately determined by observation. Since racial status is not listed on the ERAS application and only some of the applicants disclosed their race in their personal statements, other options were not available.
Over the past several years, program directors have scrutinized USMLE scores more closely. Because an increasing number of medical schools require passage of both USMLE Step 1 and USMLE Step 2 for graduation from medical school, program directors are leery of applicants who have posted marginally passing scores on Step 1 because they fear matching an applicant who may fail Step 2. In that case, the program director either attempts to fill the position post-match with an unmatched applicant or holds the position open in hopes that the applicant will pass the exam in the near future. Unfortunately, even after passage of the exam, the position may stay open, because some medical schools issue diplomas only twice yearly, and state licensing boards will not issue training licenses until the diploma has been issued.
Some hospitals have criteria for residency appointment regarding USMLE scores (e.g., passage of Step 2 or Step 3 by the end of the internship year). Students who perform marginally well on Step 1 may be seen as risky prospects for passing subsequent steps of the examination and, therefore, not be considered further.
Last, in internal medicine, another force has prompted program directors to scrutinize USMLE scores. In 1997, the American Board of Internal Medicine (ABIM) began posting the pass rate of each residency program on its Web site, giving residency applicants an additional measure by which to compare programs. A 1993 study found that a significant association existed between performance on the NBME Part I and performance on the ABIM's certifying exam.9 Presumably the same relationship exists between performance on USMLE Step 1 and performance on the ABIM exam. Therefore, program directors have attempted to boost their program's performance on the ABIM exam by recruiting applicants with higher USMLE scores. A solid pass rate on the ABIM exam can be used to recruit outstanding residents and ensure continued success of the training program.
The Council on Graduate Medical Education's 12th Report on Minorities in Medicine noted that performance on standardized tests “may be more predictive of science achievement than success as a physician.”10 Unfortunately, outcomes measures that assess a physician's competency are lacking and sorely needed. Until medical educators can develop such tools, reliance on incomplete measures of competency, such as USMLE scores, is likely to persist.
Unfortunately, the focus on USMLE scores has had a detrimental impact on minority medical students. Using the data from our residency program, setting a minimum score on the USMLE Step 1 to grant an interview would preferentially exclude African-American students. The end result of such use of USMLE Step 1 scores is that many programs, particularly the more competitive programs, will become less diverse. Program directors should be aware of this finding and policy-making bodies should discuss the implications of how USMLE scores are used.
Increased diversity in medicine has been identified as a virtue for medicine and humanity as a whole. To achieve optimal diversity, the issue of matriculating minority students and developing them into competent practitioners needs to be carefully analyzed and addressed by medical schools, policy groups, the NBME, and the specialty medical boards. Assessing the clinical competence of physicians with tools that are valid and fair remains one of medical education's greatest challenges.
1. SAT and ACT scores hold relatively stable. NCTM News Bulletin. 1999;36(4), Nov:1,4.
2. Koenig JA, Sireci SG, Wiley A. Evaluating the predictive validity of MCAT scores across diverse applicant groups. Acad Med. 1998;73:1095–106.
3. Wagoner NE, Suriano JR. Program directors' responses to a survey on variables used to select residents in a time of change. Acad Med. 1999;74:51–8.
4. Volle RL. Using National Board of Medical Examiners scores in selection of residents. JAMA. 1988;259:266.
5. Dawson B, Iwamoto CK, Ross LP, Nungester RJ, Swanson DB, Volle RL. Performance on the National Board of Medical Examiners Part I examination by men and women of different race and ethnicity. JAMA. 1994;272:674–6.
6. National Resident Matching Program. Results and Data: 2000 Match. Washington, DC: Association of American Medical Colleges, 2000.
7. Edmond MB, Roberson M, Hasan N. The dishonest dean's letter. Acad Med. 1999;74:1033–5.
8. Robinson L (ed). AAMC Data Book: Statistical Information Related to Medical School and Teaching Hospitals. Washington, DC: Association of American Medical Colleges, 2000.
9. Sosenko J, Stekel KW, Soto R, Gelbard M. NBME Examination Part I as a predictor of clinical and ABIM certifying examination performances. J Gen Intern Med. 1993;8:86–8.
10. Minorities in Medicine: 12th report of the Council on Graduate Medical Education. Available from: Council on Graduate Medical Education Web site, <http://www.cogme.gov/rpt12_2.htm
>, accessed 2/22/2000.