Age (rounded to the nearest integer) was calculated at the end of June of the students' graduation year. Gender (1 = male; 2 = female) and ethnicity were based on the students' self-reports in their records. Minority status is typically defined based on four categories: Asian, Black, Hispanic, and American Indian.2 However, the Asian category contains a heterogeneous mixture of ancestries (i.e., Chinese, Japanese, Korean, Filipino, Indian, Pakistani, Vietnamese, Native Hawaiian, other Pacific Islander, and other Asian subgroups). For this study, the Asian subgroups were disaggregated, and a total of nine ethnic groups were examined: Caucasian, Chinese, Filipino, Japanese, Other Asian, Mixed Asian, Hawaiian/part-Hawaiian, Other Pacific Islander, and Mixed/part-non-Asian (see below for description).
Because Native Hawaiians without any mixed ancestry constitute less than 1% of the population of Hawai'i,13 students were defined as being Hawaiian/part-Hawaiian if they reported having any Hawaiian ancestry. The students were further grouped to explore differences among those with diverse cultural backgrounds. Other Asians (e.g., Korean) were defined as individuals who had a single Asian ancestry but were not Chinese, Japanese, or Filipino. Mixed Asians were students with mixed Asian ancestry but no non-Asian ancestry. Other Pacific Islanders were individuals of a single Pacific Islander ancestry (e.g., Samoan) but were not Hawaiian/part-Hawaiian. Mixed/part-non-Asians were those who were of mixed ancestry, part of which was non-Asian (e.g., an individual who identified both Caucasian and Japanese ancestry).
Two measures of undergraduate performance were used: undergraduate science GPA and undergraduate cumulative GPA.
Six measures of first-try MCAT performance were used: (1) Verbal Reasoning scaled score; (2) Physical Science scaled score; (3) Biological Science scaled score; (4) Writing Sample raw score (where J = 4 to T = 14; as used by Koenig, Sireci, and Wiley10); (5) MCAT sum of scaled scores without Writing Sample; and (6) MCAT sum of scaled scores with Writing Sample.
In this study, for students who took Steps 1 or 2 more than once, only their first scores were used in the analysis.
Data were obtained from the Office of Student Affairs at JABSOM. All procedures were formally approved by the Committee on Human Studies (equivalent to an Institutional Review Board) of the University of Hawai'i at Mānoa.
Means and standard deviations were calculated for each ethnic group and for each of the predictor variables. Multivariate analysis of variance (MANOVA), followed by a series of univariate analysis of variance (ANOVA) procedures, was conducted to determine whether there were ethnic differences in ages and academic performances.
Using MANOVA significance as a test to proceed with further univariate analyses decreased the likelihood of obtaining statistically significant univariate findings by chance alone, and a statistically significant MANOVA Wilks' lambda allowed for the examination of each academic measure separately.
MCAT scores, with and without science GPA, were used as predictor variables in regression analyses designed to predict USMLE Step 1 and 2 composite scores. Given the relatively small sample sizes for some of the ethnic groups, a statistical approach similar to that used by Koenig et al.10 was employed, whereby a common regression equation was fit for all students within an ethnic group, and over- and under-prediction errors were examined for each ethnic group. The respective regression equations were used to calculate the predicted USMLE scores. If the predicted USMLE score was lower than the actual USMLE score, under-prediction occurred. Conversely, if the predicted USMLE score was higher than the actual USMLE score, over-prediction occurred.
Ethnic Differences in Ages and Academic Performances
Table 1 presents the comparisons of the different ethnic groups' demographic and academic variables. Disaggregating findings by ethnic group is important in order to discern finer differences among these groups. However, because there were only five Other Pacific Islanders, revealing their actual mean and standard deviation for each of the academic measures may compromise the confidentiality of their performances. Therefore, for Table 1 only, these five Other Pacific Islanders were combined with the Hawaiian/Part Hawaiian students.
The results indicated that Chinese and Other Asian students were younger than the Caucasian students. There was no statistically significant difference among ethnic groups by gender [χ2(7) = 5.47, p = .6027] or graduating class [χ2(28) = 34.15, p = .20]. These findings should be interpreted with caution, however, because many of the cell n sizes were less than 5.
Table 1 shows the overall F value for MCAT Writing Sample was not statistically significant. Although the overall F values for Undergraduate Science GPA and Cumulative GPA were statistically significant, Newman–Keuls subsequent test did not reveal any significant difference among the ethnic group means.
For the other academic measures, the pattern of results indicated that Chinese, Other Asians, and Caucasians tended to have higher scores, and Hawaiians/Other Pacific Islanders, and Filipinos tended to have lower scores. The variance accounted for by ethnicity ranged from .027 (MCAT Writing Sample) to a relatively substantial amount (for a demographic variable) of .194 (MCAT sum without Writing Sample).
MCAT without Science GPA Predicting USMLE Step 1 and 2 Scores
Table 2 presents the univariate linear regression analyses used to determine the predictive validity of the different MCAT scores for performances on USMLE Steps 1 and 2.
Writing Sample was not a statistically significant predictor of performances on USMLE Steps 1 and 2. A fair amount of variance was accounted for (r2 = .295 to .335) by Physical Science, Biological Science, MCAT sum without Writing Sample, and MCAT sum with Writing Sample in the prediction of performance on USMLE Step 1. Verbal Reasoning was also a significant predictor of USMLE Step 1 performance, but to a much lesser extent (r2 = .048). Compared with the USMLE Step 1 findings, the prediction of USMLE Step 2 performance was less impressive (r2 = .015 to .168), although statistically significant results were also found for all MCAT measures except Writing Sample (p = .0505 for the latter).
Based on the variance accounted for by the variables, the single best predictor of USMLE Step 1 performance was the MCAT sum without Writing Sample. However, the single best predictor of USMLE Step 2 performance was the MCAT sum with Writing Sample. In an effort to maximize accuracy in the prediction of USMLE performance, the MCAT sum without Writing Sample was used to predict USMLE Step 1 performance, and the MCAT sum with Writing Sample was used to predict USMLE Step 2 performance.
Table 3 presents the results for the under- and over-predictions by ethnic group. Although the percentages of under- and over-prediction varied by ethnic group, overall, there was no statistically significant proportional difference among the nine ethnic groups for USMLE Step 1 performance (χ2 = 6.25, p = .6190). However, the non-significant results may have been due to the small sample sizes (i.e., lower statistical power).
The same analyses were performed using the MCAT sum with Writing Sample to predict USMLE Step 2 scores. There was a tendency toward different under- and over-prediction percentages among the nine ethnic groups (χ2 = 14.4, p = .0728). Closer examination of the percentage difference between under- and over-prediction for each ethnic group revealed statistically significant over-prediction for Filipinos and a tendency toward under-prediction for Caucasians (p = .0934).
MCAT with Science GPA Predicting USMLE Step 1 and 2 Scores
In the previous analyses, the single best predictor was used to maximize the prediction of USMLE performance using only one predictor variable. To improve upon this prediction (i.e., variance accounted for), a multiple-regression approach was subsequently employed. However, because inter-correlated predictor variables would “cancel” one another out by predicting the same “shared” variation in the criterion variable when using a simultaneous multiple-regression approach, and because a stepwise multiple-regression approach would increase the likelihood of spurious results that might not be replicable, a sequential multiple-regression approach was utilized.
Given the importance placed upon Science GPA in both admissions and theory, Science GPA was also added to the model in the prediction of USMLE performance. It accounted for a statistically significant amount of variance when entered second into the model after each MCAT score. Therefore, a similar set of analyses was conducted with the addition of Science GPA as the second predictor (i.e., x2; see Table 4). MCAT sum without Writing Sample and with Science GPA were the best combined predictors of USMLE Step 1 scores. MCAT sum with Writing Sample and Science GPA were the best combined predictors of USMLE Step 2 scores.
Table 5 displays the under- and over-prediction percentages by ethnic group. Generally similar results were found in comparison with the MCAT score as the only predictor. There was an overall non-significant difference in the percentages among the nine ethnic groups (χ2 = 8.00, p = .4336) in predicting USMLE Step 1 performance. For the prediction of Step 2 performance, although there was an overall non-significant difference in the percentages among the nine ethnic groups (χ2 = 9.19, p = .3267), Filipinos were over-predicted once again.
The results of our study (1) reflect the substantial ethnic diversity among the students at JABSOM, (2) show that MCAT scores and Science GPA are important predictors of USMLE performance, and (3) demonstrate that there are ethnic differences in the degrees of prediction of USMLE performances, with particular differentiation in the prediction of USMLE Step 2 scores for Filipino medical students compared with all other ethnic groups.
The ethnic diversity of students at JABSOM reflects the richness of Hawai'i's population and culture. The 13.6% of Native Hawaiians in JABSOM's student body compares favorably to the 8.6% at the University of Hawai'i at Mānoa14 and provides the potential to increase the percentage of Native Hawaiians among practicing physicians in Hawai'i (now 2.6%).1 In addition, because the sample size for some of the ethnic groups in our study was relatively small and the influence of problem-based learning on the results is unknown, some of our findings must be interpreted with caution.
Our results appear to replicate the findings of other investigators regarding the predictive validity of MCAT and GPA scores for USMLE performance.8,10,11 We found that the single best predictor of USMLE Step 1 performance was the MCAT composite, as reflected in our sum without Writing Sample, and that the single best predictor for USMLE Step 2 performance was the MCAT composite with Writing Sample. Similarly, when predictors were combined, we found that the MCAT sum without Writing Sample and with Science GPA were the best combined predictors of USMLE Step 1 performance, and MCAT sum with Writing Sample and Science GPA were the best combined predictors of USMLE Step 2 scores. We need further research on the relationship of the composition of MCAT sum (with versus without Writing Sample) and performances in medical school and on the USMLE. In addition, the reason Writing Sample in isolation appeared to better predict USMLE Step 2 performance than Step 1 performance warrants further research. It should be noted that the variance accounted for by the Writing Sample in the univariate prediction of USMLE Step 2 was relatively small (r2 = 1.5%).
Our study adds to the existing literature because we studied an Asian/Pacific Island population not accessible elsewhere in the United States. Overall, JABSOM students' performances on USMLE Steps 1 and 2 were comparable to national averages. However, Hawaiians/other Pacific Islanders and Filipinos tended to score slightly lower on these exams. These findings are consistent with previous reports of underrepresented minorities' scoring lower than majority students on the USMLE.8,10,11
Although MCAT scores did not significantly under- or over-predict USMLE Step 1 scores for any ethnic group, the MCAT sum with Writing Sample alone, and it combined with Science GPA, did significantly over-predict USMLE Step 2 scores for Filipino medical students. The over-prediction may have been due to factors not included in our present study—cultural differences in learning styles and communication, primary languages used, reading skills (including decoding, comprehension, and automaticity), test-taking skills, and social and environmental factors (such as family support and financial responsibilities)—some of which previous authors15,16 have suggested are more predictive of overall success in medical school for minority students than for non-minority students. In this context, it is interesting to speculate on how the new Clinical Skills Examination being proposed by the USMLE will either decrease or increase the predictive validity of preadmission variables for the ethnic groups in question.
JABSOM has one of the most ethnically diverse student populations in the nation and has been successful in graduating underrepresented minorities, including Native Hawaiians and other Pacific Islanders. We are beginning to understand some of the unique precursors of academic success within and across these ethnic groups at JABSOM. We recognize that USMLE performance is only one indicator of achievement in medical school. We believe that more research on the predictors of performance of underrepresented groups in medical school is required to optimally address disparities, promote students' development, and create educational opportunities that best ensure success for all students.
1. Blaisdell RK. 1995 update on Kanaka Maoli (indigenous Hawaiian) health. Asian American and Pacific Island Journal of Health. 1996;4:160–5.
2. Association of American Medical Colleges. Minority Graduates of U.S. Medical Schools: Trends, 1950–1998. Washington, DC: Association of American Medical Colleges, 2000.
3. Little D, Izutsu S, Judd N, Else I. A medical school-based program to encourage Native Hawaiians to choose medical careers. Acad Med. 1999;74:339–41.
4. Norman GR, Schmidt HG. The psychological basis of problem-based learning: a review of the evidence. Acad Med. 1992;67:557–65.
5. Blake RL, Hosokawa MD, Riley SL. Student performances on Step 1 and Step 2 of the United States Medical Licensing Examination following implementation of a problem-based learning curriculum. Acad Med. 2000;75:66–70.
6. Colliver JA. Effectiveness of problem-based learning curricula: research and theory. Acad Med. 2000;75:259–66.
7. Huff KL, Fang D. When are students most at risk of encountering academic difficulty: a study of the 1992 matriculants to U.S. medical schools. Acad Med. 1999;74:453–60.
8. Campos-Outcalt D, Rutala PJ, Witzke DB, Fulginiti JV. Performances of underrepresented-minority students at the University of Arizona College of Medicine, 1987–1991. Acad Med. 1994;69:577–82.
9. Case SM, Swanson DB, Ripkey DR, Bowles LT, Melnick DE. Performance of the class of 1994 in the new era of USMLE. Acad Med. 1996;71(10 suppl):S91–S93.
10. Koenig JA, Sireci SG, Wiley A. Evaluating the predictive validity of MCAT scores across diverse applicant groups. Acad Med. 1998;73:1095–106.
11. Johnson DG, Lloyd SM, Jones RF, Anderson J. Predicting academic performance at a predominately black medical school. J Med Educ. 1996;61:629–39.
12. Department of Business, Economic Development and Tourism, State of Hawai'i. The State of Hawai'i Data Book, 1997: A Statistical Abstract. Honolulu, HI: Department of Business, Economic Development and Tourism, State of Hawai'i, 1998.
13. Department of Business, Economic Development and Tourism, State of Hawai'i. The State of Hawai'i Data Book, 1995: A Statistical Abstract. Honolulu, HI: Department of Business, Economic Development and Tourism, State of Hawai'i, 1995.
14. Institutional Research Office, University of Hawai'i. Fall Enrollment Report: University of Hawai'i, Fall 2000. Honolulu, HI: University of Hawai'i, 2000.
15. Kerbeshian LA. Predicting and fostering success of American Indians in medical school. Acad Med. 1989;64:396–400.
© 2003 Association of American Medical Colleges
16. Malate A. Unpublished thesis, 1997. University of Hawai'i, Honolulu, HI.