Kasuya, Richard T. MD, MSEd; Naguwa, Gwen S. MD; Guerrero, Anthony P.S. MD; Hishinuma, Earl S. PhD; Lindberg, Marlene A. PhD; Judd, Nanette K. PhD
At the University of Hawai'i John A. Burns School of Medicine (JABSOM), the unique ethnic composition of our student body reflects the diversity of Hawai'i, where no single ethnic majority group exists. Of significance, however, is that certain ethnicities remain highly underrepresented among physicians in Hawai'i. In 1995, 20% of Hawai'i's population was Native Hawaiian, while only 2.6% of the state's physicians were Native Hawaiian.1 The Association of American Medical Colleges (AAMC)2 includes Native Hawaiians on its list of underrepresented minority ethnic groups in the United States.
As part of its mission to serve the people of Hawai'i and the Pacific Basin, JABSOM actively recruits underrepresented minority students, including Native Hawaiians, Filipinos, Micronesians, Samoans, and recent Southeast Asian immigrants. Our special opportunities programs,3 in existence for the past 30 years, focus on both preparing these students for medical school and optimizing their academic success once they are accepted into medical school. According to the AAMC,2 JABSOM has graduated more Asian physicians than any other medical school in the country, and is one of 12 medical schools that have graduated nearly 30% of all underrepresented minority physicians.
In 1989, JABSOM implemented a problem-based learning (PBL) curriculum, in which essentially all of the material traditionally taught in lectures is now learned through case-based study in small groups. Studies of the success of PBL have indicated possible benefits in long-term retention of knowledge and lifelong interest in learning,4 and a recent report suggests that a PBL curriculum may have contributed to higher scores on the United States Medical Licensing Exam (USMLE).5 Nevertheless, a recent review of the outcomes of PBL in medical education has suggested the need for further research in this area.6
JABSOM provides a unique opportunity to further research issues involving minority students in a PBL curriculum. This study describes the student population graduating in the years 1996–2000. We have attempted to determine whether demographic and pre-admission performance variables [i.e., grade-point average (GPA), Medical College Admissions Test (MCAT) scores] predict USMLE scores in this group of students.
Previous studies suggest that, although most underrepresented minority students are ultimately successful in completing medical school, they are at higher risk of experiencing academic difficulty. For example, in a survival analysis of the 1992 matriculants to all U.S. medical schools, Huff and Fang7 found that underrepresented minorities were 97% more likely to encounter academic difficulty and that this risk did not stabilize until after the third year. Campos-Outcalt et al.8 found that underrepresented minorities at the University of Arizona College of Medicine achieved lower National Board of Medical Examiners (NBME) scores than did majority students. Case et al.9 reported different USMLE Step 1 first-try pass rates as follows: white = 93.4%; Asian = 86.8%; African American = 58.2%; Hispanic = 77.5%; and Other/unknown = 81.5%. Koenig et al.10 reported pass rates on the USMLE for four ethnic groups: African American = 69.3%; Asian = 93.7%; Caucasian = 96.8%; and Hispanic/Latino = 88.9%. To our knowledge, there has been no study specifically describing USMLE performances for Native Hawaiians, Filipinos, Micronesians, and other Polynesians as separate entities within the “Asian–Pacific Islander” group.
Investigators have identified ethnic differences in the prediction of USMLE scores based on previous measures of performance. In Koenig et al.'s study,10 significantly fewer Asian and Hispanic/Latino students passed the USMLE Step 1 on the first try than were predicted to by their scores on the MCAT. In contrast, Caucasian students tended to perform better than expected; that is, more Caucasians passed than would be expected based on their MCAT scores, although the difference between the percentage of students passing and what would be expected by chance was not statistically significant. Johnson et al.11 had previously reported that college GPAs and MCAT scores were less predictive of NBME scores for students at a predominantly African American medical school than for students at predominantly Caucasian American medical schools.
The purposes of this study were (1) to compare USMLE performances of students of various ethnicities, predominantly Pacific Islander and Asian, and (2) to examine the predictive validity of MCAT scores for USMLE performance among these students.
A total of 268 medical students graduated from JABSOM from 1996 to 2000. Ten students (nine members of the 1996 cohort who took the older version of the MCAT, and one student without a USMLE Step 1 score) were not included in the study. Therefore, the sample for this study consisted of 258 students.
Table 1 describes the sample's demographics (i.e., ethnicity, age, gender, cohort), as well as academic measures (i.e., undergraduate GPA, MCAT scores, USMLE scores). Compared with the proportions of major ethnic groups in the state of Hawai'i,12 JABSOM had higher percentages of Chinese and Japanese students and lower percentages of Hawaiian/part-Hawaiian, Caucasian, and Filipino students.
Age (rounded to the nearest integer) was calculated at the end of June of the students' graduation year. Gender (1 = male; 2 = female) and ethnicity were based on the students' self-reports in their records. Minority status is typically defined based on four categories: Asian, Black, Hispanic, and American Indian.2 However, the Asian category contains a heterogeneous mixture of ancestries (i.e., Chinese, Japanese, Korean, Filipino, Indian, Pakistani, Vietnamese, Native Hawaiian, other Pacific Islander, and other Asian subgroups). For this study, the Asian subgroups were disaggregated, and a total of nine ethnic groups were examined: Caucasian, Chinese, Filipino, Japanese, Other Asian, Mixed Asian, Hawaiian/part-Hawaiian, Other Pacific Islander, and Mixed/part-non-Asian (see below for description).
Because Native Hawaiians without any mixed ancestry constitute less than 1% of the population of Hawai'i,13 students were defined as being Hawaiian/part-Hawaiian if they reported having any Hawaiian ancestry. The students were further grouped to explore differences among those with diverse cultural backgrounds. Other Asians (e.g., Korean) were defined as individuals who had a single Asian ancestry but were not Chinese, Japanese, or Filipino. Mixed Asians were students with mixed Asian ancestry but no non-Asian ancestry. Other Pacific Islanders were individuals of a single Pacific Islander ancestry (e.g., Samoan) but were not Hawaiian/part-Hawaiian. Mixed/part-non-Asians were those who were of mixed ancestry, part of which was non-Asian (e.g., an individual who identified both Caucasian and Japanese ancestry).
Two measures of undergraduate performance were used: undergraduate science GPA and undergraduate cumulative GPA.
Six measures of first-try MCAT performance were used: (1) Verbal Reasoning scaled score; (2) Physical Science scaled score; (3) Biological Science scaled score; (4) Writing Sample raw score (where J = 4 to T = 14; as used by Koenig, Sireci, and Wiley10); (5) MCAT sum of scaled scores without Writing Sample; and (6) MCAT sum of scaled scores with Writing Sample.
In this study, for students who took Steps 1 or 2 more than once, only their first scores were used in the analysis.
Data were obtained from the Office of Student Affairs at JABSOM. All procedures were formally approved by the Committee on Human Studies (equivalent to an Institutional Review Board) of the University of Hawai'i at Mānoa.
Means and standard deviations were calculated for each ethnic group and for each of the predictor variables. Multivariate analysis of variance (MANOVA), followed by a series of univariate analysis of variance (ANOVA) procedures, was conducted to determine whether there were ethnic differences in ages and academic performances.
Using MANOVA significance as a test to proceed with further univariate analyses decreased the likelihood of obtaining statistically significant univariate findings by chance alone, and a statistically significant MANOVA Wilks' lambda allowed for the examination of each academic measure separately.
MCAT scores, with and without science GPA, were used as predictor variables in regression analyses designed to predict USMLE Step 1 and 2 composite scores. Given the relatively small sample sizes for some of the ethnic groups, a statistical approach similar to that used by Koenig et al.10 was employed, whereby a common regression equation was fit for all students within an ethnic group, and over- and under-prediction errors were examined for each ethnic group. The respective regression equations were used to calculate the predicted USMLE scores. If the predicted USMLE score was lower than the actual USMLE score, under-prediction occurred. Conversely, if the predicted USMLE score was higher than the actual USMLE score, over-prediction occurred.
Ethnic Differences in Ages and Academic Performances
Table 1 presents the comparisons of the different ethnic groups' demographic and academic variables. Disaggregating findings by ethnic group is important in order to discern finer differences among these groups. However, because there were only five Other Pacific Islanders, revealing their actual mean and standard deviation for each of the academic measures may compromise the confidentiality of their performances. Therefore, for Table 1 only, these five Other Pacific Islanders were combined with the Hawaiian/Part Hawaiian students.
The results indicated that Chinese and Other Asian students were younger than the Caucasian students. There was no statistically significant difference among ethnic groups by gender [χ2(7) = 5.47, p = .6027] or graduating class [χ2(28) = 34.15, p = .20]. These findings should be interpreted with caution, however, because many of the cell n sizes were less than 5.
Table 1 shows the overall F value for MCAT Writing Sample was not statistically significant. Although the overall F values for Undergraduate Science GPA and Cumulative GPA were statistically significant, Newman–Keuls subsequent test did not reveal any significant difference among the ethnic group means.
For the other academic measures, the pattern of results indicated that Chinese, Other Asians, and Caucasians tended to have higher scores, and Hawaiians/Other Pacific Islanders, and Filipinos tended to have lower scores. The variance accounted for by ethnicity ranged from .027 (MCAT Writing Sample) to a relatively substantial amount (for a demographic variable) of .194 (MCAT sum without Writing Sample).
MCAT without Science GPA Predicting USMLE Step 1 and 2 Scores
Table 2 presents the univariate linear regression analyses used to determine the predictive validity of the different MCAT scores for performances on USMLE Steps 1 and 2.
Writing Sample was not a statistically significant predictor of performances on USMLE Steps 1 and 2. A fair amount of variance was accounted for (r2 = .295 to .335) by Physical Science, Biological Science, MCAT sum without Writing Sample, and MCAT sum with Writing Sample in the prediction of performance on USMLE Step 1. Verbal Reasoning was also a significant predictor of USMLE Step 1 performance, but to a much lesser extent (r2 = .048). Compared with the USMLE Step 1 findings, the prediction of USMLE Step 2 performance was less impressive (r2 = .015 to .168), although statistically significant results were also found for all MCAT measures except Writing Sample (p = .0505 for the latter).
Based on the variance accounted for by the variables, the single best predictor of USMLE Step 1 performance was the MCAT sum without Writing Sample. However, the single best predictor of USMLE Step 2 performance was the MCAT sum with Writing Sample. In an effort to maximize accuracy in the prediction of USMLE performance, the MCAT sum without Writing Sample was used to predict USMLE Step 1 performance, and the MCAT sum with Writing Sample was used to predict USMLE Step 2 performance.
Table 3 presents the results for the under- and over-predictions by ethnic group. Although the percentages of under- and over-prediction varied by ethnic group, overall, there was no statistically significant proportional difference among the nine ethnic groups for USMLE Step 1 performance (χ2 = 6.25, p = .6190). However, the non-significant results may have been due to the small sample sizes (i.e., lower statistical power).
The same analyses were performed using the MCAT sum with Writing Sample to predict USMLE Step 2 scores. There was a tendency toward different under- and over-prediction percentages among the nine ethnic groups (χ2 = 14.4, p = .0728). Closer examination of the percentage difference between under- and over-prediction for each ethnic group revealed statistically significant over-prediction for Filipinos and a tendency toward under-prediction for Caucasians (p = .0934).
MCAT with Science GPA Predicting USMLE Step 1 and 2 Scores
In the previous analyses, the single best predictor was used to maximize the prediction of USMLE performance using only one predictor variable. To improve upon this prediction (i.e., variance accounted for), a multiple-regression approach was subsequently employed. However, because inter-correlated predictor variables would “cancel” one another out by predicting the same “shared” variation in the criterion variable when using a simultaneous multiple-regression approach, and because a stepwise multiple-regression approach would increase the likelihood of spurious results that might not be replicable, a sequential multiple-regression approach was utilized.
Given the importance placed upon Science GPA in both admissions and theory, Science GPA was also added to the model in the prediction of USMLE performance. It accounted for a statistically significant amount of variance when entered second into the model after each MCAT score. Therefore, a similar set of analyses was conducted with the addition of Science GPA as the second predictor (i.e., x2; see Table 4). MCAT sum without Writing Sample and with Science GPA were the best combined predictors of USMLE Step 1 scores. MCAT sum with Writing Sample and Science GPA were the best combined predictors of USMLE Step 2 scores.
Table 5 displays the under- and over-prediction percentages by ethnic group. Generally similar results were found in comparison with the MCAT score as the only predictor. There was an overall non-significant difference in the percentages among the nine ethnic groups (χ2 = 8.00, p = .4336) in predicting USMLE Step 1 performance. For the prediction of Step 2 performance, although there was an overall non-significant difference in the percentages among the nine ethnic groups (χ2 = 9.19, p = .3267), Filipinos were over-predicted once again.
The results of our study (1) reflect the substantial ethnic diversity among the students at JABSOM, (2) show that MCAT scores and Science GPA are important predictors of USMLE performance, and (3) demonstrate that there are ethnic differences in the degrees of prediction of USMLE performances, with particular differentiation in the prediction of USMLE Step 2 scores for Filipino medical students compared with all other ethnic groups.
The ethnic diversity of students at JABSOM reflects the richness of Hawai'i's population and culture. The 13.6% of Native Hawaiians in JABSOM's student body compares favorably to the 8.6% at the University of Hawai'i at Mānoa14 and provides the potential to increase the percentage of Native Hawaiians among practicing physicians in Hawai'i (now 2.6%).1 In addition, because the sample size for some of the ethnic groups in our study was relatively small and the influence of problem-based learning on the results is unknown, some of our findings must be interpreted with caution.
Our results appear to replicate the findings of other investigators regarding the predictive validity of MCAT and GPA scores for USMLE performance.8,10,11 We found that the single best predictor of USMLE Step 1 performance was the MCAT composite, as reflected in our sum without Writing Sample, and that the single best predictor for USMLE Step 2 performance was the MCAT composite with Writing Sample. Similarly, when predictors were combined, we found that the MCAT sum without Writing Sample and with Science GPA were the best combined predictors of USMLE Step 1 performance, and MCAT sum with Writing Sample and Science GPA were the best combined predictors of USMLE Step 2 scores. We need further research on the relationship of the composition of MCAT sum (with versus without Writing Sample) and performances in medical school and on the USMLE. In addition, the reason Writing Sample in isolation appeared to better predict USMLE Step 2 performance than Step 1 performance warrants further research. It should be noted that the variance accounted for by the Writing Sample in the univariate prediction of USMLE Step 2 was relatively small (r2 = 1.5%).
Our study adds to the existing literature because we studied an Asian/Pacific Island population not accessible elsewhere in the United States. Overall, JABSOM students' performances on USMLE Steps 1 and 2 were comparable to national averages. However, Hawaiians/other Pacific Islanders and Filipinos tended to score slightly lower on these exams. These findings are consistent with previous reports of underrepresented minorities' scoring lower than majority students on the USMLE.8,10,11
Although MCAT scores did not significantly under- or over-predict USMLE Step 1 scores for any ethnic group, the MCAT sum with Writing Sample alone, and it combined with Science GPA, did significantly over-predict USMLE Step 2 scores for Filipino medical students. The over-prediction may have been due to factors not included in our present study—cultural differences in learning styles and communication, primary languages used, reading skills (including decoding, comprehension, and automaticity), test-taking skills, and social and environmental factors (such as family support and financial responsibilities)—some of which previous authors15,16 have suggested are more predictive of overall success in medical school for minority students than for non-minority students. In this context, it is interesting to speculate on how the new Clinical Skills Examination being proposed by the USMLE will either decrease or increase the predictive validity of preadmission variables for the ethnic groups in question.
JABSOM has one of the most ethnically diverse student populations in the nation and has been successful in graduating underrepresented minorities, including Native Hawaiians and other Pacific Islanders. We are beginning to understand some of the unique precursors of academic success within and across these ethnic groups at JABSOM. We recognize that USMLE performance is only one indicator of achievement in medical school. We believe that more research on the predictors of performance of underrepresented groups in medical school is required to optimally address disparities, promote students' development, and create educational opportunities that best ensure success for all students.