Skip Navigation LinksHome > May 2003 - Volume 78 - Issue 5 > USMLE Performances in a Predominantly Asian and Pacific Isla...
Academic Medicine:
Special Theme Research Reports

USMLE Performances in a Predominantly Asian and Pacific Islander Population of Medical Students in a Problem‐based Learning Curriculum

Kasuya, Richard T. MD, MSEd; Naguwa, Gwen S. MD; Guerrero, Anthony P.S. MD; Hishinuma, Earl S. PhD; Lindberg, Marlene A. PhD; Judd, Nanette K. PhD

Free Access
Article Outline
Collapse Box

Author Information

Dr. Kasuya is director, Office of Medical Education, and assistant dean for medical education; Dr. Naguwa is associate professor, Department of Pediatrics/Office of Medical Education, and director of medical student education in pediatrics; Dr. Guerrero is assistant professor, Departments of Pediatrics and Psychiatry, and director of medical student education in psychiatry; Dr. Hishinuma is associate professor and vice chair of research, Department of Psychiatry, and associate director of the Native Hawaiian Mental Health Research Development Program; Dr. Lindberg is director, Office of Medical Education Program Evaluation and Student Assessment; Dr. Judd is specialist and director, Imi Ho'ola Post Baccalaureate Program; all with the University of Hawai'i John A. Burns School of Medicine, Honolulu, Hawai'i.

Correspondence and requests for reprints should be addressed to Dr. Kasuya, Office of Medical Education, University of Hawai'i John A. Burns School of Medicine, 1960 East–West Road, Honolulu, HI 96822; e-mail: 〈kasuya@hawaii.edu〉.

Collapse Box

Abstract

Purpose: To compare the USMLE performances of students of various ethnicities, predominantly Pacific Islander and Asian, at one medical school and to examine the predictive validity of MCAT scores for USMLE performance.

Method: A total of 258 students in the graduating classes of 1996–2000 at the University of Hawai'i School of Medicine were classified by ethnicity. Demographic and performance characteristics of the groups were examined, and MCAT scores with and without undergraduate science GPA were used to predict USMLE performance. Under- and over-prediction rates were computed for each ethnic group.

Results: Ethnic groups did not differ significantly by gender or undergraduate GPA. Chinese, Caucasian, and Other Asian students tended to have higher MCAT scores than Hawaiian/other Pacific Islander, and Filipino students. Ethnic groups did not differ significantly in prediction of USMLE Step 1 performance. For Step 2, MCAT scores significantly over-predicted performance of Filipino students and tended to under-predict performance of Caucasian students.

Conclusion: Although MCAT scores and science GPA were good predictors of USMLE performance, ethnic differences were found in the degrees of their predictive validity. These findings both replicate and extend results of earlier studies, and again point to the importance of exploring additional predictor variables. The authors encourage future research on the effects of the following factors on success in medical school: reading and test-taking skills, socio-cultural and environmental influences on learning, communication styles, primary language use, family support, and family responsibilities.

At the University of Hawai'i John A. Burns School of Medicine (JABSOM), the unique ethnic composition of our student body reflects the diversity of Hawai'i, where no single ethnic majority group exists. Of significance, however, is that certain ethnicities remain highly underrepresented among physicians in Hawai'i. In 1995, 20% of Hawai'i's population was Native Hawaiian, while only 2.6% of the state's physicians were Native Hawaiian.1 The Association of American Medical Colleges (AAMC)2 includes Native Hawaiians on its list of underrepresented minority ethnic groups in the United States.

As part of its mission to serve the people of Hawai'i and the Pacific Basin, JABSOM actively recruits underrepresented minority students, including Native Hawaiians, Filipinos, Micronesians, Samoans, and recent Southeast Asian immigrants. Our special opportunities programs,3 in existence for the past 30 years, focus on both preparing these students for medical school and optimizing their academic success once they are accepted into medical school. According to the AAMC,2 JABSOM has graduated more Asian physicians than any other medical school in the country, and is one of 12 medical schools that have graduated nearly 30% of all underrepresented minority physicians.

In 1989, JABSOM implemented a problem-based learning (PBL) curriculum, in which essentially all of the material traditionally taught in lectures is now learned through case-based study in small groups. Studies of the success of PBL have indicated possible benefits in long-term retention of knowledge and lifelong interest in learning,4 and a recent report suggests that a PBL curriculum may have contributed to higher scores on the United States Medical Licensing Exam (USMLE).5 Nevertheless, a recent review of the outcomes of PBL in medical education has suggested the need for further research in this area.6

JABSOM provides a unique opportunity to further research issues involving minority students in a PBL curriculum. This study describes the student population graduating in the years 1996–2000. We have attempted to determine whether demographic and pre-admission performance variables [i.e., grade-point average (GPA), Medical College Admissions Test (MCAT) scores] predict USMLE scores in this group of students.

Previous studies suggest that, although most underrepresented minority students are ultimately successful in completing medical school, they are at higher risk of experiencing academic difficulty. For example, in a survival analysis of the 1992 matriculants to all U.S. medical schools, Huff and Fang7 found that underrepresented minorities were 97% more likely to encounter academic difficulty and that this risk did not stabilize until after the third year. Campos-Outcalt et al.8 found that underrepresented minorities at the University of Arizona College of Medicine achieved lower National Board of Medical Examiners (NBME) scores than did majority students. Case et al.9 reported different USMLE Step 1 first-try pass rates as follows: white = 93.4%; Asian = 86.8%; African American = 58.2%; Hispanic = 77.5%; and Other/unknown = 81.5%. Koenig et al.10 reported pass rates on the USMLE for four ethnic groups: African American = 69.3%; Asian = 93.7%; Caucasian = 96.8%; and Hispanic/Latino = 88.9%. To our knowledge, there has been no study specifically describing USMLE performances for Native Hawaiians, Filipinos, Micronesians, and other Polynesians as separate entities within the “Asian–Pacific Islander” group.

Investigators have identified ethnic differences in the prediction of USMLE scores based on previous measures of performance. In Koenig et al.'s study,10 significantly fewer Asian and Hispanic/Latino students passed the USMLE Step 1 on the first try than were predicted to by their scores on the MCAT. In contrast, Caucasian students tended to perform better than expected; that is, more Caucasians passed than would be expected based on their MCAT scores, although the difference between the percentage of students passing and what would be expected by chance was not statistically significant. Johnson et al.11 had previously reported that college GPAs and MCAT scores were less predictive of NBME scores for students at a predominantly African American medical school than for students at predominantly Caucasian American medical schools.

The purposes of this study were (1) to compare USMLE performances of students of various ethnicities, predominantly Pacific Islander and Asian, and (2) to examine the predictive validity of MCAT scores for USMLE performance among these students.

Back to Top | Article Outline

METHOD

Sample

A total of 268 medical students graduated from JABSOM from 1996 to 2000. Ten students (nine members of the 1996 cohort who took the older version of the MCAT, and one student without a USMLE Step 1 score) were not included in the study. Therefore, the sample for this study consisted of 258 students.

Table 1 describes the sample's demographics (i.e., ethnicity, age, gender, cohort), as well as academic measures (i.e., undergraduate GPA, MCAT scores, USMLE scores). Compared with the proportions of major ethnic groups in the state of Hawai'i,12 JABSOM had higher percentages of Chinese and Japanese students and lower percentages of Hawaiian/part-Hawaiian, Caucasian, and Filipino students.

Table 1
Table 1
Image Tools
Table 1
Table 1
Image Tools
Back to Top | Article Outline
Measures
Demographic

Age (rounded to the nearest integer) was calculated at the end of June of the students' graduation year. Gender (1 = male; 2 = female) and ethnicity were based on the students' self-reports in their records. Minority status is typically defined based on four categories: Asian, Black, Hispanic, and American Indian.2 However, the Asian category contains a heterogeneous mixture of ancestries (i.e., Chinese, Japanese, Korean, Filipino, Indian, Pakistani, Vietnamese, Native Hawaiian, other Pacific Islander, and other Asian subgroups). For this study, the Asian subgroups were disaggregated, and a total of nine ethnic groups were examined: Caucasian, Chinese, Filipino, Japanese, Other Asian, Mixed Asian, Hawaiian/part-Hawaiian, Other Pacific Islander, and Mixed/part-non-Asian (see below for description).

Because Native Hawaiians without any mixed ancestry constitute less than 1% of the population of Hawai'i,13 students were defined as being Hawaiian/part-Hawaiian if they reported having any Hawaiian ancestry. The students were further grouped to explore differences among those with diverse cultural backgrounds. Other Asians (e.g., Korean) were defined as individuals who had a single Asian ancestry but were not Chinese, Japanese, or Filipino. Mixed Asians were students with mixed Asian ancestry but no non-Asian ancestry. Other Pacific Islanders were individuals of a single Pacific Islander ancestry (e.g., Samoan) but were not Hawaiian/part-Hawaiian. Mixed/part-non-Asians were those who were of mixed ancestry, part of which was non-Asian (e.g., an individual who identified both Caucasian and Japanese ancestry).

Back to Top | Article Outline
Undergraduate GPAs

Two measures of undergraduate performance were used: undergraduate science GPA and undergraduate cumulative GPA.

Back to Top | Article Outline
MCAT

Six measures of first-try MCAT performance were used: (1) Verbal Reasoning scaled score; (2) Physical Science scaled score; (3) Biological Science scaled score; (4) Writing Sample raw score (where J = 4 to T = 14; as used by Koenig, Sireci, and Wiley10); (5) MCAT sum of scaled scores without Writing Sample; and (6) MCAT sum of scaled scores with Writing Sample.

Back to Top | Article Outline
USMLE

In this study, for students who took Steps 1 or 2 more than once, only their first scores were used in the analysis.

Back to Top | Article Outline
Procedures

Data were obtained from the Office of Student Affairs at JABSOM. All procedures were formally approved by the Committee on Human Studies (equivalent to an Institutional Review Board) of the University of Hawai'i at Mānoa.

Back to Top | Article Outline
Statistical Analyses

Means and standard deviations were calculated for each ethnic group and for each of the predictor variables. Multivariate analysis of variance (MANOVA), followed by a series of univariate analysis of variance (ANOVA) procedures, was conducted to determine whether there were ethnic differences in ages and academic performances.

Using MANOVA significance as a test to proceed with further univariate analyses decreased the likelihood of obtaining statistically significant univariate findings by chance alone, and a statistically significant MANOVA Wilks' lambda allowed for the examination of each academic measure separately.

MCAT scores, with and without science GPA, were used as predictor variables in regression analyses designed to predict USMLE Step 1 and 2 composite scores. Given the relatively small sample sizes for some of the ethnic groups, a statistical approach similar to that used by Koenig et al.10 was employed, whereby a common regression equation was fit for all students within an ethnic group, and over- and under-prediction errors were examined for each ethnic group. The respective regression equations were used to calculate the predicted USMLE scores. If the predicted USMLE score was lower than the actual USMLE score, under-prediction occurred. Conversely, if the predicted USMLE score was higher than the actual USMLE score, over-prediction occurred.

Back to Top | Article Outline

RESULTS

Ethnic Differences in Ages and Academic Performances

Table 1 presents the comparisons of the different ethnic groups' demographic and academic variables. Disaggregating findings by ethnic group is important in order to discern finer differences among these groups. However, because there were only five Other Pacific Islanders, revealing their actual mean and standard deviation for each of the academic measures may compromise the confidentiality of their performances. Therefore, for Table 1 only, these five Other Pacific Islanders were combined with the Hawaiian/Part Hawaiian students.

The results indicated that Chinese and Other Asian students were younger than the Caucasian students. There was no statistically significant difference among ethnic groups by gender [χ2(7) = 5.47, p = .6027] or graduating class [χ2(28) = 34.15, p = .20]. These findings should be interpreted with caution, however, because many of the cell n sizes were less than 5.

Table 1 shows the overall F value for MCAT Writing Sample was not statistically significant. Although the overall F values for Undergraduate Science GPA and Cumulative GPA were statistically significant, Newman–Keuls subsequent test did not reveal any significant difference among the ethnic group means.

For the other academic measures, the pattern of results indicated that Chinese, Other Asians, and Caucasians tended to have higher scores, and Hawaiians/Other Pacific Islanders, and Filipinos tended to have lower scores. The variance accounted for by ethnicity ranged from .027 (MCAT Writing Sample) to a relatively substantial amount (for a demographic variable) of .194 (MCAT sum without Writing Sample).

Back to Top | Article Outline
MCAT without Science GPA Predicting USMLE Step 1 and 2 Scores

Table 2 presents the univariate linear regression analyses used to determine the predictive validity of the different MCAT scores for performances on USMLE Steps 1 and 2.

Table 2
Table 2
Image Tools

Writing Sample was not a statistically significant predictor of performances on USMLE Steps 1 and 2. A fair amount of variance was accounted for (r2 = .295 to .335) by Physical Science, Biological Science, MCAT sum without Writing Sample, and MCAT sum with Writing Sample in the prediction of performance on USMLE Step 1. Verbal Reasoning was also a significant predictor of USMLE Step 1 performance, but to a much lesser extent (r2 = .048). Compared with the USMLE Step 1 findings, the prediction of USMLE Step 2 performance was less impressive (r2 = .015 to .168), although statistically significant results were also found for all MCAT measures except Writing Sample (p = .0505 for the latter).

Based on the variance accounted for by the variables, the single best predictor of USMLE Step 1 performance was the MCAT sum without Writing Sample. However, the single best predictor of USMLE Step 2 performance was the MCAT sum with Writing Sample. In an effort to maximize accuracy in the prediction of USMLE performance, the MCAT sum without Writing Sample was used to predict USMLE Step 1 performance, and the MCAT sum with Writing Sample was used to predict USMLE Step 2 performance.

Table 3 presents the results for the under- and over-predictions by ethnic group. Although the percentages of under- and over-prediction varied by ethnic group, overall, there was no statistically significant proportional difference among the nine ethnic groups for USMLE Step 1 performance (χ2[8] = 6.25, p = .6190). However, the non-significant results may have been due to the small sample sizes (i.e., lower statistical power).

Table 3
Table 3
Image Tools

The same analyses were performed using the MCAT sum with Writing Sample to predict USMLE Step 2 scores. There was a tendency toward different under- and over-prediction percentages among the nine ethnic groups (χ2[8] = 14.4, p = .0728). Closer examination of the percentage difference between under- and over-prediction for each ethnic group revealed statistically significant over-prediction for Filipinos and a tendency toward under-prediction for Caucasians (p = .0934).

Back to Top | Article Outline
MCAT with Science GPA Predicting USMLE Step 1 and 2 Scores

In the previous analyses, the single best predictor was used to maximize the prediction of USMLE performance using only one predictor variable. To improve upon this prediction (i.e., variance accounted for), a multiple-regression approach was subsequently employed. However, because inter-correlated predictor variables would “cancel” one another out by predicting the same “shared” variation in the criterion variable when using a simultaneous multiple-regression approach, and because a stepwise multiple-regression approach would increase the likelihood of spurious results that might not be replicable, a sequential multiple-regression approach was utilized.

Given the importance placed upon Science GPA in both admissions and theory, Science GPA was also added to the model in the prediction of USMLE performance. It accounted for a statistically significant amount of variance when entered second into the model after each MCAT score. Therefore, a similar set of analyses was conducted with the addition of Science GPA as the second predictor (i.e., x2; see Table 4). MCAT sum without Writing Sample and with Science GPA were the best combined predictors of USMLE Step 1 scores. MCAT sum with Writing Sample and Science GPA were the best combined predictors of USMLE Step 2 scores.

Table 4
Table 4
Image Tools

Table 5 displays the under- and over-prediction percentages by ethnic group. Generally similar results were found in comparison with the MCAT score as the only predictor. There was an overall non-significant difference in the percentages among the nine ethnic groups (χ2[8] = 8.00, p = .4336) in predicting USMLE Step 1 performance. For the prediction of Step 2 performance, although there was an overall non-significant difference in the percentages among the nine ethnic groups (χ2[8] = 9.19, p = .3267), Filipinos were over-predicted once again.

Table 5
Table 5
Image Tools
Back to Top | Article Outline

DISCUSSION

The results of our study (1) reflect the substantial ethnic diversity among the students at JABSOM, (2) show that MCAT scores and Science GPA are important predictors of USMLE performance, and (3) demonstrate that there are ethnic differences in the degrees of prediction of USMLE performances, with particular differentiation in the prediction of USMLE Step 2 scores for Filipino medical students compared with all other ethnic groups.

The ethnic diversity of students at JABSOM reflects the richness of Hawai'i's population and culture. The 13.6% of Native Hawaiians in JABSOM's student body compares favorably to the 8.6% at the University of Hawai'i at Mānoa14 and provides the potential to increase the percentage of Native Hawaiians among practicing physicians in Hawai'i (now 2.6%).1 In addition, because the sample size for some of the ethnic groups in our study was relatively small and the influence of problem-based learning on the results is unknown, some of our findings must be interpreted with caution.

Our results appear to replicate the findings of other investigators regarding the predictive validity of MCAT and GPA scores for USMLE performance.8,10,11 We found that the single best predictor of USMLE Step 1 performance was the MCAT composite, as reflected in our sum without Writing Sample, and that the single best predictor for USMLE Step 2 performance was the MCAT composite with Writing Sample. Similarly, when predictors were combined, we found that the MCAT sum without Writing Sample and with Science GPA were the best combined predictors of USMLE Step 1 performance, and MCAT sum with Writing Sample and Science GPA were the best combined predictors of USMLE Step 2 scores. We need further research on the relationship of the composition of MCAT sum (with versus without Writing Sample) and performances in medical school and on the USMLE. In addition, the reason Writing Sample in isolation appeared to better predict USMLE Step 2 performance than Step 1 performance warrants further research. It should be noted that the variance accounted for by the Writing Sample in the univariate prediction of USMLE Step 2 was relatively small (r2 = 1.5%).

Our study adds to the existing literature because we studied an Asian/Pacific Island population not accessible elsewhere in the United States. Overall, JABSOM students' performances on USMLE Steps 1 and 2 were comparable to national averages. However, Hawaiians/other Pacific Islanders and Filipinos tended to score slightly lower on these exams. These findings are consistent with previous reports of underrepresented minorities' scoring lower than majority students on the USMLE.8,10,11

Although MCAT scores did not significantly under- or over-predict USMLE Step 1 scores for any ethnic group, the MCAT sum with Writing Sample alone, and it combined with Science GPA, did significantly over-predict USMLE Step 2 scores for Filipino medical students. The over-prediction may have been due to factors not included in our present study—cultural differences in learning styles and communication, primary languages used, reading skills (including decoding, comprehension, and automaticity), test-taking skills, and social and environmental factors (such as family support and financial responsibilities)—some of which previous authors15,16 have suggested are more predictive of overall success in medical school for minority students than for non-minority students. In this context, it is interesting to speculate on how the new Clinical Skills Examination being proposed by the USMLE will either decrease or increase the predictive validity of preadmission variables for the ethnic groups in question.

JABSOM has one of the most ethnically diverse student populations in the nation and has been successful in graduating underrepresented minorities, including Native Hawaiians and other Pacific Islanders. We are beginning to understand some of the unique precursors of academic success within and across these ethnic groups at JABSOM. We recognize that USMLE performance is only one indicator of achievement in medical school. We believe that more research on the predictors of performance of underrepresented groups in medical school is required to optimally address disparities, promote students' development, and create educational opportunities that best ensure success for all students.

Back to Top | Article Outline

REFERENCES

1. Blaisdell RK. 1995 update on Kanaka Maoli (indigenous Hawaiian) health. Asian American and Pacific Island Journal of Health. 1996;4:160–5.

2. Association of American Medical Colleges. Minority Graduates of U.S. Medical Schools: Trends, 1950–1998. Washington, DC: Association of American Medical Colleges, 2000.

3. Little D, Izutsu S, Judd N, Else I. A medical school-based program to encourage Native Hawaiians to choose medical careers. Acad Med. 1999;74:339–41.

4. Norman GR, Schmidt HG. The psychological basis of problem-based learning: a review of the evidence. Acad Med. 1992;67:557–65.

5. Blake RL, Hosokawa MD, Riley SL. Student performances on Step 1 and Step 2 of the United States Medical Licensing Examination following implementation of a problem-based learning curriculum. Acad Med. 2000;75:66–70.

6. Colliver JA. Effectiveness of problem-based learning curricula: research and theory. Acad Med. 2000;75:259–66.

7. Huff KL, Fang D. When are students most at risk of encountering academic difficulty: a study of the 1992 matriculants to U.S. medical schools. Acad Med. 1999;74:453–60.

8. Campos-Outcalt D, Rutala PJ, Witzke DB, Fulginiti JV. Performances of underrepresented-minority students at the University of Arizona College of Medicine, 1987–1991. Acad Med. 1994;69:577–82.

9. Case SM, Swanson DB, Ripkey DR, Bowles LT, Melnick DE. Performance of the class of 1994 in the new era of USMLE. Acad Med. 1996;71(10 suppl):S91–S93.

10. Koenig JA, Sireci SG, Wiley A. Evaluating the predictive validity of MCAT scores across diverse applicant groups. Acad Med. 1998;73:1095–106.

11. Johnson DG, Lloyd SM, Jones RF, Anderson J. Predicting academic performance at a predominately black medical school. J Med Educ. 1996;61:629–39.

12. Department of Business, Economic Development and Tourism, State of Hawai'i. The State of Hawai'i Data Book, 1997: A Statistical Abstract. Honolulu, HI: Department of Business, Economic Development and Tourism, State of Hawai'i, 1998.

13. Department of Business, Economic Development and Tourism, State of Hawai'i. The State of Hawai'i Data Book, 1995: A Statistical Abstract. Honolulu, HI: Department of Business, Economic Development and Tourism, State of Hawai'i, 1995.

14. Institutional Research Office, University of Hawai'i. Fall Enrollment Report: University of Hawai'i, Fall 2000. Honolulu, HI: University of Hawai'i, 2000.

15. Kerbeshian LA. Predicting and fostering success of American Indians in medical school. Acad Med. 1989;64:396–400.

16. Malate A. Unpublished thesis, 1997. University of Hawai'i, Honolulu, HI.

Cited By:

This article has been cited 7 time(s).

Jama-Journal of the American Medical Association
Educational epidemiology - Reply
Carney, PA; Pias, CF; Nierenberg, DW; Brooks, WB; Stukel, TA; Keller, A
Jama-Journal of the American Medical Association, 292(): 2970-2971.

Annals Academy of Medicine Singapore
Logistic regression analysis to predict medical licensing examination of Thailand (MLET) step1 success or failure
Wanvarie, S; Sathapatayavongs, B
Annals Academy of Medicine Singapore, 36(9): 770-773.

Advances in Health Sciences Education
Impact of preadmission variables on USMLE step 1 and step 2 performance
Kleshinski, J; Khuder, SA; Shapiro, JI; Gold, JP
Advances in Health Sciences Education, 14(1): 69-78.
10.1007/s10459-007-9087-x
CrossRef
Aggression and Violent Behavior
Striving for a culturally responsive process in training health professionals on Asian American and Pacific Islander youth violence prevention
Guerrero, APS; Goebert, DA; Alicata, DA; Bell, CK
Aggression and Violent Behavior, 14(6): 499-505.
10.1016/j.avb.2009.07.011
CrossRef
Educational Psychology
School achievement differences among Chinese and Filipino American students: acculturation and the family
Eng, S; Kanitkar, K; Cleveland, HH; Herbert, R; Fischer, J; Wiersma, JD
Educational Psychology, 28(5): 535-550.
10.1080/01443410701861308
CrossRef
Academic Medicine
Evaluation and Treatment of Students With Difficulties Passing the Step Examinations
Laatsch, L
Academic Medicine, 84(5): 677-683.
10.1097/ACM.0b013e31819faae1
PDF (59) | CrossRef
Academic Medicine
The Predictive Validity of the MCAT for Medical School Performance and Medical Board Licensing Examinations: A Meta-Analysis of the Published Research
Donnon, T; Paolucci, EO; Violato, C
Academic Medicine, 82(1): 100-106.
10.1097/01.ACM.0000249878.25186.b7
PDF (581) | CrossRef
Back to Top | Article Outline

© 2003 Association of American Medical Colleges

Login

Article Tools

Images

Share