Secondary Logo

Share this article on:

A Comparison of the Characteristics and Examination Performances of U.S. and Non-U.S. Citizen International Medical Graduates who sought Educational Commission for Foreign Medical Graduates Certification: 1995–2004

Boulet, John R.; Swanson, David B.; Cooper, Richard A.; Norcini, John J.; McKinley, Danette W.

Section Editor(s): McIlroy, Jodi PhD; Hemmer, Paul MD

Predicting the Licensing Process

Background International medical graduates (IMGs) make up a substantial part of the U.S. physician workforce. Unfortunately, little is known about the demographic and performance characteristics of these doctors, especially over time.

Method Educational Commission for Foreign Medical Graduates (ECFMG) applicants from 1995–2004 were studied. The characteristics and examination performances of U.S. citizen and non-U.S. citizen IMGs were contrasted.

Results ECFMG applications from U.S. citizen IMGs have been increasing. Compared to non-U.S. citizen IMGs, these individuals perform less well on some of the certification examinations, especially in the last 5 years, but are more likely to eventually achieve certification. Based on those medical students/graduates who took the examinations, the performance of U.S. citizen IMGs varied considerably from one medical school to another.

Conclusion The composition of the pool of ECFMG-certified IMGs available for graduate medical education has changed over the past 10 years. Shifts in the characteristics and abilities of IMGs are likely to have some future impact on health care delivery in the United States.

Correspondence: John (Jack) Boulet, PhD, Research and Data Resources Foundation for Advancement of International Medical Education and Research (FAIMER), 3624 Market Street, Philadelphia, PA, 19104; e-mail: (jboulet@ecfmg.org).

The Educational Commission for Foreign Medical Graduates (ECFMG) is responsible for the certification of all international medical graduates (IMGs) who wish to enter graduate medical education (GME) training programs in the United States. As part of the certification process, IMGs must take and pass a number of examinations, including United States Medical Licensing Examination (USMLE) Step 1 (basic science), Step 2 CK (clinical knowledge), and Step 2 CS (clinical skills).1 From 1998 to 2004, passing the ECFMG Clinical Skills Assessment (CSA), the forerunner to Step 2 CS, was also a certification requirement. The variability of medical school educational programs, including student selection criteria, educational content, curriculum length, and the availability of clinical experiences, would suggest the potential for performance differences among internationally-trained doctors who seek graduate medical education (GME) opportunities in the United States. Given the large, and recently increasing, number of IMGs seeking ECFMG certification,2 knowing more about differences in the characteristics and proficiencies of various defined applicant groups is important.

Although examination data for IMGs is available, relatively little empirical research has been conducted to look specifically at the basic and clinical science abilities of various IMG groups over time. Moreover, although extensive research has been conducted to support the psychometric adequacy of clinical skills assessments,3,4 studies of performance differences in recent examinee cohorts are lacking. In 1997, a series of papers outlining the performance of international medical school graduates on the USMLE were published.5–7 For examinees taking Steps 1 and 2 CK for the first time in the 1994 and 1995 academic years, U.S. medical students outperformed non-US citizen IMGs (non-USIMGs) who, in turn, outperformed U.S. citizen IMGs (USIMGs). Among IMG subgroups, first-takers, younger examinees, recent graduates, and native English speakers tended to perform better overall than their counterparts.

Over the past several years, based on social, economic, and political factors, the demographic makeup of the ECFMG applicant pool has varied considerably. Most recently, applications from U.S. citizens who attended medical schools outside the United States and Canada (USIMGs) have increased considerably.2,8 Unfortunately, although application and certification trends have been documented, there has been relatively little topical empirical research conducted to summarize and compare, either cross-sectionally or longitudinally, the performance of defined IMG cohorts. The purpose of this investigation was twofold: (1) to document changes in the composition the ECFMG applicant pool from 1995 to 2004; and (2) to compare USIMG and non-USIMG performance on the requisite certification examinations.

Back to Top | Article Outline

Method

Data sources

Demographic data were assembled from ECFMG certification records for the period from 1995 to 2004, inclusive. Candidates were assigned to an examinee cohort based upon the year in which the earliest application was filed for any examination required for certification. Examination records were available for USMLE Steps 1 and 2 CK throughout the study period, the ECFMG CSA (1998–2004), and Step 2 CS (2004–05). Pass/fail results were tallied for all administrations, and examinees were assigned an initial and ultimate (across all attempts) pass/fail status for each examination component they had taken.

Back to Top | Article Outline

Examinations required for certification

Passing scores on USMLE Step 1 (basic science) and Step 2 CK (clinical knowledge) are required for ECFMG certification. These examinations can be taken at numerous test centers located around the world. In 1998, the ECFMG introduced the CSA. This performance-based standardized patient (SP) assessment, offered only in the United States, was used to establish whether IMGs possessed the clinical skills necessary to enter graduate training positions in the United States.9 As part of the certification process, IMGs were required to pass two conjunctively scored elements of the CSA, the integrated clinical encounter (history taking, physical examination, written communication) and doctor–patient communication (interpersonal skills and spoken English proficiency). The CSA was replaced by Step 2 CS in June 2004. The Step 2 CS examination, offered at 5 test sites in the United States, is similar to the CSA but includes three conjunctive components, each of which must be passed; integrated clinical encounter, communication and interpersonal skills, and spoken English proficiency.10 For the purpose of this investigation, the results from the two clinical skills examinations (CSA, Step 2 CS) were combined, yielding seven years of assessment data (1998–2004).

The rules governing the administration of the requisite ECFMG certification examinations have changed somewhat over the past 10 years, including the sequencing and the timing that governs repeat attempts. Presently, provided that they meet other eligibility requirements (e.g., verification of the medical school diploma), IMGs can take the Step examinations in any order.11 Multiple attempts are allowed on each examination, but not within 60 days of the previous unsuccessful administration.

Back to Top | Article Outline

Study groups

International medical graduates (IMGs) are defined as individuals who attended medical school programs outside the United States and Canada that are not accredited by the Liaison Committee on Medical Education (LCME). A U.S. citizen IMG (USIMG) was defined as someone whose citizenship at entry to medical school was American; all others are non-U.S. citizen IMGs (non-USIMGs). Both USIMGs and non-USIMGs attend medical schools located throughout the world.

Back to Top | Article Outline

Results

Between 1995 and 2004, there were 126,268 ECFMG applicants, with over 14% (n = 17,970) coming from USIMGs.* In 2004, over 20% (n = 2,252) of all applicants were USIMGs. USIMG applicants attended over 650 different medical schools, located in 112 countries. Non-USIMG applicants attended 1,479 medical schools located in 160 different countries. Compared with non-USIMGs, USIMG applicants were younger (mean age = 28.5, SD = 5.6 vs. 29.1, SD = 5.5) and more likely to be male (63.5% vs. 59.0%), Hispanic or Latino (12.0% vs. 7.4%), have English as a native language (69.8% vs. 9.3%), and attend schools where English was the language of instruction (91.0% vs. 63.5%).

Based on the regional location of the medical school, the Caribbean has consistently supplied the greatest numbers of USIMGs. In the past 5 years (2000–2004), the number of USIMG applicants from medical schools in this region has grown appreciably, from 920 in 2000 to 1,453 in 2004. Interestingly, Caribbean medical schools have also been attended by substantial numbers of non-USIMGs. In 2004, ECFMG received 701 applications from non-U.S. citizens attending, or having attended, medical schools in the Caribbean. For USIMGs, over 50% of the applications were submitted by students/graduates who were attending (or had attended) 1 of 4 schools: Ross University (n = 3,410), St. George's University (n = 3,198), American University of the Caribbean School of Medicine (n = 1,556), and Universidad Autonoma de Guadalajara (n = 1,555). The numbers of examinees from Ross and St. George's during this period represent class sizes that are larger than those of any U.S. school. For non-USIMGs, Dow Medical College, located in India, provided the most applicants in the 10-year study period (n = 1,354).

Based on 1995–2004 applicants, the first-time and ultimate pass rates for USMLE Step 1, Step 2 CK, and CSA/Step 2 CS takers are presented in Table 1. For Step 1, there were 110,624 first-time attempts for IMGs who applied in the 10-year period. Overall, non-USIMGs were more likely to pass Step 1 on their first attempt. Although USIMGs and non-USIMGs had similar pass rates on first attempts between 1995 and 1999 (not shown), starting in 2000, non-USIMGs were much more likely than USIMGs to pass Step 1 on their first attempt. In 2004, nearly 6,300 (72%) of the 8,733 initial Step 1 attempts of non-USIMGs resulted in a pass; for USIMGs, only 56% passed on the first attempt. Ultimately, based on all 1995–2004 applicants, over 84% of the USIMGs eventually passed Step 1; as of December 2005, 78.3% of non-USIMG applicants had passed Step 1.

Table 1

Table 1

On Step 2 CK, differences in first-attempt pass rates varied considerably between USIMGs and non-USIMGs over the 10-year study period. For example, in 1996, based on the applicants accrued from 1995 through 1996, 52.5% of non-USIMG first attempts resulted in a pass. In contrast, 64.6% of USIMG first-attempts resulted in a pass. Most recently (2004 first-time test takers in the applicant cohort), the pass rate for non-USIMGs (81.0%) was considerably higher than that for USIMGs (68.6%). Nevertheless, based on first attempts, and aggregated over the 10-year applicant period, USIMGs (n = 14,507) had a first-time pass rate of 70.4%. In contrast, the first-time pass rate for non-USIMGs (n = 85,150) was 66.1%. However, more recently (2001–04), the difference between the first-time pass rate for non-USIMGs and USIMGs grew from 2% (2001, non-USIMG = 81.8%, USIMG = 79.8%) to over 12% (2004, non-USIMG = 81.0%, USIMG = 68.6%). Ultimately, based on all 1995–2004 applicants, over 89% of the USIMGs eventually passed Step 2 CK. In contrast, as of December, 2005, only 81% of non-USIMG applicants had passed Step 2 CK.

In terms of clinical skills proficiency (CSA or Step 2 CS), USIMGs were more likely to pass on first attempt (91.6%) than non-USIMGs (84.2%). Since 1998, when the CSA was first administered, USIMGs have consistently had lower yearly first-time fail rates than non-USIMGs. In 2004, of the 1,587 first-time CSA or Step 2 CS attempts by USIMGs, only 160 failed (10.1%). Ultimately, based on data collected through December 2005 for all 1995–2004 applicants, 98.6% of all USIMGs eventually passed the clinical skills examination. This compares with 96.1% of non-USIMGs.

For Step 1 repeaters (individuals who did not pass on the initial attempt and took the exam again), the USIMGs, on average, made more attempts (mean = 2.9, SD = 1.4) than non-USIMGs (mean = 2.6, SD = 1.1). On Step 2 CK, the mean number of USIMG repeater attempts was 2.7 (SD = 1.1). This was also higher than that for non-USIMGs (mean = 2.5, SD = 1.0). In contrast, USIMG repeaters made fewer attempts on CSA/Step 2 CS (mean USIMG = 2.1 [SD = 0.5]; mean non-USIMG = 2.2 [SD = 0.5]). Based on the 10 medical schools most frequently attended by USIMGs applicants in the study period, the first attempt pass rates on all three examinations varied considerably from school to school: Step 1, 16% to 89%; Step 2 CK, 32% to 94%; CSA/Step 2 CS, 78% to 96%.

Data from 1995–1999 applicants (n = 76,656; USIMG = 8,313, non-USIMG = 68,343) were investigated to determine “ultimate” ECFMG certification rates. The remaining cohort (2000–2004) was not included because in can take several years to complete the ECFMG certification process, potentially biasing the results. Based on the 1995–1999 applicants, nearly 73% (n = 6,036) of the USIMGs and just under 50% (n = 34,073) of the non-USIMGs achieved certification (see Table 1).

Back to Top | Article Outline

Discussion

Currently, IMGs make up over 25% of the U.S. practicing physician workforce12 and nearly 27% of all GME residency positions.13,14 Although efforts are under way to expand enrollment at U.S. medical schools,15 it is likely that the need for IMGs, both in residency and practice positions, will continue for some time. In the past 5 years (2001–2004), there has been a steady increase in the number of applicants for ECFMG certification, both from USIMGs and non-USIMGs. Depending on a number of factors, including the successful completion of the certification process and GME, and various immigration issues, many of these individuals are certain to end up in the U.S. physician workforce.

The composition of the ECFMG applicant pool has changed quite dramatically, especially over the past 5 years. There has been a steady increase in the number of U.S. citizens starting the certification process, many of whom attended, or are attending, medical schools in the Caribbean. Although this may be attributed to a number of factors, including a potential undersupply of physicians in the United States,16,17 several important issues come to the fore. First, will these individuals, perhaps because they have no visa restrictions and are more likely to speak English as their native language, continue to have greater success than non-USIMGs in filling residency positions that are not secured by U.S. medical graduates? Second, given that the demographic profiles of USIMGs and non-USIMGs continue to be different (e.g., gender, age, language), will this have some impact on the long-term supply of health care services or the cultural-linguistic match18 between physicians and patients? Last, given the differences in examination performance between USIMGs and non-USIMGs, and the greater likelihood of USIMGs attaining certification and securing GME positions,2 will the quality of the pool of doctors entering GME suffer? Although all IMGs must ultimately pass the requisite certification examinations to be eligible to enter GME, those with higher scores, or fewer attempts, may eventually perform better. To fully answer these, and other, important questions, a more focused, stratified, look at current IMG examination performances and practice patterns is needed.

From a medical school or even country-based perspective, the summaries of performances presented in this manuscript are certainly biased in one way or another. Not all IMGs from a given region, country, or medical school attempt ECFMG certification. Furthermore, depending on whether the ECFMG certification exams are required for medical school graduation, some groups may be more motivated than others. Nevertheless, given the relatively large numbers of test takers, and our desire to simply compare and contrast specific cohorts who start the ECFMG certification process, some general conclusions can be drawn. First, aggregated over the 10-year period and based on initial attempts, non-USIMGs outperform USIMGs on Step 1 but not on Step 2 CK or CSA/Step 2 CS. However, over the past 4 years, with the exception of CSA/Step 2 CS, non-USIMGs have consistently, and increasingly, outperformed USIMGs on both Step 1 and Step 2 CK. Interestingly, this trend coincides with the growth in the number of USIMGs seeking ECFMG certification. Although additional research is certainly warranted, after accounting for U.S. medical school enrollment, there may not be a sufficient number of U.S. citizens who are qualified and wish to pursue medical degrees.19

The differences in first-time pass rates between USIMGs and non-USIMGs on the clinical skills examinations were also noteworthy. Language and cultural factors seem likely to play a role in explaining the findings. Spoken English proficiency is specifically measured on both the CSA and Step 2 CS; a U.S. citizen whose native language is English should have little difficulty on this part of the assessment. Furthermore, many of the USIMGs secure clinical training experiences in the United States as part of their medical school program. Prior experience dealing with American patients could also provide some advantage on both the CSA and the Step 2 CS.

Based on our findings, a number of more focused investigations are called for. Even though performance differences were found between USIMGs and non-USIMGs, especially more recently, relatively little is known about their academic training programs. Interestingly, the analysis of first-time pass rates for USIMGs at individual schools was quite variable. For both Step 1 and Step 2 CK, there was over a 60% difference in first attempt pass rates between the highest and lowest ranked schools. While there was less school-to-school variability on CSA/Step 2 CS, this can probably be attributed to a more homogeneous examinee population (IMGs tend to take the clinical skills assessment last, having already passed Step 1 and Step 2 CK) and the likelihood that USIMGs are proficient in the English language. Although these data do not necessarily imply that the medical training in one institution is better than another, the root causes of this variability are certainly worth exploring.

The heterogeneity of the ECFMG applicant pool, both in terms of demographic and performance characteristics, and especially over time, suggests that more detailed studies, including focused investigations of medical school selection criteria, curricula, and graduation requirements are needed. Also, little is known about the preparation activities and motivations of the individual examinees who choose to take the certification examinations. Other factors that could impact performance include self-selection and the order in which the examinations are taken. The greater number of attempts on Step 1 and 2 CK, although related to pass rates, would suggest that the USIMGs are more persistent in meeting ECFMG certification requirements.

Based on our data, it is clear that USIMGs are playing an increasing role in the in the supply of U.S. physicians and that the quality of those who complete the ECFMG certification process deserves closer scrutiny.

Back to Top | Article Outline

References

1 Educational Commission for Foreign Medical Graduates. ECFMG Certification Fact Sheet. Philadelphia: ECFMG, 2004.
2 Boulet J, Norcini JJ, Whelan GP, Hallock JA, Seeling SS. The international medical graduate pipeline: Recent trends in ECFMG certification and residency training. Health Affairs. 2006;25:469–77.
3 Norcini J, Boulet J. Methodological issues in the use of standardized patients for assessment. Teach Learn Med. 2003;15:293–7.
4 Whelan GP, Boulet JR, McKinley DW, et al. Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA). Med Teach. 2005;27:200–6.
5 Case SM, Swanson DB, Ripkey DR, Bowles LT, Melnick DE. Preliminary descriptive analyses of the performance of US citizens attending foreign medical schools on USMLE Steps 1 and 2. In: Scherpbier A, van der Steeg A, van der Vleuten C, Rethans J, van der Steeg A, eds. Advances in Medical Education. The Netherlands: Kluwer Academic Publishers, 1997: 135–8.
6 Swanson DB, Case SM, Ripkey DR, et al. Performance of examinees from foreign schools on the basic science component of the United States Medical Licensing Examination. In: Scherpbier A, van der Steeg A, van der Vleuten C, Rethans J, van der Steeg A, eds. Advances in Medical Education. The Netherlands: Kluwer Academic Publishers, 1997: 187–90.
7 Ripkey DR, Case SM, Swanson DB, et al. Performance of examinees from foreign schools on the clinical science component of the United States Medical Licensing Examination. In: Scherpbier A, van der Steeg A, van der Vleuten C, Rethans J, van der Steeg A, editors. Advances in Medical Education. The Netherlands: Kluwer Academic Publishers, 1997: 175–8.
8 Educational Commission for Foreign Medical Graduates. 2004 Annual Report. Philadelphia: ECFMG, 2005.
9 Whelan G. High-stakes medical performance testing: the Clinical Skills Assessment program. JAMA. 2000;283:1748.
10 Dillon GF, Boulet JR, Hawkins RE, Swanson DB. Simulations in the United States Medical Licensing Examination (USMLE). Qual Saf Health Care. 2004;13:i41–5.
11 Educational Commission for Foreign Medical Graduates. 2005 Information Booklet. Philadelphia: ECFMG, 2004.
12 Physician characteristics and distribution in the US, 2003-2004 Edition. In: Pasko T, Smart DR, eds. Chicago: American Medical Association Press, 2003.
13 Brotherton SE, Rockey PH, Etzel SI. US graduate medical education, 2003–2004. JAMA. 2004;292:1032–7.
14 Brotherton SE, Rockey PH, Etzel SI. US graduate medical education, 2004–2005: trends in primary care specialties. JAMA. 2005;294:1075–82.
15 Association of American Medical Colleges. The physician workforce: Position statement. Washington, DC: Association of American Medical Colleges, 2005.
16 Cooper RA. Weighing the evidence for expanding physician supply. Ann Intern Med. 2004;141:705–14.
17 Cooper RA, Getzen TE, McKee HJ, Laud P. Economic and demographic trends signal an impending physician shortage. Health Aff (Millwood). 2002;21:140–154.
18 Hayes-Bautista DE, Hsu P, Hayes-Bautista M, et al. Latino physician supply in California: sources, locations, and projections. Acad Med. 2000;75:727–36.
19 Cooper RA. Medical schools and their applicants: an analysis. Health Aff (Millwood). 2003;22:71–84.

*Citizenship at medical school information was missing for 407 applicants.
Cited Here...

The ultimate pass rate is based on all applicants in the 1995–2004 period. Because applicants may take the required examinations numerous times, and later applicants (e.g., 2004) have not had time to make numerous attempts, it is necessarily an underestimate of the true ultimate pass rate for 1995–2004 applicants.
Cited Here...

© 2006 Association of American Medical Colleges