Analysis of Performance Trends from 2010–2019 on the American Board of Internal Medicine Nephrology Certifying Exam

Abstract

Background 

The pass rate on the American Board of Internal Medicine (ABIM) nephrology certifying exam has declined and is among the lowest of all internal medicine (IM) subspecialties. In recent years, there have also been fewer applicants for the nephrology fellowship match.

Methods 

This retrospective observational study assessed how changes between 2010 and 2019 in characteristics of 4094 graduates of US ACGME-accredited nephrology fellowship programs taking the ABIM nephrology certifying exam for the first time, and how characteristics of their fellowship programs were associated with exam performance. The primary outcome measure was performance on the nephrology certifying exam. Fellowship program pass rates over the decade were also studied.

Results 

Lower IM certifying exam score, older age, female sex, international medical graduate (IMG) status, and having trained at a smaller nephrology fellowship program were associated with poorer nephrology certifying exam performance. The mean IM certifying exam percentile score among those who subsequently took the nephrology certifying exam decreased from 56.7 (SD, 27.9) to 46.1 (SD, 28.7) from 2010 to 2019. When examining individuals with comparable IM certifying exam performance, IMGs performed less well than United States medical graduates (USMGs) on the nephrology certifying exam. In 2019, only 57% of nephrology fellowship programs had aggregate 3-year certifying exam pass rates ≥80% among their graduates.

Conclusions 

Changes in IM certifying exam performance, certain trainee demographics, and poorer performance among those from smaller fellowship programs explain much of the decline in nephrology certifying exam performance. IM certifying exam performance was the dominant determinant.

Since 2010, the number of applicants for nephrology fellowship training positions in the United States has steadily declined, whereas the number of available positions has increased.¹ In 2010 there were 576 applicants for 374 positions (a ratio of 1.5), whereas in 2019 there were 325 applicants for 464 positions (a ratio of 0.7). Thus, nephrology fellowships have gone from being highly selective and competitive, on the basis of the ratio of the number of applicants to the number of available positions, to being among the least competitive internal medicine (IM) subspecialty disciplines. This has led to much discussion in the nephrology community regarding reasons for the decline and consequences of changes in the characteristics of those entering nephrology fellowship training.

Another important change affecting nephrology training has been a marked decline in the pass rate among those taking the American Board of Internal Medicine (ABIM) certifying examination (exam) in nephrology for the first time, which is now among the lowest of all IM subspecialties, with a pass rate of 74% in 2019 and 80% in 2020.² Over the same period, initial certifying exam pass rates for all other IM subspecialties remained stable or increased. Pass rates in most other IM subspecialties were similar to previous years in 2019 and 2020. The low pass rate on the nephrology exam is important not only for the individuals who do not pass the exam, but may also affect training programs because current Accreditation Council for Graduate Medical Education (ACGME) program requirements for nephrology and other IM programs specify that in the preceding 3-year period “the program’s aggregate pass rate of those taking the examination for the first time must be higher than the bottom fifth percentile of programs in that subspecialty,” and “any program whose graduates over the time period specified in the requirement have achieved an 80 percent pass rate will have met this requirement, no matter the percentile rank of the program for pass rate in that subspecialty.”³ Before July 1, 2020, the ACGME program requirement for IM subspecialties was an aggregate 5-year pass rate of ≥80%. In 2014, 15% of US nephrology fellowship programs were below this benchmark, more than double that seen in prior years.⁴ Effective July 1, 2020 the requirement was changed to above the fifth percentile or an 80% pass rate in the preceding 3 years.

This study was undertaken to evaluate changes in nephrology fellow and nephrology fellowship program characteristics between 2010 and 2019, to assess their association with performance among first-time takers of the ABIM nephrology certifying exam. We also explored nephrology programs’ performance over this period, to determine how many had pass rates that fell below ACGME standards.

Methods

Using the ABIM administrative database to identify the study cohort and obtain examinee performance, training, demographic characteristics, and nephrology fellowship program characteristics, we identified physicians who took the ABIM nephrology certifying exam for the first time between 2010 and 2019 and who graduated from US nephrology fellowship programs, which were ACGME accredited, during the same period. Exam performance data in both IM and nephrology were compiled and merged with ABIM administrative data related to physician demographic and training program information. We examined the changes in IM certifying exam performance, demographic, and program characteristics over time, the association between those factors and nephrology certifying exam scores. The Institutional Review Board of the University of Pennsylvania approved this study as exempt.

Outcomes and Other Measures

The primary outcome measure was performance on the nephrology certifying exam. This exam is designed to evaluate the knowledge, diagnostic reasoning, and clinical judgment expected of the certified nephrologist in the broad domain of the discipline.⁵ ABIM exams are constructed using an Automated Test Assembly software program with an optimization algorithm to ensure that exam forms across administrations are comparably constructed with regard to blueprint coverage, difficulty, discrimination, relevance, and other statistical constraints. The exam blueprint is reviewed periodically by physicians with clinical expertise in the discipline for importance and relevance. After each exam administration, a process known as statistical equating is undertaken to ensure scores are comparable across different years and different forms of the assessment.⁶ The ABIM form optimization and equating processes are part of the Standards for Best Assessment Practices (see standards 5.12–5.15).⁷

We used percentile score of the performance among all first-time nephrology certifying exam takers in each year. Our measure of clinical knowledge before entering a nephrology fellowship program was performance on the IM initial certifying exam, for which we used the percentile score derived from of the performance of all IM exam takers in each year.

Trainee and Nephrology Fellowship Program Characteristics

Trainee level variables were self-described and included age, sex, medical school type (US/Canada medical school, international medical school, and osteopathic medical school), and whether they were born in the United States or Canada, or abroad. We used median age at the time each individual first took the nephrology exam as the threshold to define two age groups, ≤33 years and >33 years. Nephrology program level characteristics included program type (e.g., university or nonuniversity as classified by FREIDA⁸), trainees’ average IM score by tertile groups, geographic location, and program size. Because program size can vary from year to year and some programs opened or closed within the decade, we defined larger programs as having a mean of three or more test takers per year for years that had at least one test taker. This threshold was chosen because the median value of the number of first-time test takers was three in 7 of the 10 years (Supplemental Figure 1) and the median value of the mean number of test takers per year per program was 2.8. A program’s 3-year ABIM nephrology certifying exam aggregate pass rate, that is, the percent of all first-time test takers in each rolling 3-year period who passed the nephrology certifying exam was also calculated, for each period in which there was at least one test taker in each of the 3 years. A program’s mean percentile score was likewise calculated for each rolling 3-year period.

Statistical Analysis

We reported the descriptive statistics of test takers’ exam percentile scores, demographics, and program characteristics. Additionally, to compare the difference between the first and last year of the study decade, we used chi-squared and two-tailed t tests to compare the differences between first-time test takers in 2010 and in 2019. With the physician as the unit of analysis, we regressed nephrology certifying exam percentile scores against each examinee’s IM certifying percentile score, measures of examinee demographics, training, country of birth, and measures of fellowship program characteristics, described previously. A variable for year was included to control for the secular trend of nephrology exam percentile scores. To evaluate the difference between larger and smaller programs over time, we included the interaction between program size and year indicators. To adjust for the clustering effect of trainees within fellowship programs, generalized estimating equation analysis with Huber–White standard error estimates was performed; 95% confidence intervals (95% CIs) were calculated. Lastly, to understand the contribution of the key covariates to the change in nephrology exam performance over the study period, we conducted a postregression simulation for 2019 data. We estimated the change in the nephrology percentile score that might be expected if we set the mean value seen in 2010 for each individual covariate that had a statistically significant estimated coefficient, whereas all other covariates remained at 2019 levels. Two-sided P<0.05 indicated significance. Stata, version 14 (StataCorp LLC) was used for all statistical analyses.

Results

A total of 4207 physicians took the nephrology certifying exam for the first time in the years 2010–2019. Of these, 4094 completed their nephrology fellowship training in 150 ACGME-accredited US programs. The final study sample included 4079 examinees, after excluding 15 with missing IM exam performance data who took the exam before 1998 (Figure 1).

Figure 1.: Flow diagram of study sample identification.

Physician characteristics are shown in Table 1 for the entire study population and for years 2010 and 2019 separately. Of the 4079 first-time test takers, 37.8% were female, 64.7% were international medical graduates (IMGs), the median age when taking the exam was 33 years, and 38.5% were trained in fellowship programs with fewer than three trainees per year on average. There were no statistically significant differences between 2010 and 2019 in program characteristics, but examinees differed in most demographic variables and their IM exam scores. Compared with 2010 examinees, 2019 examinees were more likely to be older than 33 years (52.4% in 2019 versus 34.1% in 2010) and to be IMGs (69.8% in 2019 versus 57.4% in 2010).

Table 1. - First-time takers of the nephology initial certifying exam (2010–2019): Test taker demographics and nephrology fellowship program characteristics

Characteristic	Full Sample (n=4079)	Year 2010 (n=399)	Year 2019 (n=361)	P value a
Nephrology percentile score, mean (SD) b	49.1 (28.8)	48.9 (28.7)	49.5 (28.7)	0.76
IM percentile score, mean (SD)	51.8 (27.9)	56.7 (27.9)	46.1 (28.7)	<0.001
Age >33 yrs when took nephrology exam, %	43.0	34.1	52.4	<0.001
Female, %	37.8	38.1	36.6	0.66
Medical school type, %				<0.001
DO	5.1	4.3	8.0
International medical school	64.7	57.4	69.8
US/Canada medical school	30.2	38.4	22.2
Non-US/Canada born, %	69.3	64.2	85.3	<0.001
Program type, %				0.97
Nonuniversity based	17.4	16.8	16.9
University based	82.6	83.2	83.1
Region, %				0.32
Midwest	19.6	21.1	16.3
Northeast	34.1	36.6	34.4
South	31.9	29.6	34.1
West	13.8	12.3	14.4
Puerto Rico	0.7	0.5	0.8
Smaller programs (<3 takers per year c , 2010–19), %	38.5	38.9	35.7	0.38

DO, osteopathic medical school graduates.

^aTest the difference between 2010 and 2019; calculated from chi-squared tests for the equality of percentages and two-tailed t tests for the equality of means between year 2010 and 2019 examinees.

^bMean nephrology percentile score would be 50 by definition if all test takers were included in the study. However, our study included only first-time test takers in 150 US programs.

^cSmaller programs are defined as those having an average of <3 test takers per year for years that they had at least one test taker.

The mean IM certifying exam percentile score among those who subsequently took the nephrology certifying exam for the first time decreased throughout the study period (Figure 2) with the 2010 and 2019 examinees being statistically different in IM percentile score (mean, 56.7; SD, 27.9 for 2010 examinees versus 46.1, SD 28.7, for 2019 examinees, P<0.001). Figure 2 also shows there was a steady decrease in both the number of US medical school graduates (USMGs) and the fraction of USMGs among all examinees (153, 38%, in 2010 versus 80, 22%, in 2019). In Figure 2 and Figure 3, data for IM exam performance are shown on the basis of the year each examinee took the nephrology initial certification exam, regardless of the year in which the IM exam was taken. As shown in Figure 3A, over the last decade, the mean IM certifying exam percentile score among those subsequently taking the nephrology certifying exam declined among IMGs, but remained relatively stable among USMGs. Figure 3B shows the nephrology certifying exam mean percentile score declined among IMGs, but increased among USMGs. The number of graduates of osteopathic medical schools taking the exam each year was small, so trends in performance are difficult to interpret, but scores of this group appeared stable over the study period.

Figure 2.: Number of first-time test takers for 2010–2019 nephrology initial certifying exam by medical school type and trend in average IM certifying exam percentile score. The mean IM certifying exam percentile score is shown for the year in which the nephrology initial certification exam was taken, regardless of the year the IM certifying exam. DO, osteopathic medical school graduates.

Figure 3.: Mean IM and Nephrology certifying exam percentile score trend by medical school type and nephrology fellowship program size. (A and B) Medical school type and (C and D) nephrology fellowship program size. Larger program: ≥3 first-time test takers per year on average; smaller program: <3 first-time test takers per year on average for each year there was at least one test taker from the program. The mean IM certifying exam percentile score is shown for the year in which the nephrology initial certification exam was taken, regardless of the year the IM certifying exam.

Figure 3, C and D shows the yearly trend in IM and nephrology percentile scores for larger and smaller nephrology fellowship programs, with larger programs defined as those having a mean of three or more test takers per year. Between 2010 and 2015, examinees from smaller nephrology fellowship programs performed similarly to those from larger programs on the IM certifying exam. After 2015, mean percentile scores on the IM certifying exam of examinees of both program types declined, with examinees from smaller programs having lower IM exam percentile scores. Between 2010 and 2019, mean percentile scores on the nephrology certifying exam declined from 49% to 40% among those who trained in smaller programs, but increased from 49% to 55% among those from larger programs.

In the regression model (Table 2), after adjusting for other covariates, we found that a one-point higher IM percentile score was significantly associated with a 0.58 point (95% CI, 0.55 to 0.61) improvement in nephrology exam percentile score, namely, one SD increase in IM percentile score (27.9) was associated with a 16.2 point increase in nephrology exam percentile score. Those examinees who were older than 33 scored 7.81 points lower than their younger counterparts (95% CI, −9.40 to −6.22). Females scored 4.96 points lower compared with males (95% CI, −6.60 to −3.32). Compared with examinees who were US or Canadian medical school graduates and US or Canadian born, those with other medical school type and birth country combinations performed worse, ranging from 2.45 to 9.86 points lower. Program type, program geographic location, and program mean IM exam percentile scores were also associated with nephrology exam performance. In 2010, there was no difference in nephrology exam performance between test takers from large and small fellowship programs (difference, −1.86; 95% CI, −7.03 to 3.31), the overall test, a chi-square test with degree of freedom 9, of the interaction between program size and year for the study decade was statistically significant (chi-square [9], 18.41; P=0.03), which indicates there was a difference between larger and smaller programs, with graduates of smaller programs performing less well in later years.

Table 2. - Predictors of initial nephrology certifying exam score among first-time test takers

Covariate	Coefficient (95% CI)	P value
IM percentile score	0.58 (0.55 to 0.61)	<0.001
Age ≥33 yrs when took nephrology exam	−7.81 (−9.40 to −6.22)	<0.001
Female	−4.96 (−6.60 to −3.32)	<0.001
Medical school type by birth country
US medical school US born	0 (Reference)
US medical school foreign born	−2.45 (−5.19 to 0.28)	0.08
International medical school US born	−9.86 (−13.42 to −6.3)	<0.001
International medical school foreign born	−5.00 (−6.75 to −3.26)	<0.001
DO US born	−4.69 (−8.10 to −1.28)	0.007
DO foreign born	−6.66 (−10.75 to −2.58)	0.001
Program type
Nonuniversity based	0 (Reference)
University based	3.24 (0.04 to 6.44)	0.05
Region
Midwest	0 (Reference)
Northeast	−3.64 (−6.37 to −0.92)	0.009
South	−4.53 (−7.1 to −1.96)	0.001
West	−0.93 (−4.43 to 2.57)	0.60
Puerto Rico	−18.77 (−23.75 to −13.8)	<0.001
Program group by average IM score
Top	0 (Reference)
Middle	−2.91 (−5.59 to −0.23)	0.03
Bottom	−3.85 (−6.4 to −1.30)	0.003
Program size
Smaller (<3 per year on average)	0 (Reference)
Larger (≥3 per year on average)	−1.86 (−7.03 to 3.31)	0.48
Year
2010	0 (Reference)
2011	−0.63 (−5.91 to 4.64)	0.81
2012	−2.22 (−7.36 to 2.92)	0.40
2013	−0.34 (−5.52 to 4.84)	0.90
2014	−1.09 (−6.10 to 3.91)	0.67
2015	2.50 (−2.71 to 7.70)	0.35
2016	4.79 (−0.72 to 10.29)	0.09
2017	3.17 (−2.04 to 8.39)	0.23
2018	7.80 (2.44 to 13.15)	0.004
2019	1.90 (−3.42 to 7.23)	0.48
Program size X year
Large 2011	3.07 (−3.33 to 9.48)	0.35
Large 2012	6.32 (0.10 to 12.54)	0.05
Large 2013	6.15 (−0.24 to 12.55)	0.06
Large 2014	6.15 (−0.05 to 12.35)	0.05
Large 2015	2.23 (−4.41 to 8.87)	0.51
Large 2016	1.43 (−5.03 to 7.89)	0.66
Large 2017	6.49 (0.11 to 12.87)	0.05
Large 2018	1.09 (−5.58 to 7.76)	0.75
Large 2019	11.19 (4.52 to 17.86)	0.001

DO, osteopathic medical school graduates.

If 2019 nephrology test takers had the same mean IM percentile score as in 2010, the mean 2019 nephrology percentile scores would have been 6.15 points (95% CI, 5.88 to 6.42) higher than it actually was. If the 2019 test takers had the same percent of older examinees as in 2010, the mean 2019 nephrology percentile score would have been 1.43 points (95% CI, 1.14 to 1.72) higher. If 2019 test takers had the same medical school/birth country composition as in 2010, the mean 2019 nephrology percentile score would have been 1.07 points (95% CI, 0.68 to 1.46) higher. If 2019 test takers had the same sex composition as 2010, the mean 2019 nephrology percentile score would have been only 0.08 points (95% CI, 0.05 to 0.10) lower. Because the fraction of all test takers who were women was relatively constant throughout the study period (38.1% in 2010 versus 36.6% in 2019), sex had negligible effect on the observed change between 2010 and 2019 in nephrology certifying exam performance.

To ensure our analysis using the first and last years of the study decade accurately reflected the years in between, and were not due to peculiarities of one or both of the individual years, we conducted, as a sensitivity analysis, a 5-year mean change prediction by using the mean value or share of the first 5 years (2010–2014) and that of the last 5 years (2015–2019). The sensitivity analysis showed the actual fraction of the 10-year predicted change in the first 5 years was 0.55, 0.47, and 0.46 for IM percentile score, age, and medical school or birth country, respectively, of the total 10-year change. If there had been a perfect linear trend without any year-to-year noise over a 10-year period, the predicted change in each of the two 5-year periods would be half of the change predicted over the 10 years. Thus, this analysis did not change the findings compared with using just 2010 and 2019.

Figure 4 shows the 3-year aggregate pass rate for the 150 ACGME-accredited fellowship programs for years ending in 2012–2019, and the number of programs below the bottom fifth percentile. Some nephrology fellowship programs did not have at least one examinee in 3 consecutive years, so their aggregate pass rate could not be calculated; these are denoted as not applicable. There were 41 such programs between 2017 and 2019. Figure 4 also shows the number of programs with 3-year aggregate pass rates <80% by program size; 53%–67% of such programs were smaller programs, as defined above, depending on the year.

Figure 4.: Nephrology initial certifying exam 3-year aggregate and individual nephrology fellowship program pass rate 2012–2019. Columns show number of programs with aggregate 3-year pass rates <80% and ≤5th percentile among all programs, <80% and >5th percentile among all programs, and ≥80%. Only those ≤5th percentile do not meet ACGME program requirements. NA, programs did not have at least one test taker in 3 consecutive years and aggregate pass rates could not be calculated.

The mean 3-year aggregate nephrology program pass rate was 91% in 2012 but dropped to 80% in 2019. Only 6–8 programs per year from 2012 to 2019 had aggregate 3-year program pass rates at or below the fifth percentile among all applicable nephrology programs. Many more programs, 41 (38%) in 2019, had aggregate pass rates above the fifth percentile but <80% over 3 years. In 2012, 108 (86%) programs had 3-year program aggregate pass rates ≥80% whereas in 2019, only 62 (57%) programs had 3-year program pass rates ≥80%.

Discussion

The purpose of this study was to examine changes over the decade 2010–2019 in nephrology trainee and nephrology fellowship program characteristics and their association with performance among first-time test takers of the ABIM nephrology certifying exam. We also examined the effect of the declining nephrology certifying exam pass rate on 3-year cumulative program pass rates relative to current ACGME board pass rate program requirements.³

When considering certifying exam performance, it is important to note all ABIM certifying exams adhere to standard principles for exam development.⁹ As described above, each exam within a discipline is constructed to ensure psychometrically equivalent difficulty compared with prior exams so a decline in performance over time principally reflects test taker knowledge, rather than characteristics of the exam.^6,7 Because the same principles and practices are applied when creating and scoring all ABIM certifying exams, seeing changes in nephrology exam performance but not other disciplines further suggests the change in nephrology exam performance is not due to changes in characteristics of the nephrology exam.

In 2009, those subsequently taking the nephrology certifying exam performed highest on the IM exam compared with any other subspecialty, whereas in 2019, they performed lower than any other subspecialty (internal ABIM data). None of the other IM subspecialty disciplines experienced such a drastic decrease in performance over this period. After adjusted regression, IM percentile score remained the most important predictor of subsequent nephrology exam performance, whereas age (older than the median age 33), female sex, and not being a USMG were associated with poorer performance. Graduates of university programs performed better than those from nonuniversity programs, whereas regional variation remained after regression adjustment. Graduates of the least competitive nephrology fellowship programs, defined as those with the lowest tertile of mean IM certifying exam scores, performed worse on the nephrology certifying exam than those from the most competitive programs. We also found an interaction between year and program size.

That IM certifying exam score is a strong predictor of nephrology certifying exam score is understandable. Both the IM and nephrology certifying exams are designed to evaluate knowledge, diagnostic reasoning, and clinical judgement. Nephrology is also one of the primary medical content categories for the IM exam.¹⁰ Our finding that trainees older than the median age of 33 years performed less well than those who were younger may relate to these trainees having pursued nontraditional training or career paths before nephrology fellowship (such as clinical training in their home country before coming to the United States or prior work as a hospitalist), or having a need to attend to family obligations leading to training delays. Other studies have also found age to be a negative predictor on similar standardized exams.¹¹ The fact that graduates of the least competitive nephrology fellowship programs performed worse after regression adjustment indicates there might be a peer effect, that is, training with abler trainees is beneficial to all trainees, or a training effect, that is, more competitive programs may have certain advantages in terms of program infrastructure, available clinical and education opportunities, and faculty.

Changes in certain characteristics of the 2019 examinee cohort, compared with that of 2010, were identified, and may help explain the decline in nephrology certifying exam performance over this time frame. The mean percentile score on the IM certifying exam among those who subsequently took the nephrology certifying exam declined by >10 percentile points, from 56.7 in 2010 to 46.1 in 2019. This factor alone is estimated to account for a 6.15 point decline in the 2019 cohort’s nephrology exam percentile score. The second-largest decline attributed to the covariates in the regression model was the increased numbers of trainees >33 years of age in 2019. This, however, accounted for only a 1.43 point decline, emphasizing the overwhelming importance of the decrease in IM exam performance to the observed decrease in nephrology exam performance. Also, much less important was the increase in non-US born/non-US medical school status. As noted, sex was not an important factor in the performance decline between 2010 and 2019. Over the decade, the significant decline in mean IM exam percentile score among the 2019 cohort reflected the reality that nephrology had become among the least competitive IM subspecialty disciplines in recent years, that is, a likely “recruitment effect.” Because individual performance on the IM exam is not available to fellowship program directors, further studies identifying other IM resident characteristics that associate with IM exam performance and subsequent success in nephrology training are needed.

Over the last decade, among first-time test takers of the ABIM nephrology certifying exam, the share of IMGs has increased. In 2019, IMGs comprised nearly 70% of those taking the nephrology exam for the first time, an increase of more than eight percentage points from 2010. On the basis of our regression analyses, when we compare IMGs to USMGs who were of equal ability on the basis of their IM performance, IMGs still performed less well on the nephrology certifying exam throughout the decade. This suggests a nephrology fellowship program training issue in that as a group IMGs appear to have benefited less from their nephrology training experiences compared with USMGs. This speaks to an urgent need to identify nephrology fellowship program characteristics and actions that might help IMGs thrive in fellowship training to close this gap in performance.

Our regression analyses also showed that when comparing women to men who were of equal ability on the basis of their IM performance, women performed less well on the nephrology certifying exam. We are unaware of published comparisons of performance among men and women for any other IM subspecialty, so whether this finding is peculiar to nephrology cannot be stated. However, as noted previously, the change in nephrology certifying exam performance among women from 2010 to 2019 was not significant and so the overall contribution of sex in the model, although statistically significant, is not likely of any educational significance.

One of the most salient findings of our study was that although IM exam percentile scores declined among those entering both larger and smaller nephrology fellowship programs, the decline was steeper among those who trained in smaller nephrology programs. After regression adjustment, there was no difference in 2010 in nephrology exam performance between larger and smaller programs, but comparing 2010 to 2019, the difference was more than 11 percentile points, favoring larger programs. This widened gap in nephrology exam performance between smaller and larger programs suggests there may be both recruitment and training effects; that is, IM percentile scores among smaller programs have declined to a greater extent than for larger programs (a recruitment issue), but have even greater dissimilarity in nephrology exam performance. Reasons for the disparity in IM exam percentile score and diverging outcomes in nephrology exam performance among smaller and larger programs require further study so this gap can be addressed. Little is known in general about size of graduate medical education training program and board certifying exam pass rates. For example, in other studies, residency program size did not correlate with pediatric board pass rates¹² but did with family medicine program size.¹³ Findings for IM board pass rates show some evidence for a weak association between pass rates and program size.^14,15 We are not aware of published studies of IM fellowships and program size.

Using the current ACGME program requirements, we found the percent of US nephrology fellowship programs with cumulative 3-year pass rates ≥80% has also decreased markedly over the decade (from 86% to 57%). We also suggest this absolute standard (pass rate ≥80% over 3 years), rather than the relative standard of being above the fifth percentile in pass rate for the discipline, is a more meaningful benchmark in identifying poorly performing nephrology fellowship programs, given the relatively large number of programs above the fifth percentile but below the 80% level.

Our study has limitations. First, we excluded examinees trained outside the 150 ACGME-accredited US nephrology training programs; therefore, our results might not be generalizable to trainees trained outside those programs, for example, in Canada, where trainees are also eligible for ABIM certification in nephrology. However, our study included approximately 97% of all first-time nephrology certifying exam takers over the last decade. Second, there may be residual confounding, such as additional physician variables and nephrology fellowship program, characteristics that could explain the nephrology exam performance that were not available for this study. Third, the outcome variable, nephrology exam percentile score, is on the basis of responses to questions on the nephrology certifying exam, a simulated clinical practice environment, rather than direct patient care. However, assessment of clinical decision-making using written vignettes has been demonstrated to mirror similar assessments that use standardized patients.¹⁶

This is one of the first studies assessing trainee and fellowship program characteristics associated with ABIM nephrology certifying exam performance. We were able to describe trends and identify several factors associated with poor performance on the exam over the past decade. It is important to emphasize that from this study we cannot make inferences about the observed changes in exam performance and any aspect of the quality of care provided by those who pass the nephrology certifying exam. We believe our findings will be useful to the broad nephrology community, the ACGME, program directors in nephrology, and applicants to nephrology fellowships. We also hope our findings will stimulate further research into nephrology training and how fellowship programs can improve the training experiences and success of their fellows.

Disclosures

B. Jaar is Chair of the ABIM Nephrology Exam Committee; reports receiving honoraria from ABIM, Nephrology; reports being a scientific advisor or member of ABIM, BMC Medicine, BMC Nephrology, the Clinical Journal of the American Society of Nephrology, and the National Kidney Foundation; and reports other interests/relationships with UpToDate royalties as an author. J. S. Berns was previously Chair of the ABIM Council and the ABIM Nephrology Board; reports consultancy agreements with Rubicon, Inc.; reports receiving honoraria from ABIM and UpToDate; and reports being a scientific advisor or member of American Journal of Kidney Diseases as Deputy Editor. All other authors are employees of ABIM.

Funding

None.