The ASN In-Training Examination Needs More Time, Not a New Paradigm

Until last year, the nephrology community had but a single uniform measure of competency achieved as the result of nephrology fellowship training: The American Board of Internal Medicine (ABIM) subspecialty examination, which can be taken only after completion of nephrology subspecialty training. There had also been little published recently regarding current national nephrology fellowship training practices and outcomes. We now have a nephrology in-training examination (ITE) and several recent articles addressing nephrology training in the United States (^1–5). Specific details of the ITE, first administered in 2009, were recently published (⁶). In this issue of CJASN, an experienced clinician and training program director criticizes several aspects of both the ITE and the ABIM certifying examinations (⁷). The criticisms of the ITE are mainly twofold; first, on average, only approximately 70% of questions on the ITE were answered correctly, and, second, the specific questions on the ITE are not revealed to program directors and trainees. Dr. Brown also expresses concern that the scores of second-year fellows at his program did not “accurately depict their excellent clinical knowledge” and notes a relatively small increase in scores between first- and second-year fellows around the United States. Because others have recently written about ABIM examinations and the maintenance of certification process (^8–13), I confine my comments to the ITE examination.

Is the “new testing paradigm” proposed for the ITE by Dr. Brown a good one? Documentation of the successes and inadequacies of our training programs and of the strengths and weaknesses of the knowledge and clinical skills of nephrologists who enter practice should be multimodal and an ongoing effort. The ITE provides insight into the strengths and weaknesses for trainees and program directors in 11 specific content areas that in general mirror those of the ABIM examination but it is just one component of a trainee's assessment; other tools, such as direct observation, review of patient charts, clinical care, patient outcomes, patient surveys, case discussions, oral examinations, use of standardized patients and simulators, and global assessments, should complement the ITE in all programs and are required by the Accreditation Council for Graduate Medical Education. There is inherent imprecision in this assessment given the relatively small number of questions devoted to each topic area, but there were not as few as suggested by Dr. Brown: There were four questions on ethics; 13 to 14 on clinical pharmacology, electrolyte disorders, and transplantation; and 15 to 19 on hypertension, intensive care unit nephrology/acute renal failure, glomerular and vascular diseases, and chronic kidney disease. New testing modules for assessment of competency related to urinalysis and renal pathology will be added to the examination in 2011.

In addressing Dr. Brown's concern that the overall correct response rate was low and may perhaps not reflect the full skill set of his trainees, I offer the following comments. The ITE is a “low risk” examination, in that passing or not is not an issue and the specific achieved score has little real consequence; as such, fellows may not have devoted their full effort to achieving their best possible performance, and, unlike the ABIM examination, most fellows probably do not study for the ITE—mine did not. Clearly, some of the questions were difficult, with a mean difficulty of 0.69 (proportion of examinees who answered correctly), but an “easy” examination would not discriminate among test-takers' abilities. Perhaps also our fellows do not know as much as we think they do or lack the synthesis and complex decision-making skills that we think they have. Rather than criticize the ITE because fellows' scores do not seem to match up with a program director's subjective assessment of what his or her fellows know and their ability to use that knowledge in clinical decision making, the ITE should be viewed as a more objective assessment tool that can provide useful information beyond our own personal impressions.

The reported ITE difficulty and relevance are also not as concerning as suggested by Dr. Brown if one considers only those who responded to the voluntary posttest survey; 87% reported that the ITE was at an appropriate degree of difficulty, 91% reported that the content was very or moderately relevant, and no content area was believed to be over- or underemphasized—I would consider this a resounding success for the first rendition of such a complex undertaking delivered to a rather diverse audience. All six program directors who took the examination were satisfied with the examination difficulty and scope. I also contend that what first- and second-year renal fellows believe to be “relevant” should probably not be a major driving force in determining the content or structure of the ITE. What is perceived as being relevant to a first- or second-year renal fellow may well differ from the view of program directors. Also, a fellow who has his or her sights set on a clinical private practice position is likely to have a different view of a question's “relevance” than one who plans to pursue a basic or clinical science academic career.

The relatively small increase in ITE examination scores between first- and second-year fellows (64 versus 69% correct scores) may or may not be a matter of concern and may be explained by a variety of factors, for instance the front-loading of clinical activity in the first year in most programs. If this is a concern, then we may want to reconsider our training paradigm rather than our testing paradigm.

I disagree with the suggestion that the American Society of Nephrology (ASN) ITE Committee should provide the examination's questions and answers. Although doing so might add some value, I do not think that this would change the usefulness of the ITE in a meaningful way; program directors and trainees would know only the relatively few specific questions on a single year's examination. This seems less useful to me than having a more global sense of strengths and weaknesses in broader content areas. Any gain, though, would not offset the huge disadvantage of the need to create an entirely new examination each year—a virtually impossible task. As it is, the current test development schedule requires more than 1 year to prepare each nephrology ITE. More detail of the ITE testing areas and item educational objectives will be provided to program directors in the future, however.

I also strongly disagree with the suggestion that program directors collectively write the ITE by submitting new questions each year. Writing reasonably decent ABIM-style examination questions is difficult; writing good examination questions is really difficulty, and writing excellent questions seems to be nearly impossible! Experience tells us that writing high-quality questions takes practice, a commitment to learning the test-writing process, and willingness to revise questions—sometimes many times. I am absolutely certain that the ITE would be nearly useless and perhaps provide more misinformation than useful information if the current rigorous test development process were abandoned (and equally certain that the other ITE steering committee members agree with me). Having been on the ASN ITE steering and question-writing committees, I know that many “great questions” end up discarded or revised beyond recognition. Selecting and refining questions for the examination is an arduous process (⁶), which, to the extent possible, ensures that examination questions are well written, test both factual knowledge and clinical decision making, and are at an appropriate level of difficulty (recognizing that examinees span a spectrum between first and second year of fellowship).

It may be that there is room for improvement in how we qualify nephrologists and others for subspecialty certification, but before tossing out the ITE for a new testing paradigm, we should allow the examination to mature and program directors should be afforded the opportunity to use the results provided to assess and improve their training programs for the next several years. Compared to the 2009 ITE, the 2010 exam proved to be slightly more difficult, had better item discrimination and overall reliability, and had a larger item correct response difference between first and second year fellows. Eventually, correlation between ITE scores and ABIM test scores and other measures of competence and clinical skill can be studied (¹⁴). Then, with data in hand, consideration of a new testing paradigm might be appropriate. For now, I am delighted that the ASN and Drs. Mark Rosenberg and Mitchell Rosner spearheaded development of this useful, even if imperfect, new assessment tool.

Disclosures

None.