Share this article on:

Reconceptualizing Variable Rater Assessments as Both an Educational and Clinical Care Problem

Kogan, Jennifer R. MD; Conforti, Lisa N. MPH; Iobst, William F. MD; Holmboe, Eric S. MD

doi: 10.1097/ACM.0000000000000221

The public is calling for the U.S. health care and medical education system to be accountable for ensuring high-quality, safe, effective, patient-centered care. As medical education shifts to a competency-based training paradigm, clinician educators’ assessment of and feedback to trainees about their developing clinical skills becomes paramount. However, there is substantial variability in the accuracy, reliability, and validity of the assessments faculty make when they directly observe trainees with patients. These difficulties have been treated primarily as a rater cognition problem focusing on the inability of the assessor to make reliable and valid assessments of the trainee.

The authors’ purpose is to reconceptualize the rater cognition problem as both an educational and clinical care problem. The variable quality of faculty assessments is not just a psychometric predicament but also an issue that has implications for decisions regarding trainee supervision and the delivery of quality patient care. The authors suggest that the frame of reference for rating performance during workplace-based assessments be the ability to provide safe, effective, patient-centered care. The authors developed the Accountable Assessment for Quality Care and Supervision equation to remind faculty that supervision is a dynamic, complex process essential for patients to receive high-quality care. This fundamental shift in how assessment is conceptualized requires new models of faculty development and emphasizes the essential and irreplaceable importance of the clinician educator in trainee assessment.

Dr. Kogan is associate professor, Department of Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania.

Ms. Conforti is research associate for academic programs, American Board of Internal Medicine, Philadelphia, Pennsylvania.

Dr. Iobst is vice president of academic affairs and clinical affairs and vice dean, Commonwealth Medical College, Scranton, Pennsylvania. During the preparation of this article, he was vice president of academic affairs, American Board of Internal Medicine, Philadelphia, Pennsylvania.

Dr. Holmboe is senior vice president of milestones development and evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois. During the preparation of this article, he was chief medical officer, American Board of Internal Medicine, Philadelphia, Pennsylvania.

Please see the end of this article for information about the authors.

Funding/Support: American Board of Internal Medicine.

Other disclosures: Dr. Holmboe receives royalties from Mosby-Elsevier.

Ethical approval: Reported as not applicable.

Correspondence should be addressed to Dr. Kogan, 3701 Market St., Suite 640, Philadelphia, PA 19104; telephone: (215) 615-0503; fax: (215) 662-7979; e-mail:

© 2014 by the Association of American Medical Colleges