Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools. The authors developed an online KFE to assess third-year medical students’ decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick’s unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables.
From February 2012 through January 2013, 759 students (at eight U.S. medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores.
Five hundred fifteen (67.9%) students consented to participate. Across KFE forms, mean scores ranged from 54.6% to 60.3% (standard deviation 8.4–9.6%), and Phi-coefficients ranged from 0.36 to 0.52. Adding five cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P < .01).
These results provide strong evidence for response-process and relationship-to-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.
V.J. Lang is associate professor of medicine, director of the medicine subinternship, and senior associate division chief, Hospital Medicine, University of Rochester School of Medicine and Dentistry, Rochester, New York.
N.B. Berman is professor of pediatrics and of medical education, Geisel School of Medicine at Dartmouth, Hanover, New Hampshire.
K. Bronander is professor of medicine and medical director of simulation, University of Nevada, Reno School of Medicine, Reno, Nevada.
H. Harrell is professor of medicine and codirector of the medicine clerkship, University of Florida, Gainesville, Florida.
S. Hingle is professor of medicine, director of the Year 3 curriculum, and director of faculty development, Southern Illinois University School of Medicine, Springfield, Illinois.
A. Holthouser is professor of medicine and pediatrics and senior associate dean for medical education, University of Louisville, Louisville, Kentucky.
D. Leizman is associate professor of medicine and clerkship director for internal medicine, Case Western Reserve University, University Hospital, Cleveland Medical Center, Cleveland, Ohio.
C.D. Packer is professor of medicine, Case Western Reserve University, and clerkship director for internal medicine, Louis Stokes Cleveland Veterans Affairs Medical Center, Cleveland, Ohio.
Y.S. Park is associate professor, Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois.
T.R. Vu is associate professor of clinical medicine, Indiana University School of Medicine, Indianapolis, Indiana.
R. Yudkowsky is professor, Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois.
S. Monteiro is assistant professor, Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada.
G. Bordage is professor, Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois.
Funding/Support: Project support was provided by Aquifer Internal Medicine (Hanover, New Hampshire), a nonprofit organization that distributes the key features examination described in this report.
Other disclosures: Dr. Valarie J. Lang, Dr. Kirk Bronander, and Dr. Heather Harrell are supported for their work with Aquifer Internal Medicine (Hanover, New Hampshire), a nonprofit organization that distributes the key features examination described in this report.
Ethical approval: The authors procured institutional review board approval to conduct the study described in this report from Case Western Reserve University, University of Florida, Indiana University School of Medicine, University of Louisville, University of Nevada at Reno School of Medicine, University of Rochester School of Medicine & Dentistry, Southern Illinois University, University of Illinois at Chicago, and University of Illinois at Peoria.
Disclaimer: The work presented here is that of the authors and does not necessarily represent the views or policy of the United States Veterans Affairs Administration.
Previous presentations: Portions of the study were presented at Academic Internal Medicine Week (New Orleans, Louisiana, October 2013); the Ottawa Conference (Ottawa, Ontario, Canada, April 2014); and the University of Illinois at Chicago Masters in Health Professions Education conference (Chicago, Illinois, July 2015).
Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A611.
Correspondence should be addressed to Valerie J. Lang, University of Rochester School of Medicine and Dentistry, 601 Elmwood Ave., Box MED-HMD, Rochester, NY 14642; telephone: (585) 273-4881; e-mail: Valerie_Lang@urmc.rochester.edu; Twitter: @ValerieLang9.