Secondary Logo

Share this article on:

Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship

Lang, Valerie J., MD, MHPE; Berman, Norman B., MD; Bronander, Kirk, MD; Harrell, Heather, MD; Hingle, Susan, MD; Holthouser, Amy, MD; Leizman, Debra, MD; Packer, Clifford D., MD; Park, Yoon Soo, PhD; Vu, T. Robert, MD; Yudkowsky, Rachel, MD, MHPE; Monteiro, Sandra, PhD; Bordage, Georges, MD, PhD

doi: 10.1097/ACM.0000000000002506
Research Report: PDF Only

Purpose: Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools. The authors developed an online KFE to assess third-year medical students’ decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick’s unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables.

Method: From February 2012 through January 2013, 759 students (at 8 U.S. medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores.

Results: Five hundred fifteen (67.9%) students consented to participate. Across KFE forms, mean scores ranged from 54.5% to 60.3% (SD 8.4 9 – 9.6%), and Phi-coefficients ranged from 0.36 to 0.52. Adding 5 cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P < 0.01).

Conclusions: These results provide strong evidence for response-process and relationship-to-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.

V.J. Lang is associate professor of medicine, director, Medicine Sub-Internship, and senior associate division chief, Hospital Medicine, University of Rochester School of Medicine and Dentistry, Rochester, New York.

N.B. Berman is professor of pediatrics and of medical education, Geisel School of Medicine at Dartmouth, Hanover, New Hampshire.

K. Bronander is professor of medicine and medical director, Simulation, University of Nevada, Reno School of Medicine, Reno, Nevada.

H. Harrell is professor of medicine and co-director, Medicine Clerkship, University of Florida, Gainesville, Florida.

S. Hingle is professor of medicine, director, Year 3 Curriculum, and director, Faculty Development, Southern Illinois University School of Medicine, Springfield, Illinois.

A. Holthouser is professor of medicine and pediatrics, senior associate dean, Medical Education, University of Louisville, Louisville, Kentucky.

D. Leizman is associate professor of medicine, clerkship director, Internal Medicine, Case Western Reserve University, University Hospital, Cleveland Medical Center, Cleveland, Ohio.

C.D. Packer is professor of medicine, Case Western Reserve University, and clerkship director, Internal Medicine, Louis Stokes Cleveland Veterans Affairs Medical Center, Cleveland, Ohio.

Y.S. Park is associate professor, Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois.

T.R. Vu is associate professor of clinical medicine, Indiana University School of Medicine, Indianapolis, Indiana.

R. Yudkowsky is professor, Department of Medical Education, University of Illinois at Chicago in Chicago, Illinois.

S. Monteiro is assistant professor, Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada.

G. Bordage is professor, Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois.

Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A611.

Acknowledgements: The authors wish to thank Regina Kovach, MD, L. James Nixon, MD, MHPE, Janet Jokela, MD, and Debra Stottlemyer, MD for their contributions to exam development; Saad Alvi, MD for data gathering; and Audra Bucklin for project management.

Funding/Support: Project support was provided by Aquifer Internal Medicine (Hanover, New Hampshire), a non-profit organization that distributes the key features examination described in this report.

Other disclosures: Dr. Valarie J. Lang, Dr. Kirk Bronander, and Dr. Heather Harrell are supported for their work with Aquifer Internal Medicine (Hanover, New Hampshire), a non-profit organization that distributes the key features examination described in this report.

Ethical approval: The authors procured institutional review board approval to conduct the study described in this report from Case Western Reserve University, University of Florida, Indiana University School of Medicine, University of Louisville, University of Nevada at Reno School of Medicine, University of Rochester School of Medicine & Dentistry, Southern Illinois University, University of Illinois at Chicago, and University of Illinois at Peoria.

Disclaimers: The work presented here is that of the authors and does not necessarily represent the views or policy of the United State Veterans Affairs Administration.

Previous presentations: Portions of the study were presented at Academic Internal Medicine Week (New Orleans, Louisiana, October 2013); the Ottawa Conference (Ottawa, Canada, April 2014); and the University of Illinois at Chicago Masters in Health Professions Education conference (Chicago, Illinois, July 2015).

Correspondence should be addressed to Valerie J. Lang, University of Rochester School of Medicine and Dentistry, 601 Elmwood Avenue, Box MED-HMD, Rochester, NY 14642; telephone: (585) 273-4881; e-mail: Valerie_Lang@urmc.rochester.edu; Twitter: @ValerieLang9.

© 2018 by the Association of American Medical Colleges