Purpose: High-quality checklists are essential to performance test score validity. Prior research found that physical exam checklists of items that clinically discriminated between competing diagnoses provided more generalizable scores than all-encompassing thoroughness checklists. The purpose of this study was to compare validity evidence for clinically discriminating versus thoroughness checklists, hypothesizing that evidence would favor the former.
Method: Faculty at four Chicago-area medical schools developed six standardized patient (SP) cases with checklists of about 20 items (“thoroughness [long] checklists”). Four clinicians identified a subset of items that clinically discriminated between competing diagnoses of each case (“clinically discriminating [short] checklists”). Cases were administered to 155 University of Illinois at Chicago fourth-year medical students during their 2011 Clinical Skills Examination (CSE). Validity evidence was compared for CSE scores based on thoroughness versus clinically discriminating checklist items.
Results: Validity evidence favoring clinically discriminating checklists included response process: greater SP checklist accuracy (kappa = 0.75 for long and 0.84 for short checklists, P < .05); internal structure: better item discrimination (0.28 long, 0.42 short, P < .001); internal consistency reliability (0.80 long, 0.92 short); standard error of measurement (z score 8.87 long, 8.05 short); and generalizability (G = 0.504 long, 0.533 short). There were no significant differences overall in relevance ratings, item difficulty, or cut scores of long versus short checklist items.
Conclusions: Limiting checklist items to those affecting diagnostic decisions resulted in better accuracy and psychometric indices. Thoroughness items performed without thinking do not reflect clinical reasoning ability and contribute construct-irrelevant variance to scores.
Dr. Yudkowsky is associate professor, Department of Medical Education, and director, Dr. Allan L. and Mary L. Graham Clinical Performance Center, University of Illinois at Chicago, Chicago, Illinois.
Dr. Park is assistant professor, Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois.
Dr. Riddle is assistant professor, Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois.
Ms. Palladino is a student, University of Illinois College of Pharmacy. At the time of the study she was a graduate research assistant, Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois.
Dr. Bordage is professor, Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois.
Funding/Support: This study was funded in part by a grant from the National Board of Medical Examiners (NBME), Edward J. Stemmler MD Medical Education Research Fund. The project does not necessarily reflect NBME policy, and NBME support provides no official endorsement.
Other disclosures: None reported.
Ethical approval: Ethical approval was obtained from the institutional review board at the University of Illinois at Chicago (no. 2010-0362).
Previous presentations: Oral abstracts summarizing portions of this study were presented at the Annual Conference of the Association of Standardized Patient Educators, Atlanta, Georgia, June 2013; and at the Annual Clinical Skills Chicago Style Conference, Chicago, Illinois, August 2013.
Correspondence should be addressed to Dr. Yudkowsky, UIC-COM Department of Medical Education 986 CMET, 808 S. Wood MC 591, Chicago, IL 60612; telephone: (312) 996-3598; e-mail: email@example.com.