Share this article on:

Validity Evidence for a Patient Note Scoring Rubric Based on the New Patient Note Format of the United States Medical Licensing Examination

Park, Yoon Soo PhD; Lineberry, Matthew PhD; Hyderi, Abbas MD, MPH; Bordage, Georges MD, PhD; Riddle, Janet MD; Yudkowsky, Rachel MD, MHPE

doi: 10.1097/ACM.0b013e3182a34b1e
Rime

Purpose: This study examines validity evidence for the Patient Note Scoring Rubric, which was developed for a local graduation competency exam (GCE) to assess patient notes written in the new United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills format. The rubric was designed to measure three dimensions: Documentation, justified differential diagnosis (DDX), and Workup.

Method: Analyses used GCE data from 170 fourth-year medical students who completed five standardized patient (SP) cases in May 2012. Five physician raters each scored all responses for one case. Internal structure was examined using correlations between dimensions and between cases; a generalizability study was also conducted. Relationship to other variables was examined by correlating patient note scores with SP encounter scores. Consequence was assessed by comparing pass–fail rates between the rubric and the previous global rating. Response process was examined using rater feedback.

Results: Correlations between scores from different dimensions ranged between 0.33 and 0.44. Reliability of scores based on the phi coefficient was 0.43; 15 cases were required to reach a phi coefficient of 0.70. Evidence of case specificity was found. Documentation scores were moderately correlated with SP scores for data gathering (r = 0.47, P < .001). There was no meaningful change in pass–fail rates. Raters’ feedback indicated that they required more training for scoring the DDX and Workup dimensions.

Conclusions: There is initial validity evidence for use of this rubric to score local clinical exams that are based on the new USMLE patient note format.

Dr. Park is assistant professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois.

Dr. Lineberry is assistant professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois.

Dr. Hyderi is associate dean for curriculum and associate professor, Department of Family Medicine, University of Illinois at Chicago College of Medicine, Chicago, Illinois.

Dr. Bordage is professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois.

Dr. Riddle is director of faculty development and research assistant professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois.

Dr. Yudkowsky is director, Graham Clinical Performance Center, and associate professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois.

Funding/Support: None.

Other disclosures: None.

Ethical approval: This study was approved by the institutional review board of the University of Illinois at Chicago.

Correspondence should be addressed to Dr. Park, Department of Medical Education (MC 591), College of Medicine, University of Illinois at Chicago, 808 S. Wood St., 963 CMET, Chicago, IL 60612-7309; telephone: (312) 355-5406; fax: (312) 413-2048; e-mail: yspark2@uic.edu.

© 2013 by the Association of American Medical Colleges