Academic Medicine

Skip Navigation LinksHome > October 2011 - Volume 86 - Issue 10 > Evaluating Construct Equivalence and Criterion-Related Valid...
Academic Medicine:
doi: 10.1097/ACM.0b013e31822bc0a4

Evaluating Construct Equivalence and Criterion-Related Validity for Repeat Examinees on a Standardized Patient Examination

Raymond, Mark R. PhD; Kahraman, Nilufer PhD; Swygert, Kimberly A. PhD; Balog, Kevin P. MS

Collapse Box


Purpose: Prior studies report large score gains for examinees who fail and later repeat standardized patient (SP) assessments. Although research indicates that score gains on SP exams cannot be attributed to memorizing previous cases, no studies have investigated the empirical validity of scores for repeat examinees. This report compares single-take and repeat examinees in terms of both internal (construct) validity and external (criterion-related) validity.

Method: Data consisted of test scores for examinees who took the United States Medical Licensing Examination Step 2 Clinical Skills (CS) exam between July 16, 2007, and September 12, 2009. The sample included 12,090 examinees who completed Step 2 CS on one occasion and another 4,030 examinees who completed the exam on two occasions. The internal measures included four separately scored performance domains of the Step 2 CS examination, whereas the external measures consisted of scores on three written assessments of medical knowledge (Step 1, Step 2 clinical knowledge, and Step 3). The authors subjected the four Step 2 CS domains to confirmatory factor analysis and evaluated correlations between Step 2 CS scores and the three written assessments for single-take and repeat examinees.

Results: The factor structure for repeat examinees on their first attempt was markedly different from the factor structure for single-take examinees, but it became more similar to that for single-take examinees by their second attempt. Scores on the second attempt correlated more highly with all three external measures.

Conclusions: The findings support the validity of scores for repeat examinees on their second attempt.

© 2011 Association of American Medical Colleges


Article Tools


Article Level Metrics