Letters to the Editor
To the Editor:
In their Invited Commentary, Drs. Katsufrakis and Chaudhry state that, to their knowledge, no study has ever been done to answer the question of whether the United States Medical Licensing Examination (USMLE) Step 1 scores predict success in residency.1 However, multiple studies have demonstrated that Step 1 scores are not correlated with clinical skills during residency or program directors’ (PDs’) ratings of residents.2,3
What does Step 1 predict? Step 1 scores significantly below the mean predict increased likelihood of failing specialty board exams.4 But students who score near or above the mean have extremely high board passage rates.
If Step 1 does not predict clinical performance, and even middling scores predict high board passage rates, why do PDs value high Step 1 scores? It is a convenient way to compare applicants, Katsufrakis and Chaudhry explain. But importantly, Step 1 was very specifically not designed to compare residency applicants.5 The standard error of difference for Step 1 is 8 points, meaning 2 scores must differ by at least 16 points for us to be confident they are truly different.6 Furthermore, Step 1 is designed to be more reliable close to the pass/fail cut point and less reliable further from the cut point.5 This makes Step 1 a good tool to use for dichotomous licensing decisions, but a bad tool to use for comparing applicants with Step 1 scores far above the cut point, as is done when selecting residents.6
What can the National Board of Medical Examiners do? Stop releasing scores near or above the mean as they do not have a valid use. Stop defending inappropriate uses of Step 1 scores. Start reporting scores in a way that indicates their true level of precision. Start protecting students.
1. Katsufrakis PJ, Chaudhry HJ. Improving residency selection requires close study and better understanding of stakeholder needs. Acad Med. 2019;94:305–308.
2. McGaghie WC, Cohen ER, Wayne DB. Are United States Medical Licensing Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med. 2011;86:48–52.
3. Wagner JG, Schneberk T, Zobrist M, et al. What predicts performance? A multicenter study examining the association between resident performance, rank list position, and United States Medical Licensing Examination Step 1 scores. J Emerg Med. 2017;52:332–340.
4. de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting performance on the American Board of Surgery qualifying and certifying examinations: A multi-institutional study. Arch Surg. 2010;145:852–856.
5. DeChamplain AFD. A primer on classical test theory and item response theory for assessments in medical education. Med Educ. 2010;44:109–117.