Share this article on:

Caution Needed When Predicting Problem Residents

Prasad, Vinay MD

doi: 10.1097/ACM.0b013e3181ff64ac
Letters to the Editor

Resident, Department of Medicine, Northwestern University, Chicago, Illinois; v-prasad@md.northwestern.edu.

Back to Top | Article Outline

To the Editor:

In their article “Can we predict problem residents?” Brenner and colleagues1 demonstrate an association between the presence of negative comments in the deans' letters of U.S. medical graduates (USMGs) and residents' having future problems in their psychiatry residency programs. To quote the authors, “The presence of these negative comments showed a strong correlation with problem outcomes among USMGs (χ2 = 7.5241, P < .01).”

Applying the study's results, however, may pose difficulties. Although not strictly appropriate for case–control studies, a back-of-the-envelope calculation can be useful here. Assuming that the rate of false-positives in the control group is similar to that among all residents, one can calculate the sensitivity, specificity, and the positive and negative likelihood ratios (LRs) to determine whether the presence of negative dean's letter comments is a predictor of future problems among USMGs. Those numbers are sensitivity = 0.62, specificity = 0.72, +LR = 2.20, and −LR = 0.53.

What do a +LR of 2.20 and a −LR of 0.53 mean? In cases with intermediate pretest probability (20%–80%), they change posttest probability modestly. And in extreme cases, with high or low pretest probability, their effect is negligible.2

In the case of problem residents, as the authors encountered 40 in 20 years of residency training, the pretest probability is low. In programs and fields where problem residents are infrequent, it may be lower. Thus, the posttest probability remains low irrespective of negative comments. A similar case can be made for the negative LR of 0.53 in predicting who won't be a problem resident, as most are not.

Problem residents are devastating, but they are also exceptional cases. Use of negative comments on the dean's letter as a screening tool may, unfortunately, result in the loss of many ultimately good applicants even if also identifying the rare problem one. This approach may be worth it, given the damage done by problem residents, and the use of negative dean's letter comments may be warranted in broader admission formulas (as the authors report), but it may be challenging and costly in specialty fields with difficulty attracting good applicants.

Vinay Prasad, MD

Resident, Department of Medicine, Northwestern University, Chicago, Illinois; v-prasad@md.northwestern.edu.

Back to Top | Article Outline

References

1Brenner AM, Mathai S, Jain S, Mohl PC. Can we predict “problem residents”? Acad Med. 2010;85:1147–1151. http://journals.lww.com/academicmedicine/Fulltext/2010/07000/Can_We_Predict__Problem_Residents__.15.aspx. Accessed September 28, 2010.
2Grimesa DA, Schulza KF. Refining clinical diagnosis with likelihood ratios. Lancet. 2005;365:1500–1505.
© 2011 Association of American Medical Colleges