Approximately 7% to 28% of medical trainees, regardless of their level of training or specialty, will require remediation in the form of an individualized learning plan to achieve competence.1–4 Among medical students, reported deficits include inability to integrate large amounts of material, poor time and stress management, and poor test-taking skills; among residents, they include insufficient medical knowledge, poor clinical judgment, inefficient use of time, inappropriate interactions with patients and colleagues, and unacceptable moral behaviors.4–6 Despite the magnitude of this problem, it remains a challenge for medical educators to identify underperforming learners, isolate learners’ areas of deficiency, create and implement effective remediation plans, and conduct unbiased reassessments of learners’ progress.7
In 2009, Hauer and colleagues’8 thematic review of the literature on the remediation of medical students, residents, fellows, and practicing physicians revealed that only a few small studies had been published on the subject. Moreover, the reviewed studies were predominantly focused on the remediation of specific skill sets and did not demonstrate standardized methods that would provide the evidence necessary to guide best practices in remediation. The authors called for more research on this topic, specifically for studies with higher power and clearly defined methods and outcomes.
In that review, Hauer et al8 also proposed a model of remediation that included assessment of the learner’s competence, diagnosis of the learner’s deficiency, and development of an individualized learning plan incorporating deliberate practice, feedback, reflection, and, finally, a focused reassessment.8 Their model is derived from prior work on expert performance showing that expertise is earned through intentionally practicing tasks beyond one’s level of comfort and competence under the guidance of a coach. The coach’s role is to provide feedback and teach self-reflection.9
The remediation program at the University of Colorado School of Medicine was created in 2006 and is based on the aforementioned principles of deliberate practice, feedback, and reflection.10 Here, we report the findings of the prospective observational study we conducted to identify the deficits of the learners referred to the remediation program during its first six years and predictors of poor academic outcomes. After describing our remediation program, we describe and compare the types of deficits addressed during remediation, the faculty time required for remediation, and learners’ academic outcomes. We also analyze learner feedback on the remediation program. The goal of our study was to help educators better predict the needs of struggling learners and guide future resource allocation and program development.
The Remediation Program at the University of Colorado School of Medicine
Learners at all levels of training—medical students, residents, fellows, and attending physicians—self-refer or are referred to the remediation program director (J.G.) for assessment and remediation. Medical students are referred by their clerkship or course directors, and residents and fellows are referred by their program directors. Learners at these levels are referred because they have received repetitive negative comments on rotation evaluations, have failed or are in danger of failing a rotation, or are no longer in good academic standing and have been placed on a letter of warning or focused review. Attending physicians participate in the remediation program through self-referral only.
The remediation program faculty represent multiple medical specialties and are medical education specialists who have been recognized locally and nationally for their teaching abilities. One of these remediation specialists conducts a semistructured intake interview with each referred learner. This interview has been designed to address the Accreditation Council for Graduate Medical Education competencies,11 with some of the competencies subdivided to enable faculty to better customize remediation plans. It explores areas including the learner’s medical knowledge, clinical skills, clinical reasoning, time management and organization, interpersonal skills, communication skills, professionalism, and mental well-being. (The interview guide is available as Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A179.) For the purposes of our program, mental well-being includes psychiatric diagnoses, substance abuse, learning disabilities, and psychosocial stressors.
After the interview, if the learner is a student, the student affairs dean convenes a “Success Team”; if the learner is a resident, fellow, or attending, the remediation specialist convenes this team. The Success Team comprises the learner and a remediation specialist; it may or may not include faculty from the referring clerkship/course or the learner’s specialty, a psychiatrist or mental health professional, and the student affairs dean or program director. This team reviews a summary of the interview and the learner’s academic record, plus available reports from the Colorado Physicians’ Health Program, neuropsychiatric testing, direct observation, and additional evaluations as needed. Using the gathered data, the team determines the likely areas of deficiency and which deficit to target first during the remediation process. In general, our remediation program addresses deficits individually so as not to overwhelm the learner. Whenever possible, the Success Team makes decisions about the learner on the basis of facts or group consensus to help mitigate biases.
The Success Team then creates and implements a remediation plan to address the identified deficit. The plan includes deliberate practice, timely and regular feedback, and a chance for the learner to reflect on his or her performance.8 Remediation plans are conducted according to the guidelines in Remediation of the Struggling Medical Learner.12 (For a sample remediation plan, see Appendix 1.) The remediation specialist is charged with facilitating the remediation plan with other involved faculty and also providing individual remedial skills teaching and feedback as needed. For students, the student affairs dean is notified of the plan and receives progress updates at weekly meetings with the remediation specialist. For residents and fellows, program directors are notified of the plan and receive weekly progress updates via telephone or e-mail. Once the first deficit has been addressed, learning plans are subsequently created for each additional deficit, if more than one is present. Occasionally, while one deficit is being addressed, improvement is noted in other identified deficits.
The Success Team chooses a reassess ment method based on the deficit and notifies the learner. Reassessments are performed by faculty members from other departments or hospital sites who are unaware of the learner’s remediation status and consist of routine elements of the course or rotation, such as end-of-rotation global assessments, direct observations, National Board of Medical Examiners shelf exams or other multiple-choice question exams, or standardized patient encounters. The course, clerkship, or program director receives the results and makes the ultimate decision about success or failure of the remediation efforts. Attendings are reassessed both through their annual departmental review process and by the institution-wide medical board that assesses practice privileges.
The decision to place a learner on proba tion after the initiation of remediation is made by the promotions committee (for a medical student), program director (for a resident or fellow), or division head (for an attending physician); it is never made by members of the learner’s Success Team. The team’s initial learner assessment and remediation plan, as well as reassessment results, are made avail able to those determining probationary status.
From July 2006 to June 2012, 151 learners were referred to the University of Colorado School of Medicine remediation program. The remediation program director (J.G.) kept a confidential record of the learners enrolled in the program, their levels of training, their identified deficits, their remediation plan details, and their academic outcomes through graduation or completion of this study in October 2012. We asked faculty assigned remediation tasks to record time spent face-to-face with the learner. (This did not include time for planning, assessment, or preparing the remediation plan.) Within an individual’s remediation, time per deficit was not tracked.
We e-mailed each of the learners a single invitation to participate in a Web-based survey, hosted at SurveyMonkey (www.surveymonkey.com), one month after he or she completed the remediation program. Participation in the survey was voluntary and anonymous. The survey included five items concerning the learner’s opinion of the program and postprogram perception of skills, which were rated using a five-point Likert scale (ranging from strongly disagree = 1 to strongly agree = 5). These items were followed by two free-text comment boxes for feedback on the benefits of the program and ways in which it could be improved. At the end of each academic year, a research assistant collated all of the survey data and deidentified content of the free-text responses. This study was approved by the Colorado Multiple Institutional Review Board.
We calculated descriptive statistics; to ensure confidentiality, we combined data for fellows and attendings as “postresidency learners.” We analyzed between-group comparisons using the chi-square test. We used the Fisher exact test to examine the relationships between gender, level of training, and specific deficits. We used analysis of variance (ANOVA) to explore faculty time on the basis of deficit. To identify predictors of negative academic outcomes—such as not matching into a residency position, being put on probation, or failing to graduate—we conducted logistic regression analysis and the Fisher exact test using gender, deficit, and faculty time. We considered a P value < .10 on chi-square, the Fisher exact test, and ANOVA to be significant for entry into the logistic regression model; for all other analyses, we considered a P value < .05 to be significant. Two research assistants independently reviewed the qualitative data collected via the free-text comments and categorized the data into themes. They then discussed the themes and combined them after reaching consensus.
Of the 151 learners referred to the remediation program, 72 (48%) were medical students, 65 (43%) were residents, and 14 (9%) were postresidency learners. Two of the referred learners did not complete the program: One was placed on probation, and the other withdrew from the training program. Overall, more men (n = 89; 59%) than women (n = 62; 41%) were referred (P = .04). Almost all of the referred learners—including all residents, fellows, and attendings, and most medical students (68/72; 94%)—had failed an exam, a course, or a rotation, or were failing a midcourse assessment or annual review. Ten (6.6%) of the 151 learners self-referred to the program, all for at least poor medical knowledge after failing a standardized exam. Self-referred learners were otherwise statistically the same as referred learners in terms of deficits identified, faculty face time required for remediation, and academic outcomes. None of the learners were on probation at the time of referral.
Deficits and faculty time
Most learners had more than one deficit, with a mean (standard deviation [SD]) of 2.14 (1.37) deficits for medical students, 1.59 (0.77) for residents, and 1.80 (1.15) for fellows and attendings. Medical knowledge, clinical reasoning, and professionalism were the most common deficits identified (Figure 1). Difficulties with mental well-being—including psychiatric diagnoses, substance abuse, learning disabilities, and psychosocial stressors—were signifi cantly more prevalent among medical students than other learners (P = .03). Men were more likely than women to have difficulties with mental well-being (P = .06) and significantly more likely to have communication deficits (P = .01; see Supplemental Digital Figure 1 at http://links.lww.com/ACADMED/A179). The prevalence of professionalism deficits increased steadily from one level of training to the next.
Overall, the mean (SD) number of hours of faculty face time required for remediation was 18.8 (23.8; skewness 1.9) per learner. Medical students required 17.8 (26.7) hours, residents 19.8 (26.5) hours, and postresidency learners 15.7 (15.6) hours (P = .89). Because many learners had more than one deficit, those learners with the comorbid deficit of either clinical reasoning (P < .001) or mental well-being (P = .03) required significantly more faculty time than learners without either of these deficits (Table 1).
Fourteen (9%) of the 151 learners were placed on probation during remediation: 4 (6%) of the 72 medical students, 7 (11%) of the 65 residents, and 4 (29%) of the 14 postresidency learners. Our logistic regression analysis identified poor professionalism as the only predictor of being placed on probation (P < .001). In addition, 3 (4%) of the medical students did not match into any residency program. All 3 displayed interpersonal skills deficits.
By completion of the study in October 2012, 136 (90%) of the 151 referred learners had graduated from their training program, were in good academic standing, or, in the case of attendings, were practicing medicine without restrictions (Table 2). Ten of the graduates (7% of learners) had transferred to another training program and graduated from that program. Fifteen learners (10%) were on probation or restricted practice, had transferred to another training program and failed to graduate from that program, or had withdrawn from their training program. No learners were dismissed.
Neither number of deficits nor combination of deficits was predictive of academic outcome. Having been placed on probation was the greatest predictor of a poor academic outcome such as restricted practice, transferring but not graduating, or withdrawing (P < .0001). Faculty face time significantly reduced the odds of probation by 3.1% per hour (95% CI, 0.09–0.63) and of all negative outcomes by 2.6% per hour (95% CI, 0.96–0.99).
Of the 151 referred learners, 120 (79%) responded to the survey after completing the remediation program. The majority of respondents agreed or strongly agreed that the program had helped them address challenges or gain skills that would help them succeed, and that they had been supported by the remediation program (Figure 2). Of the 69 respondents who wrote comments, 53 (77%) expressed appreciation for the availability and approachability of faculty, 47 (68%) commented on the productive and honest feedback they received, 22 (32%) noted that the remediation plan was customized to their needs, 16 (23%) appreciated the extra teaching, and 9 (13%) appreciated the recognition of the improvements they had made. Suggestions for improvements to the program included cutting down on clinical responsibilities during remediation (n = 44; 64%), knowing more about the program in advance of the referral (n = 32; 46%), and knowing about the program earlier in training (n = 15; 22%).
Published studies on the success of programmatic remediation efforts are small, and their reported outcomes are highly variable. One study showed that 80% of the medical students who required remediation in their third-year medicine clerkship went on to have a successful intern year, receiving only favorable comments on their evaluations.13 Another study, however, reported that 15 of 17 surgical residents with performance problems still had those problems by the end of the training program.7 Our study on the remediation program at the University of Colorado School of Medicine is the largest and most comprehensive of its kind to date. We found that deficits among learners varied by learner level, that certain deficits were correlated with the amount of time needed for remediation as well as probationary status, and that increases in faculty time significantly reduced the odds of a negative outcome.
While medical knowledge, clinical reasoning, and professionalism were the most common deficits identified in this study, underperforming learners also struggled with a broad spectrum of competency-based deficits, and most had more than one deficit. Several prior studies have acknowledged that the first recognized deficit is often linked to broader concerns involving multiple deficits.1,14–16 A recent study17 found that the average number of deficits per resident was 2.6, which is just slightly higher than the 1.59 per resident in our study. Although the terms used to define the deficits in that study were slightly different, the same themes arose, with problems with medical expertise, professionalism, and communication being the most prevalent.17
Medical students in this study were more likely than other learners to be identified as having mental well-being issues. This may be because students are under the stressors of having their career paths determined by their grades and receive higher-stakes evaluations than do learners later in training. The difference may also reflect a cultural shift toward acknowledging, treating, and reporting mental illness. A prior study15 noted that 32.6% of residents in difficulty had depression, anxiety, or personality disorders that contributed to their poor performance, which is slightly higher than but consistent with our finding of 29% of referred residents.
Professionalism deficits consistently and steadily increased for each level of training. There are several possible explanations for this finding: Expectations of professional behavior change as a physician takes on more responsibility; burnout rates increase from medical school throughout residency into practice18–20; peers, who may help mask unprofessional behavior, are less present during clinical care; and attendings and fellows may feel that they can be less restrained in their behavior.
Remediation of struggling learners requires substantial resources.15 In our study, the remediation of clinical reasoning and communication deficits took the most faculty face time. To develop clinical reasoning skills, an underperforming learner requires ample face time with faculty to learn how to develop a framework for clinical problem solving and then to test this framework on a wide variety of cases in which the learner vocalizes his or her thinking to more seasoned clinicians and teachers. Faculty aid the learner in following the necessary steps of the decision-making process, identify errors, and correct faulty reasoning. The remediation of communication deficits requires faculty to observe numerous sessions in which the learner interacts with real or standardized patients so that faculty may provide feedback and determine whether the learner is incorporating that feedback in subsequent encounters.
Learners who struggled with mental well-being required significantly more faculty face time than other learners. Such issues often limit the pace at which a learner can acquire new information and skills because they limit the ability to remain on task while studying and learning.21–23
Among the learners placed on probation, we found poor professionalism to be the only consistent and strong predictor of this outcome. This finding may reflect tolerance for other areas of incompetence if the learner is trying and displays some insight, or it may reflect the difficulty of addressing professionalism issues and their persistence despite remediation. Such persistence of poor professional behavior in this subset of learners may also explain why those placed on probation were more likely than others to have poor academic outcomes. This is consistent with prior studies showing a correlation among practicing physicians between disciplinary action by a state medical board and prior unprofessional behavior as a medical student.24,25
Remediation time, as measured in faculty face time, was shown in this study to reduce negative academic outcomes and to reduce the odds of a learner being placed on probation. Prior work has shown that without specialized remediation, struggling learners simply spend more time using previously unsuccessful learning methods rather than changing their strategies, and, therefore, are more likely to fail again.26,27 One way to decrease faculty face time would be to provide remediation to learners in small groups, which would also enhance sharing of and reflecting on ideas, allow learners to critique each other’s work, and provide a means for peer-to-peer emotional support.28
It can be challenging, however, to define “success” with remediation. For some learners, success means being able to practice medicine safely and independently, whereas for others, it means changing career paths. Of the learners who entered our remediation program, 90% achieved what we considered to be successful outcomes. The remaining 10% of learners were placed on probation or were otherwise restricted, transferred to another training program and failed to graduate, or withdrew from their training program. None of the learners in our study were dismissed. Our results are slightly better than those reported by Dupras et al,15 who found that 12.6% of residents were placed on probation, 4.7% were dismissed, and 7.9% resigned. These differences may be due to varying definitions and the lack of standardization at a national level of who should be placed on probation.
Not only did remediation work, but learners also felt that the program had benefited them as individuals. Most of the learners who offered suggestions for improvements identified issues that could be addressed prior to enrollment in the program. These included providing more information about the program so that it would become a well-known resource on campus.
This study had several limitations. It was conducted at a single institution with a program dedicated to underperforming learners and a specific culture of learners. It is possible that not all learners in need of remediation were identified and included in the program, although attempts were made to be all-inclusive. Learners with minor deficits were more likely to have been excluded. Further, there is no standard definition of what constitutes a “deficit.” In this study, deficits were linked to the most appropriate category by a group of authors with expertise in remediation; however, these categories are not mutually exclusive, leading to challenges with misclassification bias.29 Additional factors such as learner age, referral source, presence of documented disability, and willingness to participate were not included in the data collection.
In summary, although remediation of struggling learners requires substantial resources, it can be very successful. Additional studies are needed to compare remediation strategies, assess how to optimize faculty time, and explore whether there is a maximum amount of time one learner should be offered for remediation. Until then, remediation programs should maximize underperforming learners’ potential by following the basic tenets of early identification, diagnosis of specific areas of deficiency, and design of remediation plans that are based on learners’ greatest deficits and incorporate deliberate practice, feedback, and reflection.
Acknowledgments: The authors wish to thank David Weitzenkamp, PhD, for assistance with data analysis.
1. Reamy BV, Harman JH. Residents in trouble: An in-depth assessment of the 25-year experience of a single family medicine residency. Fam Med. 2006;38:252–257
2. Schwind CJ, Williams RG, Boehler ML, Dunnington GL. Do individual attendings’ post-rotation performance ratings detect residents’ clinical performance deficiencies? Acad Med. 2004;79:453–457
3. Yao DC, Wright SM. National survey of internal medicine residency program directors regarding problem residents. JAMA. 2000;284:1099–1104
4. Paul G, Hinman G, Dottl S, Passon J. Academic development: A survey of academic difficulties experienced by medical students and support services provided. Teach Learn Med. 2009;21:254–260
5. Olmesdahl PJ. The establishment of student needs: An important internal factor affecting course outcome. Med Teach. 1999;21:174–179
6. Yao DC, Wright SM. The challenge of problem residents. J Gen Intern Med. 2001;16:486–492
7. Williams RG, Roberts NK, Schwind CJ, Dunnington GL. The nature of general surgery resident performance problems. Surgery. 2009;145:651–658
8. Hauer KE, Ciccone A, Henzel TR, et al. Remediation of the deficiencies of physicians across the continuum from medical school to practice: A thematic review of the literature. Acad Med. 2009;84:1822–1832
9. Ericsson AK, Prietula MJ, Cokely ET. The making of an expert. Harv Bus Rev. 2007;85:114–121, 193
10. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(10 suppl):S70–S81
11. Accreditation Council for Graduate Medical Education. . The Outcomes Project. 1999 www.acgme.org/outcome/comp/compFull.asp
. Published Accessed December 15, 2012. [No longer available.]
12. Guerrasio J Remediation of the Struggling Medical Learner. 2013 Irwin, Pa Association for Hospital Medical Education Press
13. Lavin B, Pangaro L. Internship ratings as a validity outcome measure for an evaluation system to identify inadequate clerkship performance. Acad Med. 1998;73:998–1002
14. Hauer KE, Teherani A, Kerr KM, O’Sullivan PS, Irby DM. Student performance problems in medical school clinical skills assessments. Acad Med. 2007;82(10 suppl):S69–S72
15. Dupras DM, Edson RS, Halvorsen AJ, Hopkins RH Jr, McDonald FS. “Problem residents”: Prevalence, problems and remediation in the era of core competencies. Am J Med. 2012;125:421–425
16. Sayer M, Chaput De Saintonge M, Evans D, Wood D. Support for students with academic difficulties. Med Educ. 2002;36:643–650
17. Zbieranowski I, Takahashi SG, Verma S, Spadafora SM. Remediation of residents in difficulty: A retrospective 10-year review of the experience of a postgraduate board of examiners. Acad Med. 2013;88:111–116
18. Dyrbye LN, Moutier C, Durning SJ, et al. The problems program directors inherit: Medical student distress at the time of graduation. Med Teach. 2011;33:756–758
19. Martins AE, Davenport MC, Del Valle MP, et al. Impact of a brief intervention on the burnout levels of pediatric residents. J Pediatr (Rio J). 2011;87:493–498
20. Prins JT, Gazendam-Donofrio SM, Tubben BJ, van der Heijden FM, van de Wiel HB, Hoekstra-Weebers JE. Burnout in medical residents: A review. Med Educ. 2007;41:788–800
21. Pelley JW Success Types for Medical Students. 1997 Lubbock, Tex Texas Tech University Extended Studies
22. Eysenck NW. Anxiety, learning and memory: A reconceptualization. J Res Pers. 1979;13:363–385
23. Zeidner M. Test Anxiety: The State of the Art. 1998 New York, NY Plenum
24. Papadakis MA, Teherani A, Banach MA, et al. Disciplinary action by medical boards and prior behavior in medical school. N Engl J Med. 2005;353:2673–2682
25. Papadakis MA, Arnold GK, Blank LL, Holmboe ES, Lipner RS. Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards. Ann Intern Med. 2008;148:869–876
26. Loyens SMM, Rikers EMJP, Schmidt HG. The impact of students’ conception of constructivist assumptions on academic achievement and drop-out. Stud High Educ. 2007;32:581–602
27. Mattick K, Knight L. High-quality learning: Harder to achieve than we think? Med Educ. 2007;41:638–644
28. Winston KA, Van der Vleuten CP, Scherpbier AJ. An investigation into the design and effectiveness of a mandatory cognitive skills programme for at-risk medical students. Med Teach. 2010;32:236–243
29. Cox SM, Goepfert A, Hicks PFincher RME. Chapter 10: Working with students with difficulties: Academic and nonacademic. Guidebook for Clerkship Directors. 20053rd ed San Antonio, Tex Alliance for Clinical Education In:
Example of a Remediation Plan as Implemented Through the Remediation Program at the University of Colorado School of Medicine