Barsuk, Jeffrey H.; Cohen, Elaine R.; McGaghie, William C.; Wayne, Diane B.
Iatrogenic complications occur in 2.9% to 3.7% of hospitalized patients.1 Medical procedures are a common cause of these complications.1,2 Many bedside procedures are attempted by unsupervised medical trainees, subjecting patients to excess risk.3 Simulation-based education allows for acquisition of clinical skills in a safe, forgiving environment. This method improves procedural skills in a variety of medical specialties.4–7
Deliberate practice of clinical skills with individualized performance feedback8 is a critical feature of simulation-based education. Mastery learning7,9 is a rigorous form of competency-based education where skills are measured against high achievement standards. All learners must reach mastery standards, although training time varies. In simulation-based mastery learning (SBML), trainees must meet or exceed the mastery standard in a simulated environment before performing procedures on patients.
Prior research demonstrates that SBML improves trainee skill and quality of patient care in advanced cardiac life support (ACLS)9,10 and central venous catheter (CVC) insertion.11–13 ACLS skills were retained for 14 months in one study,14 but skill retention after SBML for CVC insertion is unknown. The aim of this study was to evaluate CVC insertion skill retention six months and one year after SBML.
This is a longitudinal cohort study of CVC insertion skill retention after SBML. The study was performed at Northwestern Memorial Hospital (NMH), from July 2007 to July 2009. The Northwestern University institutional review board approved the study. Trainees provided informed consent before participating.
Second- and third-year internal medicine residents completed SBML of internal jugular (IJ) and subclavian (SC) CVC insertion using a simulator and ultrasound.11,12 Residents were pretested on the simulator using a 27-item skills checklist. Subsequently, residents received two, 2-hour education sessions featuring a lecture, ultrasound training, deliberate practice with the CVC simulator, and directed feedback. Education sessions contained standardized didactic material on CVC indications and complications, and a stepwise demonstration of IJ and SC CVC insertions. After training, residents were posttested and required to meet or exceed a minimum passing score (MPS) on the simulator using the skills checklist. The MPS was set for IJ and SC procedures by an expert panel.15
We identified 61 individuals who completed SBML and remained at NMH during the study period. These physicians were completing internal medicine residency or had advanced to other graduate medical education programs or faculty positions. Physicians were invited to participate in the current study to assess CVC insertion skill retention six months and one year after training. Posttest scores at the end of SBML were compared with follow-up scores.
Posttest and follow-up skills examinations used the same CVC simulator, ultrasound device, and checklist. Examinations were graded by a single unblinded instructor (J.H.B.) and videotaped. A random sample of 108 of 326 (33%) posttests, six-month, and one-year follow-up examinations were rescored by a second rater (D.B.W.) to estimate interrater reliability and rule out single-rater bias. The second rater was blind to results of the first checklist scoring and examination occasion. Demographic data including age, gender, year of training, and scores on the United States Medical Licensing Examination (USMLE) Steps 1 and 2 were obtained. Participants reported confidence in CVC insertion using a 100-point scale (0 = not confident and 100 = very confident) and clinical experience with the procedure.
Primary outcome measures were checklist scores at posttest, six months, and one year. Skill retention was assessed two ways. First, the proportion of subjects who met or exceeded the MPS at each examination was compared. Second, group mean examination scores were compared. Secondary outcome measures were procedural self-confidence and CVC insertion experience. Correlations were calculated between secondary outcomes and skill retention.
Checklist interrater reliability was estimated using the Kappa (κ) coefficient,16 chance adjusted using the Brennan and Prediger formula.17 The proportion of trainees who met or exceeded the MPS at posttest was compared with six-month and one-year follow-up scores using the chi-square statistic. Mean scores for each examination occasion were compared using one-way ANOVA. Relationships between age, self-confidence, CVC insertion experience, and USMLE Step 1 and 2 scores were associated with performance at six months and one year using Spearman correlations. Gender, ethnicity, and year of training were compared with follow-up performance using the chi-square statistic.
Forty-nine of 61 subjects (80.3%) consented to participate in the study. Sixteen (32.7%) were examined at both six months and one year. Twelve subjects (19.7%) declined to participate because of scheduling conflicts or graduation. The MPS was previously set at 79.1% by a multidisciplinary expert panel.15 Demographic characteristics regarding subjects who declined to participate were similar to those who consented. There were no between-group differences regarding age, gender, or proportion completing residency, pursuing fellowship training, or in faculty positions.
Interrater reliability measured by the mean kappa coefficient was high (κn = 0.87) across the 27 checklist items. Pretest checklist scores were poor. Mean IJ score was 49.1% (SD = 24.3), while mean SC score was 44.1% (SD = 26.9). Only 12.2% of residents met or exceeded the MPS for IJ and 8.2% for SC (Figure 1). At posttest, all residents met or exceeded (mastered) the MPS for IJ and SC.
There was a statistically significant difference between the proportion of subjects who met or exceeded the MPS at posttest, six-month, and one-year follow-up. As shown in Figure 1, for IJ CVC insertion, 49/49 (100%) subjects met the MPS at posttest, 28/34 (82.4%) met the MPS at six months, and 27/31 (87.1%) met the MPS at one year (P = .013). For SC CVC insertion, 49/49 (100%) subjects met the MPS at posttest, 29/34 (85.3%) met the MPS at six months, and 26/31 (83.9%) met the MPS at one-year follow-up (P = .016). The examinees who failed to meet the MPS were different at each follow-up occasion.
There was also a statistically significant difference between mean scores at posttest, six-month, and one-year follow-up. The mean score on the 27-item IJ checklist was 96.5% (SD = 4.7) at posttest, 84.6% (SD = 18.9) at six months, and 87.9% (SD = 16.1) at one year (P < .001). For SC, residents scored 94.6% (SD = 10.6) at posttest, 88.2% (SD = 15.8) at six months, and 88.2% (SD = 16.8) at one year (P = .002). However, in contrast to pretest performance, all follow-up examination means were above the MPS.
Trainees reported high self-confidence about CVC insertion. Age, self-confidence, CVC insertion experience, and USMLE Step 1 and 2 scores showed no practical correlation with follow-up skill examination scores and overall (Table 1). Two of 30 correlations (6%) are statistically significant, a frequency barely exceeding chance for this dataset. Gender and year of training were not associated with follow-up skill performance.
This study demonstrates substantial skill retention six months and one year after SBML. A small number of poor performers account for the significant decrease in mean performance and proportion meeting the MPS. Figure 1 shows that between 82.4% and 87.1% of trainees exceeded the MPS and maintained high performance one year after training. Use of the passing rate is an appropriate method to document skill retention because it represents the number of residents who could perform a simulated CVC insertion competently.15 Use of this approach has been validated by clinical studies showing that residents who completed CVC SBML delivered higher-quality clinical care than residents who did not complete this training. Specifically, they required fewer needle passes to insert a CVC, caused fewer arterial punctures, and had higher success rates.11,12 Rates of catheter-related bloodstream infections also declined dramatically.13
CVC insertion skill retention has not been reported previously. Millington and colleagues18 showed retention of CVC insertion knowledge and procedural self-confidence 18 months after training but did not assess procedural skill. Our finding that skills acquired after SBML are retained at one year are similar to results from studies of ACLS14 and from studies of high-risk obstetric deliveries.19 However, other studies of simulation-based interventions have demonstrated significant skill decay over time.20 Our intervention features a standardized curriculum with rigorous assessments that verify skill acquisition. We believe this design is responsible for skill retention and recommend a mastery learning approach for medical procedure training.
Self-confidence, CVC insertion experience, and year of training did not predict performance. This is consistent with findings from earlier work.6,7,9,11,12 Linking improved self-confidence to improved clinical outcomes is important because self-assessment does not always correlate with performance.21 Substantial evidence shows that cumulative experience is not a proxy for clinical skill.22 The likely reason is that sustained deliberate practice rarely occurs in clinical settings. Procedural skills should be documented through rigorous assessment rather than reliance on clinical experience or reported self-confidence because we cannot reliably predict who will pass or fail follow-up examinations.
This study has several limitations. First, it was conducted at one institution with a small sample size. Second, not all eligible subjects participated. Third, the first rater performing checklist scoring was not blind to follow-up status. This was addressed by including a second rater blinded to posttest or follow-up status using a random sample of examinations from all three testing occasions. The high interrater reliability demonstrated in this study is consistent with other reports published by our research group.6,7,9,11,12,14 Finally, we did not evaluate clinical care or patient outcomes. Further study is needed as SBML for CVC insertion has been shown to improve patient care and outcomes.11–13
In conclusion, this study demonstrates that SBML produces high performance that is resistant to decay across a one-year period. Skill retention by individual residents cannot be predicted by posttest performance, self-confidence, or clinical experience. We have no explanation for this outcome, so periodic testing and refresher training for invasive procedures are recommended. On the basis of these findings, we continue to require that all residents meet or exceed the MPS on the CVC simulator before inserting CVCs on actual patients.
The authors thank the Northwestern University internal medicine residents for their dedication to education and patient care. The authors also acknowledge Drs. Douglas E. Vaughan and Charles M. Watts for their support and encouragement of this work.
Dr. McGaghie's contribution was supported in part by the Jacob R. Suker, MD, Professorship in Medical Education at Northwestern University and by grant UL 1 RR025741 from the National Center for Research Resources, National Institutes of Health. The National Institutes of Health had no role in the preparation, review, or approval of the manuscript.
Northwestern University institutional review board approved the study.
1Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med. 1991;324:370–376.
2Duffy FD, Holmboe ES. What procedures should internists do? Ann Intern Med. 2007;146:392–393.
3Lucas BP, Asbury JK, Wang Y, et al. Impact of a bedside procedure service on general medicine inpatients: A firm-based trial. J Hosp Med. 2007;2:143–149.
4Andreatta PB, Woodrum DT, Birkmeyer JD, et al. Laparoscopic skills are improved with LapMentor training: Results of a randomized, double-blinded study. Ann Surg. 2006;243:854–860.
5Blum MG, Powers TW, Sundaresan S. Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy. Ann Thorac Surg. 2004;78:287–291.
6Wayne DB, Butter J, Siddall VJ, et al. Simulation-based training of internal medicine residents in advanced cardiac life support protocols: A randomized trial. Teach Learn Med. 2005;17:210–216.
7Wayne DB, Barsuk JH, O'Leary KJ, Fudala MJ, McGaghie WC. Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice. J Hosp Med. 2008;3: 48–54.
8Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(10 suppl):S70–S81.
9Wayne DB, Butter J, Siddall VJ, et al. Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. J Gen Intern Med. 2006;21:251–256.
10Wayne DB, Didwania A, Feinglass J, Fudala MJ, Barsuk JH, McGaghie WC. Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: A case-control study. Chest. 2008;133:56–61.
11Barsuk JH, McGaghie WC, Cohen ER, Balachandran JS, Wayne DB. Use of simulation-based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit. J Hosp Med. 2009;4:397–403.
12Barsuk JH, McGaghie WC, Cohen ER, O'Leary KJ, Wayne DB. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med. 2009;37:2697–2701.
13Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Use of simulation-based education to reduce catheter-related bloodstream infections. Arch Intern Med. 2009;169:1420–1423.
14Wayne DB, Siddall VJ, Butter J, et al. A longitudinal study of internal medicine residents' retention of advanced cardiac life support skills. Acad Med. 2006;81(10 suppl):S9–S12.
15Wayne DB, Barsuk JH, Cohen E, McGaghie WC. Do baseline data influence standard setting for a clinical skills examination? Acad Med. 2007;82(10 suppl):S105–S108.
16Fleiss JL, Levin BA, Paik MC. Statistical Methods for Rates and Proportions. 3rd ed. Hoboken, NJ: Wiley; 2003.
17Brennan RL, Prediger DJ. Coefficient kappa: Some uses, misuses and alternatives. Educ Psychol Meas. 1981;41:687–699.
18Millington SJ, Wong RY, Kassen BO, Roberts JM, Ma IW. Improving internal medicine residents' performance, knowledge, and confidence in central venous catheterization using simulators. J Hosp Med. 2009;4:410–416.
19Crofts JF, Bartlett C, Ellis D, Hunt LP, Fox R, Draycott TJ. Management of shoulder dystocia: Skill retention 6 and 12 months after training. Obstet Gynecol. 2007;110:1069–1074.
20Duran R, Aladag N, Vatansever U, Kucukugurluoglu Y, Sut N, Acunas B. Proficiency and knowledge gained and retained by pediatric residents after neonatal resuscitation course. Pediatr Int. 2008;50:644–647.
21Wayne DB, Butter J, Siddall VJ, et al. Graduating internal medicine residents' self-assessment and performance of advanced cardiac life support skills. Med Teach. 2006;28:365–369.
22Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: The relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142:260–273.