Postgraduate Education Outcomes
A Longitudinal Study of Internal Medicine Residents’ Retention of Advanced Cardiac Life Support Skills
Wayne, Diane B.; Siddall, Viva J.; Butter, John; Fudala, Monica J.; Wade, Leonard D.; Feinglass, Joe; McGaghie, William C.
Section Editor(s): Karini, Reena MD; Raible, Michelle MD
Correspondence: Diane B. Wayne, MD, Northwestern University Feinberg School of Medicine, Department of Medicine, 251 E. Huron Street, Galter 3-150, Chicago, IL, 60011; e-mail: (email@example.com).
Background: Internal medicine residents must be competent in Advanced Cardiac Life Support (ACLS) for board certification. Traditional ACLS courses have limited ability to enable residents to achieve and maintain skills. Educational programs featuring reliable measurements and improved retention of skills would be useful for residency education.
Method: We developed a training program using a medical simulator, small-group teaching and deliberate practice. Residents received traditional ACLS education and subsequently participated in four two-hour educational sessions using the simulator. Resident performance in six simulated ACLS scenarios was assessed using a standardized checklist.
Results: After the program, resident ACLS skill improved significantly. The cohort was followed prospectively for 14 months and the skills did not decay.
Conclusions: Use of a simulation-based educational program enabled us to achieve and maintain high levels of resident performance in simulated ACLS events. Given the limitations of traditional methods to train, assess and maintain competence, simulation technology can be a useful adjunct in high-quality ACLS education.
The American Board of Internal Medicine requires candidates for certification to be judged competent by their residency program director in advanced cardiac life support (ACLS).1 Residents usually fulfill this requirement by completing an American Heart Association (AHA) approved provider course that typically involves reading, lecture, and practical instruction about the recognition and management of ACLS protocols and procedures. Despite recommendations about ACLS skill renewal on a two-year cycle,2 several authors have argued that frequent refresher courses should be given to maintain health care providers’ ACLS skill and knowledge.3–6 Such courses aim to prevent rapid clinical skill decay that has not only been demonstrated for ACLS,7–9 but also for other clinical skills including advanced trauma life support,10–12 and airway management.13 Meta-analysis demonstrates that professional skills decay rapidly without practice or use,14 a likely condition for the rare in-hospital cardiac events that prompt an ACLS response.15
Given concerns about skill acquisition and retention, as well as recent reports regarding the quality of rescuscitation efforts even by well-trained personnel,16,17 our research group developed educational protocols and assessments of residents’ skill in six ACLS scenarios using simulation technology. Using a simulation-based educational intervention featuring deliberate practice,18 a randomized trial produced a 38% increase in residents’ ACLS skills compared to wait-list controls.19 This finding was replicated with crossover of the randomized groups and demonstrated persistence of skill in short-term follow-up.19 A subsequent study used a multidisciplinary expert panel to set minimum passing standards for ACLS skill acquisition.20 The ACLS standards were then used with a fresh group of residents in a mastery learning study where all residents received the educational intervention and demonstrated competence in six ACLS scenarios. In that project the amount of time needed to reach ACLS skill mastery was allowed to vary while residents’ skill outcomes were identical clinically.21
This report presents longitudinal data collected on two occasions addressing retention of ACLS skills among the 38 internal medicine residents enrolled in the earlier randomized trial.19 The present study was passive because no educational intervention beyond clinical experience was introduced. We hypothesized that the residents’ measured ACLS skills would decay significantly (approximately 20%) over 14 months without an educational refresher as shown in several earlier studies.22–24
The study used a longitudinal, observational design without a control group25 to evaluate the retention of ACLS skills acquired by postgraduate year (PGY)-2 internal medicine residents in the context of a randomized trial reported earlier.19 The ACLS skills involve residents’ responses to six simulated scenarios: the asystole, ventricular fibrillation, supraventricular tachycardia, ventricular tachycardia, symptomatic bradycardia, and pulseless electrical activity. Primary measurements were obtained at baseline (endpoint of the randomized trial) and on two subsequent occasions six and 14 months later.
Study participants included 38 residents who were enrolled at Northwestern University’s Chicago campus internal medicine residency program. Demographic characteristics about the residents are described elsewhere.19 The residents completed the PGY-2 training year from July 2003 to June 2004; the PGY-3 year ran from July 2004 to June 2005. All subjects completed an AHA ACLS course at the beginning of training (July 2002) and again in May, 2004. The Northwestern University Feinberg School of Medicine Institutional Review Board approved the study. Participants provided informed consent separately for the randomized trial and the longitudinal retention study. There was no resident attrition in either study.
Operational details about the randomized trial addressing ACLS skill acquisition among the residents are reported in another article.19 In brief, the study showed that four two-hour simulation-based educational interventions produced large and consistent increases in residents’ ACLS clinical skills and boosted resident morale about responding to simulated hospital codes. Both the baseline and follow-up studies were conducted in Northwestern Memorial Hospital’s Patient Safety Simulation Center utilizing a life-size Human Patient Simulator developed by Medical Education Technologies, Inc., Sarasota, Florida. The mannequin realistically portrays physiologic and pharmacologic responses seen in common clinical situations using computer software. Features include palpable pulses, reactive pupils, audible heart sounds and appropriate responses of the respiratory and cardiovascular system to medications and other interventions.
Primary measurements were observational checklists, based on AHA protocols,26 of resident performance in response to simulated scenarios in asystole, ventricular fibrillation, supraventricular tachycardia, ventricular tachycardia, symptomatic bradycardia and pulseless electrical activity. The checklists were developed and tested using rigorous standards.27 The checklists yielded highly reliable data. Checklist measurements of residents’ ACLS skills for this study were obtained at the conclusion of the randomized trial in March 2004. This was the longitudinal study baseline. Subsequent measurements were obtained during September 2004 (6 months) and May 2005 (14 months) when residents were assigned to a one-hour testing session in the simulation center.
Evaluations of each resident’s adherence to ACLS protocols on a randomly selected set of three of the six simulated scenarios were recorded by one of two faculty raters on the checklists during the baseline and two follow-up testing sessions. The randomization was done to assure that there was no systematic bias in the assignment of scenarios to residents. A 50% random sample of the scenarios within each testing session was rescored by a second faculty rater from videotapes to assess interrater reliability. The rescoring was blind to the results of the first checklist recording.
Raw checklist scores ranged from 16 to 31 items for the six ACLS simulations. To achieve equal weighting of performance across all simulations, scores were computed as percent correct for each of the simulations and summed across the three simulations randomly assigned to each resident. The total scores thus ranged from 0 to 300.
Checklist score interrater reliability for the longitudinal study was estimated at baseline and at the two follow-up occasions using the Kappa (κ) coefficient28 adjusted using the formula of Brennan and Prediger.29 ANOVA was used to assess retention of ACLS skills performance from baseline across the two follow-up occasions.30 All analyses were performed using SPSS statistical software (version 14.0 for Windows).
Outcomes of the longitudinal ACLS follow-up study are presented in Table 1 and displayed graphically in Figure 1. The descriptive statistics and visual display suggest the longitudinal data are highly reliable on grounds of interrater agreement (κ) and are very stable from baseline to follow-up one (6 months) to follow-up two (14 months). This observation is verified by ANOVA which shows no statistically significant difference in residents’ ACLS performance across the three testing occasions (p > .5).
These data clearly show that ACLS skills acquired by 38 Northwestern University internal medicine residents following an educational intervention using simulation technology and deliberate practice did not decay significantly over a 14-month time span. This finding is in sharp contrast with earlier research that shows ACLS skills are not retained without sustained practice and retesting.5–9,22–24,31 At the 2001 Utstein Conference on CPR Education, recommendations were made to simplify CPR education to improve provider skill and the quality of resuscitation efforts.32 Simulation technology was advocated as a useful method to provide small-group facilitated teaching, and a call was made for research into improved resuscitation performance with a suggested follow-up of six months. Our program has been in existence for three years and has been successfully utilized in three residency classes with a fourth currently in progress (n = 159).19–21 Thus, we report an educational intervention in ACLS skills using deliberate practice and small group teaching that maintains a high level of performance over fourteen months, is feasible for internal medicine residents and is rated highly by learners.
Several factors may be responsible for our results including the very high level of ACLS skill acquisition achieved by the residents in the previous randomized trial,19 resident motivation to maintain high performance, and resident familiarity with the simulation-based testing environment that was used for all performance evaluations. Our program includes features of other successful initiatives including the use of simulated events,33 reliable assessment measures,34 and the multiple teaching and practice sessions.35 We do not believe that the single day ACLS provider course in May 2004 or additional clinical experience rather than the simulation training accounts for the high achievement at the time of residency graduation. Baseline data from graduating residents without simulator training shows much weaker results (on average, 17% poorer performance across the six simulations) despite three years of patient care experiences and successful completion of two ACLS courses.36 Future research will study possible sources of variation in resident performance data as well as the generalizability of our results to trainees in other programs.
Data for one of the six scenarios (bradycardia) showed a modest skills decay over the 14-month study period. We believe that this is most likely due to the small number of items as well as the difficulty of the case, requiring the use of multiple medications and pieces of equipment. However, the decay was mitigated in our overall results given the stability of the other five scenarios.
This study has several limitations. It was conducted in one residency program at a single academic medical center. The sample size (n = 38) was relatively small but similar in size to other studies of skill acquisition and retention in basic life support skills.37,38 A computer-enhanced simulation mannequin was used for ACLS skill acquisition, short-run outcome evaluation, and long-run skill retention assessments. These limitations do not, however, diminish the pronounced and lasting impact that the simulation-based ACLS educational program produced among the internal medicine residents.
This study demonstrated that ACLS skills acquired by internal medicine residents in a simulation-based educational program used as an adjunct to traditional ACLS provider courses are robust to deterioration over a 14-month time span. The skills were retained without further simulation training or other educational interventions. The project was implemented successfully in a complex residency schedule and features reliable assessment of ACLS competence. New ACLS guidelines published in December 200539 have streamlined several algorithms and emphasized high quality CPR skills as the basis for high quality ACLS. In an accompanying editorial, Hazini and colleagues call for the “…training of lay rescuers and health care providers in simple, high quality CPR skills that can be easily taught, remembered and implemented to save lives.”40 After our three-year experience with medical simulation in teaching and assessing ACLS skills, we believe programs such as ours can be a useful adjunct to traditional didactic courses to develop and maintain high quality resuscitation skills for prolonged periods of time.
This research was supported in part by the Excellence in Academic Medicine Act under the State of Illinois Department of Public Aid administered through Northwestern Memorial Hospital. We thank the Northwestern University internal medicine residents for their dedication to patient care and education. We acknowledge Charles Watts, MD and J. Larry Jameson, MD, PhD for their support and encouragement of this work.
2 Cummins RO, Sanders A, Mancini E, Hazinski MF. In-hospital resuscitation: a statement for healthcare professionals from the American Heart Association Emergency Cardiac Care Committee and the Advanced Cardiac Life Support, Basic Life Support, Pediatric Resuscitation, and Program Administration Subcommittees. Circulation. 1997;95:2211–2.
3 Makker R, Gray-Siracusa K, Evers M. Evaluation of advanced cardiac life support in a community teaching hospital by use of actual cardiac arrests. Heart Lung. 1995;24:116–20.
4 Kaye W. Research on ACLS training: which methods improve skill and knowledge retention? Resp Care. 1995;40:538–46.
5 Wollard M, Whitfeild R, Smith A, et al. Skill acquisition and retention in automated external defribrillator (AED) use and CPR by lay responders: a prospective study. Resuscitation. 2004;60:17–28.
6 Kaye W, Mancini ME, Rallis SF. Advanced cardiac life support refresher course using standardized objective-based mega code testing. Crit Care Med. 1987;15:55–60.
7 O’Steen DS, Kee CC, Minick MP. The retention of advanced cardiac life support knowledge among registered nurses. J Nurs Staff Devel. 1996;12:66–72.
8 Brennan RT, Braslow A. Skill mastery in public CPR classess. Am J Emerg Med. 1998;16:653–7.
9 Moser DK, Dracup K, Guzy PM, Taylor SE, Breu C. Cardiopulmonary resuscitation skills retention in family members of cardiac patients. Am J Emer Med. 1990;8:498–503.
10 Ali J, Cohen R, Adam R, et al. Attrition of cognitive and trauma management skills after the advanced trauma life support (ATLS) course. J Trauma: Injury Infec Crit Care. 1996;40:860–6.
11 Blumenfeld A, Abraham RB, Stein M, et al. Cognitive knowledge decline after advanced trauma life support courses. J Trauma: Injury Infec Crit Care. 1998;44:513–6.
12 Azcona LA, Gutierrez GEO, Fernandez CJP, et al. Attrition of advanced trauma life support (ATLS) skills among ATLS instructors and providers in Mexico. J Am Coll Surg. 2002;195:372–7.
13 Kovacs G, Bullock G, Ackroyd-Stolarz S, Cain E, Petrie D. A randomized controlled trial on the effect of educational interventions in promoting airway management skill maintenance. Ann Emer Med. 2000;36:301–9.
14 Arthur W Jr, Bennett W Jr, Stanush PL, McNelly TL. Factors that influence skill decay and retention: a quantitative review and analysis. Hum Perform. 1998;11:57–101.
15 Peberdy MA, Kaye W, Ornato JP, et al. Cardiopulmonary resuscitation of adults in the hospital: a report of 14,720 cardiac arrests from the National Registry of Cardiopulmonary Resuscitation. Resuscitation. 2003;58:297–308.
16 Wik L, Kramer-Johansen J, Myklebust H, et al. Quality of cardiopulmonary resuscitation during out-of-hospital cardiac arrest. JAMA. 2005;293:299–304.
17 Abella BS, Alvarado JP, Myklebust H, et al. Quality of cardiopulmonary resuscitation during in-hospital cardiac arrest. JAMA. 2005;293:305–10.
18 Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79:S70–S81.
19 Wayne DB, Butter J, Siddall VJ, et al. Simulation-based training of internal medicine residents in advanced cardiac life support protocols: a randomized trial. Teach Learn Med. 2005;17:210–6.
20 Wayne DB, Fudala MJ, Butter J, et al. Comparison of two standard-setting methods for advanced cardiac life support training. Acad Med. 2005;80:S63–6.
21 Wayne DB, Butter J, Siddall VJ, et al. Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. J Gen Intern Med. 2006;21:251–6.
22 Donnelly P, Assar D, Lester C. A comparison of manikin CPR performance by lay persons trained in three variations of basic life support guidelines. Resuscitation. 2000;45:195–9.
23 Chamberlain D, Smith A, Woollard M, et al. Trials of teaching methods in basic life support (3): comparison of simulated CPR performance after first training and at 6 months, with a note on the value of re-training. Resuscitation. 2002;53:179–87.
24 Anthonypillai F. Retention of advanced cardiopulmonary resuscitation knowledge by intensive care trained nurses. Intensive Crit Care Nurs. 1992;8:180–4.
25 Shadish WR, Cook TD, Campbell DT. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin, 2002.
26 Cummins RO. ACLS Provider Manual. Dallas, TX: American Heart Association, 2001.
27 McGaghie WC, Renner BR, Kowlowitz V, et al. Development and evaluation of musculoskeletal performance measures for an objective structured clinical examination. Teach Learn Med. 1994;6:59–63.
28 Fleiss JL. Statistical Methods for Rates and Proportions. 2nd Ed. New York: Wiley; 1981.
29 Brennan RL, Prediger DJ. Coefficient kappa: some uses, misuses, and alternatives. Educ Psychol Meas. 1981;41:687–99.
30 Maxwell SE, Delany HD. Designing Experiments and Analyzing Data. Mahwah, NJ: Laurence Erlbaum Associates, 2004.
31 Kaye W, Mancini ME. Retention of cardiopulmonary resuscitation skills by physicians, registered nurses, and the general public. Crit Care Med. 1986;14:620–2.
32 Chamberlain DA, Hazinski MF. Education in resuscitation: an ILCOR symposium: Utstein Abbey: Stavanger, Norway: June 22–24, 2001. Circulation. 2003;108:2575–94.
33 Palmisano JM, Akingbola OA, Moler FW, Custer JR. Simulated pediatric cardiopulmonary resuscitation: initial events and response times of a hospital arrest team. Respir Care. 1994;39:725–9.
34 Brennan RT, Braslow A, Batcheller AM, Kaye W. A reliable and valid method for evaluating cardiopulmonary resuscitation training outcomes. Resuscitation. 1996;32:85–93.
35 Smith A, Colquhoun M, Woollard M, et al. Trials of teaching methods in basic life support (4): comparison of simulated CPR performance at unannounced home testing after conventional or staged training. Resuscitation. 2004;61:41–7.
36 Wayne DB, Butter J, Fudala M, et al. Graduating internal medicine residents’ self-assessment and performance of advanced cardiac life support skills. Med Teach. 2006;28:365–69.
37 Handley JA, Handley AJ. Four-step CPR—improving skill retention. Resuscitation. 1998;36:3–8.
38 Wik L, Myklebust H, Auestad BH, Steen PA. Retention of basic life support skills 6 months after training with an automated voice advisory manikin system without instructor involvement. Resuscitation. 2002;52:273–9.
39 2005 American Heart Association Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care. Circulation. 2005;112:IV-1–IV-203.
40 Hazini MF, Nadkarni VM, Hickey RW, et al. Major changes in the 2005 AHA guidelines for CPR and ECC: reaching the tipping point for change. Circulation. 2005;112:IV-206–IV-211.
© 2006 Association of American Medical Colleges