Purpose: To determine whether medical student intubation proficiency with a neonatal mannequin differs according to weekly or consecutive day practice sessions during a six-week pediatric clerkship.
Method: From July 2010 through June 2011, the authors prospectively randomized 110 third-year medical students into three neonatal intubation practice groups: standard (control; no practice sessions), weekly (practice once/week for four consecutive weeks), or consecutive day (practice once/day for four consecutive days). At baseline, students performed intubation during individual sessions using a neonatal mannequin (SimNewB). Two reviewers, blinded to practice group, viewed videotapes of intubations and independently scored students on equipment selection, procedural skill steps, length of intubation attempts (in seconds), and the number of attempts (up to three) needed for a successful intubation. Videotaped individual final assessment intubation sessions during week six were evaluated in the same manner.
Results: Students in the weekly and consecutive day practice groups performed better at the final assessment on all variables than students in the standard group (P < .001), but over six weeks, the authors detected no differences between the two distributed practice formats for any outcomes of interest.
Conclusions: Practice improved all aspects of neonatal intubation performance, including choosing the correct equipment, properly performing the skill steps, length of time to successful intubation, and success rate, for novice health care providers in a simulation setting. Over six weeks, neither practice format proved superior, but it remains unclear whether one format is superior for learning and skill retention over the long term or in actual practice.
Dr. Ernst is associate professor and director of medical education in newborn medicine, Department of Pediatrics, Division of Neonatal–Perinatal Medicine, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma.
Dr. Cline was, at the time of this study, neonatology fellow, Department of Pediatrics, Division of Neonatal–Perinatal Medicine, University of Oklahoma Health Sciences Center, Oklahoma City, and she is currently a clinical neonatologist in private practice, CoxHealth South, Springfield, Missouri.
Dr. Dannaway is assistant professor and assistant director, Neonatal Fellowship Program, Department of Pediatrics, Division of Neonatal–Perinatal Medicine, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma.
Ms. Davis is a graduate student, Division of Biostatistics and Epidemiology, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma.
Dr. Anderson is assistant professor and statistician, Division of Biostatistics and Epidemiology, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma.
Dr. Atchley was, at the time of this study, neonatology fellow, and she is currently assistant professor, Department of Pediatrics, Division of Neonatal–Perinatal Medicine, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma.
Dr. Thompson is assistant dean for medical education and is affiliated with the Department of Pediatrics, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma.
Funding/Support: None reported.
Other disclosures: None reported.
Ethical approval: Ethical approval has been granted via exemption by the University of Oklahoma Health Sciences Center institutional review board.
Previous presentations: This work was presented as an oral presentation at the Southern Society for Pediatric Research Annual Meeting in New Orleans, Louisiana, on February 19, 2011. It was also presented, in poster format, on May 3, 2011, at the Pediatric Academic Societies Annual Meeting in Denver, Colorado, and on October 13, 2011, at the American Academy of Pediatric National Convention and Exhibition in Boston, Massachusetts.
Correspondence should be addressed to Dr. Ernst, University of Oklahoma Health Sciences Center, Division of Neonatology, 1200 Everett Dr., 7th Floor North Pavilion, Oklahoma City, OK 73104; telephone: (405) 271-5215; e-mail: Kimberly-Ernst@ouhsc.edu.
Approximately 10% of newborns require some assistance to begin breathing at birth, and about 1% require extensive resuscitative measures.1 With the changes in clinical practice over the last decade, including the increased use of noninvasive methods of respiratory support, as well as duty hours regulations, trainees have fewer opportunities during residency to learn neonatal intubation skills on actual patients and even more scarce opportunities after graduation to use those skills in clinical practice because of the explosion of hospitalist services.2–5 Because simulation provides residency trainees and experienced practitioners with a controlled, risk-free means of practicing skill sets, it is now an essential component of both contemporary training and continuing education.6–8
The Clinical Skills Education and Testing Center, a state-of-the-art simulation facility at the University of Oklahoma Health Sciences Center, provides opportunities for health care providers and trainees to gain valuable experience in clinical procedures and team-based clinical exercises. During the 2010–2011 academic year, we used the resources at the Clinical Skills Center to augment our third-year medical student pediatric clerkship. Students practiced basic resuscitation, placing an emergency umbilical line, and neonatal intubation, using the SimNewB mannequin (Laerdal Medical, AS, Stavanger, Norway) during an introductory course in neonatal resuscitation skills.
The purpose of our study was to determine whether novice medical students’ intubation proficiency on a neonatal simulator mannequin differed according to weekly or consecutive day practice sessions during a six-week pediatric clerkship.
All third-year medical students enrolled at the University of Oklahoma College of Medicine, Oklahoma City Campus, from July 2010 through June 2011, participated in the study during their six-week pediatric clerkship. The University of Oklahoma Health Sciences Center institutional review board granted an exemption because of the study’s educational nature. We orally informed students that they were participating in a randomized educational study and would be videotaped. We explained that their filmed performance of a clinical procedure would be reviewed for the study and not be used for any other purpose. Neither the decision to participate nor actual performance of the procedure affected students’ clerkship evaluation. We did not offer any incentives for participating. We excluded the data of any students who did not complete all assigned practice and/or assessment sessions. Because of students’ participation in a planned follow-up study regarding extinction of neonatal intubation skills, students were not given access to additional practice sessions until they had completed the follow-up study unless they declined to participate in that study (in which case they were allowed access to the mannequin for additional practice and received feedback on their performance after their final assessment session).
During their six-week pediatrics clerkship, groups of two to four students rotate weekly in the newborn nursery. We randomized students (in their newborn-nursery groups) to one of three practice schedules: standard (control), weekly, or consecutive day. Consecutive day randomization also included randomization of the practice week to ensure that participants were evenly distributed across the six weeks (Table 1).
Baseline assessment (Week one)
During their pediatric clerkship orientation, we asked all students to complete a questionnaire to determine previous intubation experience including prior career and medical school experiences. We showed the entire group of students a 10-minute instructional video, prepared expressly for the clerkship, on neonatal intubation that included the items we would be evaluating during the study: proper equipment selection, intubation technique, and confirmation of correct endotracheal tube placement. Additionally, each individual student read a short scenario depicting a newly born, term infant delivered with respiratory failure who had already undergone the basic steps of newborn resuscitation as well as bag-mask ventilation. After reading the scenario, each student stepped into the simulated delivery room, where the study commenced.
The proctor (D.C.D. or C.B.A.) asked the student to choose the equipment needed for intubation and postintubation support from a selection of apparatus (i.e., three different endotracheal tubes varying in size and cuff status, two laryngeal blades differing in size, a laryngoscope handle, a self-inflating bag, an oxygen tank, and various distractor items [suction tubing, a meconium aspirator]). The reviewers (K.D.E. and W.L.C.) assessed students on their ability to select the proper equipment for the scenario, assemble it correctly, and ensure proper functioning (e.g., is the laryngoscope light lit?), all of which—along with standing in the customary position to receive the infant—had been demonstrated in the instructional video. The accurate completion of these steps resulted in up to a total of 11 possible preparation score points, one for each step completed correctly.
After the participant prepared the equipment, the proctor placed the SimNewB mannequin sideways on the warmer. The reviewers (K.D.E. and W.L.C.) assessed students on their ability to perform several skill steps, including positioning the infant, holding the laryngoscope properly, inserting the endotracheal tube correctly, stabilizing the endotracheal tube manually, ventilating the bag, and placing the endotracheal tube into the trachea. We considered performing all six of these steps correctly to be a “successful” intubation attempt. We counted each unique insertion of the laryngoscope blade into the mouth as one intubation attempt. The proctor confirmed each attempt as either successful (chest rise with bag ventilation) or unsuccessful. Because of testing time constraints and “real-world” practice, we gave students a maximum of three attempts before we considered their overall effort unsuccessful and informed them that a senior resident had arrived to “rescue” them. The accurate completion of each intubation step resulted in up to a total of eight possible performance score points.
We measured the time of each individual attempt from the moment when the tip of the laryngoscope blade entered the mouth to the moment the student removed the tip from the mouth. We calculated a total intubation time (in seconds) as the sum of each attempt time for up to three attempts. For students who were successful, we counted this sum as the time to successful intubation. For each student who was unsuccessful, we calculated a total imputed intubation time as the individual student’s actual total intubation time plus the average time computed for all the students who were successful on the first attempt. Our assumption in devising this conservative time penalty was that the unsuccessful candidate would have been successful with one additional attempt. This formula allowed us to evaluate all the students, which we deemed necessary because so few students were successful at baseline (week one).
We videotaped all assessment sessions. Two investigators (K.D.E. and W.L.C.), blinded to practice group assignments, reviewed the videos separately and independently scored each student. Each student’s final score was the average of the two reviewers’ preparation scores, the highest performance score of their (up to) three attempts, and total intubation time.
After the baseline assessment, students in the standard group had no additional practice sessions during the six-week clerkship. Weekly students practiced once a week for four consecutive weeks in weeks two, three, four, and five. Consecutive day students practiced four consecutive days during a single week (see Table 1). Individual, required “self-study” practice sessions lasted up to 10 minutes. No formalized instruction or feedback occurred during these sessions.
Final assessment (Week six)
At the final assessment, we asked students to complete another questionnaire to obtain information regarding any intubations they had observed or performed over the six-week study period. Each student then completed his or her final intubation assessment which was videotaped and scored as previously described for the baseline assessment.
We used SAS version 9.2 software (SAS Institute Cary, North Carolina) to perform our statistical analyses, and we defined significance as P < .05. We evaluated, for each practice group, the preparation score, performance score, time to successful intubation, and total imputed intubation time for normality, and we reported these using medians (along with 25th and 75th percentiles). We assessed differences among the practice groups using a Kruskal–Wallis test, and we compared baseline and final assessments within each practice group using a Wilcoxon signed rank test. To further compare and uncover any possible significant differences in the weekly and consecutive day groups, we used the Bonferroni-corrected Wilcoxon signed rank test. We compared the intergroup percentage of students successful for each number of intubation attempts using a chi-square test or Fisher exact test. Finally, we calculated intraclass correlation with 95% confidence interval (CI) to determine the level of agreement between the two independent reviewers.
We randomized 121 third-year medical students into three groups, but eventually excluded 11 (3 from the standard group and 4 from both the weekly and consecutive day groups) because they did not complete all of the practice and/or evaluation sessions. Of the remaining 110 students, 41 were in the standard group, 32 in the weekly group, and 37 in the consecutive day group. At baseline, the three groups had similar prior career and educational experiences; only 1 student (in the standard group) had previously attempted an infant intubation (Table 2).
We found the two reviewers’ scores to be highly correlated, with an intraclass correlation of 0.85 (95% CI: 0.79–0.89) for preparation scores, 0.86 (95% CI: 0.80–0.90) for performance scores, and 0.999 (95% CI: 0.9988–0.9999) for total intubation time.
Preparation scores, performance scores, time to successful intubation, and total imputed intubation times (Table 3) were similar across all three groups at baseline: 24 of the standard students (58%), 15 of the weekly students (47%), and 20 of the consecutive day students (54%) were successful within three attempts (P = .33). At the final assessment, all three groups had improved, but the standard group had improved to a 66% success rate (27 students), whereas both the weekly and consecutive day groups had both improved to an overall success rate of 97% (31 and 36 students, respectively) within three attempts (P < .001 among all three groups; P ≈ 1.00 between the weekly and consecutive day groups).
We observed the largest improvement in first-attempt success improvements from baseline to the final assessment: from 3 students to 11 (20% increase) in the standard group, from 6 students to 26 (62% increase) in the weekly group, and from 4 students to 29 (67% increase) in the consecutive day group (P < .001 for all groups). Furthermore, first-attempt intubation times also improved between the baseline and final assessments for students in the two practice groups. On average, weekly students’ times decreased 27 seconds from 42.5 to 15.5 seconds, and consecutive day students’ times decreased 11.3 seconds from 31.3 to 20.0 seconds, while standard students’ times actually increased by 6.5 seconds from 23.5 to 30.0 seconds (P < .001).
At the final assessment, we detected a significant difference for all of the outcomes, and both groups who experienced practice sessions received higher preparation and performances scores plus lower time to successful intubation and total imputed intubation times when compared with the standard group. Although the two practice groups performed significantly better than the standard group, we observed no difference between the two practice formats for any of the outcome variables.
Research has shown that experience with live patients improves intubation proficiency. Further, experience with live patients is considered the gold standard for improving procedural skills; however, practice on high-fidelity medical simulators has also been shown to improve learning outcomes in proportion to the amount of practice.9,10 Our results not only align with previous studies that suggest that repetitive practice can enhance the effects of training but also provide data describing the proficiency of third-year medical students in performing neonatal intubation using the Laerdal SimNewB mannequin. Whereas both practice groups reduced their time-to-successful intubation by 75% and increased their first-attempt success by 70%, we detected no clear difference between the two distributed practice formats. Although some studies have also suggested that longer teaching sessions improve intubation skills, and others have compared guided versus experiential learning to increase the effects of training,2,11–13 our study demonstrates that short, self-taught practice can also improve performance, at least in the short term.
The results of a 2010 study by Bismilla and colleagues14 showed that both pediatric and neonatal trainees failed to meet the Neonatal Resuscitation Program standard for neonatal intubation compared with respiratory therapists. Advanced neonatal fellow trainees were 69% successful in placing the nasotracheal tube and took about 56 seconds versus respiratory therapists, who were 100% successful and took about 25 seconds.14 In their 2012 study, Downes and colleagues15 found that pediatric residents had successful neonatal intubations only 21% of the time, while respiratory therapists were successful 50% of the time. The anesthesia literature suggests that proficiency in adult intubation requires at least 45 successful attempts,16–18 whereas a more recent observation of one pediatric trainee over five years showed that 100 neonatal intubation experiences were necessary to attain 90% proficiency.3 Our findings, which show that repeated exposure to neonatal intubation increases the number of successful intubations and decreases time to intubation, align with these previous observations and findings. Students in both practice groups, but not the standard group, significantly decreased their median time to successful neonatal intubation: The weekly group decreased their time to 20.5 seconds, and the consecutive day group decreased their time to 22.5 seconds. Both of these average times fall within the Neonatal Resuscitation Program’s recommended time-to-intubation of 30 seconds or less. Furthermore, the first-attempt success rate improved to 97% for the students in both practice groups versus 66% for the students in our control group, who had no extra practice.
Data are lacking on the transfer of simulation-learned skills from the simulator to real patients, especially in the neonatal setting. A 2013 randomized, prospective study by Kessler and colleagues19 confirmed that a single simulation-based session was insufficient to affect pediatric intern success in the real world for lumbar punctures and intravenous line placement. As Finan and colleagues20 suggested, using a newborn mannequin with less-than-optimal anatomical, visual, and tactile cues has its obvious disadvantages when that experience needs to be translated into real-world situations. Obviously, it is difficult to replicate living breathing human infants and all of the visual cues that an experienced resuscitator uses to evaluate a baby. Simulator mannequins must have skin that is durable, unlike that of a very preterm infant, which affects jaw-opening and laryngoscope-maneuvering techniques. Furthermore, an appropriately sized, 24-week-gestational-age (500 grams) infant mannequin has obvious technical limitations—both electronic and physical (e.g., such as the inability to include the mechanical parts necessary for both breath and heart sounds in an appropriately small mannequin)—that decreases both its realism and its functionality. Although we recognize the limitations of simulators, we still feel it is worthwhile to provide trainees with the opportunity to learn the mechanics of new skills free of the many other factors (e.g., environmental stressors and difficult airway anatomies) that affect performance.
Although our students have enjoyed a history of high academic achievement and received just-in-time video training, they had difficulty assimilating the information into their actual performance. Many demonstrated a significant lack of problem-solving skills, something extremely important to clinicians in the medical field. Several students, despite being told they could move the mannequin, proceeded to attempt intubation from the side or foot of the mannequin (we had placed the equipment table purposely to block more direct access). Other students stated that “the equipment is broken” when, in fact, we had purposely loosened the light bulb to test whether the students checked to be sure bulb was tightened, one of the required preparation steps. Furthermore, the protocol stated that if a student did not keep the endotracheal tube secured after placement, then it would fall out (i.e., the proctor would remove it). Several students repeated the same mistake—not securing the tube while turning their back to grab the ventilation bag—at each intubation attempt, despite being told each time that the tube had fallen out. As we witnessed these events through the study’s progression, we wished we had developed an analytical “finesse” score both to help determine which participants had the most difficulty with problem solving and to examine how finesse may have affected not only the students’ overall scores for this study but also their future clinical procedural success. We believe that some students would probably have better success in certain fields that are less technical in nature.
We suspect that our inability to find a significant difference between the two practice formats at the final assessment may be due to six weeks not being a prolonged enough period of time to differentiate retention of the skill. As our study has shown the benefit of neonatal intubation practice, our next challenge will be assessing how quickly intubation skill performance declines after initial training, and whether one of the two practice formats affects retention of the skill over the longer term. In 2008, Duran and colleagues21 found that neonatal intubation skill deteriorated significantly at one year after assessment. However, the participants in the 2008 study were pediatric residents who were not naïve to neonatal intubation and had other practice opportunities before their follow-up evaluation. We feel that a follow-up study of our naïve participants during their fourth year of medical school would be beneficial in determining the timing of neonatal intubation skill deterioration and may elucidate whether weekly or consecutive day practice affects this deterioration timing. This information may be helpful in suggesting the optimal timing for neonatal intubation refresher courses.
Practice improves all aspects of intubation performance, including choosing the correct equipment, properly performing the skill steps, length-of-time to successful intubation, success rate, and number of attempts needed for a successful intubation, for novice health care providers using a neonatal mannequin in a simulation setting. Although neither weekly nor consecutive day practice proved to be superior to the other in a six-week period, it is possible that one distributed practice format may be optimal for skill retention over a more prolonged period of time or when the skill set is transferred to the clinical setting. Future studies should include a longer period of evaluation both to determine the optimal practice format as well as to discover the rate of skill deterioration, which, in turn, may inform the appropriate timing of refresher courses.
Acknowledgments: The authors wish to thank Nick Bowen for his technical assistance in video recording the sessions and the Clinical Skills Education and Testing Center, which provided the physical space and other resources for the project.
1. Perlman JM, Risser R. Cardiopulmonary resuscitation in the delivery room. Associated clinical events. Arch Pediatr Adolesc Med. 1995;149:20–25
2. O’Donnell CPF, Kamlin COF, Davis PG, Morley CJ. Endotracheal intubation attempts during neonatal resuscitation: Success rates, duration, and adverse effects. Pediatrics. 2006;117:e16–e21
3. Doglioni N, Cavallin F, Zanardo V, Trevisanuto D. Intubation training in neonatal patients: A review of one trainee’s first 150 procedures. J Matern Fetal Neonatal Med. 2012;25:1302–1304
4. Falck AJ, Escobedo MB, Baillargeon JG, Villard LG, Gunkel JH. Proficiency of pediatric residents in performing neonatal endotracheal intubation. Pediatrics. 2003;112(6 pt 1):1242–1247
5. Mason WT, Strike PW. See one, do one, teach one—is this still how it works? A comparison of the medical and nursing professions in the teaching of practical procedures. Med Teach. 2003;25:664–666
6. Brim NM, Venkatan SK, Gordon JA, Alexander EK. Long-term educational impact of a simulator curriculum on medical student education in an internal medicine clerkship. Simul Healthc. 2010;5:75–81
7. Dayal AK, Fisher N, Magrane D, Goffman D, Bernstein PS, Katz NT. Simulation training improves medical students’ learning experiences when performing real vaginal deliveries. Simul Healthc. 2009;4:155–159
8. Okuda Y, Bryson EO, DeMaria S Jr, et al. The utility of simulation in medical education: What is the evidence? Mt Sinai J Med. 2009;76:330–343
9. Leone TA, Rich W, Finer NN. Neonatal intubation: Success of pediatric trainees. J Pediatr. 2005;146:638–641
10. McGaghie WC, Issenberg SB, Petrusa ER, Scalese RJ. Effect of practice on standardised learning outcomes in simulation-based medical education. Med Educ. 2006;40:792–797
11. Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review. Med Teach. 2005;27:10–28
12. Owen H, Plummer JL. Improving learning of a clinical skill: The first year’s experience of teaching endotracheal intubation in a clinical simulation facility. Med Educ. 2002;36:635–642
13. Ti LK, Chen FG, Tan GM, et al. Experiential learning improves the learning and retention of endotracheal intubation. Med Educ. 2009;43:654–660
14. Bismilla Z, Finan E, McNamara PJ, LeBlanc V, Jefferies A, Whyte H. Failure of pediatric and neonatal trainees to meet Canadian Neonatal Resuscitation Program standards for neonatal intubation. J Perinatol. 2010;30:182–187
15. Downes KJ, Narendran V, Meinzen-Derr J, McClanahan S, Akinbi HT. The lost art of intubation: Assessing opportunities for residents to perform neonatal intubation. J Perinatol. 2012;32:927–932
16. deOlivera Filbo GR. The construction of learning curves for basic skills in anesthetic procedures: An application for the cumulative sum method. Anesth Analg. 2002;95:411–416
17. Mulcaster JT, Mills J, Hung OR, et al. Laryngoscopic intubation: Learning and performance. Anesthesiology. 2003;98:23–27
18. Konrad C, Schüpfer G, Wietlisbach M, Gerber H. Learning manual skills in anesthesiology: Is there a recommended number of cases for anesthetic procedures? Anesth Analg. 1998;86:635–639
19. Kessler DO, Arteaga G, Ching K, et al. Interns’ success with clinical procedures in infants after simulation training. Pediatrics. 2013;131:e811–e820
20. Finan E, Bismilla Z, Campbell C, Leblanc V, Jefferies A, Whyte HE. Improved procedural performance following a simulation training session may not be transferable to the clinical environment. J Perinatol. 2012;32:539–544
21. Duran R, Aladağ N, Vatansever U, Küçükuğurluoğlu Y, Süt N, Acunaş B. Proficiency and knowledge gained and retained by pediatric residents after neonatal resuscitation course. Pediatr Int. 2008;50:644–647