The Effectiveness of Video-Assisted Debriefing Versus Oral Debriefing Alone at Improving Neonatal Resuscitation Performance

A Randomized Trial

Abstract

Introduction 

Debriefing is a critical component of effective simulation-based medical education. The optimal format in which to conduct debriefing is unknown. The use of video review has been promoted as a means of enhancing debriefing, and video-assisted debriefing is widely used in simulation training. Few empirical studies have evaluated the impact of video-assisted debriefing, and the results of those studies have been mixed. The objective of this study was to compare the effectiveness of video-assisted debriefing to oral debriefing alone at improving performance in neonatal resuscitation.

Methods 

Thirty residents, divided into 15 teams of 2 members each, participated in the study. Each team completed a series of 3 neonatal resuscitation simulations. Each simulation was followed by a facilitated debriefing. Teams were randomly assigned to receive either oral debriefing alone or video-assisted debriefing after each simulation. Objective measures of performance and times to complete critical tasks in resuscitation were evaluated by blinded video review on the first (pretest) and the third (posttest) simulations using a previously validated tool.

Results 

Overall neonatal resuscitation performance scores improved in both groups [mean (SD), 83% (14%) for oral pretest vs. 91% (7%) for oral posttest (P = 0.005); 81% (16%) for video pretest vs. 93% (10%) for video posttest (P < 0.001)]. There was no difference in performance scores between the 2 groups on either the pretest or posttest [overall posttest scores, 91.3% for oral vs. 93.4% for video (P = 0.59)]. Times to complete the critical tasks of resuscitation also did not differ significantly between the 2 study groups. The educational effect of the video-assisted debriefing versus oral debriefing alone was small (d = 0.08).

Conclusions 

Using this study design, we failed to show a significant educational benefit of video-assisted debriefing. Although our results suggest that the use of video-assisted debriefing may not offer significant advantage over oral debriefing alone, exactly why this is the case remains obscure. Further research is needed to define the optimal role of video review during simulation debriefing in neonatal resuscitation.

With the release of the sixth edition of the Neonatal Resuscitation Program (NRP) course, simulation-based training has become the standard method of teaching neonatal resuscitation.¹ The classroom portion of the sixth edition of the NRP course focuses almost exclusively on teaching hands-on skills through simulated scenarios and debriefing.² Early research on simulation-based training in neonatal resuscitation showed it to be feasible and well received by participants.³ There are limited data, however, on the effectiveness of simulation-based training at improving neonatal resuscitation performance, both in simulated and real clinical scenarios.⁴ Prior studies reporting outcomes after a single simulation-based training session have failed to demonstrate significant improvements in neonatal resuscitation performance.^5–7 A recent report on the use of deliberate practice with simulation demonstrated significant improvements in overall neonatal resuscitation performance, improvements in positive-pressure ventilation skills, and a decrease in time to emergent umbilical access and administration of intravenous (IV) medications.⁸

Facilitated debriefing is a critical component of effective learning in simulation-based education.^9–11 However, the optimal method by which to conduct facilitated debriefing remains to be defined.^11–13 Video review has been promoted as a modality to improve the quality of facilitated debriefing, and the new NRP Instructor DVD provides detailed instructions and examples on how to conduct a facilitated debriefing using video.¹⁴ A recent review on debriefing as part of the learning process in simulation-based education found that additional research is needed to investigate learning outcomes based on video playback versus no video playback during debriefing.¹¹ Prior experimental outcomes comparing the benefits of video-assisted debriefing to oral debriefing alone have been mixed. Studies of simulation-based training in anesthesia have failed to show the benefits of video-assisted debriefing when compared with oral debriefing, for both technical performance and nontechnical skills.^15,16 Chronister and Brown¹⁷ found faster response times to initiate cardiopulmonary resuscitation and time to defibrillation in a group of nursing students provided with video-assisted debriefing as compared with a group that received oral debriefing only, but the overall performance did not differ between the 2 groups. To the authors’ knowledge, there have been no previous studies comparing the effectiveness of oral debriefing versus video-assisted debriefing at improving neonatal resuscitation performance. To evaluate the educational benefit of video-assisted debriefing to oral debriefing in simulation-based neonatal resuscitation training, we performed the following study. Our hypothesis was that video-assisted debriefing would be more effective at improving neonatal resuscitation performance when compared with traditional oral debriefing alone.

MATERIALS AND METHODS

Design

The study followed a prospective randomized design. The Sim-PICO framework (P = population, I = intervention, C = comparator, O = outcome)¹¹ of the study is provided in Figure 1. The residents were randomized to receive oral debriefing or video-assisted debriefing while completing a series of 3 neonatal resuscitation simulations. Three simulation sessions were chosen to establish a baseline level of performance, provide an opportunity for practice, and evaluate the effect of the practice. Measurements of neonatal resuscitation performance and times to complete critical tasks in resuscitation were compared on the first (pretest), the second, and the third (posttest) simulation sessions.

FIGURE 1: Sim-PICO (P = population, I = intervention, C = comparator, O = outcome) format of the study.¹¹ The vertical columns in the matrix represent the generic dimensions of debriefing. In this study, the intervention was video-assisted debriefing, and the comparator was oral debriefing alone. The aim of the study was to answer a question related to the “what” of debriefing. The outcome was the difference in posttest scores on the NRPE and times to complete critical tasks in resuscitation. Note that the outcome crosses multiple dimensions (columns) because the other factors can influence the outcome. However, as noted in the matrix, the what column is what was altered and tested in this study.

Participants

All 38 pediatric and family medicine residents at Tripler Army Medical Center were invited to be the subjects in the study. Tripler Army Medical Center is a 450-bed tertiary care center in Honolulu, with approximately 3000 deliveries and 300 neonatal intensive care unit admissions per year. The participating subjects were randomly paired into teams of 2 members each. Each subject was teamed with another subject from the same year group and within the same training program if possible. Each team completed a series of 3 standardized neonatal resuscitation simulations. Each simulation was followed by a facilitated debriefing. The teams were randomly assigned to receive either oral debriefing alone or video-assisted debriefing after each simulation session. The team members and debriefing method remained the same throughout all 3 simulations, and all scoring was based on team, not individual, performance. Informed consent was obtained from all subjects. The study protocol was approved by the institutional review board at Tripler Army Medical Center. Investigators adhered to the policies for protection of human subjects as prescribed in Title 45 Code of Federal Regulations, Part 46.

Simulation Environment

All simulations were performed in a realistic simulated delivery room. The subjects received assistance during each simulation from one of the researchers (D.C.) who acted as a labor and delivery nurse. During the simulations, the “nurse” was able to perform some steps of NRP, with explicit direction from the residents, including checking heart rate, providing chest compressions, and drawing up medications. The nurse did not provide direction in NRP, make treatment decisions, or assist with medication dosages.

Simulation Sessions

The simulator used was a SimBaby (Laerdal Medical, Wappingers Falls, NY), which had been specially modified to include an integrated umbilical catheterization task trainer.¹⁸ Before each session, the subjects received a brief introduction to the Laerdal SimBaby, including the simulator’s visual (perioral cyanosis and chest wall movement), auditory (breath and heart sounds), and tactile (brachial and femoral pulses) cues along with a brief review of the functionality of the umbilical catheterization task trainer.

Simulations sessions were scheduled on separate days with a goal of no longer than 2 months between simulation sessions to avoid natural decline in skills over time. Simulation sessions were spaced apart in time, rather than repeated on the same day, to take advantage of the “spacing effect,” a well-described phenomenon in educational psychology and medical simulation that, for a given amount of study time, spaced study yields better learning than does massed.^19,20

The initial condition of the infant in all simulations was limp, apneic, cyanotic, and bradycardic (heart rate <60 beats per minute). All simulations required intubation and IV epinephrine. The second and the third simulations also required a normal saline bolus. Pulse oximetry and cardiorespiratory monitors were not used during the simulations because these were not standard practice at all neonatal resuscitations at our institution at the time of the study and the subjects were asked to rely solely on physiologic cues (auscultation of breath sounds, palpation of pulse, etc) from the simulator.

At the start of each simulation, the team was called into the simulated delivery room and given a brief history leading up to the delivery. The teams were then allowed 60 seconds to prepare for the resuscitation and ready their equipment. All simulations were timed, and a maximum time of 10 minutes was allowed for each simulation session, regardless of how far in the resuscitation the team had progressed. Simulation sessions could be stopped earlier than 10 minutes if all appropriate steps in the resuscitation had been completed. At the time of the study, the subjects did not receive simulation training in neonatal resuscitation in any venue other than this research study. No subjects reported prior experience using the Laerdal SimBaby.

Debriefing

Every simulation was facilitated by a single investigator (T.S.) who also led the debriefing session immediately after the simulation. Debriefings followed the “debriefing with good judgment” model of Rudolph et al.²¹ Each debriefing consisted of 3 phases: reaction, analysis, and summary.²² In the reaction phase, the facilitator inquired how the subjects felt the simulation went and allowed them to “blow off steam.” During the analysis phase of the debriefing, the facilitator provided a “low” level of facilitation, as defined by Dismukes and Smith,²³ wherein he guided the debriefing through the steps of the resuscitation in chronologic order to allow for a discussion of all relevant learning points that arose during each step in the resuscitation. During the analysis discussion, the facilitator attempted to investigate the basis for performance gaps using advocacy statements paired with inquiry statements as described by Rudolph et al.²¹ The debriefing sessions were intended to provide directed and specific formative assessment to the participants, with an express intent of improving performance in subsequent sessions.²² In the summary phase of the debriefing, the lessons learned during the analysis phase where distilled and specific measures could be changed to improved performance in the future were highlighted.

The subjects in the oral debriefing group relied solely on mental recall of the events during the analysis phase of the debriefing session. The subjects in the video-assisted debriefing group were allowed to watch the entire video of the resuscitation during the analysis phase of the debriefing session, with the facilitator stopping the video to allow discussion of relevant learning points. When needed, the video was rewound and reviewed again to highlight specific performance gaps. As needed, the subjects in both groups received hands-on instruction in technical skills, including airway management, umbilical catheterization, and medication administration, during the debriefing sessions.

To help standardize the debriefings, a debriefing form (Appendix A) was used during the debriefings of both the oral and video-assisted groups. Notes taken on the form during the simulation session were used during the debriefing session to highlight areas of good performance and areas for improvement. To further standardize the debriefings, all debriefing sessions in both the oral and video-assisted groups were timed and limited to 20 minutes in length.

The Neonatal Resuscitation Performance Evaluation Tool

Neonatal resuscitation performance was scored using a previously validated scoring instrument, known as the Neonatal Resuscitation Performance Evaluation (NRPE) tool.⁸ This instrument was chosen because of its ability to give mean performance scores in the various subdomains of neonatal resuscitation, a feature not found in other published neonatal resuscitation performance scoring instruments.^5,24 The NRPE tool was developed by modification of a tool previously used to evaluate performance in actual recorded neonatal resuscitations.²⁵ The validity of the data derived from the NRPE tool was determined using several lines of evidence as described by Downing,²⁶ which included content, consequence, response process, relationship to other variables, and internal structure validity. The NRPE tool shows strong evidence of content and consequence validity.⁸ The evidence of response process validity, for example, protecting data integrity by controlling for potential sources of error associated with test administration, was provided in our study by standardizing the simulation environment and scenarios, maintaining the same team members throughout the study, and providing good quality videos to a blinded reviewer. The relationship of the NRPE tool to other variables was evident in that the modified NRPE tool measured 12 (80%) of 15 specific performance metrics noted on the NRP “megacode” performance checklist included in the 2006 Neonatal Resuscitation Instructor Manual.²⁴ Internal structure validity of the data derived from the NRPE tool was evident in that the tool is able to reliably differentiate teams with more experience in neonatal resuscitation from those with less experience (P = 0.003).⁸ Prior reliability testing of the NRPE tool showed a good interrater reliability for the various subdomains of resuscitation, with a mean Cohen κ of 0.63.⁸

Evaluation of Resuscitation Performance

Evaluation of NRP performance on the first (pretest), the second, and the third (posttest) simulations was conducted by way of video review by a blinded reviewer (A.S.-C.) using the NRPE tool. The reviewer was blinded to the subject year group, simulation session number, and study group assignment. Neonatal Resuscitation Program performance was measured in 7 subdomains including preparation and initial steps, communication of heart rate, positive-pressure ventilation, chest compressions, intubation, medication administration, and umbilical vessel catheterization. Each subdomain included up to 6 task-specific performance metrics. Each metric was given a score of 1 if completed correctly or 0 if not completed or incorrectly performed. The scores for each domain were determined by calculating the number of points achieved divided by the total points possible in that subdomain that yielded 0% to 100%. An overall NRP performance score was calculated by dividing the total accrued points in all subdomains by the total points possible for the resuscitation scenario. Times to achieve critical elements of resuscitation were also determined by video review.

Statistical Analysis

An a priori sample size estimate, based on an estimated initial overall performance of 70% with an SD of 15%, gave an n of 8 teams in each study group to show an educationally meaningful difference in performance between the 2 study groups with a power of 0.8 and an α of 0.05. This sample size estimate was supported by literature in the field of education, indicating that an effect size greater than 1 SD is large and acceptable for a given teaching intervention.²⁷ Changes in performance scores from the first to the third simulation within each group were analyzed using the Friedman repeated-measures analysis of variance on ranks. Comparisons of NRP performance scores between the 2 groups on the pretest and posttest were conducted using the Mann-Whitney rank sum test. Changes in times within each group were evaluated using 1-way repeated-measures analysis of variance. Comparisons of demographic data, and times to complete critical tasks, between the pretest and the posttest of oral debriefing and video-assisted debriefing groups were conducted using an unpaired 2-sided t test or Mann-Whitney rank sum test, as appropriate. P < 0.05 was considered statistically significant. The educational effect size of the video-assisted debriefing intervention was calculated based on posttest overall performance means and SDs using Cohen d. By convention, an effect size of 0.2 or lower was considered to be “small”; higher than 0.5, “medium”; and ≥ 0.8 or higher, “large.”²⁷ Data were analyzed using SigmaPlot 11.0 (Systat Software, Inc).

RESULTS

Thirty subjects, divided into 15 teams of 2 members each, participated in the study. Seven groups received oral debriefing. Eight groups received video-assisted debriefing. The 2 study groups did not differ significantly from one another at baseline. Table 1 provides demographic data of the study subjects.

Table 1: Demographic Data of Study Participants

The teams completed the series of 3 simulations over a period of 9 months (September 2007 to June 2008). The time between simulation sessions 1 and 2 averaged 123 (SD, 65) days for oral debriefing and 80 (SD, 50) days for the video-assisted group (P = 0.18). The time between simulation sessions 2 and 3 averaged 64 (SD, 38) days for oral debriefing and 68 (SD, 21) days for the video-assisted group (P = 0.81).

Table 2 shows NRP performance scores on the pretest and posttest. Overall neonatal resuscitation performance scores improved in both groups [oral pretest, 83% (SD, 14%) vs. oral posttest, 91% (SD, 6%) (P = 0.009); video pretest, 81% (SD, 16%) vs. video posttest, 93% (SD, 10%) (P < 0.001)]. The video-assisted debriefing group had a larger improvement in overall neonatal resuscitation scores when compared with the oral debriefing group (video, 12% vs. oral, 8%), but the difference was not statistically significant (P = 0.59). There were no differences in performance scores for any of the 7 subdomains of neonatal resuscitation, or for overall performance, between the 2 groups on the pretest or posttest (P ≥ 0.2). The calculated educational effect size of video review during debriefing was small (d = 0.08).

Table 2: Comparison of Neonatal Resuscitation Performance Scores

Table 3 shows the times to complete critical tasks of neonatal resuscitation on the pretest and posttest. The average times to provide the first positive-pressure breath, achieve successful intubation, achieve vascular access, administer the first IV medication, and give the first IV epinephrine dose decreased on the posttest in both groups. The only statistically significant changes were in time to achieve vascular access and administer the first IV medication in the oral group (IV access, P = 0.039; first IV medication, P = 0.03). There was no significant difference in the time to complete any critical tasks between the 2 study groups on either the pretest or posttest.

Table 3: Comparison of Times to Complete Critical Steps in Neonatal Resuscitation

DISCUSSION

The purpose of this study was to investigate the additional beneficial effects of video-assisted debriefing, as compared with traditional oral debriefing alone, at improving neonatal resuscitation performance. Simulations were conducted using a sophisticated infant simulator in a realistic delivery room environment providing a “high-fidelity” experience to the learners. The debriefing methods used were based on previously published and widely taught methods. A debriefing form was used to guide the debriefings. The video review was provided as an adjunct to the oral debriefing and included rewinding and review of pertinent learning points on an individualized basis, tailored to the team’s performance. Using this design, we were unable to show a significant benefit of video-assisted debriefing over oral debriefing alone at improving neonatal resuscitation performance or time to complete critical tasks of resuscitation, and the added educational benefit of video review during debriefing was small. To the authors’ knowledge, this is the first study to compare oral debriefing with video-assisted debriefing in neonatal resuscitation simulation-based training.

Despite its widespread use, there is limited empiric evidence to support video review during the facilitated debriefing in simulation-based medical education.^11,13 A review of the literature revealed only 3 previously published investigational studies examining the benefits of video-assisted debriefing. In a study of anesthesia residents’ crisis management skills, Savoldelli et al¹⁵ randomized residents to receive no debriefing, oral debriefing alone, or video-assisted debriefing after participating in a series of 2 intraoperative cardiac arrest simulations. On the second simulation, crisis management skills improved significantly in both the oral debriefing and video-assisted debriefing groups, not in the group that received no debriefing. There was no difference in score improvement between the oral and video-assisted debriefing groups. Byrne et al¹⁶ compared the performance of 2 groups of anesthetists managing a simulated anesthetic crisis. One group received a short verbal explanation of the clinical problem, and the other group received a verbal explanation in addition to video review. After 5 simulation sessions, there was no difference between the 2 groups in either chart errors or time to solve the crisis. Most recently, Chronister and Brown¹⁷ performed a comparative, crossover study of nursing student’s performance and response times during a cardiopulmonary arrest simulation after receiving either oral debriefing or video-assisted debriefing. Results showed faster response times to initiate cardiopulmonary resuscitation and time to shock in the video-assisted debriefing group; however, there was no difference in performance between the 2 groups. We believe that the current study adds to the existing literature on the use of video review during simulation-based medical education.

The use of video review during feedback and debriefing has been examined in several areas outside medicine including sports, education, music, and the military. In sports, video-assisted feedback has been shown to be superior to self-guided and oral feedback alone at improving psychomotor performance.²⁸ Video feedback has also been rated more favorably than descriptive feedback alone by athletes.²⁹ In education, video feedback has been shown to have a positive effect on teacher interaction skills in a range of contact professions.³⁰ The aggregate effect size of video review in education has been reported as small to medium (0.4).³¹ In music, real-time video feedback of vocal performance has been associated with positive impacts on teaching behaviors and student experiences.³² In the military, the use of high-technology, multivariant video review has been used for decades as part of the army’s after-action review process.³³

Given its generally accepted utility and positive impact in other sectors, it is unclear why video-assisted debriefing has not proven itself to be more beneficial in simulation-based medical education. As described by Fanning and Gaba,¹³ the benefit of video review during medical simulation debriefing is highly dependent on the skills and techniques of the facilitator, rather than the video footage itself. This makes an analysis of the debriefing methods used in comparative investigations critically important. During the debriefings in this study, we used a debriefing form as a guide during the debriefing discussion. This type of “scripted debriefing” may have influenced our negative results. Further research is needed to examine the impact of scripting during debriefing in medical simulation.^11,34 In the current study, the video was watched in its entirety and used as a backdrop to the oral discussion with rewinding and review of certain segments to highlight areas of good or suboptimal performance on an individualized basis. This was done with the intent to provide maximal expose of the study intervention (video review) to the subjects. Perhaps watching only snippets of the video would be more educationally beneficial. However, in the study by Savoldelli et al,¹⁵ only selected video segments, chosen to illustrate the instructor’s constructive criticisms, were reviewed with similar negative results and a trend toward greater improvement in the oral debriefing group. The authors of that study postulated that the video review may have interfered with the facilitator’s feedback by changing the content of the feedback, may have resulted in “information overload” for the learners, or may have been a distraction during the debriefing session, causing learners to pay less attention to the instructor’s comments and criticisms.¹⁵ Timing is another element of the debriefing, which requires further study.¹¹ In this study, debriefing followed immediately after the simulation. As such, the events of the simulation were likely still fresh in the mind of the participants. This may allow oral debriefing alone to be sufficient to explore the events. We do not know if the video review would have had more impact if the debriefings were held some time later or if including the option of making the videos available to students for review later, on their own time, would have proven more educationally beneficial. In addition, providing more time during the debriefing sessions to review the videos (eg, 30 minutes instead of 20 minutes) may have altered our results.

Despite the negative results of the current study, we do not feel that our findings imply that video review during simulation debriefing has no implicit benefits. On the contrary, we believe that video review has many important uses during debriefing sessions, and we commonly use the technique during simulation-based training in neonatal resuscitation at our institution. A specific example of the utility of video review is the ability to clear up controversies of what actions occurred, or did not occur, during the simulation session by viewing the tape. This ability to definitively clear up controversies between the learner and the facilitator can help reduce hindsight bias and may prove invaluable at times during facilitated debriefings. Another advantage of video review is the ability to allow participants to view for themselves personal quirks and mannerisms (eg, laughing under stress) that they themselves may not be aware of. As anyone who has experienced it can tell you, it is one thing to be told you acted in a certain way during a simulation session and quite another thing to see the action on video. Although clearly beneficial to learners, the effects of these specific benefits of video review may not be directly measurable in the current study design.

Limitations

Our study has several limitations. Our experimental design did not include a control group that received no debriefing. The decision to omit a control group was based on evidence from prior studies and published commentaries indicating that omitting debriefing from simulation-based training significantly limits the educational benefits to the participants.^13,15,22 A second limitation is the higher-than-expected baseline performance. In our institution, all pediatric residents are required to become NRP instructors before graduation, which explains the high number of NRP instructors in our study. This may limit the generalizability of our results. Another potential limitation is the relatively low subject number. The sample size for the study was chosen to provide the ability to measure a large educational effect size of the intervention, defined as a difference between the 2 groups greater than 1 SD from the pretest mean.²⁷ Arguably, this standard may be too strict to apply to the current study, especially given the small educational effect size that we found. A larger sample would be needed to adequately power the study to determine a smaller educational effect size.³⁵ However, the practical significance of such an investigation would need to be established.³⁶ In addition, the scoring instrument that we used combines elements of both cognitive and psychomotor skills. This makes it difficulty to determine which of these domains (if either) were impacted more by the use of video review. These limitations must be kept in mind when interpreting our results.

The strengths of this study include the use of the same team members for all simulation sessions, the use of standardized scenarios and well-accepted debriefing methods, and the use of a blinded reviewer using a tool that yields reliable scores that enables one to make valid judgments of neonatal resuscitation performance.

CONCLUSIONS

Using our methodology, video-assisted debriefing did not seem to offer significant educational benefits over oral debriefing alone during simulation-based training in neonatal resuscitation. This does not negate the possibility that, under different circumstance, video review may have educational benefits. Clearly, additional research is needed to define the optimal use of video review and to determine what benefits (if any) exist in using video review during simulation debriefing. Research questions that remain unanswered are many and include the following: Does scripting of the debriefing modulate the benefits of video review? What is the best “dose” of video review during debriefing? Does video review have a different impact on technical versus nontechnical skills? What effect does the timing of video review after the simulation have? What are the qualitative effects of video review on simulation participants? All these are important questions, and their answers will provide needed insight to simulation educators on the optimal use of video review during simulation debriefing.