Simulation-based training has become an integral practice in improving skill levels in procedures such as laparoscopy,1 endoscopy2 and bronchoscopy.3 Various task-training simulation devices have been introduced in anaesthesia subspecialties such as ultrasound-guided regional anaesthesia simulation,4,5 transthoracic echocardiography6 and virtual reality simulation in fibre-optic intubation.7 These are considered more effective than traditional teaching for acquiring procedural skills and are ideally suited to goal-directed learning in resident education.8
The ability to intubate using a flexible optical bronchoscopic (FOB) is an advanced core skill for airway management training. It is a complex psychomotor skill which requires hand–eye coordination. Mastering FOB intubation is an essential requirement of anaesthesia resident education.9–11 However, residency training programmes are inconsistent in training for acquisition of FOB intubation skill. The traditional approach of ‘see one, do one, teach one’ is considered obsolete and newer simulation-based approaches are advocated.12
Teaching of FOB intubation skill includes the following – informational (upper airway anatomy, bronchoscope structure & function and fundamentals of FOB intubation); simulation (static manikins, virtual reality computer and virtual reality simulator); supervised-graded clinical experience.13
In the last 2 decades, advanced high-fidelity simulators specifically designed for teaching FOB intubation have become available. They have been proposed to obviate the need for animal airway models and cadavers for practice skills.14 The AccuTouch bronchoscopy simulator (Immersion Medical Systems, Gaithersburg, Maryland, USA), a full-fledged high-fidelity nonportable simulator, was first made available around 2001. There are conflicting reports as to its efficacy – some have shown a significant improvement in FOB intubation time by novice trainees,14 whereas others found no added benefit compared with low-fidelity simulation training.15,16 The ORSIM simulator (ORSIM Airway Simulation Ltd., Auckland, New Zealand) is a recently introduced portable laptop based, less expensive, high-fidelity simulator.
We hypothesised that training on the ORSIM bronchoscopy simulator would be superior to didactic-only teaching as regards the performance of FOB intubation in patients. The primary objective of this randomised controlled trial was to compare the effects of simulator based and didactic-only training on the performance of FOB intubation in patients.
Materials and methods
The ORSIM simulator is a portable, high-fidelity, virtual reality bronchoscopy simulator. It consists of three components – a virtual bronchoscope, an interface into which the bronchoscope is inserted, and a laptop computer (Fig. 1). After the bronchoscope is inserted into the interface module, the operator's hand motion is tracked and translated into simulated endoscopic views on the computer screen as the operator ‘navigates’ the simulated airway. Baker et al.17 provided initial evidence on the validity and reliability of the ORSIM bronchoscopy simulator, supporting its potential value in training and assessment in airway management.
This study was approved by the University Health Network, Toronto, Canada's Research Ethics Board (REB # 13-6231-BE; Chairperson Dr Alan Barolet) on 7 November 2013. The trial was registered at clinicaltrials.gov (NCT02699242). Written informed consent was obtained from all participants participating in the trial. Requirement for consent from patients was waived by the ethics committee. This was a single-centre, randomised controlled clinical trial at the Toronto Western Hospital. Inclusion criteria for subject eligibility were anaesthesia or medical residents, or anaesthesia assistants who had performed less than five FOB intubations in patients. Those who had previous experience with a bronchoscopy simulator were excluded. The participants generally had experience with airway management but not with the FOB intubations.
Random allocation sequence numbers were generated by a computer in blocks of eight. These were stored securely in sequentially numbered, opaque sealed envelopes opened 30 min before the scheduled cases. Participants were enrolled by a research coordinator. Simulator training was provided to the simulation group (Group SIM) by the research fellow. The intra-operative data were collected by the research coordinator blinded to the study group.
FOB intubation was performed on patients who were scheduled for elective surgery and required tracheal intubation. Patients with anticipated or known difficult airway, BMI more than 35 kg m−2 or at risk of pulmonary aspiration were not included. Patients were under the care of an anaesthetist not involved with the study. Monitoring included noninvasive blood pressure, oxygen saturation (SpO2), ECG and capnography (ETCO2). After preoxygenation for 3 min, with visualisation of the ETCO2 waveform, general anaesthesia was induced using a standard technique, consisting of lidocaine 1 mg kg−1, propofol 2 to 3 mg kg−1, fentanyl 2 to 3 μg kg−1 and rocuronium 0.6 mg kg−1. Patients were mask ventilated for 2 min after induction, and nasal O2 supplementation at 10 l min−1 was continued during the FOB intubation. Assistance with tongue pull and jaw thrust was provided during the FOB intubation. Criteria for termination of the FOB intubation were attempt time exceeding 4 min, airway tissue trauma during the intubation attempt, attending anaesthetist not willing to continue with the study, or SpO2 of 90% or less.
During the FOB intubation process, endoscopic images from the FOB were recorded onto a SD card. An external digital camera mounted on a tripod was used to capture video of the subject during the FOB intubation. Intubation time was defined as time from introduction of the FOB into mouth until first ETCO2 visualisation.
Pretraining (first) intubation
Each participant performed two FOB intubations. Before the first intubation, each participant was given a didactic training session on airway anatomy and step-by-step instructions on FOB intubation and bronchoscope manipulation. This included a 15-min PowerPoint presentation on FOB intubation to provide a basic level of knowledge for all participants. Participants performed the first tracheal intubation within 1 to 3 days after completing their didactic training (Fig. 2).
Half an hour before the start of the first FOB intubation attempt participants were assigned randomly to either the ORSIM simulator training (Group SIM) or the didactic teaching only [control group (Group CON)] groups. After this first intubation attempt, participants in Group SIM received training with the ORSIM simulator for 1 h: this included simulated clinical scenarios along with oral and nasal bronchoscopy navigation. Participants in Group CON received no further training.
Posttraining (second) intubation
Within 1 week of the training phase, each participant performed their second FOB intubation in patients and was evaluated as in the first FOB intubation.
The recorded videos were evaluated by two anaesthetists blinded to group allocation. The evaluators were experienced in the use of the FOB Global Rating Scale (GRS) and checklist scoring methodology. The primary outcome included previously validated15 measurement tools in FOB intubation: a 40-point GRS (Appendix 1, http://links.lww.com/EJA/A175) and an 11-item checklist (Appendix 2, http://links.lww.com/EJA/A175). Intubation time (s) and success (tracheal intubation in <4 min) were also determined. The primary outcome comparison was the posttraining vs. pretraining GRS and checklist scores within Group SIM and Group CON.
Baseline data between the groups were compared using the t test for the continuous variables, and the Chi-squared test for the categorical variables.
For comparison of variables within Group SIM and CON (GRS, checklist, intubation time), paired tests were performed. The Shapiro–Wilk statistic was used to determine if the data of continuous variables were normally distributed. For variables with normal distribution of data, paired t tests were used; for variables with a skewed distribution, Wilcoxon signed-ranks tests were used. A P value of less than 0.05 was considered significant.
Sample size calculation
The effect size of training with the simulator compared with the control group was calculated at 1.92 SDs using the GRS.15 To detect an effect size of 1.5 SD, a power of 0.80 and an alpha of 0.05, the calculation indicated a sample size of 15 per group (total 30) was required. To take account of a possible dropout rate of 10%, a final sample size of 34 was used.
The intraclass correlation coefficient (ICC), an index of interrater agreement as the ratio of variance between participants due to error variance, was calculated (with 95% confidence intervals) based on average-rating.19 Cohen's kappa (κ) coefficient, a measure of interrater agreement for categorical data, was rated as follows: κ, 0.81 to 1.0, near-perfect agreement; 0.61 to 0.80, substantial agreement; 0.41 to 0.60, moderate agreement; 0.21 to 0.40, fair agreement; 0.00 to 0.20, slight agreement and less than 0.00, poor agreement.20
Statistical analysis was undertaken using statistical software (IBM SPSS Statistics V20.0.0; IBM Corporation, Armonk, New York, USA).
The study participants included anaesthesia or medicine residents and anaesthesia assistants. The study took place over a 16-month period from February 2015 to May 2016. A total of 34 participants (18, Group SIM: 16, Group CON) enrolled and started the study. During the intubation attempts by three participants the attending staff anaesthetist elected to take over the intubation due to perceived failure to progress or concerns about patient safety. Therefore, data from these three participants, two from the Group CON and one from the Group SIM, were not analysed. Thirty-one participants completed both intubations, 16 in Group SIM and 15 in Group CON. There were no oesophageal intubations or other complications.
Baseline data included patient's age, sex, BMI, the bronchoscopist's age group, previous experience with bronchoscopy and background training. These baseline characteristics were comparable between the groups (Table 1).
ICC with 95% CI, which was calculated between the two raters for each pair of the variables, showed a high degree of agreement in both the GRS and checklist scores (Table 2).
Intubation time and the GRS were found to be normally distributed variables, thus paired t tests were used. The checklist scores had a skewed distribution, so Wilcoxon signed-ranks test was used.
Within Group CON, there was no significant difference between pre and posttraining intubation time, GRS or checklist score (Table 3). In Group SIM, there was significant improvement between the pretraining and posttraining GRS [22.9 ± 8.1 vs. 28.2 ± 7.3, mean difference (95% CI) 5.3 (0.3 to 10.3), P = 0.04] and intubation time [177.6 ± 77.6 vs. 119.3 ± 52.2 s, mean difference −58.4 (−100.3 to −16.5) s, P = 0.01]. Posttraining and pretraining checklist scores were similar in both groups.
Our study showed that the GRS and the intubation time were significantly improved posttraining in Group SIM, but were unchanged in Group CON. The checklist score was not significantly changed posttraining in either group.
Four clinical trials have evaluated the effect of simulation training on FOB intubation in patients. Naik et al.18 studied residents trained either on a low-fidelity simulator (choose the hole) or with didactic lectures. The simulation group performed significantly better in FOB intubation time (81 vs. 210 s; P < 0.01), GRS (P < 0.01), checklist score (P < 0.05) and success rate (75 vs. 33%, P < 0.005), compared with the didactic group. Rowe and Cohen21 studied novice residents in either a simulation group with the AccuTouch simulator or a control group without simulator training. In the simulation group, the posttraining FOB intubation time in paediatric patients was significantly improved compared with the pretraining values (0.88 to 5.15 min, P < 0.001). Intubation time was unchanged in the control group. GRS and checklist score were not assessed. Two clinical trials compared the effect of low and high-fidelity simulation training on FOB intubation in patients. In the Chandra et al.15 study, respiratory therapists were randomised to a low-fidelity (choose the hole) or a high-fidelity model (AccuTouch) training followed by two consecutive FOB intubations in anesthetised patients. No difference in GRS, checklist score or intubation time was found between the groups. Crabtree et al.22 randomised respiratory therapists into two groups, high and low-fidelity simulation. The participants were evaluated on performance of FOB intubation in models and in anaesthetised patients. No correlation was found in intubation time, GRS or checklist score in the FOB intubation performed on models and in patients. However, the FOB intubation performance parameters in patients were not compared between the groups.
Our results are consistent with the findings of Naik and Rowe.18,21 The GRS and intubation time were improved by simulation training as compared with didactic training. The two studies15,22 comparing low-fidelity simulation with high-fidelity simulation training did not find any difference in performance. However, there are no control groups with didactic training alone to compare with.
Several nonhuman trials conducted simulation training variously with virtual-reality software,23,24 virtual-reality application7 or virtual-reality simulators,14 and were evaluated on simulators, manikins or cadavers. In a study by Giglioli et al.23 residents practiced with Virtual Fibreoptic Intubation (VFI) software on a computer. When tested on a manikin the VFI group scored better on the GRS and checklist score (P < 0.05), and procedure time (42 vs. 78 s, P = 0.05), compared with the control group. Boet et al.24 observed higher intubation success rate (81 vs. 52%, P < 0.05) in a manikin after VFI training of medical students. De Oliveira et al. developed iPhone-based virtual-reality software (iLarynx; Apple Inc., Cupertino, CA, USA) that simulated hand movements for performance of the FOB skills. Medical students trained on this software were evaluated using a manikin. The iLarynx group had a lower failure rate (2/10 vs. 8/10, P = 0.001) and fewer failed intubation attempts (4 vs. 24, P < 0.005) compared with the control group.7 Goldmann et al. studied novice anaesthesia residents who trained on a virtual reality AccuTouch simulator. Simulation training improved intubation time in the simulation scenario (114 vs. 75 s, P = 0.001), and in the cadaver (24 vs. 86 s, P < 0.001) compared with no simulation training.14
Several uncontrolled observational studies explored the usefulness of the ORSIM simulator training. Badiger et al. trained 38 anaesthesia registrars attending an airway course on the ORSIM simulator and asked them to grade their simulation training experience.25 Most respondents provided a high (8.5 out of 10) overall score for ORSIM as an educational tool. Ninety-three participants in two airway management courses practiced on three simulation models-ORSIM, Dexter Trainer and Oxford Box, and completed questionnaires regarding their usefulness.26 The ORSIM was rated highest in the understanding of airway anatomy and improved psychomotor skills (P < 0.01): 82% preferred ORSIM to improve their FOB intubation skills. Medical students trained on ORSIM simulator improved their success rate (100 vs. 54%) and bronchoscopy time (68 vs. 108 s) when comparing their third with their first attempts.27
The FOB simulation training has also been studied in nonanaesthesia settings. In a workshop using the AccuTouch simulator, residents in emergency medicine performed difficult paediatric airway simulations before and after computer-based tutorial and practice.28 Post training, there was a significant decrease in procedure time, endoscope collisions and increased efficiency scores. The effectiveness of simulation-based bronchoscopy training remains uncertain though it has been in use for almost 2 decades.29–34 Emphasis is placed on incorporating simulation among multiple tools in bronchoscopy training.35
The use of models or cadavers in the assessment of FOB intubation has limited application. In such nonhuman studies, observed higher success rates has not always translated directly into better clinical performance.36 This may be due to challenges during clinical FOB navigation, management of secretions in the airway and passage of a tracheal tube over the FOB in live patients.10 Nontechnical skills such as task management, communications and decision making, may contribute to performance of the FOB intubation in patients.
First, our results are from a single tertiary care institution and may not be generalisable. Second, our study had a relatively small sample size although statistical significance was reached for the primary outcome variable in our study. Third, differences in skill and experience in airway management of the FOB participants could have been a confounding factor influencing our primary outcome. However, the number of the FOB intubation performed prior to the study was similar between the groups. Fourth, the interval between the first and second intubation varied between a day and a week. A shorter interval may have benefitted the participant's performance. Fifth, 1 h simulation training may have been insufficient to translate into benefits in clinical performance. Some of the previous studies allowed longer simulation training periods.
Our study showed that the virtual reality FOB simulation training using the ORSIM simulator resulted in improved performance in FOB intubation in patients. FOB simulation training with the ORSIM simulator may be a useful adjunct in acquiring the FOB intubation skills and could be incorporated into core residency training. Questions remain regarding the optimal duration of simulator training and optimal interval for retraining.
Acknowledgements relating to this article
Assistance with the study: we wish to acknowledge Dr Michael M Murphy, Chair, Dept. Anesthesiology & Pain Medicine, University of Alberta, Edmonton, for providing the ORSIM equipment free of charge for the study. We also wish to thank all the respiratory therapists of the Toronto Western Hospital for acknowledging requests to provide timely services and fibreoptic bronchoscope during study.
Financial support and sponsorship: Supported in part by the Department of Anaesthesia, Toronto Western Hospital, University of Toronto, Ontario, Canada.
Conflicts of interest: none.
Presentation: preliminary data for this study were presented as an abstract at World Airway Management Meeting (WAMM) Dublin, Ireland 13 November 2015. Abstract and poster presentation at the Society of Airway Management (SAM), Newport Beach, California, USA, 14 to 17 September 2017.
1. Willis RE, Van Sickle KR. Current status of simulation-based training in graduate medical education. Surg Clin North Am
2. Ekkelenkamp VE, Koch AD, de Man RA, et al. Training and competence assessment in GI endoscopy: a systematic review. Gut
3. Kennedy CC, Maldonado F, Cook DA. Simulation-based bronchoscopy training: systematic review and meta-analysis. Chest
4. Adhikary SD, Hadzic A, McQuillan PM. Simulator for teaching hand-eye coordination during ultrasound-guided regional anaesthesia. Br J Anaesth
5. Liu Y, Glass NL, Glover CD, et al. Comparison of the development of performance skills in ultrasound-guided regional anesthesia simulations with different phantom models. Simul Healthc
6. Neelankavil J, Howard-Quijano K, Hsieh TC, et al. Transthoracic echocardiography simulation is an efficient method to train anesthesiologists in basic transthoracic echocardiography skills. Anesth Analg
7. De Oliveira GS Jr, Glassenberg R, Chang R, et al. Virtual airway simulation to improve dexterity among novices performing fibreoptic intubation. Anaesthesia
8. Murray DJ. Progress in simulation education: developing an anesthesia curriculum. Curr Opin Anaesthesiol
9. Koppel JN, Reed AP. Formal instruction in difficult airway management: a survey of anesthesiology residency programs. Anesthesiology
10. Dawson AJ, Marsland C, Baker P, et al. Fibreoptic intubation skills among anaesthetists in New Zealand. Anaesth Intensive Care
11. McNarry AF, Dovell T, Dancey FM, et al. Perception of training needs and opportunities in advanced airway skills: a survey of British and Irish trainees. Eur J Anaesthesiol
12. Krage R, Erwteman M. State-of-the-art usage of simulation in anesthesia: skills and teamwork. Curr Opin Anaesthesiol
13. Russo SG, Dierdorf SF. Teaching airway management outside the operating room. Benumof and Hagberg's airway management. 3rd ed.2013; Philadelphia: Elsevier Saunders, 1079–1080.
14. Goldmann K, Steinfeldt T. Acquisition of basic fiberoptic intubation skills with a virtual reality airway simulator. J Clin Anesth
15. Chandra DB, Savoldelli GL, Joo HS, et al. Fiberoptic oral intubation: the effect of model fidelity on training for transfer to patient care. Anesthesiology
16. Dalal PG, Dalal GB, Pott L, et al. Learning curves of novice anesthesiology residents performing simulated fibreoptic upper airway endoscopy. Can J Anesth
17. Baker PA, Weller JM, Baker MJ, et al. Evaluating the ORSIM®
simulator for assessment of anaesthetists’ skills in flexible bronchoscopy: aspects of validity and reliability. Br J Anaesth
2016; 117 (Suppl 1):i87–i91.
18. Naik VN, Matsumoto ED, Houston PL, et al. Fiberoptic orotracheal intubation on anesthetized patients: do manipulation skills learned on a simple model transfer into the operating room? Anesthesiology
19. Bould MD, Crabtree NA, Naik VN. Assessment of procedural skills in anaesthesia. Br J Anaesth
20. Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics
21. Rowe R, Cohen RA. An evaluation of a virtual reality airway simulator. Anesth Analg
22. Crabtree NA, Chandra DB, Weiss ID, et al. Fibreoptic airway training: correlation of simulator performance and clinical skill. Can J Anaesth
23. Giglioli S, Boet S, De Gaudio AR, et al. Self-directed deliberate practice with virtual fiberoptic intubation improves initial skills for anesthesia residents. Minerva Anestesiol
24. Boet S, Bould MD, Schaeffer R, et al. Learning fibreoptic intubation with a virtual computer program transfers to ‘hands on’ improvement. Eur J Anaesthesiol
25. Badiger S, Fearnley A, Ahmad I. Fibreoptic tracheal intubation training using bronchoscopy simulation. Eur J Anaesthesiol
26. Vaidyanath C, Sharma M, Mendonca C. Fibreoptic airway endoscopy training: comparison of three different trainer models. Eur J Anaesthesiol
27. Vaidyanath C, Sharma M, Mistry V, et al. ORSIM™ bronchoscopy simulator improves psychomotor skills for fibreoptic intubation amongst novices. Proceedings of the Anaesthesia Research Society Meeting. Br J of Anaesth
28. Binstadt E, Donner S, Nelson J, et al. Simulator training improves fiber-optic intubation proficiency among emergency medicine residents. Acad Emerg Med
29. Krogh CL, Konge L, Bjurström J, et al. Training on a new, portable, simple simulator transfers to performance of complex bronchoscopy procedures. Clin Respir J
30. Ost D, DeRosiers A, Britt EJ, et al. Assessment of a bronchoscopy simulator. Am J Respir Crit Care Med
31. Davoudi M, Osann K, Colt HG. Validation of two instruments to assess technical bronchoscopy skill using virtual reality simulation. Respiration
32. Konge L, Arendrup H, von Buchwald C, et al. Virtual reality simulation of basic pulmonary procedures. J Bronchology Interv Pulmonol
33. Blum MG, Powers TW, Sundaresan S. Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy. Ann Thorac Surg
34. Wahidi MM, Silvestri GA, Coakley RD, et al. A prospective multicenter study of competency metrics and educational interventions in the learning of bronchoscopy among new pulmonary fellows. Chest
35. Ernst A, Wahidi MM, Read CA, et al. Adult bronchoscopy training: current state and suggestions for the future: CHEST expert panel report. Chest
36. Colt HG. Simulation in bronchoscopy training: are we there yet? Curr Respir Care Rep