Secondary Logo

Journal Logo

Construct Validity of the Simbionix Bronch Mentor Simulator for Essential Bronchoscopic Skills

Pastis, Nicholas J. MD, FCCP*; Vanderbilt, Allison A. EdD; Tanner, Nichole T. MD, MSCR*; Silvestri, Gerard A. MD, MS*; Huggins, John T. MD*; Svigals, Zach MD*; Shepherd, Ray W. MD

Journal of Bronchology & Interventional Pulmonology: October 2014 - Volume 21 - Issue 4 - p 314–321
doi: 10.1097/LBR.0000000000000111
Original Investigations
Free

Background: Although simulation-based bronchoscopy has been shown to be an effective training modality, formal assessment should still be performed as new technology emerges. We sought to validate a simulator in essential bronchoscopic tasks, and survey perceptions of bronchoscopists on simulation.

Methods: A cohort study at 2 medical centers used 3 groups to assess construct validity of the Simbionix Bronchoscopy Simulator: 7 first-year fellows with <10 bronchoscopies each (novice), 6 pulmonologists with 200 to 1000 bronchoscopies each (experienced), and 7 pulmonologists with >1000 bronchoscopies each (expert). Participants were tested in 4 tasks (1: scope manipulation, 2: guided anatomic navigation, 3: airway anatomy, and 4: lymph node anatomy). Participants were scored and surveyed on their impressions of simulation. The means and Kruskal-Wallis test among groups were compared by task item (P<0.05).

Results: There were statistically significant differences in mean ranks among groups for tasks 1 and 3. For task 1, final score, time, mid-lumen time, and wall hits were discriminative (P=0.006, 0.006, 0.012, and 0.014, respectively). For task 3, time, bronchial segments identified, bronchial segments incorrectly identified, and bronchial segments skipped were discriminative (P=0.04, 0.012, 0.013, and 0.013, respectively). There was no statistically significant difference for task 2 and task 4. All participants agreed that simulation training is helpful and should be incorporated into bronchoscopic training.

Conclusions: The simulator demonstrated validity in differentiating skill in scope manipulation and airway anatomy, but did not discriminate skill levels in anatomic orientation or identification of lymph nodes. Bronchoscopy simulation was viewed as helpful by all levels and should be considered before performance on patients.

*Division of Pulmonary/Critical Care, Medical University of South Carolina, Charleston, SC

Virginia Commonwealth University School of Medicine, Richmond, VA

Disclosure: There is no conflict of interest or other disclosures.

Reprints: Nicholas J. Pastis, MD, FCCP, Division of Pulmonary/Critical Care, Medical University of South Carolina, 96 Jonathan Lucas St., 812 CSB, Charleston, SC 29414 (e-mail: pastisn@musc.edu).

Received March 24, 2014

Accepted August 6, 2014

Fueled by increased patient safety and satisfaction concerns and in some cases, reduced trainee work hours with less procedural volume, simulation-based training is emerging as an effective, zero-risk environment to teach procedures.1,2 At present, studies that demonstrate the efficacy, cost-effectiveness, and improved patient comfort and safety with simulation training for various other medical and surgical procedures3–6 outnumber such studies of bronchoscopy simulation for trainees.7,8

Although bronchoscopy training contributes greatly to the foundation of knowledge for pulmonary fellows, there is no standardized training approach and no universal simulation modality to facilitate learning the essential skills before performing bronchoscopy on live patients. Training for this procedure still depends largely on the apprenticeship model, and competency is assessed based on subjective measurement by faculty teachers and on the completion of at least 100 bronchoscopies. Minimum threshold numbers of procedures have been proposed (primarily by an expert opinion) to achieve competency in bronchoscopy9,10; however, the reliability of such numbers has been called into question.11,12

Previously, Ost et al8 validated the accuracy of a bronchoscopy simulator to assess experience level, and by training new fellows on the bronchoscopy simulator, there was a more rapid acquisition of skill compared with conventional training. The exercise used in this study emphasized proper technique for airway inspection and identification of airway anatomy.

A trainee’s first encounter with airway anatomy identification on a live patient may lead to a longer procedure with more scope to wall contact and ultimately, increased procedural discomfort and complications. In a prospective study of new pulmonary fellows by Wahidi and colleagues, the use of simulation-based bronchoscopy training resulted in accelerated acquisition of bronchoscopy skills among first-year pulmonary fellows as assessed by a validated tool for bronchoscopy skill assessment, the Bronchoscopy Skills and Tasks Assessment Tool. In this study, the educational intervention that incorporated 20 simulated bronchoscopies before the fifth bronchoscopy of a real patient led to a statistically significant improvement in the speed of skill acquisition.7

Currently, effective modalities exist to simulate bronchoscopy including inanimate airway models, extracted and preserved animal lungs, live animals, and airway simulation software,8,13–19 yet there is no standard method recommended for trainees. Despite its effectiveness, several important questions regarding simulation remain unanswered: What is the optimal way to standardize and implement simulation across fellowships? Should competence in the most essential tasks be achieved before bronchoscopy on live patients and which tasks are most essential?

Before a new bronchoscopy simulator can achieve predictive validity to determine whether a trainee’s competence on the simulator will translate to competence on patients, the first step is to establish that the simulator differentiates different degrees of competence. The aim of this study was to determine whether a high-fidelity bronchoscopy simulator, the Simbionix Bronch Mentor Simulator, could discriminate between various degrees of skill in 4 essential bronchoscopy tasks, among 3 different cohorts of bronchoscopists: novice, experienced, and expert. Unlike other studies on bronchoscopy simulators, this study separated the simulation experience into different essential skills. In addition, the didactic value of the simulator was evaluated through written surveys by these 3 cohorts.

Back to Top | Article Outline

METHODS

The BRONCH Mentor is a multidisciplinary simulator that provides an anatomically correct working environment using a Pentax diagnostic bronchoscope with tactile feedback, and realistic visualization displayed on a 24-inch touch screen. Although this device offers a wide variety of bronchoscopic tools, such as biopsy forceps, cytology brush, aspirating needle, balloon, and electrocautery, these were not used as part of this study. This simulator was designed to provide a bronchoscopic training curriculum that consists of basic skills as well as a full complement of more advanced procedures to promote the ongoing development of the user. Only 4 basic skills were evaluated in this study.

Skill task 1 was basic scope manipulation. The goal was to acquire basic bronchoscope control capabilities. By learning to navigate the bronchoscope in a “cyber environment,” the user develops hand-eye coordination (ie, staying in the center and avoiding wall contact while following an object along a randomly selected narrowing lumen) (Fig. 1). Performance metrics included the following: final score 0 to 100, total navigation time, % time at mid-lumen (mid-lumen=medial 50% of the gap between lumen to scope’s edge), % time in contact with wall, number of wall contacts (total), and distribution of wall contacts (wide lumen, medium lumen, and narrow lumen) (Fig. 1).

FIGURE 1

FIGURE 1

Skill task 2 was guided anatomic navigation. The goal was to learn and practice correct scope navigation (ie, keeping the scope view aligned so that anterior/posterior and lateral/medial in the view corresponds with the orientation of the simulated patient) while performing bronchial navigation maneuvers to achieve a complete airway inspection. For directional guidance, figures within the lumen corresponded to the scope’s view finder, indicating the correct scope roll and flexion/extension for each bifurcation. When the scope’s view finder was correctly aligned with the figures in the airway, the figures changed color and thus indicated that the maneuver was performed satisfactorily. The performance metrics were total performance time, % time at mid-lumen, % time in contact with the wall, % time with clear visibility, list of bifurcations where maneuver was performed and a lobe or segment was entered satisfactorily (on first, second, or third and up attempt), list of bifurcations where maneuver was performed unsatisfactorily, and list of bifurcations skipped (Fig. 2).

FIGURE 2

FIGURE 2

Skill task 3 was lung anatomy and bronchial segments. The goal was to gain familiarity with bronchial anatomy and identify each bronchial segment by name. The user identified the bronchial segments by focusing the scope on question marks within the segments, and selecting the correct name for each. Two naming conventions (verbal and numeric) are available and 1 is selected before starting. Three identification tries are allowed for each segment; after a third mistake the software reveals the segment’s name. Performance metrics included: naming convention used (descriptive-numeric/advanced-numeric), bronchial segments correctly identified (at first/second/third try), bronchial segments incorrectly identified, bronchial segments skipped, total performance time, % time at mid-lumen, and % time in contact with wall (Fig. 3).

FIGURE 3

FIGURE 3

Skill task 4 was lymph node (LN) anatomy, and the goal was to gain familiarity with adjoining mediastinal LNs, under the International Association for the Study of Lung Cancer map.20 By focusing the scope on arrows next to airway walls, LNs were revealed behind the walls and a menu was opened to select the correct name. Three identification tries are allowed for each LN, and after the third mistake, the software reveals LN name. Performance metrics included the following: LNs correctly identified (correct at first/second/third try), LNs incorrectly identified, LNs skipped, total performance time, % time at mid-lumen, and % time in contact with wall (Fig. 4).

FIGURE 4

FIGURE 4

The objective of this prospective observational trial was to determine whether a bronchoscopy simulator could distinguish between novice, experienced, and expert bronchoscopists on the basis of their scores on 4 essential bronchoscopic skills. The study was conducted at 2 university hospitals, both of which had pulmonary fellowship programs, and the study was approved by the institutional review board of the Medical University of South Carolina and the Virginia Commonwealth University Medical Center. Informed consent was obtained from all participants.

Participants were allocated to 3 groups to assess the validity and didactic value of the Simbionix Bronchoscopy Simulator. The first group, the novices, was defined as physicians who performed <10 bronchoscopies. They were all first-year pulmonary fellows. The second group was comprised of experienced physicians who were faculty members in academic Pulmonary/Critical Care and performed between 200 and 1000 bronchoscopies. The third group consisted of experts who were also physicians from academic Pulmonary/Critical Care who performed over 1000 bronchoscopies. Four of 7 experts had formal interventional pulmonology training in the past. The authors acknowledge the controversy behind numbers-based competency which is not evidence based, but groups were defined based on previous validation work which used number of procedures.21 The threshold numbers defining each skill level were higher in this study than in other studies on bronchoscopy simulation to account for the increased bronchoscopy volume of those who are on dedicated procedural services and perform the majority of bronchoscopies at their institution. Each participant performed the simulated bronchoscopic tasks in a closed room and was alone except for 1 of the investigators who provided instructions for operating the simulator and recorded results.

All participants signed informed consent and were asked to complete a questionnaire on demographics and their general medical and bronchoscopy experience. It included the number of bronchoscopies performed annually and the number of years registered as a bronchoscopist.

After the simulator run, participants were asked to answer questions about their appreciation of the education value of the bronchoscopy exercises performed. Appreciation is expressed on a 4-point Likert scale varying from not at all to absolutely.22–24 Questions were asked concerning the realism of imaging, simulator setup, and bronchoscopic control. Experts were asked whether the Bronchoscopy Mentor could be used as a teaching device for novice bronchoscopists and whether experience on the simulator could be useful in practice.

Back to Top | Article Outline

Statistical Analysis

Descriptive statistics and Kruskal-Wallis tests were performed for the statistical analysis of the data. A separate analysis between groups was performed using a 2-tailed Mann-Whitney exact U test. A P-value of <0.05 was considered significant.

Back to Top | Article Outline

RESULTS

The means and Kruskal-Wallis test among groups were compared by task item (Table 1). There were statistically significant differences in mean ranks among the 3 groups for tasks 1 and 3. For task 1, mean ranks for final score, total time, % time at mid-lumen, and total wall hits were most favorable (P=0.006,0.006,0.012, and 0.014, respectively) for expert, then experienced, and then novice bronchoscopists. For task 3, mean ranks for total time, number of bronchial segments identified on first attempt, number of bronchial segments incorrectly identified after 3 attempts, and bronchial segments skipped were most favorable (P=0.04, 0.012, 0.013, and 0.013, respectively) for expert, then experienced, and then novice bronchoscopists. There was no statistically significant difference for task 2 and task 4.

TABLE 1

TABLE 1

All novices strongly agreed and all participants agreed that simulation training is helpful and should become part of bronchoscopic training. All novices strongly agreed and all 21 participants at least agreed that simulation training is helpful and should become part of bronchoscopic training (Tables 2 and 3). The most frequent answers to the free text part of the survey included the following: for the basic skill missing—needle technique (3 responses); liked most: airway anatomy (10 responses); liked least: orientation (11 responses). Cronbach α was calculated for the novice group (α=0.519), for the intermediate group (α=0.851), and for the expert group (α=0.179), therefore, only the survey of the intermediate group showed internal consistency.

TABLE 2

TABLE 2

TABLE 3

TABLE 3

Back to Top | Article Outline

DISCUSSION

This study demonstrates the validity and potential utility of a bronchoscopy simulator in 2 critical skills germane to bronchoscopy: basic scope manipulation to remain centered in the airways and correct identification of the endobronchial anatomy. In addition, all novices strongly agreed and all other participants at least agreed that simulation training is helpful and should become part of bronchoscopic training.

As long as scope manipulation and airway anatomy are not unrealistically simplistic or unrealistically difficult, experts should perform better than experienced who should perform better than novice bronchoscopists. Hence, it is logical that the most “life-like” simulation for these most fundamental tasks were easily validated among bronchoscopists with different skill levels. Before any diagnostic test is performed, it is essential that all bronchoscopists be able to properly manipulate a flexible bronchoscope and identify endobronchial anatomy. However, the negative results of this study beg the question: why did testing on scope orientation (presumably a very basic and essential task) not achieve statistical significance in discriminating among different skill levels? One possibility is that the sensitivity for detecting correct orientation on the simulator was programmed too high or incorrectly. Alternatively, this may be an area in bronchoscopy training which is not taught in a standardized manner and graduates of fellowships may orient the bronchoscope in varying directions as they perform airway inspections.

As for LN anatomy, it is not as surprising that this simulator was not validated for this task. Previous studies have suggested that only a minority of pulmonologists routinely perform transbronchial needle aspiration during their training.25,26 Currently, while the majority of fellowship programs have endobronchial ultrasound, only a minority of programs have endobronchial ultrasound competency metrics for graduating fellows.27 Although it is true that lung cancer staging is a requirement of all fellows, without routinely performing bronchoscopic staging, it may be difficult to correlate the International Association of Lung Cancer (IASLC) LN map with the endobronchial view of where LN stations are supposed to reside.

This study provides only the first step for evaluating this bronchoscopy simulator. It establishes construct validity, that is, it measures what it purports to measure for 2 simulated tasks, but it does not validate translational impact, that is, it does not establish that success in these tasks translates to success in real patients. In addition, limitations of this study include a small sample size and lack of validation for the competency numbers used for this study.

Future directions in bronchoscopy simulation will include cost-benefit analysis of purchasing simulators. In addition, while the “see one, do one, teach one” method has become obsolete and the door for simulation training is now open, further studies will be needed to develop standardized means of implementing simulation in a more universal manner among training programs. Models like the one used in this study should be perfected and used more rigorously to evaluate new technologies in simulation before considering them as adjuncts to bronchoscopy training on live patients.

Back to Top | Article Outline

CONCLUSIONS

The bronchoscopy simulator demonstrated construct validity in discriminating skill level in scope manipulation and airway anatomy. By using a model of validation, certain exercises such as orientation on the simulator may be identified, which may need modifications before widespread use. Although not validated in its ability to differentiate skill levels in identification of LN stations, this may not have been a function of failure of this technology, but rather gaps in skill identifying LN anatomy among practitioners. Bronchoscopy simulation was viewed as a helpful by all levels and should be considered to teach scope manipulation and airway anatomy before performance on live patients.

Back to Top | Article Outline

REFERENCES

1. Ziv A, Wolpe P, Small S, et al.. Simulation-based medical education: an ethical imperative. Acad Med. 2003;78:783–788.
2. Dong Y, Suri H, Cook D, et al.. Simulation-based objective assessment discerns clinical proficiency in central line placement: a construct validation. Chest. 2010;137:1050–1056.
3. Martin M, Vashisht B, Frezza E, et al.. Competency-based instruction in critical invasive skills improves both resident performance and patient safety. Surgery. 1998;124:313–317.
4. Scott D, Bergen P, Rege R, et al.. Laparoscopic training on bench models: better and more cost effective than operating room experience? J Am Coll Surg. 2000;191:272–283.
5. Sedlack R, Kolars J, Alexander J. Computer simulation training enhances patient comfort during endoscopy. Clin Gastroenterol Hepatol. 2004;2:348–352.
6. Seymour N, Gallagher A, Roman S, et al.. Virtual reality training improves operating room performance results of a randomized, double-blinded study. Ann Surg. 2002;236:458–463.
7. Wahidi M, Silvestri G, Coakley R, et al.. A prospective multicenter study of competency metrics and educational interventions in the learning of bronchoscopy among new pulmonary fellows. Chest. 2010;137:1040–1049.
8. Ost D, DeRosiers A, Britt E, et al.. Assessment of a bronchoscopy simulator. Am J Respir Crit Care Med. 2001;164:2248–2255.
9. Ernst A, Silvestri G, Johnstone D, et al.. Interventional pulmonary procedures, guidelines from the American College of Chest Physicians. Chest. 2003;123:1693–1717.
10. Torrington K. Bronchoscopy training and competency: how many are enough? Chest. 2000;118:572–573.
11. Alvarez F, Burger C, Grinton S, et al.. Competencies in pulmonary procedures. Chest. 2004;125:800–801.
12. Prakash U. Guidelines for training and practice of interventional pulmonology. By the numbers? J Bronchol. 2003;10:169–173.
13. Blum M, Powers T, Sundaresan S. Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy. Ann Thorac Surg. 2004;78:287–291.
14. Chen J, Hsu H, Lai I, et al.. National Taiwan University Endoscopic Simulation Collaborative Study Group (NTUSEC). Validation of a computer-based bronchoscopy simulator developed in Taiwan. J Formos Med Assoc. 2006;105:569–576.
15. Colt H, Crawford S, Galbraith OI. Virtual reality bronchoscopy simulation: a revolution in procedural training. Chest. 2001;120:1333–1339.
16. Di Domenico S, Simonassi C, Chessa L. Inexpensive anatomic trainer for bronchoscopy. Interact Cardiovasc Thorac Surg. 2007;6:567–569.
17. Moorthy K, Smith S, Brown T, et al.. Evaluation of virtual reality bronchoscopy as a learning and assessment tool. Respiration. 2003;70:195–199.
18. Rowe R, Cohen R. An evaluation of a virtual reality airway simulator. Anesth Analg. 2002;95:62–66.
19. Davoudi M, Colt H. Bronchoscopic simulation: a brief review. Adv Health Sci Educ Theory Pract. 2009;14:287–296.
20. Goldstraw P. International Association for the Study of Lung Cancer. Staging Handbook in Thoracic Oncology. 2009:1st ed.Denver:Editorial Rx Press;104.
21. Koch A, Buzink S, Heemskerk J, et al.. Expert and construct validity of the Simbionix GI Mentor II endoscopy simulator for colonoscopy. Surg Endosc. 2007;22:158–162.
22. Ritter E, McClusky D, Lederman A, et al.. Objective psychomotor skills assessment of experienced and novice flexible endoscopists with a virtual reality simulator. J Gastrointest Surg. 2003;7:871–878.
23. Likert R. A technique for the measurement of attitudes. Arch Psych. 1932;140:5–55.
24. Thomas-Gibson S, Williams C. Colonoscopy training new approaches, old problems. Gastrointest Endosc Clin N Am. 2005;15:813–827.
25. Haponik E, Shure D. Underutilization of transbronchial needle aspiration: experiences of current pulmonary fellows. Chest. 1997;112:251–253.
26. Pastis N, Nietert P, Silvestri G. Variation in training for interventional pulmonary procedures among US pulmonary/critical care fellowships: a survey of fellowship directors. Chest. 2005;127:1614–1621.
27. Tanner N, Pastis N, Silvestri G. Training for linear endobronchial ultrasound among US pulmonary/critical care fellowships: a survey of fellowship directors. Chest. 2013;143:423–428.
Keywords:

simulation; bronchoscopy; validation

© 2014 by Lippincott Williams & Wilkins.