Secondary Logo

Share this article on:

EBUS-STAT Subscore Analysis to Predict the Efficacy and Assess the Validity of Virtual Reality Simulation for EBUS-TBNA Training Among Experienced Bronchoscopists

Scarlata, Simone MD; Palermo, Patrizio MD; Candoli, Piero MD; Tofani, Ariela MD; Petitti, Tommasangelo MD; Corbetta, Lorenzo MD

Journal of Bronchology & Interventional Pulmonology: April 2017 - Volume 24 - Issue 2 - p 110–116
doi: 10.1097/LBR.0000000000000349
Original Investigations

Background: Linear endobronchial ultrasound transbronchial needle aspiration (EBUS-TBNA) represents a pivotal innovation in interventional pulmonology; determining the best approach to guarantee systematic and efficient training is expected to become a main issue in the forthcoming years. Virtual reality simulators have been proposed as potential EBUS-TBNA training instruments, to avoid unskilled beginners practicing directly in real-life settings. A validated and perfected simulation program could be used before allowing beginners to practice on patients. Our goal was to test the reliability of the EBUS-Skills and Task Assessment Tool (STAT) and its subscores for measuring the competence of experienced bronchoscopists approaching EBUS-guided TBNA, using only the virtual reality simulator as both a training and an assessment tool.

Methods: Fifteen experienced bronchoscopists, with poor or no experience in EBUS-TBNA, participated in this study. They were all administered the Italian version of the EBUS-STAT evaluation tool, during a high-fidelity virtual reality simulation. This was followed by a single 7-hour theoretical and practical (on simulators) session on EBUS-TBNA, at the end of which their skills were reassessed by EBUS-STAT.

Results: An overall, significant improvement in EBUS-TBNA skills was observed, thereby confirming that (a) virtual reality simulation can facilitate practical learning among practitioners, and (b) EBUS-STAT is capable of detecting these improvements. The test’s overall ability to detect differences was negatively influenced by the minimal variation of the scores relating to items 1 and 2, was not influenced by the training, and improved significantly when the 2 items were not considered. Apart from these 2 items, all the remaining subscores were equally capable of revealing improvements in the learner. Lastly, we found that trainees with presimulation EBUS-STAT scores above 79 did not show any significant improvement after virtual reality training, suggesting that this score represents a cutoff value capable of predicting the likelihood that simulation can be beneficial.

Conclusions: Virtual reality simulation is capable of providing a practical learning tool for practitioners with previous experience in flexible bronchoscopy, and the EBUS-STAT questionnaire is capable of detecting these changes. A pretraining EBUS-STAT score below 79 is a good indicator of those candidates who will benefit from the simulation training. Further studies are needed to verify whether a modified version of the questionnaire would be capable of improving its performance among experienced bronchoscopists.

*Geriatrics—Unit of Respiratory Pathophysiology and Thoracic Endoscopy, Campus Bio Medico University and Teaching Hospital

Public Health and Statistics Unit, Campus Bio Medico University, Rome

Pneumology Outpatient Clinic, Casa di Cura convenzionata San Marco, Latina

Pulmonary and Thoracic Endoscopy Unit, AUSL, Ravenna

§Unit of Diagnostic and Interventional Bronchology, Careggi University Hospital, Florence, Italy

Disclosure: There is no conflict of interest or other disclosures.

Reprints: Simone Scarlata, MD, Unit of Respiratory Pathophysiology and Thoracic Endoscopy, Campus Bio Medico University and Teaching Hospital, Via Alvaro del Portillo 200, Rome 00128, Italy (e-mail: s.scarlata@unicampus.it).

Received April 23, 2016

Accepted November 1, 2016

Linear endobronchial ultrasound transbronchial needle aspiration (EBUS-TBNA) represents a pivotal innovation in interventional pulmonology, allowing the respiratory physician to achieve an outstanding rate of diagnostic accuracy in mediastinal staging of lung cancer and, to a lesser extent, in benign and hematologic thoracic disorders.1 The complexity of this advanced diagnostic procedure can make the learning curve for beginners a difficult, costly, and time-consuming process.2 Determining the best approach to guarantee systematic and efficient training is expected to become a main issue in the forthcoming years.

Unfortunately, traditional training practices (ie, supervision by seniors over the beginner in a real-life setting) are not optimal for EBUS training, as they are associated with longer procedure time, higher amount of sedation, and increased rate of complications.3 Konge et al4 have recently argued that a systematic training program in endosonography should be based on 3 dedicated steps: (a) theoretical knowledge; (b) training on simulators; and (c) supervised performance on the patient. Other authors are currently suggesting virtual reality simulators as potential instruments able to provide training on EBUS-TBNA, thereby avoiding unskilled beginners practicing directly in real-life settings.5 A practical proposal would be to envisage a validated and perfected simulation program that practitioners would train on before applying the procedure on patients.

Previous reports have demonstrated that there is a significant heterogeneity in the EBUS-TBNA learning curves of trainers and beginners,6 highlighting the need for validated, objective, multistep procedures to measure individual competence. For this purpose, tools capable of assessing both the operator’s relevant knowledge and his/her technical skills are now available. Indeed, theoretical tests have already been performed and validation obtained. In particular, Savran and colleagues have devised a theoretical test in endosonography consisting of multiple-choice questions. Validity evidence was gathered, and the test demonstrated its validity in terms of both content and construct.7 Among the practical/theoretical tests, the Endobronchial Ultrasound Skills and Task Assessment Tool (EBUS-STAT) has recently been tested and validated,8 but its usefulness in assessing the skills of trained bronchoscopists performing a virtual reality simulation training for EBUS-TBNA (before starting their practice on patients) has never been proved. Moreover, it is not clear whether existing evidence can be generalized to other settings, nor whether the individual components and internal consistency of the EBUS-STAT questionnaire are capable of performing equivalently and of detecting improvements.

The aim of this study therefore is to test the reliability of the EBUS-STAT assessment tool and its subscores in measuring the skills of experienced bronchoscopists as they approach EBUS-guided TBNA, using only the virtual reality simulator for the purpose of training and subsequent assessment.

Back to Top | Article Outline

METHODS

Assessment Protocol

Study participants were recruited from among the 16 graduate physicians attending a 1-year postgraduate training program in Interventional Pulmonology provided by the University of Florence, Italy. At the beginning of their training course participants were asked to self-evaluate their ability, confidence, and experience (approximate number of performed procedures) in every field of interventional pulmonology, including a simple diagnostic flexible bronchoscopy and the EBUS procedure. Self-evaluation consisted in filling out a multiple-choice questionnaire. All trainees but 2 had reached the minimum competence level of 100 bronchoscopies, as suggested by the European Respiratory Society.9 Furthermore, 6 participants reported having occasionally performed an EBUS and having prior theoretical knowledge but defined their experience and confidence as poor. After the self-evaluation, their skills were objectively assessed by three different evaluators who administered the Italian version of the EBUS-STAT evaluation tool during a high-fidelity virtual reality simulation using a dedicated device (Bronch Mentor EBUS Simulator—Simbionix Ltd, Airport City, Israel). Importantly, the 3 evaluators had never worked together or shared EBUS training and experience previously. After the baseline evaluation had been completed, participants underwent a 4-hour theoretical session that provided them with a comprehensive background on ultrasound physics, technical and practical tips, including an overall description of scopes, needles, sampling technique, ultrasonographic landmarks for the systematic assessment of mediastinal nodal stations, as well as a brief update on the existing evidence in the literature supporting the EBUS approach in clinical practice. Afterwards, they all spent the next 3 hours practicing with the simulator under skilled and experienced supervision. Worth noting is the fact that all participants, including those waiting for their turn with the virtual simulator or those who had already completed theirs, continued to play an active part in the training and were allowed to interact with the supervisor for further explanations. Lastly, each participant’s skills with the EBUS-TBNA were reassessed by readministering the EBUS-STAT questionnaire.

Back to Top | Article Outline

Baseline Self-Evaluation Questionnaire

Before the training, every participant was asked to self-evaluate his/her theoretical knowledge on endoscopic ultrasound and skills in using the EBUS scope. Each trainee was also asked how many flexible bronchoscopies and EBUS procedures he/she had previously performed. The administered questionnaire was used to classify participants according to their dexterity in guiding the bronchoscope through the upper and lower airways and to estimate their confidence with the EBUS procedure.

Back to Top | Article Outline

The EBUS-STAT

This is a 10-section assessment specifically designed for convex probe EBUS-TBNA procedure. Items 1 to 7 test technical ability, of which items 1 to 2 relate to general skills and 3-7 to specific EBUS-TBNA skills; items 8 to 10, on the other hand, assess skills with computed tomography (CT) scans and EBUS image recognition and correlation.10 Item number 6, specifically, explores the trainee’s familiarity with the ultrasound machine’s keyboard functions. This tool has already been tested and validated on real patients8 but never tested before to assess acquired learning abilities after a training using a high-fidelity simulation device.

Back to Top | Article Outline

Analytical Approach

Baseline competence level and the association between the measured level of dexterity and the accuracy of self-assessment were determined. Differences between the mean EBUS-STAT scores at baseline and those recorded after the training session were assessed through Student t test for paired variables. Global and partial scores were compared. To discriminate the weighted role of items in the technical skills subgroup, statistically significant differences were identified for each individual item, before and after the training exercise. A P-value ≤0.05 was considered statistically significant.

Taking into account the ceiling effect in dexterity improvement observed in more expert bronchoscopists practicing on virtual reality simulators,5 we tried to identify a pre-EBUS-STAT score that was a predictor for ineffectiveness of the simulation training. To do so we studied the linear relationship between precourse EBUS-STAT scores and the difference between precourse and postcourse EBUS-STAT scores, after evaluating the reliability of the linear regression model assumptions.

Back to Top | Article Outline

RESULTS

Fifteen trainees successfully completed the protocol. Another performed the baseline evaluation and attended the training but refused to repeat the assessment at the end of the training and was therefore excluded from the study.

Back to Top | Article Outline

Baseline Characterization of Trainee Ability by Self-Evaluation and EBUS-STAT

Most participants displayed a good knowledge of flexible bronchoscopy technical skills, with a vast majority (ie, all but 2) having already performed at least 100 procedures; they were capable of identifying anatomic landmarks inside the bronchial airways, although their theoretical and practical skills with EBUS-TBNA were generally poor. The mean global score at baseline evaluation was 68.6 (SD, 11.7). Considering that the maximum score is 100, this characterizes participants’ level of competence as intermediate. Participants obtained very high scores in the first 2 items, thus confirming that they were competent and confident in basic bronchoscopic skills (items 1 and 2: 7.3±1.7). The average score on technical skills (items 1 to 7) was 35.4 out of 55 (SD, 6.9), whereas the average score on the test assessing CT scans and EBUS image recognition and correlation (items 8 to 10) was 20.7 out of 30 (SD, 4.2). Conversely, items 3 to 10 displayed a wide range of different levels, with much lower scores on average (Table 1).

TABLE 1

TABLE 1

Back to Top | Article Outline

Comparison of the EBUS-STAT Scores After a High-fidelity Simulation Training

Table 1 sums up and compares the mean scores obtained before and after the training program.

An overall statistically significant improvement is clearly demonstrated by the global score increasing from 68.6±11.7 to 79.3±6.8 (P=0.006), confirming that the EBUS-STAT questionnaire is capable of identifying the improvement in trainees’ skills achieved through the virtual reality simulation. Interestingly, when we compared the values of the subgroups we found no significant differences in the scores assessing the trainees’ practical skills (items 1 to 7) (37.7±4.8 posttraining vs. 35.4±6.9 pretraining; P=0.134), although we did observe a significant increase in the EBUS and CT scan recognition scores (items 8 to 10) (24.9±2.5 posttraining vs. 20.7±4.2 pretraining; P=0.002). To test the masking effect of having a basic knowledge of flexible bronchoscopy on the cumulative score of items 1 to 7, we reevaluated the differences after excluding the first 2 items and found that mean scores of items 3 to 7 were statistically different (45.6±1.7 posttraining vs. 40.6±2.1 pretraining; P=0,03). Similarly, a statistically significant difference was also found between the scores of items 3 to 10 (55.7±3.9 posttraining vs. 48.9±4.0 pretraining; P<0.001). As confirmatory proof, the differences between the scores obtained on items 1 and 2 were compared and, as expected, they were not found to be statistically significant (8.9±0.5 vs. 7.3±1.7; P=0.08).

A comparison of each trainee’s presimulation and postsimulation scores is summarized in Figure 1. As already previously described in the literature, participants with lower scores at the baseline evaluation showed the most marked improvement after the training experience; conversely, the 1-day training course did not determine significant skills improvement in those participants who already had a high level of confidence with the EBUS-TBNA procedure before the test. Figure 2 provides a graphic comparison of the partial scores relating to practical skills (items 1 to 7) (A), theoretical competence (items 8 to 10) (B), and practical skills, excluding those items (1 and 2) evaluating the trainee’s confidence with general flexible bronchoscopy (C and D).

FIGURE 1

FIGURE 1

FIGURE 2

FIGURE 2

We found that EBUS-STAT values above 79 were predictive of poor improvement after simulation training, thus confirming the existence of a ceiling effect in bronchoscopy simulation (Fig. 3). The linear regression model, in fact, showed F(1,13)=36.86, P<0.0001, consistent with the approximated EBUS-STAT pretest value of 79. Details of the model are shown in Table 2.

FIGURE 3

FIGURE 3

TABLE 2

TABLE 2

In particular, pretraining EBUS-STAT values above 79 were associated with a considerable likelihood of remaining unchanged or even getting worse after simulation. In contrast, for each point below the cutoff value of 79 in the baseline EBUS-STAT score, the posttraining score increased by 0.846 points.

Back to Top | Article Outline

DISCUSSION

Our study provides initial evidence of the ability of the EBUS-STAT to identify the improvement effect provided by a single-day high-fidelity simulation training in EBUS-TBNA.

Furthermore, the study also shows that EBUS-TBNA-related knowledge and skills, assessed by EBUS-STAT, is a method that is equally effective in revealing practical and technical improvements also in a group of trainees who are experienced in flexible bronchoscopy; this is achieved despite the fact that the trainee’s performance and response to the questionnaire may be negatively affected by his/her considerable dexterity in flexible bronchoscopy.

Although Davoudi et al8 have already demonstrated the general reliability of the test, the reproducibility of this finding in a subset that is “naive” to EBUS, but does have experience and confidence in flexible bronchoscopy, cannot be taken for granted.

Davoudi and colleagues, in fact, observed that experienced trainees were those showing the worse correlation coefficient when compared with beginners or intermediate practitioners.11 They identified, as the external cause leading to this finding, the general tendency of the self-assessment tool to misclassify the pretest knowledge of the participant, both overestimating or underestimating his/her real, objective, performance. This phenomenon, representing an open and unsolved issue, is well known and described.11

Our study was able to confirm the ability of the EBUS-STAT test to assess performance and reveal the improved skillfulness provided by the virtual reality-based simulation program. A virtual reality-based training model for EBUS-TBNA has already been described and validated by Konge et al.12 However, the generalizability of their model was not tested elsewhere or used for assessment on patients. Moreover, the authors used a different evaluation protocol, one that does not display exactly the same features as EBUS-STAT.

We observed a significant improvement in the global EBUS-STAT score after a single-day theoretical and practical training with virtual reality simulators, confirming previous reports.5 It must be reported, however, that—although very intense—the length of the training we performed was very short and this must be taken into account in evaluating the learning curve of an EBUS procedure, which is generally considerably longer than a single day, even among highly skilled pulmonologists. For this reason, future studies should also investigate the ideal number of consecutive sessions needed to optimize trainees’ learning, as well as to assess their retained knowledge, by repeating the EBUS-STAT questionnaire at longer intervals.

We were also able to confirm the validity of the EBUS-STAT as a mastery test: the ceiling effect—whereby less experienced trainees displayed a greater improvement, whereas those who had scored higher at baseline improved their mean scores less—suggests that the EBUS-STAT can be a potential screening tool capable of identifying the skilled endoscopists who are less likely to benefit from virtual reality simulation, but who are probably ready to raise their training level to real patients. Interestingly, we were able to provide a regression equation that determines which EBUS-STAT score at baseline predicts a significant improvement in the trainee’s confidence with EBUS-TBNA. We are not aware of any other previously published study providing such information. Conversely, the test showed a huge improvement in those trainees belonging to the lowest score group at baseline, confirming that the virtual reality simulation is a valuable, complementary, and not alternative instrument to expedite the learning curve of physicians approaching endobronchial ultrasound. It is well known, in fact, that traditional learning models are conditioned by the availability of patients and by the clinical features of the endoscopic procedure, which are often too challenging for inexperienced trainees and can therefore influence the trainer’s role in supervising and empowering the trainees. Furthermore, simulation-based models make it possible to repeat a specific procedure several times in a friendly and stress-free environment, facilitating in-depth understanding of the procedure and practical skill consolidation.5 It is noteworthy that we were able to identify a potential cutoff value of the EBUS-STAT score able to predict the level of improvement in trainees (very good for baseline scores below 79 and definitely poor for higher values). This finding, if externally validated, might become extremely useful in the future assessment of EBUS-TBNA practitioners, providing an instrument for external and objective certification of dexterity and competence: this will allow for optimization of teaching resources useful in the early recognition of those trainees requiring simulation training before attempting real-life EBUS-TBNA, distinguishing them from those less likely to benefit from simulation training and who are already skilled enough to proceed straight to the real patient.

Interestingly, we lamented an interevaluator judgment heterogeneity that, in a few cases, resulted in a lower score. We attributed this finding to important differences in the familiarity that different evaluators had with the test, which is likely to have had a negative effect in this result.13 This finding makes it unlikely that there was a bias related to evaluators’ judgments being artificially homogenous and suggests that interobserver reliability was maintained, despite a considerable heterogeneity among the supervisors. Furthermore, as suggested by Konge and Annema,14 we did not consider the theoretical part of the assessment tool, as it contains predetermined answers, necessarily resulting in a 100% interrater agreement and causing an artificial increase in the agreement itself.

It is worth highlighting that all approaches based on the use of predetermined items, including ours using EBUS-STAT, suffer from a basic limitation: as it is impossible to repeat the test more than a few times, it is not possible to test multiple procedures, nor to explore the variance in individual performances.

Nonetheless, the EBUS-STAT proved to be extremely precise in detecting ability changes before and after the simulation training. We believe that the test sensitivity could be improved, when applied in a setting of already experienced flexible bronchoscopy endoscopists, by removing the first two items investigating their general anatomic and technical knowledge in endoscopy, as the replies to these items might reduce the discriminative power of the test. However, our observation is far from being conclusive and deserves further studies designed for this aim.

In conclusion, our experience confirms the usefulness of an assessment test such as the EBUS-STAT in detecting trainees’ improvement in performing the EBUS-TBNA procedure after a simulation training course among experienced bronchoscopists. Our data also suggest that the EBUS-STAT could play a role in assessing adequate competence levels after simulation training in novice and intermediate trainees, before allowing them to practice on patients. Conversely, EBUS-STAT may also act as a screening tool to identify already skilled practitioners, who can be allowed to skip simulation and practice directly on the patient.

Our results support the need for a specifically designed study to verify whether the test could gain in accuracy if the first 2 items are eliminated and to confirm that simulation training is of limited use with bronchoscopists who possess previous experience in EBUS-TBNA.

Furthermore, our study adds further evidence in favor of the use of high-fidelity virtual reality simulation in the training for EBUS-TBNA, as a means of speeding up the trainees’ learning curve and reducing the impact of teaching in clinical practice, in terms of both cost savings and guaranteeing safety.

Lastly, we have provided an EBUS-STAT cutoff value potentially useful in discriminating between trainees likely to benefit from virtual reality simulation and those unlikely to do so.

This preliminary finding, although promising, requires further evaluation and external validation before it can become suitable for practical application.

Back to Top | Article Outline

REFERENCES

1. Grosu HB, Iliesiu M, Caraway NP, et al. Endobronchial ultrasound-guided transbronchial needle aspiration for the diagnosis and subtyping of lymphoma. Ann Am Thorac Soc. 2015;12:1336–1344.
2. Wahidi MM, Hulett C, Pastis N, et al. Learning experience of linear endobronchial ultrasound among pulmonary trainees. Chest. 2014;145:574–578.
3. Stather DR, Maceachern P, Chee A, et al. Trainee impact on advanced diagnostic bronchoscopy: an analysis of 607 consecutive procedures in an interventional pulmonary practice. Respirology. 2013;18:179–184.
4. Konge L, Colella S, Vilmann P, et al. How to learn and to perform endoscopic ultrasound and endobronchial ultrasound for lung cancer staging: a structured guide and review. Endosc Ultrasound. 2015;4:4–9.
5. Konge L, Clementsen PF, Ringsted C, et al. Simulator training for endobronchial ultrasound: a randomised controlled trial. Eur Respir J. 2015;46:1140–1149.
6. Stather DR, Chee A, MacEachern P, et al. Endobronchial ultrasound learning curve in interventional pulmonary fellows. Respirology. 2015;20:333–339.
7. Savran MM, Clementsen PF, Annema JT, et al. Development and validation of a theoretical test in endosonography for pulmonary diseases. Respiration. 2014;88:67–73.
8. Davoudi M, Colt HG, Osann KE, et al. Endobronchial ultrasound skills and tasks assessment tool: assessing the validity evidence for a test of endobronchial ultrasound-guided transbronchial needle aspiration operator skill. Am J Respir Crit Care Med. 2012;186:773–779.
9. Bolliger CT, Mathur PN, Beamis JF, et al. European Respiratory Society/American Thoracic Society. ERS/ATS statement on interventional pulmonology. Eur Respir J. 2002;19:356–373.
10. Bronchoscopy International. The Bronchoscopy Education Project (Part II): Endobronchial Ultrasound Bronchoscopy Competence Program. Available at: http://www.aabronchology.org/PDF/bronch_course/. Accessed July 31, 2012.
11. Davis DA, Mazmanian PE, Fordis M, et al. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296:1094–1102.
12. Konge L, Annema J, Clementsen P, et al. Using virtual-reality simulation to assess performance in endobronchial ultrasound. Respiration. 2013;86:59–65.
13. Yudkowsky R, Downing SM, Tekian ADowning SM, Yudkowsky R. Standard setting. Assessment in Health Professions Education. New York, NY: Routledge; 2009:119–148.
14. Konge L, Annema J. Assessment of endobronchial ultrasound-guided transbronchial needle aspiration performance. Am J Respir Crit Care Med. 2013;188:254.
Keywords:

interventional pulmonology; training; EBUS-TBNA; virtual reality simulation; bronchoscopy

Copyright © 2017 Wolters Kluwer Health, Inc. All rights reserved.