1. Introduction
The development of necessary technical motor and spatial skills is 1 cornerstone of clinical training, and unsurprisingly, a significant obstacle to qualifying as a competent interventional radiologist. Complex, minimally invasive procedures, such as endovascular coiling of cerebral aneurysms, require extensive procedural training which is increasingly limited by restricted working hours,[1,2] elevated operating costs[3] and shortened diagnostic exposure.[4,5] These pressures, compounded by the push for competency-based medical education[6] and ever-rising standards of care, have made traditional apprenticeship training methods less effective.[7]
Simulation is offered as a potential solution to training concerns, however, despite leaps in technological advancements, its adaptation has been limited. Validation studies have provided evidence for increased training efficacy using simulation,[8] however, the development has faced challenges in explaining how it can replace the current gold standard - learning on patients. Some of the highest fidelity simulators on the market, including the VIST-C simulator system by Mentice and the ANGIO Mentor by Simbionix, have made strong advances in replicating the haptics and procedural dynamics of cerebral angiography (CA) to bridge this gap. Research in simulation-based angiography training has shown to improve some aspects of technical performance, such as fluoroscopy use[9,10] and fluidity of motion,[11,12] however, the effectiveness of its integration can be improved with a better understanding of the core skills it impacts.
Diagnostic angiography is used to endovascularly locate the aneurysm and identify its risk for hemorrhage and need for intervention. Subsequently, if necessary, based on the assessment, interventional angiography (IA) allows interventionalists to treat the aneurysm, usually through the deposition of electrolytic coils which fill the space to reduce blood (Fig. 1). In the clinical setting, aneurysm coiling involves a wide array of technical skills necessary for successful intervention. Finely tuned motor skills allow interventionalists to manipulate endovascular wires while matured spatial skills allow them to interpret 2-D imaging necessary for navigation. During aneurysm intervention, careful placement, electrolytic detachment and stabilization of coils requires a firm grasp of the tool’s mechanical dynamics.[13] The ability to mentally rotate fluoroscopic images allow not only for the accurate navigation of endovascular environments, but also for the appropriate placement of coils within a 3D space. As a result, mental rotation tests are administered in conjunction with technical surgical assessments to help explain performance differences and training needs.
;)
Figure 1.: A microcatheter (blue) is placed within the aneurysm for secure release of electrolytically detached coils (silver) which stabilize the aneurysm and prevent hemorrhage.
The complexity of the coiling process has left the procedure without standardized performance standards, and the measure of its success in the hands of the interventionalist. A thorough analysis of trainable technical skills in coiling would provide some clarity and direction, which is imperative for the creation of standardized performance assessments and thresholds.
Technical skills in aneurysm coiling are the backbone of the graded success of the procedure and the resulting patient outcomes. Procedural time is a basic marker used to quantify performance in this field as it suggests higher clinical competency and results in lower operating costs. [9,14] Of course, procedure times must be examined in conjunction with the accuracy and error rates of the task. Subsets of skills that generate faster interventionalists are not well studied however and could explain which competencies are necessary for such improvement. Since the placement of coils in the aneurysm space is such a careful and repetitive task, small improvements in each cycle can accumulate into larger gains.
The appropriate number of coils deposited into the aneurysm has a direct relationship to the duration of the coiling process,[15] however, it is not currently measured as a marker for operating quality. Furthermore, time delays associated with exchange of incompatible coils can further inform these markers. Prevention of adverse outcomes, such as coil protrusion[16] and aneurysm perforation,[17] can also grade operating quality through improved patient outcomes.
A query still stands: is it possible to make implementation of simulation-based training more purposeful by better understanding which technical skills it most helps develop? The objective of the study was to quantify the degree of core technical skill development in novice cerebral aneurysm coiling trainees when exposed to low risk, high frequency simulation-based training. The aim was to clarify technical skills involved in the intervention and quantify performance improvements in self-regulated simulation practice.
2. Methods
2.1. Participants
To ensure training novelty, only participants with minimal or no knowledge of endovascular skills and basic vascular background were included. The Schulich School of Medicine and Dentistry Hippocratic Council, a student-run society for medical students at the University of Western Ontario, London, Ontario, Canada, distributed a call-out for participation via an internal messaging system. A total of 20 students replied to the flyer and initiated training in the study. At the end of the 4-month study period, 12 of the initial 20 participants completed the all phases of the study, with 8 participants voluntarily dropping from the commitment. This study was approved by and completed in accordance with the Western University Health Science Research Ethics Board.
2.2. Participant orientation
All participants completed an orientation module that included study background information and demographic survey; Timed mental rotations test; Anatomy and angiography learning module and knowledge test; And CA and simulator tutorial.
- The demographic survey included questions about gender, age, handedness, surgical experience, and CA familiarity. This survey was used to identify and exclude participants who did not meet the requirements of inexperience for participation.
- A 2-part, timed mental rotations test (MRT) (Vandenberg and Kuse,1978) was distributed upon completion of the intake demographic survey. Each 3-minute, 12-question part required participants to identify 2 rotated matches out of 4 options to a sample image. Participants who scored 50% or lower (=<12/24) were considered low MRT, and those who scored above 50% (>12/24) were considered high MRT for the performance comparison analysis.
- All participants were given free time to individually review a standardized anatomy and angiography learning module. This training package included information on procedurally-relevant vascular anatomy, including names and positions of vessels from the aorta to the circle of Willis, aneurysm locations and treatments, and CA techniques and imaging. Once participants felt comfortable with the reviewed material, an untimed 10-question anatomy and angiography test was distributed. A score of 80% was required to be included in the study. In the result of a failing score, the participant would be required to restart the learning module.
- An instructional simulation booklet was created for participants to ensure consistency in familiarity with the equipment. The booklet included information about all accessible buttons on the physical simulation console, visual layout of the monitors, and functionality of pedals. Participants were also provided with a guided tour through the simulation software, including information available in the booklet.
2.3. Simulator type
The ANGIO Mentor Express (3D SYSTEMS) was used for all training and testing protocols listed in the study. This haptic virtual reality simulator provides tactile and visual feedback representative of the environment found in the Angio Suite, including imaging monitors, pedals and endovascular access site (Fig. 2).
Figure 2.: The ANGIO Mentor Express simulation set up provides the haptic representation of endovascular access (simulator console, left), imaging monitors (right), and imaging pedals (not pictured).
2.4. Simulation environment
The simulation equipment was set up in a private office within the university hospital to minimize distractions, such as noise. The simulator and associated monitors were set up on an office desk which could be operated in a seated position, as intended and prepared by simulation manufacturer. Foot pedals were available underneath the desk. The researcher was seated behind the participant during data collection.
2.5. Simulated case
ANGIO Mentor cases included in the license were used to create the training and assessment scenarios. The cases include a patient file with basic clinical information regarding suspected aneurysm location and associate complications. A left-sided internal carotid artery aneurysm case (Case File #1) was used for the training scenarios and a right-sided middle cerebral artery aneurysm case (Case File #5) was used for the assessment scenarios. These cases were chosen to ensure that training scenarios were simpler and used different vascular anatomy compared to the assessment scenarios.
A booklet with the procedural outline, including tools, actions, and objectives, was available to guide case completion. An endovascular guidewire and catheter were set up partially inserted into the simulator. During IA, the insertion of the guiding catheter into the simulator was assisted by the researcher as it was not required during diagnostic angiography (DA).
Simulated scenarios commenced with a prepared virtual patient in supine position and imaging focused on the thoracic aorta where inserted tools are first simulated. Following the case outline and instructional booklet, participants navigated through the endovascular environment by inserting and rotating wires and catheter within the simulator console.
Assistance, such as answering questions and giving direction, was only included during the training scenario. The assessment scenario progress and completion were independently guided by the participant.
2.6. Instructional design
Two separate training-assessment stages were used for data collection: DA and; Aneurysm coiling. Diagnostic angiography stage included 8 weekly sessions, each comprised of a training and subsequent assessment scenario. Upon the completion of all DA sessions, the aneurysm coiling stage commenced with an introduction to coiling module, followed by 6 weekly sessions using the training/assessment paradigm described previously. In all sessions, each set of cases could be performed only once, and no limit to case completion length was set. Due to the lack of availability of cases, there was no variation in clinical or scenario context between sessions.
A basic performance overview was made available to the participants upon completion of the assessment scenario, which included procedural pace, aneurysm filling density, coil deployment quantities, imaging frequency and contrast use. A brief informal discussion about the statistics took place with the researcher at this time. A video recording of scenario was captured but was not used during debriefing.
2.7. Outcomes
All technical performance data was extracted from the simulation software. Similar to other studies assessing the simulator, an overall procedural speed/pace was used as a superficial marker for performance improvement. In addition, a variety of indices of performance were also used to assess, on a deeper level, the potential reasons for changes seen in procedural pace. These indices included number of coils used to fill the aneurysm, including number of coils successfully deposited and coils discarded due to incompatibility. Concurrently, aneurysm packing density was used to ensure sufficient aneurysm volume was filled with coils. Major errors, such as coil protrusion into parent artery and aneurysm wall perforation, were quantified alongside other markers.
During the DA stage, only procedural pace was assessed. This data was used to ensure participants had necessary baseline performance to proceed to IA.
3. Results
The Statistical Package for Social Sciences (SPSS 18.0; SPSS Inc, Chicago, Illinois) was used to analyze the data. A 1-way repeated measure analysis of variance (ANOVA) was used for procedural time, coil use efficiency, and aneurysm perforation rate. A 2-way repeated measures ANOVA was used for MRT comparisons.
All participants successfully passed the qualifying anatomy and angiography module on the first attempt. The following subsections describe the different aspects of performance on the overall procedure.
3.1. Overall pace
Similar to results seen in other studies assessing overall simulation-based performance, [9,10,18,19] DA procedural time dropped significantly (P < .05) over a span of 8 training sessions (Fig. 3). After reaching a procedural plateau in the DA stage, participants had high procedural times and standard deviation at the first IA session, also seen at the first DA training session. Participants completed the first full IA1 session in an average of 42 minutes and improved to 24 minutes by the last session IA6 (P < .05) (Fig. 4).
Figure 3.: A significant decrease in procedural time was observed in the diagnostic angiography sessions, consistent with results seen in previous studies. Concurrently, a plateau of performance is observed between DA6-DA8. DA = diagnostic angiography.
Figure 4.: A significant decrease in procedural time is seen in 6 sessions of simulation-based training and testing. Slower novice performers (3rd quartile minus median = red) improve their pace to cluster with the higher performance group (1st quartile minus median = green) by the end of training.
3.2. Coiling success
At IA1, an average of 12.45 coils were used to fill the aneurysm completely and 2.64 coils were being discarded due to poor fit, resulting in an 82% success rate. At the completion of the study (IA6), an average of 11.08 coils were needed to fill the aneurysm and only 1.46 coils were being wasted, resulting in an 88% success rate (Fig. 5).
Figure 5.: The number of coils used to fill the aneurysm decreased from 15 coils at the first interventional angiography session to 12.5 by the last. This reduction, although not statistically significant, may have been inclusive of reductions in coils waste and coils packed.
Participants were able to maintain a 30% packing density in the aneurysm space throughout all sessions, consistent with density requirements in a real clinical case.
3.3. Coiling errors
Participants improved placing coils within the aneurysm boundaries without protrusions into the parent artery, with 4 recorded protrusions at IA1 (2.5%/case) and no protrusions at IA6 (Fig. 6).
Figure 6.: The total number of protrusions of coil out into the parent artery reduced from 4 across all participants to 0 by the end of the study.
Aneurysm perforation rates maintained at 77% for the first 4 sessions (IA1–4), but improved to 54% by IA6. By the completion of the study, only 46% of participants were successfully coiling the aneurysm without perforating the vascular wall during intervention (Fig. 7).
Figure 7.: The rate of aneurysm perforations reduced from 77% at IA1 to 54% at IA6. IA = interventional angiography.
3.4. Spatial ability
Low MRT individuals performed significantly (P < .05) slower than high MRT individuals, starting aneurysm coiling in IA1 at 22 minutes and IA6 at 13 minutes, compared to high MRT at 12 minutes and 7 minutes, respectively (Fig.8).
Figure 8.: The time to aneurysm was significantly lower for high MRT individuals compared to low MRT individuals on most sessions. This result is consistent with previous studies showing a performance advantage novice with high MRT scores have on spatially complex tasks, such as angiography. MRT = mental rotation test.
4. Discussion
4.1. Technical skills in coiling
Technical skills necessary for minimally invasive procedures are incredibly fine-tuned and precise, creating the need for rigorous hands-on training. The results in this study advocate for the potential of high-fidelity simulation training to offset the duration of introductory practice in the operating room. As seen in previous work, the procedural pace is markedly improved within the simulation environment and may be associated with increased performance in key high-risk maneuvers during coiling. Since aneurysm coiling uses progressively smaller coils to fill the space, the ability of novices in this study to maintain filling capacity while reducing the number of coils inserted and discarded implies a fine proficiency in estimating the needs of the aneurysm environment. At an average of 11 coils used to fill the narrow 9 mm aneurysm, their performance in this regard falls within the range of proficient interventionalists, who average 8.2 ± 3.8 coils/aneurysm of this scale (8.6 +/− 1.6 mm). The novices ability to gauge the 30% filling setpoint correctly, which clinically is seen as a sufficiently filled aneurysm, reinforces their apparent technical skill maturity.
The decreasing rate of committed coiling errors observed throughout the study was encouraging, however, it is hard to translate these results to clinical training as these incidence rates would not be permissible in the Angio Suite. The elimination of accidental coil protrusions into the parent artery from 0.33 rate seen in IA1 indicates an improved ability to manipulate the microcatheter to better position the expanding coils. Concurrently, the decreasing rates of aneurysm wall perforation are encouraging, but the residual error rate may be limited by the self-regulated practice established in the methodology and may require guided debriefing with an expert in order to improve significantly.
4.2. Role of MRT
Spatial ability has been thoroughly studied and correlated with novice surgical performance. [20,21] The markedly improved pace individuals with high MRT were able to achieve establishes MRT as another potential marker for assessing trainee needs. Future interventionalist fellows with low MRT may need modifications to the quantity and nature of the simulation-based training that is offered in order to maintain the high level of clinical performance expected in the Angio Suite.
4.3. Clinical transfer potential
Due to the potential severity of clinical errors in cerebral aneurysm coiling, it is difficult to assess the transfer potential of these technical skills in the manner in which they were trained in simulation. However, based on improvements seen in this study and the established validity of this simulator, clinical interventionalist fellows may be able to accelerate their progression towards clinical competence and seniority with a supplemental simulation-based training program.
5. Future directions
This study focused primarily on simulation-based skills acquisition and testing in novices outside the specialty, and would benefit from further investigation in a clinical setting. Primarily, it is important to identify which of the identified markers of coiling performance can be sensibly assessed in the Angio Suite and whether they can be applied as standardized measures of performance. A thorough standardized assessment of guided coiling tutorials with junior fellows including regular practice, formalized debriefing sessions, and translational technical assessments would complement this data and provide a clinical transfer pathway for future programs. Translating the skills learned in simulation to the clinical environment would support the value and importance of simulation in this specialty. Moreover, it would provide definitive clarity on the true utility of ever-advancing simulation-based tools, and their most efficacious use in clinical training.
Author contributions
Conceptualization: Oleksiy Zaika, Roy Eagleson, Sandrine de Ribaupierre.
Data curation: Oleksiy Zaika.
Formal analysis: Oleksiy Zaika, Mel Boulton, Roy Eagleson, Sandrine de Ribaupierre.
Funding acquisition: Oleksiy Zaika, Roy Eagleson, Sandrine de Ribaupierre.
Investigation: Oleksiy Zaika, Mel Boulton, Roy Eagleson, Sandrine de Ribaupierre.
Methodology: Oleksiy Zaika, Mel Boulton, Roy Eagleson, Sandrine de Ribaupierre.
Project administration: Oleksiy Zaika, Sandrine de Ribaupierre.
Resources: Oleksiy Zaika.
Software: Sandrine de Ribaupierre.
Supervision: Mel Boulton, Roy Eagleson, Sandrine de Ribaupierre.
Validation: Oleksiy Zaika, Mel Boulton.
Visualization: Oleksiy Zaika.
Writing – original draft: Oleksiy Zaika.
Writing – review & editing: Oleksiy Zaika, Mel Boulton, Roy Eagleson, Sandrine de Ribaupierre.
References
[1]. Okuda Y, Bryson EO, Demaria S, et al. The utility of simulation in medical education: what is the evidence? J Med. 2009;76:330–43.
[2]. Dawson S. Procedural simulation: a primer abbreviation: FDA Food and Drug Administration. J Vasc Interv Radiol. 2006;17:205–13.
[3]. Bridges M, Diamond DL. The financial impact of teaching surgical residents in the operating room. Am J Surg. 1999;177:28–32.
[4]. Gould DA, Kessel DO, Healey AE, et al. Simulators in catheter-based interventional radiology: training or computer games? Clin Radiol. 2006;61:556–61.
[5]. Gould DA. Interventional radiology simulation: prepare for a virtual revolution in training. J Vasc Interv Radiol. 2007;18:483–90.
[6]. Chetlen AL, Mendiratta-Lala M, Probyn L, et al. Conventional medical education and the history of simulation in radiology. Acad Radiol. 2015;22:1252–67.
[7]. Napolitano LM, Savarise M, Paramo JC, et al. Are general surgery residents ready to practice? A survey of the American college of surgeons board of governors and young fellows association. J Am Coll Surg. 2014;218:1063–1072.e31.
[8]. Nguyen N, Eagleson R, Boulton M, et al. Realism, criterion validity, and training capability of simulated diagnostic cerebral angiography. Stud Health Technol Inform. 2014;196:297–303.
[9]. Zaika O, Nguyen N, Boulton M, et al. Evaluation of user performance in simulation - Based diagnostic cerebral angiography training. Stud Health Technol Inform. 2016;220:465–8.
[10]. Kim AH, Kendrick DE, Moorehead PA, et al. Endovascular aneurysm repair simulation can lead to decreased fluoroscopy time and accurately delineate the proximal seal zone. J Vasc Surg. 2015:251–8.
[11]. Estrada S, Malley MKO. On the Development of Objective Metrics for Surgical Skills Evaluation Based on Tool Motion.
[12]. Estrada S, Duran C, Schulz D, et al. Smoothness of surgical tool tip motion correlates to skill in endovascular tasks. 2016:1–13.
[13]. Zanaty M, Chalouhi N, Tjoumakaris SI, et al. Endovascular management of cerebral aneurysm: review of the literature. Transl Stroke Res. 2014;5:199–206.
[14]. Crofts TJ, Griffiths JM, Sharma S, et al. Surgical training: an objective assessment of recent changes for a single health board. BMJ. 1997;314:891–5.
[15]. Vanzin JR, Abud DG, Rezende MTS, et al. Number of coils necessary to treat cerebral aneurysms according to each size group: a study based on a series of 952 embolized aneurysms. Arq Neuropsiquiatr. 2012;70:520–3.
[16]. Doerfler A, Wanke I, Goericke SL, et al. Endovascular Treatment of Middle Cerebral Artery Aneurysms with Electrolytically Detachable Coils. Available at:
www.ajnr.org. [access date June 28, 2019].
[17]. Kallmes DF, Cloft HJ. PERSPECTIVES Hospitalization costs for endovascular and surgical treatment of ruptured aneurysms in the. 2012:1037–40.
[18]. Spiotta A, Rasmussen P, Masaryk T, et al. P-008 Simulated diagnostic cerebral angiography in neurosurgical training: a pilot program. J Neurointerv Surg. 2011;3:A19–A19.
[19]. Willaert WIM, Aggarwal R, Daruwalla F, et al. Simulated procedure rehearsal is more effective than a preoperative generic warm-up for endovascular procedures. Ann Surg. 2012;255:1184–9.
[20]. Keehner MM, Tendick F, Meng MV, et al. Spatial ability, experience, and skill in laparoscopic surgery. Am J Surg. 2004;188:71–5.
[21]. Clem DW, Donaldson J, Curs B, et al. Role of spatial ability as a probable ability determinant in skill acquisition for sonographic scanning. J Ultrasound Med. 2013;32:519–28. Available at:
http://www.jultrasoundmed.org/content/32/3/519.long.
