Experience With a Simulator-Based Angiography Course for Neurosurgical Residents: Beyond a Pilot Program

Fargen, Kyle M. MD, MPH*; Arthur, Adam S. MD, MPH; Bendok, Bernard R. MD§; Levy, Elad I. MD; Ringer, Andrew MD; Siddiqui, Adnan H. MD, PhD; Veznedaroglu, Erol MD#; Mocco, J MD, MS**

Section Editor(s): Harrop, James S. MD; Bendok, Bernard R. MD

doi: 10.1227/NEU.0000000000000059
Vascular Neurosurgery Simulators

BACKGROUND: Simulation is an increasingly useful means of teaching in the era of duty hour restrictions. Since the completion of our diagnostic cerebral angiography simulator curriculum pilot program, we have performed this resident course at 2 Congress of Neurological Surgeons (CNS) annual meetings with larger participant numbers.

OBJECTIVE: To report the ongoing results of these courses.

METHODS: A 120-minute simulator-based training course was performed at 2 CNS annual meetings. Precourse written and simulator skills assessments were performed, followed by instructor-guided training on an endovascular simulator. Postcourse written and simulator practical assessments were then performed and compared with precourse scores.

RESULTS: Thirty-seven neurosurgery resident participants completed the course module: 16 completed the first course provided and 21 completed the second. Posttest written scores were significantly higher than pretest scores (mean ± SEM, 8.5 ± 0.40.3 vs 4.9 ± 0.3; P < .001). Instructor assessments of practical posttest scores of participants were significantly higher than pretest practical scores for both the CNS 2011 and CNS 2012 groups (P < .001).

CONCLUSION: The expansion of a curriculum-based, cerebral angiography simulator pilot program to trainees through courses at national neurosurgical meetings demonstrated excellent results with significant improvements in written test scores and instructor assessments of participant technical skills. With ever-expanding improvements in simulation technology and realism, simulator training for cerebral angiography may become an integral component of resident training in the future.

ABBREVIATIONS: ACGME, Accreditation Council for Graduate Medical Education

CNS, Congress of Neurological Surgeons

*Department of Neurosurgery, University of Florida, Gainesville, Florida;

Semmes-Murphey Clinic/University of Tennessee, Department of Neurosurgery, Memphis, Tennessee;

§Department of Neurosurgery, Northwestern University, Chicago, Illinois;

Department of Neurosurgery, University at Buffalo, State University of New York, Buffalo, New York;

Department of Neurosurgery, Mayfield Clinic, University of Cincinnati, Cincinnati, Ohio;

#Department of Neurosurgery, Capital Health Institute for Neurosciences, Trenton, New Jersey;

**Department of Neurosurgery, Vanderbilt University, Nashville, Tennessee

Correspondence: J Mocco, MD, MS, Vanderbilt University Medical Center, Department of Neurological Surgery, T-4224 Medical Center, North Nashville, TN 37232-2380. E-mail: j.mocco@vanderbilt.edu

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.neurosurgery-online.com).

Received April 20, 2013

Accepted June 17, 2013

Article Outline

Simulation is an increasingly useful means of teaching in the era of duty hour restrictions. There are concerns that resident physician duty hour regulations imposed by the Accreditation Council for Graduate Medical Education (ACGME) will endanger the training of neurosurgery residents by reducing operative experience.1-3 Simulator-based training may help to address this problem. Currently, simulators play a limited role in neurosurgical education and are restricted to the programs that own simulator devices or to brief courses at meetings. We previously presented a curriculum-based cerebral angiography simulator pilot course as a first step in evaluating the benefits of angiographic simulator training for neurosurgical residents.4 A second simulator-based pilot training course for neurosurgical residents has also been published.5 Although different in course specifics, these pilot programs independently demonstrated the feasibility of such efforts. Additionally, both documented improvements in participant procedural skills by the conclusion of the course on the basis of either objective procedural metrics or instructor-based scoring. Since the completion of our pilot program, we have performed this resident course at 2 Congress of Neurological Surgeons (CNS) annual meetings with larger participant numbers. The purpose of this article is to report the ongoing results of these courses.

Back to Top | Article Outline


This course was performed at the 2011 and 2012 CNS annual meetings. The course was available to all neurosurgical residents on a first-come first-served basis. The course module was designed for a total length of 120 minutes. Each participating course instructor was a neurosurgeon with fellowship training in endovascular surgical neuroradiology. This is shorter in length than the pilot program presented previously4 because the course module performed at the annual meetings did not contain lectures. Course objectives are described in detail elsewhere.4

Back to Top | Article Outline


Vist-C Simulator Systems (Mentice, Evanston, Illinois), Simbionix Systems (Stryker, Kalamazoo, Michigan), and SimSuite Compass (MicroVention, Tustin, California) simulators were used for the 2 courses. Participants had sole access to their own simulators. Each simulator uses catheters and wires that are engaged along internal tracking wheels and introduced through ports, allowing the simulator to capture fine movements both in the plane of insertion and rotationally (Figure). A foot pedal simulates fluoroscopy, and bed position is controlled by a separate joystick (see Video 1, Supplemental Digital Content 1, http://www.youtube.com/watch?v=hspM51Yt-Vg, which demonstrates the use of an angiographic simulator and how it may be used to educate residents the basics of diagnostic cerebral angiography). The 3 simulators are similar in their design, tactile feedback, and imaging projection, providing a uniform teaching environment.

Back to Top | Article Outline

Course Structure

The 2-hour module consisted of several distinct activities: (1) a written pretest (5 minutes in length); (2) a succinct explanation of the simulator device (5 minutes); (3) a skills assessment pretest (10 minutes); (4) a didactic session on cerebrovascular anatomy and angiography techniques (15 minutes); (5) hands-on one-on-one training (70 minutes); (6) a written posttest evaluation (5 minutes); and (7) a skills assessment posttest (10 minutes).

The course began with all participants taking a written test (detailed in the Assessments section below) before starting a brief didactic session. This 15-minute didactic session briefly covered the fundamentals of angiography, including the specifics of brachiocephalic and intracranial vascular anatomy, descriptions of some basic techniques of wire and catheter manipulation, and the principles of fluoroscopy and contrast usage.

Next, each participant took a practical test on the simulator in which they performed diagnostic cerebral angiography on a simulated patient. This component was graded by an instructor, as detailed below. After the practical pretest, participants received 70 minutes of direct one-on-one instruction from a faculty member. During the simulator-based education portion of the course, a total of 7 simulation practice cases were used, with increasingly difficult arch types (2 type I arches, a bovine arch, a type II arch, and 2 type III arches). During this practice session, the faculty member spent the entire time with his or her assigned participant, observing the participant’s technique and providing detailed instruction and guidance. After 90 minutes of this, participants were given a written posttest (identical to the written pretest) and performed a second practical test on the simulator itself (both detailed in the Assessments section below). This completed all educational and assessment portions of the course.

Back to Top | Article Outline


Participants were evaluated with 3 assessment tools. First, participant knowledge was assessed with scored, 12-question, multiple-choice written evaluations testing general principles of angiography. These questions addressed different types of catheters, aortic arches, and angiographic anatomy and included technical questions about proper fluoroscopy, catheter and wire advancement, and safety techniques. Identical written tests were given to each participant at the beginning and end of the course; however, no direct review of the test or its questions was performed during the course, and the participants were not told that they would be receiving the same test at the end of the session. Identical tests were used so that the written test scores could be compared directly.

Second, participants were evaluated in a practical test on the basis of their performance during simulated diagnostic angiography on a single “patient.” Each participant was given 10 minutes to perform a diagnostic angiogram on a “patient” with a type 1 aortic arch and moderate left internal carotid artery stenosis just distal to the common carotid artery bifurcation. Faculty members evaluated the participants on an ordinal scale (0 = very poor and 10 = exceptional) on the basis of their technical skills in catheter navigation, use of fluoroscopy and contrast, speed, and other factors. A validated scale does not exist for measuring endovascular simulator skill; therefore, a scoring system based on both qualitative and quantitative factors, as listed above, was created for our pilot program and has been used since. Qualitative factors assessed included proper use of 2-hand technique, ease with which access was obtained into target vessels, technical control of the catheter and wire, and the occurrence of potentially dangerous maneuvers such as advancing the catheter without leading with the wire. Additionally, the simulator itself recorded critical objective data, including time from catheter insertion to angiogram completion, amount of contrast used, and amount of fluoroscopy time, which the faculty evaluated and incorporated into the subjective assessment. Participants were assessed at 2 separate intervals: during their initial simulated angiography case, which occurred after the introduction to the functional aspects of the simulator itself, but before any formal one-on-one instruction and after completing the didactic session and all one-on-one faculty-participant simulator practice sessions.

Back to Top | Article Outline



Thirty-seven neurosurgery resident participants completed the course module: 16 completed the first course provided and 21 completed the second.

Back to Top | Article Outline

Main Results

Written pretest and posttest scores and instructor scoring are given in the Table. Posttest written scores were significantly higher than pretest scores (mean ± SEM, 4.9 ± 0.3 vs 8.5 ± 0.4; P < .001). Instructor practical posttest scores of participants were significantly higher than pretest practical scores for both the CNS 2011 and CNS 2012 groups (P < .001). Posttest practical scores were significantly higher than pretest practical scores at the CNS 2011 (n = 21; mean ± SEM, 7.3 ± 0.3 vs 3.3 ± 0.5; P < .001). Similarly, posttest practical scores were significantly higher than pretest practical scores at the CNS 2012 (n = 16; mean ± SEM, 6.7 ± 0.5 vs 2.7 ± 0.5; P < .001).

Back to Top | Article Outline


Key Results

This cerebral neuroendovascular simulation curriculum continues to show a benefit for neurosurgical residents. High-throughput 120-minute courses at large neurosurgical meetings achieved improvements in both written test scores and subjective assessments of angiographic technical skill. Among 37 resident participants at 2 national meetings, written test scores and instructor-based assessments improved significantly during the 2-hour course. These data support the use of an endovascular angiographic simulator curriculum for teaching basic angiographic skills to residents in training.

The use of simulation in neuroendovascular training has risen sharply in the last decade in concert with advances in simulator technology and improvements in simulator realism. Augmented reality simulation, which intertwines virtual reality with actual catheters and wires, allows improved haptic feedback and therefore more closely simulates actual clinical procedures. Modern angiographic simulators such as those used for these courses use actual catheters and wires and simulate the movements of these devices within the arterial system based on operator manipulations. When these instruments are advanced, withdrawn, or turned, the simulator calculates the effect of these maneuvers within the cervicocerebral arterial system and then displays the calculated action of the device within a simulated fluoroscopic view of the patient. In some simulators, fluid dynamics and arterial pulsations are factored into consideration, heightening the realism of the experience. The simulators used in this study are electronic, do not use actual fluids, and are unable to simulate arterial pulsations. Flow models are simulators with transparent plastic tubing simulating the vasculature that use a water pumping system to simulate arterial pulsations. Currently, flow models are small and allow the trainee to use microcatheters for the deployment of actual stents or coils. This allows learners to directly visualize stent or coil deployment in a simulated environment with fluid pulsations. These simulators do not use simulated fluoroscopy and do not allow practice of basic endovascular catheter and wire skills. Currently, larger simulation devices have been developed that replicate aortic and cervical vasculature, but they are currently not available for routine use. As a whole, current simulators remain far from accurately replicating the angiography suite because they lack femoral access simulation, biplanar fluoroscopy, flushing techniques, accurate haptic feedback based on instrument tension within the arterial system, and significant anatomic or procedural variability as seen in real patients. As technology continues to advance, we are likely to see each of these deficiencies addressed in future simulation technologies, creating a more realistic virtual patient environment.

The correlation between clinical endovascular experience and simulator technical skills has been demonstrated in a number of studies.6-13 This association suggests that modern simulators are realistic enough that experience obtained in clinical practice translates to better simulator skills. However, simulation technology as a training tool for novice learners to improve actual procedural technical skills remains poorly documented and unvalidated. The evidence that the experience gained on simulator technology will actually translate to improved procedural skills on real patients in the angiography suite is limited. To date, 1 small study has suggested that skills gained on a simulator may lead to improved clinical skills.11 Another study has suggested that skilled independent practitioners continue to benefit from simulation courses even after years of clinical practice.14 The paucity of data in this regard is secondary to both the considerable difficulty in designing adequate studies to test this hypothesis and the number of participants required to demonstrate the putative benefit of simulation-based education.

As simulation-based education continues to evolve, a thorough understanding of the strengths and limitations of simulation devices will become necessary so that the implementation of simulator-based training into residency education can be optimized. Although the benefits of simulator training are relatively obvious, the limitations of simulator training may be less apparent. The greatest current limitation of simulator-based education is the lack of evidence supporting the translation of simulator skills to actual clinical skills. If trainees are unable to gain actual procedural skills from the teaching tool that is being used, the simulation will be of no benefit. Simulation technologies are not yet at a point where they may accurately replicate human anatomy or the exact tactile feedback obtained from using actual instruments. Furthermore, there remain considerable technical or financial challenges limiting the application of such devices to everyday practice. Cost may be prohibitive because most simulators cost thousands of dollars. In addition, simulators rarely replicate anatomic variability or pathology, limiting the trainee to repetitive study on basic, uncomplicated scenarios. Importantly, most simulators do not provide a means of generating important procedural complications for the learner to recognize, react to, and then successfully navigate. The risk-free simulation environment provided by these simulators may fail to encourage the necessary caution required during actual patient procedures because poor technique or errors do not result in real clinical consequences. However, it is exactly this risk-free environment that allows the benefit of education without patient injury, which is so beneficial to simulator-based training. Finally, the optimal length of simulator training, the point during training when it is most beneficial, and the types of simulations most helpful to trainees have yet to be defined.

The exposure to neuroendovascular surgery and cerebral angiography among neurosurgical residency programs appears to be suboptimal. Of the > 100 neurosurgical training programs in the United States, only a minority offer endovascular surgical neuroradiology fellowships, of which only a few are accredited by the ACGME. The ACGME 2009-2010 National Data Report for Neurological Surgery, a yearly document including averages of neurosurgical resident case logs across the country, reported a mean of 26.1 cases with a median of 0 cases, indicating that most residents have little or no experience with angiography.15 In addition, a survey of practicing neurosurgeons applying for the oral board identified endovascular procedures as an area of inadequate training.16 Given the importance of endovascular diagnosis and treatment in the field of neurosurgery, we believe it is imperative that resident exposure to such procedures is increased. However, expanding education in this realm is increasingly challenging, considering the continued expansion of neurosurgical specialty training in concert with the progressive reduction in the hours residents are allowed to work. Therefore, the onus is on neurosurgical educators to develop new and more efficient means of educating our residents despite less total time for learning. Simulation is likely to play a pivotal role in the modern methods of resident education.17

Back to Top | Article Outline


The small number of participants, lack of a control group, and inability to resolve the relationship between simulator skills and actual procedural skills represent the major limitations of this study. Furthermore, the lack of a validated endovascular skills scoring scale and the lack of instructor blinding during assessments may introduce bias. Finally, one-on-one instructor education may not be feasible in larger resident courses in the future.

Back to Top | Article Outline


The results of this study are pertinent and generalizable to most neurosurgical residents and training programs.

Back to Top | Article Outline


The expansion of a curriculum-based, cerebral angiography simulator pilot program to trainees through courses at national neurosurgical meetings demonstrated excellent results. There were significant improvements in written test scores and instructor assessments of participant technical skills. With ever-expanding improvements in simulation technology and realism, simulator training may become an integral component of resident training in the future. Further study into the role, benefits, and limitations of neuroendovascular simulators in neurosurgical education is necessary before angiographic simulation technology will be accepted and formally integrated into residency training.

A podcast related to this article can be accessed online (http://links.lww.com/NEU/A576).

Back to Top | Article Outline


Dr Bendok receives research grants from MicroVention, National Institutes of Health, and Erica Keeney Foundation. Dr Levy has received a research grant from Boston Scientific Corp; has received research support from Codman & Shurtleff, Inc and ev3/Covidien Vascular Therapies; is a shareholder/has ownership interest in Intratech Medical Ltd and Mynx/Access Closure; has been a consultant for Codman & Shurtleff, Inc, TheraSyn Sensors, Inc, ev3/Covidien Vascular Therapies, and Blockade Medical LLC; has received honoraria from Boston Scientific Corp; has received other financial or material support from Abbott Vascular and ev3/Covidien; and has ownership interest in Intratech Medical Ltd and Mynx/Access Closure. Dr Ringer serves as a consultant for ev3/Covidien and Stryker. Dr Siddiqui has received research grants from the National Institutes of Health (coinvestigator: NINDS 1R01NS064592-01A1, Hemodynamic Induction of Pathological Remodeling Leading to Intracranial Aneurysms) and the University at Buffalo (Research Development Award); holds financial interests in Hotspur, Intratech Medical, StimSox, Valor Medical, and Blockade Medical; serves as a consultant to Codman & Shurtleff, Inc, Concentric Medical, Covidien Vascular Therapies, GuidePoint Global Consulting, Penumbra, Inc, Stryker Neurovascular and Pulsar Vascular; belongs to the speakers’ bureaus of Codman & Shurtleff, Inc, and Genentech; serves on National Steering committees for the Penumbra, Inc. 3D Separator Trial and Covidien SWIFT PRIME Trial; serves on an advisory board for Codman & Shurtleff and Covidien Vascular Therapies; and has received honoraria from American Association of Neurological Surgeons’ courses, Annual Peripheral Angioplasty and All That Jazz Course, Penumbra, Inc, and Abbott Vascular and Codman & Shurtleff, Inc for training other neurointerventionists in carotid stenting and for training physicians in endovascular stenting for aneurysms. Dr Siddiqui receives no consulting salary arrangements. All consulting is per project and/or per hour. Dr Veznedaroglu serves as a consultant for Codman and Stryker. Dr Mocco serves as a consultant for Lazarus Effect, Reverse Medical, Pulsar, and Edge Therapeutics. He has investor interests in Blockade Medical. Dr Arthur serves as a consultant for Covidien, Johnson & Johnson, Stryker, and Terumo-Microvention. He receives research support from Siemens and Terumo-Microvention. The other authors have no personal financial or institutional interest in any of the drugs, materials, or devices described in this article.

Back to Top | Article Outline


1. Grady MS, Batjer HH, Dacey RG. Resident duty hour regulation and patient safety: establishing a balance between concerns about resident fatigue and adequate training in neurosurgery. J Neurosurg. 2009;110(5):828–836.
2. Fargen KM, Chakraborty A, Friedman WA. Results of a national neurosurgery resident survey on duty hour regulations. Neurosurgery. 2011;69(6):1162–1170.
3. Jagannathan J, Vates GE, Pouratian N, et al.. Impact of the Accreditation Council for Graduate Medical Education work-hour regulations on neurosurgical resident education and productivity. J Neurosurg. 2009;110(5):820–827.
4. Fargen KM, Siddiqui AH, Veznedaroglu E, Turner RD, Ringer AJ, Mocco J. Simulator based angiography education in neurosurgery: results of a pilot educational program. J Neurointerv Surg. 2012;4(6):438–441.
5. Spiotta AM, Rasmussen PA, Masaryk TJ, Benzel EC, Schlenk R. Simulated diagnostic cerebral angiography in neurosurgical training: a pilot program. J Neurointerv Surg. 2013;5(4):376–381.
6. Bech B, Lönn L, Falkenberg M, et al.. Construct validity and reliability of structured assessment of endovascular expertise in a simulated setting. Eur J Vasc Endovasc Surg. 2011;42(4):539–548.
7. Van Herzeele I, Aggarwal R, Choong A, Brightwell R, Vermassen FE, Cheshire NJ. Virtual reality simulation objectively differentiates level of carotid stent experience in experienced interventionalists. J Vasc Surg. 2007;46(5):855–863.
8. Van Herzeele I, Aggarwal R, Malik I, et al.. Validation of video-based skill assessment in carotid artery stenting. Eur J Vasc Endovasc Surg. 2009;38(1):1–9.
9. Dayal R, Faries PL, Lin SC, et al.. Computer simulation as a component of catheter-based training. J Vasc Surg. 2004;40(6):1112–1117.
10. Tedesco MM, Pak JJ, Harris EJ Jr, Krummel TM, Dalman RL, Lee JT. Simulation-based endovascular skills assessment: the future of credentialing? J Vasc Surg. 2008;47(5):1008–1001; discussion 1014.
11. Chaer RA, Derubertis BG, Lin SC, et al.. Simulation improves resident performance in catheter-based intervention: results of a randomized, controlled study. Ann Surg. 2006;244(3):343–352.
12. Dawson DL. Training in carotid artery stenting: do carotid simulation systems really help? Vascular. 2006;14(5):256–263.
13. Botden SM, de Hingh IH, Jakimowicz JJ. Meaningful assessment method for laparoscopic suturing training in augmented reality. Surg Endosc. 2009;23(10):2221–2228.
14. Van Herzeele I, Aggarwal R, Neequaye S, et al.. Experienced endovascular interventionalists objectively improve their skills by attending carotid artery stent training courses. Eur J Vasc Endovasc Surg. 2008;35(5):541–550.
15. Tomsick T, Broderick J, Carrozella J, et al.. Revascularization results in the Interventional Management of Stroke II trial. AJNR Am JNeuroradiol. 2008;29(3):582–587.
16. Mazzola CA, Lobel DA, Krishnamurthy S, Bloomgarden GM, Benzil DL. Efficacy of neurosurgery resident education in the new millennium: the 2008 Council of State Neurosurgical Societies post-residency survey results. Neurosurgery. 2010;67(2):225–232; discussion 232-223.
17. Bath J, Lawrence P. Why we need open simulation to train surgeons in an era of work-hour restrictions. Vascular. 2011;19(4):175–177.

Angiography; Endovascular; Neurosurgery; Resident education; Simulator

Supplemental Digital Content

Back to Top | Article Outline
Copyright © by the Congress of Neurological Surgeons