Schirmer, Clemens M. MD, PhD*,‡; Elder, J. Bradley MD§; Roitberg, Ben MD¶; Lobel, Darlene A. MD‖
Section Editor(s): Harrop, James S. MD; Bendok, Bernard R. MD
Virtual reality (VR)--based simulators offer an engaging method to complement standard neurosurgical resident training techniques. Such technology provides trainees with the opportunity to rehearse commonly encountered procedures outside a patient care setting, providing extended opportunities to enhance surgical skills and to improve patient safety profiles. The Accreditation Council for Graduate Medical Education has instituted a requirement to incorporate simulation-based training into general surgery training programs.1 Additionally, evaluation of resident skills using simulators has been proposed as the basis for credentialing for certain procedures such as carotid angioplasty and stenting.2 Even though the use of simulators has previously been identified as part of the educational agenda for neurosurgical education,3 the complex nature of neurosurgical procedures and the need for simulators that can provide essential haptic feedback have limited the implementation of simulation-based technologies into neurosurgical resident training programs. Recent technological advances in simulator development, combined with a greater understanding of how best to integrate simulator-based instruction into formal curricula, have led the Congress of Neurological Surgeons (CNS) to develop a neurosurgical simulation training program to enhance resident education in the management of neurosurgical disorders.
Ventriculostomy placement is among the most commonly encountered procedures in resident training, typically learned in the first few months of postgraduate training and often performed unsupervised thereafter. Often, placement of a ventriculostomy is taught to junior residents by more senior residents in an apprenticeship model during a neurosurgical emergency, causing significant obstacles to meaningful learning resulting from the stress of an emergency situation combined with minimal prior hands-on experience, which may increase the risk of surgical complications. Simulation-based training for ventriculostomy placement early in residency training may improve resident skills and patient outcomes by transferring the learning curve from the operating room to the simulation center, a shift not unprecedented in procedure-based medical specialties.2 Both physical and VR-based ventriculostomy models have been developed.4 Early validation data for the ImmersiveTouch virtual ventriculostomy model suggest that improved rates of ventricular cannulation were observed after use of the VR-based simulator.5
The CNS has developed a trauma module as part of the resident simulation curriculum that focuses on teaching skills necessary to performance of the most common surgical procedures in the treatment of neurotrauma-based injuries. This module was presented as part of the CNS Resident Simulation Symposium at the 2011 and 2012 CNS annual meetings. This article analyzes the deployment, performance, and validity of a VR-based ventriculostomy simulator presented as a component of the trauma module, with structured surveys, didactic and hands-on instruction, and written and practical tests.
The trauma module of the CNS Resident Simulation Symposium was developed using a curriculum-based approach to educate neurosurgical trainees in skills necessary for the management of traumatic brain injury. Simulators incorporated into this module were chosen to meet educational goals. Didactic training and hands-on training were integrated into the session. Validation of the simulators and course design were assessed with well-defined evaluation tools.
Choice of Simulators
The CNS trauma simulation team conducted an evaluation of commercially available ventriculostomy simulators. Two ventriculostomy simulators were identified, including a physical simulator (University of Florida, Gainesville, Florida) that was in development but not yet ready for commercialization.6 A VR-based ventriculostomy simulator (ImmersiveTouch, Chicago, Illinois) was chosen for the course on the basis of its prior use in resident training forums and the validation capabilities that are integrated into the system. The ventriculostomy system (Figure 1) is an augmented VR system that integrates a haptic device and a high-resolution, high-pixel-density stereoscopic display.4 Using a head and hand tracking system, the system scores several performance measures, including entry point for the burr hole, catheter trajectory, length of catheter inserted, and distance of catheter tip from the foramen of Monro.
The trauma module was designed to conform with guidelines established for all modules in the CNS Resident Simulation Symposium. This included a precourse questionnaire to assess familiarity with simulators and questions that the participants were asked to self-assess their existing experience with performing craniotomies for trauma and placing ventriculostomies. The answers were stratified into a low experience level for < 50 procedures done and < 50% reported autonomy and a high level for > 50 procedures and > 50% autonomy. Self-reported time for completing a ventriculostomy was stratified into a fast group for < 30 minutes and a slow group for < 30 minutes.
A written pretest assessed knowledge of relevant neuroanatomy and traumatic brain injury management skills. Participants then attended a didactic session to reinforce critical concepts. A written posttest, which included the same questions as the pretest, was administered at the conclusion of the module.
After participating in the didactic portion of the course, participants were assigned to the virtual ventriculostomy simulator session. This portion of the course included a 10-minute demonstration of simulator use, followed by a hands-on practical pretest to assess baseline skill level. The faculty grading this pretest assessed the participant’s knowledge of anatomy and technical skills during performance of the procedure. The pretest was followed by hands-on training with the same faculty instructor; then participants were allowed 45 minutes of independent practice time. A practical posttest was administered using the same scoring metrics applied during the practical pretest. Hands-on training was followed by an immediate postcourse evaluation with the entire cohort. At 6 months after the course, a final questionnaire was sent to participants to evaluate how attending the course may have affected their skill and comfort levels with common neurosurgical procedures.
Written pretests and posttests were assigned ordinal scores on the basis of the number of correct responses of 12 questions administered. A technical assessment form, based on the objective structured assessment of technical skill,7 was developed to evaluate hands-on skills and to validate the simulator. Additionally, certain evaluation measures were calculated by a software program as part of the ventriculostomy simulator itself, allowing precise data collection and unbiased assessment of the following components: trajectory, depth and final location of catheter, and time to complete procedure. Other performance measures included knowledge of anatomy and landmarks, dexterity in instrument use, and technical skills (Table). Data for these assessments were acquired in real time and entered into a tablet database program. For each simulation, there was a training instance or trial run, followed by one-on-one interaction with a faculty member who provided instruction and critique. This was followed by an evaluation instance or final run. Ordinal scores between 1 and 5 were given for each performance measure. Scores 2 and 4 were interpolated between the end points given in the Table for scores of 1, 3 and 5. See the Table for definition of performance measures. Participants were classified on the basis of their level of training into a junior group encompassing medical students and residents in postgraduate years (PGY-1-PGY3) and a senior group with residents in PGY-4 to PGY-7.
TABLE Ordinal Perfor...Image Tools
Ordinal performance measures were analyzed by the use of contingency tables and Pearson χ2 tests and asymptotic Cochran-Armitage (CA) test for trends. Statistical analysis was performed with SAS JMP (SAS Institute, Carey, North Carolina). Pretest and posttest data were then evaluated to assess the validity of the 2 simulators as teaching tools.
A cohort of 15 participants was enrolled in the trauma module of the CNS neurosurgery simulation course at the 2012 CNS national meeting. Fourteen participants, all at the time of the course enrolled in postgraduate training in neurological surgery in either a US- or Canada-based institution (5 participants) or at foreign institutions (9 participants) as a resident or fellow, underwent structured teaching and testing in this second iteration of the course. Participant experience ranged from PGY-1 to PGY-7 level. One person did not arrive for the course.
Seven of 12 participants (58%) submitted answers to questions in the precourse questionnaire to assess their existing experience levels.
Self-reported experience levels correlated well with the PGY level; receiver-operating curves had areas > 0.92 (P < .005), allowing the use of the PGY level as a surrogate marker for experience.
Written Test Performance
Fourteen participants took the pretest and were exposed to the structured educational materials in the form of a lecture and presentation; then, selected participants completed the hands-on simulation of a ventriculostomy placement followed by a posttest, which was completed by 12 participants (86% completion rate). The data analysis incorporates only the 12 participants who completed the entire course, including both pretests and posttests, instruction, and hands-on components.
The median number of correct answers in the pretest was 9 of 12 (75%), and the median time taken to complete the test was 4 minutes 3 seconds. After didactic instruction and completion of the simulator, the median number of correct answers increased to 10.5 of 12 (87.5%), a significant improvement (P < .05). The time taken by participants to answer the test significantly improved to a median of 3 minutes 15 seconds (22% mean improvement; P < .02; see Figure 2A and 2B).
We analyzed the subset of 5 questions concerning ventriculostomy-related knowledge. Test participants improved significantly after instruction, from 75% correct answers to 83.3% (P < .01; see Figure 2C).
Ten participants completed the virtual ventriculostomy simulation and improved in all performance measures (Figure 3). Anatomy and landmarks were correctly described by 86% of participants, significantly improved from 29% after a trial run (P = .17; CA test, P < .04). The burr hole placement was ideal by the same 86% of participants, significantly improved from 29% (these 2 participants were different from the 2 who scored well in the previous performance submeasure; P = .09; CA test, P < .03). The trajectory of the ventriculostomy insertion was deemed ideal in 57%, not significantly improved from 29% (P = .33; CA test, P = .24), again reflecting that some participants did worse after the trial run. There was a trend toward placing the ventriculostomy at the appropriate depth of exactly 5 to 6 cm; notably only 29% of participants achieved this, although this improved from 14% (P = .13; CA test, P = .08). The final location of the catheter tip was judged as ideal in 43%, significantly improved from 14% (P = .24; CA test, P = .05). Eighty-six percent of participants were able to complete the virtual procedure in < 5 minutes, whereas no participants were able to achieve this in the trial run, with 71% requiring just under 10 minutes to complete all steps, demonstrating a significant improvement (P < .004; CA test, P < .02).
Results Stratification by Training Level
We analyzed the performances in the written tests and on the simulator, hypothesizing that junior participants would initially do worse and show more improvement and vice versa for senior participants, and to confirm the finding of a plateau at the PGY-5 level in a previous study.8
Junior participants took initially less time to complete the written pretest compared with their more senior colleagues (average difference, 22 seconds). In the posttest, the senior participants took less time (mean difference, 45 seconds), for a significant improvement on the side of the senior participants (P < .02, matched pairs). Junior participants unexpectedly answered more questions correctly in the pretest (79% vs 75%) and improved more in the posttest (89% vs 81%), amounting to a significant 11% improvement (P > .01, matched pairs). There was a similar trend when only the external ventricular drain--related questions were analyzed (77.5% vs 70% in the pretest for junior vs senior participants, 10% vs 20% improvement for junior vs senior participants in the posttest).
For the ventriculostomy simulator, senior residents planned a significantly better trajectory (P < .01; CA test, P = .27) but did not otherwise outperform more junior participants in assessments of anatomic knowledge (P = .47; CA test, P = .13), burr hole placement (P = .4; CA test, P = .5), depth of insertion (P = .58; CA test, P = .39), final catheter location (P = .18; CA test, P = .19), and time to complete the procedure (P = .62; CA test, P = .42). Junior residents improved significantly in the performance measures testing the relevant anatomy (P = .21; CA test, P < .03) and the time required to complete the procedure (P = .13; CA test, P < .04). There was no significant improvement in measures for burr hole placement (P = .38; CA test, P = .1), trajectory (P = .14; CA test, P = .27), and catheter depth (P = .39; CA test, P = .37). Senior residents demonstrated greater improvement in the final catheter tip location on their second attempt (P < .05; CA test, P = .19; Figure 4A).
Data analysis of the performance of the trainees affirms construct validity of the ventriculostomy simulator. Postcourse evaluations revealed that participants enjoyed the ventriculostomy simulator and felt it was an effective teaching tool, supporting face and content validity. More definitive statistical analyses regarding face and content validity are limited because of the small number of participants completing the postcourse questionnaire.
There have been previous attempts at simulating ventriculostomy placement. A Web-based simulator introduced in 2000 provided auditory feedback; however, its use was limited by technical drawbacks, including a lack of haptic feedback and use of a mouse-driven pointer to direct the catheter trajectory.9,10 Neuronavigation has been used to simulate the catheter position and to study this position relative to the adjacent anatomy. Although this technique was shown to help young neurosurgeons understand anatomic relationships, it does not represent a true simulation of the procedure.11 A first attempt at introducing haptic technology to ventriculostomy simulators was made by a group using the Reachin VR-haptic display (Reachin Technologies AB, Stockholm, Sweden).12 A Swiss group took a different approach and developed a 1-df model to simulate the advancement of a catheter into a physical brain model using the BrainTrain simulator (Sensory Motor Systems Laboratory, Zurich, Switzerland).13 Using the physical model obviates the need for rendering VR graphics, but concerns about the internal validity of this particular simulator have been raised.14
The introduction of neurosurgical simulation programs to aid in the training of neurosurgeons and neurosurgical residents is gaining importance as a strategy to address deficiencies in certain areas of neurosurgical education3,15 and to provide supplemental training for surgical residents who now face work hour restrictions. As simulation technology improves, a growing percentage of neurosurgical training programs in the United States are likely to incorporate simulator-based training to meet these educational needs and to address patient safety concerns.16
As discussed by Lobel et al17 in the article describing the trauma craniotomy simulator in this supplement, there is a relative paucity of peer-reviewed neurosurgical literature compared with general surgery literature regarding simulation-based training in surgical procedures. As neurosurgical simulator technologies evolve, validation of these simulators as effective teaching tools is required before widespread use as adjuncts to residency training. Concepts in simulator validation previously described in the general surgery literature can be applied to neurosurgical simulator technology. These validation concepts include construct, face, and content validation techniques18,19 and have been applied to validating haptics simulators20,21 and VR devices.22 Face validity and content validity are determined by assessing how closely a simulator appears to replicate the task being simulated among novices and experts, respectively. Construct validity refers to the degree to which a simulator is able to predict the level of expertise of the participant on the basis of their performance in the simulated task. We sought here to validate the ventriculostomy portion of the trauma module described on the basis of these concepts in validation.
The VR ventriculostomy simulator was able to predict the level of expertise on the basis of the trajectory chosen for passing the catheter but not for other metrics evaluated. This may indicate that ventriculostomy is a clinical task learned early in residency or even medical school and that junior residents may benefit more than senior residents from the training paradigm described here. Alternatively, categorization of simulator participants into PGY-1 to -3 and PGY-4 to -7 groups may be too broad to sense differences in expertise. A larger cohort of participants would have increased the power of the study and may have allowed identification of the training year in which the learning curve plateaus for each of the metrics in this simulator. In a previous study, a plateau was noted at the PGY-5 year.8 Results of our study reveal a similar trend (Figure 4B and 4C), supporting construct validity of the ventriculostomy simulator.
Participants in the trauma module were asked their opinions regarding the ability of the simulator to improve their technical skills and their likelihood to recommend the trauma simulation course to other trainees. Analysis of responses showed a trend indicating that the trauma simulators engaged the participants in tasks they felt were relevant to their training, supporting the face validity of the simulator, although the number of responses to the questionnaire was insufficient for statistical evaluation.
The participants completed the ventriculostomy simulator in the neurosurgery trauma module in a reasonable period of time. Data regarding time to completion and scores from the judges indicate that the simulators generated consistent results among the participants. When repeating a simulator trial, participants typically have significantly improved performance, underscoring the potential for positive educational impact, although the durability of this effect is unknown.
In prior reports, the validity of a training module is commonly assessed by evaluating results from a small group of trainees performing the course23 and using surrogate measures of success.24 The neurosurgical trauma simulation course describes a similarly sized group of neurosurgical trainees enrolled in an all-day simulation course. On the basis of the performance of the trainees and their postcourse feedback, the validity of the ventriculostomy simulator when used within the framework of the module is suggested. However, this will need to be more conclusively demonstrated with further testing.
A number of alternative simulators exist9,10,12 that simulate either part-tasks of or the entire ventriculostomy procedure. Because of limitations of VR-based technology in earlier years, models often incorporate a physical component along with the VR components.6,13 Our simulator has a number of features that replicate tasks that increase the realism of the simulation tremendously, eg, the ability to manipulate the head position and an overlay between the stylus and the virtual catheter.
Limitations of the fidelity of the simulation include simplified steps involved in creating the skin incision because bleeding skin vessels can pose problems for junior trainees without readily available cautery. Additional limitations include the lack of a realistic bony sidewall of the burr hole, which may affect directional guidance on the catheter, and the lack of simulating the procedure of tunneling the catheter under the skin without moving the catheter tip from the final location.
A few limitations of the design of the ventriculostomy simulation module must be noted. Although some evaluation measures are amenable to concrete analysis such as knowledge of anatomy and conceptual understanding of the procedure, others, including dexterity and ventriculostomy trajectory, are sensitive to interattempt variations, and improvement in these measure may not be observed over such a short training regimen. Therefore, there is potential for participants to obtain a poorer score in the final attempt after a good score in the trial run. One method to address these concerns is the introduction of regional simulation courses in addition to the annual CNS Resident Simulation Symposium, which will improve data analysis by incorporating greater numbers of participants. This will also allow us to overcome the inherent limitation of the small numbers in our data set, which limit the power of the analysis. Despite small numbers, we attempted to stratify the data in a reasonable manner to allow analysis pertaining to simulator validity, but we underline that the small number of participants limits such a stratification. Future work will attempt to increase the number of data points and will evaluate the relative importance of distinct part-task components for the completion of the entire task.
One drawback of high-fidelity VR simulators, in general, is a rather large upfront cost. Even though maintenance and use costs are typically much lower, they are still considerable and often include fees for personnel. The estimated cost for the ventriculostomy simulator purchase is significant, with purchase options beginning at $75 000 and lease options at $20 000.
The trauma module as a component of the CNS Resident Simulation Symposium is currently undergoing evaluation by the course directors to optimize educational benefit and validation analysis and to minimize cost.
The use of simulation could, for example, be used in conjunction with credentialing models, which require junior residents to perform a number of ventriculostomies under supervision to the satisfaction of the faculty before being allowed unsupervised performance. This would follow a pathway outlined by the example of central line placement where the training algorithm changed from one of supervised learning on patients to manikin-based practice requiring proficiency before patient interaction was permitted at the earliest stages of postgraduate education.25 We could therefore imagine the deployment of this module within the framework of the national fundamentals curriculum of the Society of Neurological Surgeons boot camp courses26 in which a large number of residents meet or in dedicated neurosurgical dissection and simulation laboratories.
The trauma module of the CNS simulation course was designed to introduce neurosurgical trainees to a simulation-based curriculum to teach skills relevant to the management of traumatic brain injury. Data analysis from the 2012 course reveals that the ventriculostomy simulator, as part of the trauma module, is a valid method to enhance resident training. At both the 2011 and 2012 meetings, the number of participants in this course was necessarily limited by the availability of simulators and faculty to provide a one-on-one training experience. Further data and additional simulators at future courses will allow improved validation measures for the module.
A podcast related to this article can be accessed online (http://links.lww.com/NEU/A574).
The authors have no personal financial or institutional interest in any of the drugs, materials, or devices described in this article. Information detailed in this article has been submitted as an abstract to the 2013 CNS Annual Meeting.
2. Gallagher AG, Cates CU. Approval of virtual reality training for carotid stenting: what this means for procedural-based medicine. JAMA. 2004;292(24):3024–3026.
3. Popp AJ. Navigating the Strait of Magellan: mapping a new paradigm for neurosurgical residency training: presidential address to the Society of Neurological Surgeons, May 7, 2007. J Neurosurg. 2008;109(4):576–582.
4. Luciano C, Banerjee P, Florea L, Dawe G. Design of the ImmersiveTouch: a high-performance haptic augmented virtual reality system. Paper presented at: 11th International Conference on Human-Computer Interactions; July 22-27, 2005; Las Vegas, NV.
5. Yudkowsky R, Luciano C, Banerjee P, et al.. Practice on an augmented reality/haptic simulator and library of virtual brains improves residents’ ability to perform a ventriculostomy. Simul Healthc. 2013;8(1):25–31.
6. Hooten KG, Lister JR, Lombard G, Bova FJ, Murad GJ. Validation of a mixed reality ventriculostomy simulator. Paper presented at: Congress of Neurological Surgeons (CNS) Annual Meeting; October 6-10, 2012; Chicago, IL.
7. Faulkner H, Regehr G, Martin J, Reznick R. Validation of an objective structured assessment of technical skill for surgical residents. Acad Med. 1996;71(12):1363–1365.
8. Banerjee PP, Yudkowsky R, Lemole M Jr, Charbel FT, Luciano C. Using a high-fidelity virtual reality and haptics-based simulation to determine the “learning curve” of neurosurgery resident's surgical skills. Simul Healthc. 2007;2(2):145.
9. Phillips NI, John NW. Web-based surgical simulation for ventricular catheterization. Neurosurgery. 2000;46(4):933–936; discussion 936-937.
10. John NW, Phillips N. Surgical simulators using the http://WWW.Stud
Health Technol Inform. 2000;70:146–152.
11. Krombach G, Ganser A, Fricke C, et al.. Virtual placement of frontal ventricular catheters using frameless neuronavigation: an “unbloody training” for young neurosurgeons. Minim Invasive Neurosurg. 2000;43(4):171–175.
12. Larsen OV, Haase J, Østergaard LR, Hansen KV, Nielsen H. The Virtual Brain Project: development of a neurosurgical simulator. Stud Health Technol Inform. 2001;81:256–262.
13. Panchaphongsaphak B, Stutzer D, Schwyter E, Bernays RL, Riener R. Haptic device for a ventricular shunt insertion simulator. Stud Health Technol Inform. 2006;119:428–430.
14. Lemole GM, Banerjee PP, Luciano C, Neckrysh S, Charbel FT. Virtual reality in neurosurgical education: part-task ventriculostomy simulation with dynamic visual and haptic feedback. Neurosurgery. 2007;61(1):142–148.
15. Mazzola CA, Lobel DA, Krishnamurthy S, Bloomgarden GM, Benzil DL. Efficacy of neurosurgery resident education in the new millennium: the 2008 Council of State Neurosurgical Societies post-residency survey results. Neurosurgery. 2010;67(2):225–232; discussion 232-233.
16. Palter VN, Orzech N, Reznick RK, Grantcharov TP. Validation of a structured training and assessment curriculum for technical skill acquisition in minimally invasive surgery: a randomized controlled trial. Ann Surg. 2013;257(2):224–230.
17. Lobel DA, Elder JB, Schirmer CM, Bowyer MW, Rezai AR. A novel craniotomy simulator provides a validated method to enhance education in the management of traumatic brain injury. Neurosurgery. 2013.
18. Abboudi H, Khan MS, Aboumarzouk O, et al.. Current status of validation for robotic surgery simulators: a systematic review. BJU Int. 2013;111(2):194–205.
19. Kelly DC, Margules AC, Kundavaram CR, et al.. Face, content, and construct validation of the Da Vinci Skills Simulator. Urology. 2012;79(5):1068–1072.
20. Liss MA, Abdelshehid C, Quach S, et al.. Validation, correlation, and comparison of the da Vinci TrainerTM
and the da Vinci Surgical Skills SimulatorTM
using the MimicTM
software for urologic robotic surgical education. J Endourol. 2012;26(12):1629–1634.
21. Singapogu RB, Dubose S, Long LO, et al.. Salient haptic skills trainer: initial validation of a novel simulator for training force-based laparoscopic surgical skills. Surg Endosc. 2013;27(5):1653–1661.
22. Plooy AM, Hill A, Horswill MS, et al.. Construct validation of a physical model colonoscopy simulator. Gastrointest Endosc. 2012;76(1):144–150.
23. Shamim Khan M, Ahmed K, Gavazzi A, et al.. Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity. BJU Int. 2013;111(3):518–523.
24. Banerjee PP, Luciano CJ, Lemole GM Jr, Charbel FT, Oh MY. Accuracy of ventriculostomy catheter placement using a head- and hand-tracked high-resolution virtual reality simulator with haptic feedback. J Neurosurg. 2007;107(3):515–521.
25. Britt RC, Reed SF, Britt LD. Central line simulation: a new training algorithm. Am Surg. 2007;73(7):680–682; discussion 682-683.
26. Selden NR, Origitano TC, Burchiel KJ, et al.. A national fundamentals curriculum for neurosurgery PGY1 residents: the 2010 Society of Neurological Surgeons boot camp courses. Neurosurgery. 2012;70(4):971–981; discussion 981.