With the publication of “To Err Is Human” in 19991 and Safety at the Sharp End,2 attention was drawn to the importance of human factors in operational teams, identifying skills that were first described for crew resource management in aviation and other high-risk industries. Flin et al.2 described these nontechnical skills (NTS) as “the cognitive, social and personal resource skills that complement technical skills, and contribute to safe and efficient task performance”. In Flin's taxonomy, NTS included individual cognitive skills (eg, situation awareness, decision-making, coping with stress, and management of fatigue) and interprofessional social skills (eg, cooperation and teamwork, conflict resolution, leadership). Evidence that human factors have an impact on surgical performance3 led to the development of training programs focusing on NTS to improve patient safety.4 Using Flin's taxonomy, several tools (ie, ANTS, NOTSS, SPLINTS) have been designed to provide surgical teams with a common language to discuss and develop the human factors that are critical for patient safety.5
Several studies have demonstrated the efficiency of simulation for knowledge acquisition and for technical6–8 and nontechnical9,10 skills training in healthcare. Training healthcare professionals to manage rare or critical events in a standardized manner and without risk for the patient has become a major challenge.11 Although the benefits of simulation have been well documented, the human resources required for mannequin or standardized patient-based simulation and the availability of human resources in simulation centers remain scarce. However, various simulation methods have recently been developed using real actors, mannequins, standardized patients, computer simulators, or serious games.
With the development of technology and the “laparoscopic surgery revolution,”12 virtual reality (VR) simulators are being used more widely in both professional practice and education programs.13,14 Virtual reality is a broad concept that encompasses the following three categories of simulators: screen-based VR simulators, virtual worlds, and immersive VR environments. First, screen-based VR simulators have been used since the 90s to develop psychomotor skills for endoscopic surgery.15 They consist of an interface comprising a computer and monitors coupled to mechanical devices or haptic units.16 This kind of simulator requires very little set-up time and can be used repeatedly by learners for practice in different pathologies and with a number of anatomical variations.17 Second, virtual worlds are three-dimensional virtual environments based on multiplayer online gaming, allowing users to free themselves from geographical proximity or time constraints (individual connection and full time access).18–20 For health professionals, medical furniture, instruments, devices, tools, and characters are added to create dedicated medical virtual worlds.21 Lastly, immersive VR environments combine three-dimensional imaging, interactions with the environment, possible haptic feedback, and head-mounted displays (HMDs) or cave automatic virtual environments (CAVE, room-sized cube VR environments) to immerse the user and occlude the real world to provide a feeling of presence.13,22
The effects of VR-based training on healthcare professionals have mostly been studied in relation to the development of technical skills, either surgical (eg, procedure, planning, knowledge of instruments) or psychomotor (eg, dexterity, accuracy, speed).23,24 The use of VR simulation-based training for the development of NTS seems to be less common. In view of the recent developments described previously, a systematic review of how VR has been used for NTS skills training and assessment could provide powerful new insights into the value and efficiency of this technique. This article provides such a review, focusing on the use of VR in the evaluation and development of a predefined set of cognitive and interprofessional social skills required by healthcare professionals.
Data Source and Search Strategy
This systematic review followed the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses.25 The PICOS (Population, Intervention, Comparison, Outcomes and Study design) worksheet and search strategy26 were used to organize our research topics and terms, combining the following five concepts: population, intervention, comparison, outcome, and settings. Based on this framework, we produced a comprehensive research equation: “(nurs* OR scrub nurse OR surgeon* OR medical student OR nursing student OR health* OR clinical) AND (virtual reality OR virtual* OR virtual gaming simulation OR immers* OR serious game) AND (skills training OR training OR medical education) AND (non-technical skill* OR cognitive skill* OR team* OR leadership OR communicat* OR decision making OR task management)”. We searched two main databases, PsycInfo and Medline, because they both cover our fields of interest (healthcare, VR, and NTS) and because their query forms are very similar, allowing us to use the same research equation. The search was conducted in May 2017 with a final extraction for screening on December 11, 2017.
Inclusion of articles was based on the following criteria: (1) focus on healthcare professionals working in teams; (2) focus on VR used in an educational or training context that includes outcomes data; and (3) focus on NTS in line with Flin's predefined set of NTS referring to cognitive skills (situation awareness, decision-making, stress, and management fatigue) and interprofessional social skills (cooperation and teamwork, conflict resolution, leadership). To get the most reliable source of scientific information, only publications in peer-reviewed journals were considered. Because VR is a relatively new technology, the period chosen for study selection was between 2007 and 2017. Finally, to have an international overview, only articles written in English have been retained.
Study Selection and Eligibility Criteria
We extracted 1373 references, 460 on PsycInfo and 913 on Medline. Four others found on Google Scholar while reading on the subject were added, thus making a total of 1377 references. We removed 188 abstracts in duplicate, resulting in 1189 titles and abstracts to read for selection. At this stage of the process, 1109 references were excluded.
In accordance with the first selection criterion, study populations that did not involve health professionals (eg, pedestrians, cyclists, students, athletes, or patients) were removed, as well as those working in smaller teams or solo (eg, dentists, pharmacists, and vets).
Following the second selection criterion, articles dealing with learning methods that were not the focus of this study were deleted (eg, e-learning, e-classroom, web-based learning, telemedicine, mobile applications, social networks). According to the same criterion, some references were also excluded because the term “virtual” was used without referring to VR (eg, virtual teams, virtual communities, virtual classroom, virtual consultations, and virtual patients). There is a vast literature on the “virtual patient,” focusing on the interpersonal social skills of healthcare professionals required to facilitate accurate diagnosis, give appropriate advice, and instruct patients about treatment. Because these skills do not come within the scope of our predefined set, these references were not included in the present review.
Finally, according to the third criterion, articles that did not address NTS were excluded: knowledge representation (anatomy, physiology, histology, three-dimensional planning for radiotherapy), technical skills, interpersonal patient-physician relationship (diagnosis, notification of critical results, or pathology), psychiatric disorders (dementia, posttraumatic stress disorder), or remediation (cognitive behavioral therapy, rehabilitation) that also use VR technologies but are out of the scope of this review.
This initial selection included 80 articles with abstracts that were insufficiently clear to decide whether or not they were eligible, and these were uploaded to read the full text. However, five remained unavailable even after their main authors were contacted by e-mail, which reduced the number of articles to read to 75.
After the full-text analysis, 49 articles were excluded for the following four reasons: (1) they did not involve NTS (n = 31), (2) they did not use VR (n = 15), (3) they did not involve health professionals (n = 2), or (4) it was a review article with no experiment (n = 1).
Selection Bias (Interrater Reliability)
To check the validity of the study, a sample of the articles (30%) was independently coded by the second author. Based on the eligibility criteria, the second coder decided whether or not to examine an article in depth. After reviewing the abstract and full text, the two coders, meaning the first and second authors, agreed on 26 articles, 11 that should be selected and 15 rejected. Their opinions only diverged on the implementation of NTS in one article (3.7%), which was eventually excluded (Cohen κ = 0.92). The selection process is summarized in the flowchart (Fig. 1).
To obtain a descriptive analysis of the results of this review, we drew up a comparative table, focusing on medical specialties, VR simulator typologies, study populations, and assessed NTS. For the latter, when assessment tools were mentioned, presence or absence of their validity evidence in the virtual environment was also examined. Another criterion for the analysis was outcomes, according to Kirkpatrick's levels (see Table 1, Supplemental Digital Content 1, http://links.lww.com/SIH/A400, which shows the characteristics of the studies included in the review). This framework was chosen because the Kirkpatrick model27 is frequently used to evaluate training programs; for example, it was recently used to evaluate serious games for training healthcare professionals.28 This model has the following four levels: (1) affective reactions after the training program, (2) improvement of knowledge and skills, (3) change of behavior and transfer of skills in professional contexts, and (4) increased patient care quality and reduced costs.
Virtual reality is a recent technology, and as expected, the number of articles published per year for the last decade on this subject is rather low although it has been increasing, with a peak of seven articles in 2015. The median number of articles published per year is two (interquartile range = 3–1, min = 0, max = 7).
Regarding sample size, the average number of participants was 39.69 (min = 10, max = 148, SD = 37.33). Looking more closely at their study design, 17 studies used an experimental design (pretest/posttest, group comparisons, or control/test group comparisons),29–44 nine were observational studies,18–20,45–50 and one was based on qualitative interviews.51
Emergency medicine18,19,32,44,47 and health education20,37,41,50,51 stand out from the rest with five articles each, but the latter includes issues that potentially concern all specialties or health professionals in different sectors. Next are urology29,30,40,48 and gastroenterology33,34,45,46 with four articles each. A further eight articles dealt with interprofessionality.18,32,37,41,47–50
Most of the studies include a range of professions: nurses feature in eight articles,19,20,32,37,38,40,48,49 surgeons33,35,38,39,43,45,46 in seven, and five articles mention other professionals.18,19,32,37,38 With regard to students, 12 articles involve residents or postgraduate trainees,18,33,35,36,38–40,42,44–46,48 ten concern health students regardless of their level or specialty,30,31,39,41,44–47,50,51 and three articles mention trainees.29,34,38
Virtual Reality Systems
In the studies selected for review, two technologies are equally represented: screen-based VR simulators in 12 studies29,30,33,34,36,38–40,42,43,48,49 and virtual worlds in 12 articles.18–20,31,32,35,37,41,44,47,50,51 Immersive virtual worlds are less common, because they are found in only three studies.39,45,46
Nontechnical Skills and Assessment Metrics
Among the 26 studies, the most frequently investigated NTS is teamwork, which is mentioned in 19 articles.18,19,29–32,34,35,37,38,40–42,45,47–51 After that is communication (15/26),18,29,30,32–34,37,38,40–42,47–50 followed by situation awareness (10/26),18,20,29–31,34,42,46,48,51 decision-making (8/26),18,20,29,30,32,34,40,42 leadership (8/26),18,29,30,34,37,41,42,44 and stress management (6/26).18,36,39,43,44,46 Most studies measured more than one NTS construct, with the exception of three articles focusing specifically on stress management,36,39,43 three others on teamwork,19,35,45 and one on communication.33
Specific metrics for NTS assessment are mentioned or described in detail in 16 articles,18,20,29–37,41–44,49 10 of which18,29,30,33,34,37,41–44 mention and use specific validated assessment tools for NTS in the operating room or in a clinical environment (ie, NOTECHS, NOTSS, T-NOTECHS, CGRS, M-OSANTS, TOSCE, T-TAQ, SURG-TLX, EMCRM). These assessment tools are presented as validated for simulation,52,53 but none of them mention their validation in the very specific context of VR simulation. Four articles31,34,36,42 use scales designed for a specific NTS but are not specific to the operating room or to health professionals (ie, situation awareness, self-efficacy, anxiety). Six articles18,20,32,35,43,49 use their own tools or checklists, or open-ended questions rather than a questionnaire. Again, none of these tools or checklists take into consideration the specificity of VR simulation, but are simply transferred from real life or traditional simulation contexts.
The levels of assessment of the simulators or scenarios used in the studies reach Kirkpatrick's level 1 (affective reactions) in 19 articles18–20,29,30,32,37–48,51 and level 2 (learning: attitude, knowledge) in 18.18,20,29–37,39,41–44,49,50 Level 3 (behavior) is only reached in two articles33,34 that describe a test of skills transfer to the clinical environment to evaluate behavioral changes among trainees. No studies were found that reached level 4 (results) evaluating the effect of simulation on patient care quality and cost.
Overall, this review demonstrates that few medical specialties use VR for NTS training. The most frequently used systems are screen-based VR simulators or virtual worlds, whereas immersive virtual worlds are rarely used. The most frequently studied NTS are the interpersonal and interprofessional social skills needed for effective medical teams, including teamwork and communication, together with situation awareness, which is a crucial cognitive ability.
Objectives and Findings of the Studies
Two main categories of study objectives emerged from our review (see Table 2, Supplemental Digital Content 2, http://links.lww.com/SIH/A401, which shows the objectives, measures and main findings of the studies).
Most studies are simulator or scenario centered, with a clear goal of establishing the acceptability of the technology for NTS training. This is the goal of 20 articles.18–20,29,31,35–39,41–49,51 Because VR has only been introduced in healthcare training recently, it seems logical to assess this first. The general conclusion of these studies is that VR simulators offer promising opportunities for NTS training of health professionals. Their acceptability as an NTS training tool is validated either directly37,41 or through one or more of its predictors as defined by Nielsen's model of system acceptability54: validity or fidelity,18,29,38,39,42,45,46 usefulness or utility,41,42,44,46 efficiency, effectiveness or efficacy,19,35,47,48 and usability.20,43,47 The acceptability of VR simulators is not assessed alone but together with teamwork,19,35,37,38,41,45,47,48,51 situation awareness,20,31,46 stress management,39,43,46 self-efficacy,36 and leadership,44 or with several NTS using a dedicated assessment tool such as NOTECHS or NOTSS.18,29,42
The purpose of the six remaining articles, among the most recent ones of this review, is to propose different uses of VR simulation. One includes a VR program that focuses on communication tools as an initial introduction to team communication strategies.50 The other five are curriculum centered, their objectives being to incorporate NTS training modules using VR simulators in the medical curriculum: four of them describe distributed simulation,30,33,34,40 in other words, the use of VR simulators initially designed for teaching technical skills in wider settings or scenarios,29 and the fifth32 addresses emergency preparedness using a virtual world.
There are few medical specialties that use VR simulation for NTS and their goals vary. First, virtual worlds are used in healthcare education and in emergency medicine to provide practice opportunities for rare events or extreme situations that are difficult to set up in real life. Second, screen-based VR simulators are used for technical skills training in domains involving laparoscopic surgery or robotic-assisted surgery, such as gastroenterology33,34 and urology30,40 curricula. In the latter, NTS are introduced progressively in VR simulation scenarios as trainees develop their technical skills.55 This is congruent with the conclusions of Shamim-Khan et al (2013)40 and Rudarakanchana et al (2014)38 who observed that junior trainees focus more on technical skills and feel that the introduction of NTS adds stress and anxiety due to the need to multitask. It also follows the hierarchy of core, procedural and team skills proposed by Windsor (2009),12 who illustrates it with a musical analogy of note, melody, and harmony: just like musicians begin to learn playing a note before trying to play a melody and then join an orchestra, core skills such as knot tying, dissecting, or suturing, for example, should be trained before procedural skills (ie, how to dissect out a pathology). Then, team skills can be trained.12 Nevertheless, this progression does not enjoy consensus and must be considered for each set of NTS. For example, the addition of cognitive skills likely to help error-detection to technical skills training can result in more effective learning.56
Screen-based VR simulators or virtual worlds are thus the most frequently used systems. Immersive virtual worlds are less commonly used and are mentioned in only three articles,39,45,46 two of them using HMDs.39,46 They were published between 2015 and 2016, which corresponds to the time when HMD such as Oculus Rift or HTC Vive became mainstream, although immersive VR simulation is not a new technology in healthcare because it has been used since the early 2000s.57 It will be interesting to see how these devices will be used in the future as their comfort and fields of view improve and how they can be integrated into new surgical training modules.
One of the major interests of VR is its realism, which is stressed in several studies.29,38,39,42,45,46 Although avatars may sometimes behave awkwardly, and haptic feedback may be approximate or missing, participants still recognize the main features of their environments or work organizations. This contributes to the feeling of immersion mentioned in two articles.19,39 These two notions are of particular interest in that they help trainees gain confidence19,29,37,42 by providing them with new learning opportunities, such as discovering an emergency department without stress,47 or being able to develop the skills needed to react to disasters when training in an unfamiliar environment.32 However, the degree of realism expected in a VR simulator has to be questioned for each scenario: if more haptic feedback, such as somatosensations or kinematic interactions, is expected with a high degree of fidelity, too much realism for an avatar in virtual worlds can lead to anxiety and rejection, due to the uncanny valley phenomenon. According to this theory, participants react favorably to environments that are very similar or very dissimilar to reality, but are uncomfortable with intermediate realism.58,59
The affective component of learning has been described as one of the four key criteria for simulation-based learning, which is centered on the learners' needs and for which motivation and self-efficacy are key concepts.60 With this in mind, many VR simulators allow participants to replay their session, helping them recognize and analyze both their interactions and their emotions.61 The emotional impact of VR simulation on self-efficacy is emphasized and appreciated by trainees,44 as well as the opportunity provided by some scenarios to communicate with other disciplines before any clinical practice47 or to experience human interactions in problematic environments.51 In terms of motivation, VR simulation seems to be of particular interest because it is described as a highly rated learning experience,37 preferred to standard didactic lectures,46 and seen as excellent preparation for clinical situations.47
Another interest of VR simulation is data generation because VR simulators can track and record every action. The data are used to give learners feedback on their performance and progress over time through their profile, allowing them to verify their skills acquisition and become proactive in their learning. However, they also help educators better understand their students' learning processes, allowing them to adjust their inputs and complement their traditional teaching methods with appropriate simulations.50 In addition, because most VR simulators and scenarios can be used in different cultural and geographic environments, the collected data could also be used for further intercultural studies of NTS.50
The most frequently studied NTS are interpersonal and social skills, such as teamwork and communication. As a cognitive skill, situation awareness also features frequently, because it is a crucial personal skill, especially in dynamic environments such as medical settings where it can impact decision-making and communication.62 Although the capacity of VR environments to recreate realistic situations makes it suitable for stress management training, none of the articles selected for this review discuss fatigue management. Although VR is used to that end for patient-focused psychological therapy, healthcare professionals may not perceive fatigue and stress management as an issue warranting the use of simulation scenarios.
Most studies are set up to assess a simulator or scenario, reaching Kirkpatrick's level 1 (affective reaction) or 2 (learning). According to these articles, VR simulators offer NTS training opportunities for healthcare students, because their feasibility, usability, validity, acceptability, or effectiveness has been validated in different situations. However, their conclusions recommend further studies. The impact of VR simulation on NTS training still requires more systematic assessment in routine clinical practice to validate a possible transfer of skills for health professionals and to determine whether it achieves its ultimate goal of improving patient safety (Kirkpatrick's level 4). This is challenging because patient safety is multifactorial and because, as this review shows, validated tools to assess NTS are not systematically used, making large-scale comparisons difficult. Furthermore, few studies mention the inclusion of NTS training using VR simulators in the curriculum as a way of helping students progress.
To continue the development of VR systems for NTS training, four lines of research are required. First, studies are needed regarding the validation of specific NTS assessment tools for VR simulation. So far, these tools have not been validated, which may impact their evaluation in these specific environments. Second, studies are needed to estimate the different technical possibilities offered by VR technologies for NTS training, such as mixed and augmented reality. Third, debriefing is considered a key element for skills transfer in simulation-based training.63 Although this has been studied in different simulation contexts64 and with different frameworks,65 specific debriefing methods for VR simulation are scarce. Avatar-based debriefing,66 for example, could be developed for health education. Fourth, future studies should evaluate the effects of these training methods at different levels of learning: attitudes, skills, transfer of skills, and cost-benefit ratio. In particular, more studies are needed in the near future to investigate the transfer of skills to the operating room (Kirkpatrick's level 3) and to validate VR simulation as an efficient training tool for NTS. However, what is really missing is the impact of these tools on the quality of care for patients and on the overall cost of care (Kirkpatrick's level 4), which has not yet been established.
Because our search was limited to two databases, there is a risk that some relevant articles were missed. The decision to select only articles written in English also excludes studies published in other languages. Our search may also have been limited by the definitions of VR and NTS, as the keywords may not have used a very specific terminology, such as avatar, 3-dimensional, HMD, mixed, or augmented reality. Augmented reality, the integration of digital information into the physical world in real time, or mixed reality, the use of real objects to enhance simulations, are related to virtual environments or VR settings.67 Thus, some studies may use simulators based on VR technology but not specify it in their keywords. However, to be as comprehensive as possible for this review, the search request was replicated, replacing VR with augmented and mixed reality: 58 articles were found but none of them dealt with NTS training.
Another limitation of this review is that NTS are not always defined and explicitly operationalized in studies and are sometimes concealed behind technical skills on which they are highly dependent. Finally, a limitation concerns virtual patient simulation, focusing on the interpersonal social skills of healthcare professionals with their patient. It was decided that this did not come within the set of skills defined for this review. However, there is a vast literature on virtual patients, which are frequently used in virtual worlds and virtual immersive environments. A systematic literature review concerning the development of interpersonal social skills and communication with patients via VR could thus be examined in another article.
In conclusion, VR simulation systems are a recent development in health education. The use of VR simulators has increased for technical skills training but to a lesser extent for NTS (ie, cognitive and interprofessional social skills). This systematic review of articles published from 2007 to 2017 shows that screen-based VR simulators or virtual worlds are the most frequently used systems, and teamwork, communication, and situation awareness are the most frequently addressed NTS. The evaluation of VR systems as training tools is essential, but there has, so far, been little systematic research. Most studies evaluate the usability and acceptability of VR simulation, and few studies have measured the effects of VR simulation on NTS development.
Nevertheless, the development of VR technologies and the portability of VR systems offer a very promising outlook for the future training of healthcare professionals. The wide range of possible scenarios that can be simulated, especially for NTS training, will undoubtedly contribute to the “successful integration of simulation throughout the fabric of healthcare”.11
1. Kohn LT, Corrigan JM, Donaldson MS; Institute of Medicine. To Err Is Human: Building a Safer Health System
. Washington, DC: National Academic Press; 1999:312.
2. Flin R, O'Connor P, Crichton M. Safety at the Sharp End, A Guide to Non-technical Skills
. Farnham: Ashgate Publishing Limited; 2008:317.
3. Hull L, Arora S, Aggarwal R, Darzi A, Vincent C, Sevdalis N. The impact of nontechnical skills on technical performance in surgery: a systematic review
. J Am Coll Surg
4. Gordon M, Darbyshire D, Baker P. Non-technical skills training to enhance patient safety: a systematic review
. Med Educ
5. Flin R. Non-technical skills for anaesthetists, surgeons and scrub practitioners (ANTS, NOTSS and SPLINTS). Health Found
6. Agha RA, Fowler AJ. The role and validity of surgical simulation. Int Surg
7. Harder BN. Use of simulation in teaching and learning in health sciences: a systematic review
. J Nurs Educ
8. McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med
9. Cumin D, Boyd MJ, Webster CS, Weller JM. A systematic review
of simulation for multidisciplinary team training in operating rooms. Simul Healthc
10. von Wendt CEA, Niemi-Murola L. Simulation in interprofessional clinical education: exploring validated nontechnical skills measurement tools. Simul Healthc J Soc Simul Healthc
11. Gaba DM. The future vision of simulation in health care. Qual Saf Health Care
12. Windsor JA. Role of simulation in surgical education and training. ANZ J Surg
13. Mantovani F, Castelnuovo G, Gaggioli A, Riva G. Virtual reality training for health-care professionals. Cyberpsychol Behav
14. Palter VN, Grantcharov TP. Virtual reality in surgical skills training. Surg Clin North Am
15. van Dongen KW, Ahlberg G, Bonavina L, Carter FJ, Grantcharov TP, Hyltander A, et al. European consensus on a competency-based virtual reality training program for basic endoscopic surgical psychomotor skills. Surg Endosc
16. Van Herzeele I, Aggarwal R, Neequaye S, Darzi A, Vermassen F, Cheshire NJ. Cognitive training improves clinically relevant outcomes during simulated endovascular procedures. J Vasc Surg
17. Blackburn SC, Griffin SJ. Role of simulation in training the next generation of endoscopists. World J Gastrointest Endoscopists
18. Cohen D, Sevdalis N, Patel V, Taylor M, Lee H, Vokes M, et al. Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments. Resuscitation
19. Heinrichs WL, Youngblood P, Harter P, Kusumoto L, Dev P. Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II. Prehosp Disaster Med
20. Hudson K, Taylor LA, Kozachik SL, Shaefer SJ, Wilson ML. Second life simulation as a strategy to enhance decision-making in diabetes care: a case study. J Clin Nurs
21. Dev P, Youngblood P, Heinrichs WL, Kusumoto L. Virtual worlds and team training. Anesthesiol Clin
22. Kilmon CA, Brown L, Ghosh S, Mikitiuk A. Immersive virtual reality simulations in nursing education. Nurs Educ Perspect
23. Choudhury N, Gélinas-Phaneuf N, Delorme S, Del Maestro R. Fundamentals of neurosurgery: virtual reality tasks for training and evaluation of technical skills. World Neurosurg
24. Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, et al. Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg
25. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg
26. Rielding AM. Learning to learn [Internet]
. New York: Neal Schuman Publisher; 2017: Available at: http://uottawa.libguides.com/ld.php?content_id=13161372
. Accessed January 10, 2017.
27. Kirkpatrick DL, Kirkpatrick JD. Evaluating Training Programs the Four Levels
. 3rd ed. San Francisco Emeryville, CA: Berrett-Koehler Publishers Group West; 2006:379.
28. Wang R, DeMaria S Jr, Goldberg A, Katz D. A systematic review
of serious games in training health care professionals. Simul Healthc
29. Brewin J, Tang J, Dasgupta P, Khan MS, Ahmed K, Bello F, et al. Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology. BJU Int
30. Brunckhorst O, Shahid S, Aydin A, McIlhenny C, Khan S, Raza SJ, et al. Simulation-based ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial. Surg Endosc
31. Creutzfeldt J, Hedman L, Felländer-Tsai L. Using virtual world training to increase situation awareness during cardiopulmonary resuscitation. Stud Health Technol Inform
32. Greci LS, Ramloll R, Hurst S, Garman K, Beedasy J, Pieper EB, et al. vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE). Prehosp Disaster Med
33. Grover SC, Garg A, Scaffidi MA, Yu JJ, Plener IS, Yong E, et al. Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial. Gastrointest Endosc
34. Khan R, Scaffidi MA, Walsh CM, Lin P, Al-Mazroui A, Chana B, et al. Simulation-based training of non-technical skills in colonoscopy: protocol for a randomized controlled trial. JMIR Res Protoc
35. Khanal P, Vankipuram A, Ashby A, Vankipuram M, Gupta A, Drumm-Gurnee D, et al. Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles. J Biomed Inform
36. Maschuw K, Osei-Agyemang T, Weyers P, Danila R, Bin Dayne K, Rothmund M, et al. The impact of self-belief on laparoscopic performance of novices and experienced surgeons. World J Surg
37. Riesen E, Morley M, Clendinneng D, Ogilvie S, Ann Murray M. Improving interprofessional competence in undergraduate students using a novel blended learning approach. J Interprof Care
38. Rudarakanchana N, Van Herzeele I, Bicknell CD, Riga CV, Rolls A, Cheshire NJW, et al. Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment. Cardiovasc Intervent Radiol
39. Sankaranarayanan G, Li B, Manser K, Jones SB, Jones DB, Schwaitzberg S, et al. Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator. Surg Endosc
40. Shamim Khan M, Ahmed K, Gavazzi A, Gohil R, Thomas L, Poulsen J, et al. Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity. BJU Int
41. Sweigart LI, Umoren RA, Scott PJ, Carlton KH, Jones JA, Truman B, et al. Virtual TeamSTEPPS(®) simulations produce teamwork attitude changes among health professions students. J Nurs Educ
42. Willaert W, Aggarwal R, Harvey K, Cochennec F, Nestel D, Darzi A, et al. Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal. Eur J Vasc Endovasc Surg
43. Wucherer P, Stefan P, Abhari K, Fallavollita P, Weigl M, Lazarovici M, et al. Vertebroplasty performance on simulator for 19 surgeons using hierarchical task analysis. IEEE Trans Med Imaging
44. Youngblood P, Harter PM, Srivastava S, Moffett S, Heinrichs WL, Dev P. Design, development, and evaluation of an online virtual emergency department for training trauma teams. Simul Healthc
45. Abelson JS, Silverman E, Banfelder J, Naides A, Costa R, Dakin G. Virtual operating room for team training in surgery. Am J Surg
46. Dorozhkin D, Olasky J, Jones DB, Schwaitzberg SD, Jones SB, Cao CGL, et al. OR fire virtual training simulator: design and face validity. Surgical Endoscopy
. 2017 Sep;31(9):3527–3533.
47. King S, Chodos D, Stroulia E, Carbonaro M, MacKenzie M, Reid A, et al. Developing interprofessional health competencies in a virtual world. Med Educ Online
48. Paige J, Kozmenko V, Morgan B, Howell DS, Chauvin S, Hilton C, et al. From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation. J Surg Educ
49. White C, Chuah J, Robb A, Lok B, Lampotang S, Lizdas D, et al. Using a critical incident scenario with virtual humans to assess educational needs of nurses in a postanesthesia care unit. J Contin Educ Health Prof
50. Umoren RA, Poore JA, Sweigart L, Rybas N, Gossett E, Johnson M, et al. TeamSTEPPS virtual teams: interactive virtual team training and practice for health professional learners. Creat Nurs
51. Rogers L. Developing simulations in multi-user virtual environments to enhance healthcare education. Br J Educ Technol
52. Sevdalis N, Davis R, Koutantji M, Undre S, Darzi A, Vincent CA. Reliability of a revised NOTECHS scale for use in surgical teams. Am J Surg
53. Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D. Development of a rating system for surgeons' non-technical skills. Med Educ
54. Nielsen J. Usability Engineering
. San Diego: Academic Press; 1993.
55. Arora S, Lamb B, Undre S, Kneebone R, Darzi A, Sevdalis N. Framework for incorporating simulation into urology training. BJU Int
56. Kohls-Gatzoulis JA, Regehr G, Hutchison C. Teaching cognitive skills improves learning in surgical skills courses: a blinded, prospective, randomized study. Can J Surg
57. Krijn M, Emmelkamp PM, Biemond R, de Wilde de Ligny C, Schuemie MJ, van der Mast CA. Treatment of acrophobia in virtual reality: the role of immersion and presence. Behav Res Ther
58. Mori M, MacDorman KF, Kageki N. The uncanny valley [from the field]. IEEE Robot Autom Mag
59. Howard MC. Investigating the simulation elements of environment and control: extending the Uncanny Valley Theory to simulations. Comput Educ
60. Kneebone R. Evaluating clinical simulations for learning procedural skills: a theory-based approach. Acad Med
61. Fertleman C, Aubugeau-Williams P, Sher C, Lim AN, Lumley S, Delacroix S, et al. A discussion of virtual reality as a new tool for training healthcare professionals. Front Public Health
62. Endsley MR. Toward a theory of situation awareness in dynamic systems. Hum Factors J Hum Factors Ergon Soc
63. Fanning RM, Gaba DM. The role of debriefing in simulation-based learning. Simul Healthc
64. Cheng A, Eppich W, Grant V, Sherbino J, Zendejas B, Cook DA. Debriefing for technology-enhanced simulation: a systematic review
and meta-analysis. Med Educ
65. Sawyer T, Eppich W, Brett-Fleegler M, Grant V, Cheng A. More than one way to debrief: a critical review of healthcare simulation debriefing methods. Simul Healthc
66. Nagendran A, Pillat R, Kavanaugh A, Welch G, Hughes C. A unified framework for individualized avatar-based interactions. Presence Teleoperators Virtual Environ
67. Stone RJ, Guest R, Mahoney P, Lamb D, Gibson C. A “mixed reality” simulator concept for future medical emergency response team training. J R Army Med Corps