Virtual reality (VR) includes “a wide variety of computer-based applications commonly associated with immersive, highly visual, 3D characteristics that allow the participant to look about and navigate within a seemingly real or physical world.”1(p40) Advancements in VR have led to developing user-friendly, lightweight, affordable, and wireless VR equipment.2 This resulted in a significant surge in the number of active VR users. For instance, the number of VR users in the United States increased from 22.5 million in 2017 to 50.2 million in 2020.3 These numbers are projected to increase to 65.9 million by 2023.3 Moreover, the number of annual VR units sold worldwide increased from 5.41 million in 2019 to 6.1 million in 2021 and is projected to increase to 14.31 million in 2024,4
Recently, particularly in the context of the COVID-19 pandemic, VR became increasingly used to deliver healthcare services that could not be delivered using traditional strategies such as physical and cognitive rehabilitation, pain management, treatment of psychological disorders, and patient entertainment.5 Virtual reality use in nursing and midwifery education has gained traction globally, whereby educators started using VR to counteract the negative effects of the pandemic on student learning and competency to practice safely.6,7 Virtual reality is known to provide an immersive medium to exchange theoretical and clinical learning,8 and enhance skill proficiency and learning outcomes among nursing students, especially when access to clinical areas was restricted.9 A recent systematic review, meta-analysis, and meta-regression of 14 trials found that VR training was more effective than conventional training in delivering and enhancing procedural knowledge among nursing students.10 Other benefits to VR include the opportunity to practice and learn in a safe and interactive environment,11 and the provision of a platform to deliver lectures, ask questions, and engage in critical thinking.12 Examples of recent successful applications of VR in nursing and midwifery education include but are not limited to teaching chemotherapy administration,13 nursing skill competence training,14 peripheral intravenous catheter insertion,15 and teaching midwives about the management of third stage labor.16
According to the Agency for Healthcare Research and Quality, VR simulation uses “a variety of immersive, highly visual, 3D characteristics to replicate real-life situations and/or healthcare procedures.”17(p56) In comparison with face-to-face and manikin-based simulation methods, VR simulation enables students to practice for a longer duration and acquire additional clinical skills that are not necessarily encountered in the clinical setting.12 Virtual reality simulation is also viewed as a viable strategy for the acquisition of non-technical skills such as situation awareness, decision making, communication, and teamwork, which are key to safe practice.18,19
Despite the many advantages of VR, there is very little evidence on the use of clear theoretical and pedagogical models to inform the design and use of VR in virtual learning environments.20 Moreover, the evidence on the feasibility of incorporating this innovative learning and teaching technology in nursing and midwifery education is lacking. Therefore, prior to its full-scale integration into the curriculum as a learning, teaching, and formative and/or summative evaluation instrument, an investigation of the usability of VR simulation is warranted, while considering the context of implementation.21 In the present study, we describe the usability of VR simulation among nursing and midwifery students, as well as their level of satisfaction with VR simulation, and explore their experiences of engaging with VR simulation.
METHODS
Design
The term “usability” is defined as “the quality of being easy to use.”22 Usability testing is vital to the development of virtual systems in healthcare education.23 In the field of human-computer interaction, usability is often defined in terms of “learnability (i.e., is it easy to accomplish the task?), efficiency (once learned, is the user fast in performing the task?), memorability (is the user able to reestablish proficiency with the design after a period of stop?), errors (how many errors does the user make?) and satisfaction (how pleasant is the design?).”23(p2) These elements of usability can be assessed using various instruments. One of the most used instruments is the System Usability Scale (SUS),24,25 which was used in the present study. Usability studies, however, tend to focus on technical tasks rather than the user experience.26 Therefore, in addition to the SUS, we used a questionnaire to measure user satisfaction as well as open-ended survey questions to explore, in greater detail, elements of VR simulation that worked, elements that did not work, and recommendations for improvement.
Sample and Setting
The inclusion criteria were (1) undergraduate nursing and midwifery students; (2) studying in a public university in Cork, Ireland; (3) registered in any of the five BSN/midwifery programs (ie, general nursing, children's and general integrated nursing, intellectual disability nursing, mental health nursing, and midwifery); (4) registered in any year of study; and (5) aged 18 years and above. Virtual reality can trigger motion sickness, which is often associated with sudden movement and extreme gaming.27 This was not the case in the current study. However, for safety reasons, students who suffer from severe motion sickness were not eligible for inclusion.
Participants were recruited using convenience and snowball sampling whereby those who completed the study were encouraged to invite their classmates to participate. Of note, there is no consensus as to how many participants should be included in a usability study,28–30 with some studies recommending sample sizes as small as five participants.31,32 However, adding more participants to a usability study reduces variance in the percentage of known usability problems.29 For instance, in a structured usability test of a Web-based application, Faulkner30 ran 100 usability trials with 60 participants and found that a sample size of five participants would help detect a minimum of 55% of usability problems whereas a sample size of 40 to 50 participants would help detect a minimum of 98% of these problems. Therefore, in the present study, we aimed to recruit 40 to 50 participants.
Two designated VIrtual SImulatiON (VISION) weeks were launched in the School of Nursing and Midwifery on the weeks of March 28 and April 4, 2022. Fifteen professionally designed study invitation posters were displayed around the school. An electronic version of the posters was posted on the school's social media and was e-mailed by the school's executive assistants to all undergraduate students on four occasions. Each poster contained brief information about the study, the researchers' contact details, and a quick response code that students could scan to register their interest in participating. Using a designated study phone, a research support officer contacted the students who registered their interest to schedule a data collection date and time.
Data Collection
This study was approved by the university's Social Research Ethics Committee and Research Access Committee. Data were collected during the two VISION weeks in a designated room in the School of Nursing and Midwifery. Four research personnel who were trained in VR and who were not known to participants collected data. Participants read a study information leaflet and provided informed consent. They were assured that this study was not a formal assessment of their performance and that they should not worry about making mistakes while taking care of the deteriorating virtual patient. Participants were shown a 5-minute tutorial video on the use of the VR headset and controllers. They were then presented with the description of five VR simulation scenarios and were asked to choose one scenario (Table 1). Screenshots from the scenarios are included in Figure 1 with permission from Oxford Medical Simulation, the company that supplied the VR simulation scenarios. Recently, postgraduate nursing students reported that these scenarios were user friendly,33 and non-specialist clinicians who used these scenarios reported a significant increase in confidence and clinical performance.34
Table 1 -
Description of the Five
Virtual Reality Simulation Scenarios
Scenario |
Description |
Acute severe asthma |
Melanie is a 28-year-old woman presenting with shortness of breath. She has a past medical history of asthma and eczema and has been without her inhaler for a few days. Examination reveals wheezing, tachypnea (ie, high respiratory rate), low oxygen saturation via pulse oximetry, and respiratory failure on blood gas. |
Urosepsis and delirium |
Wilfred is a 78-year-old man who presents with a new onset of confusion, requiring a collateral history from his relatives. He is found to have a fever, suprapubic tenderness on palpation, and a urine dip positive for blood and leukocytes. His blood work and vital signs suggest urosepsis. |
Diabetic ketoacidosis |
George is a 28-year-old man presenting with abdominal pain and fatigue. He has a history of type 1 Diabetes and poor insulin compliance due to high stress at work. He presents with dehydration, a high blood sugar, and ketones in the blood and urine. |
Chronic obstructive pulmonary disease and community-acquired pneumonia |
Wilfred is an 80-year-old man who presents with an increased shortness of breath. He has a history of COPD (chronic obstructive pulmonary disease) and a recent lower respiratory infection. He has low oxygen saturations and respiratory failure, increased white cells, raised lactate, and crackles on auscultation. |
Alcohol withdrawal with suicidal ideation |
Boris is a 40-year-old man who presents with flu-like symptoms and tremors. On further exam, he is sweaty and photophobic (ie, eyes sensitive to light) and admits to a history of alcohol abuse with sudden cessation a few days ago. His examination shows an increasing CIWA score (ie, instrument that assesses alcohol withdrawal). A further interview of his psychosocial status reveals a history of suicidality and ongoing risk factors for suicide. |
Abbreviation: CIWA, clinical institute withdrawal assessment for alcohol.
FIGURE 1: Screenshots from the different virtual reality simulation scenarios.
Each VR simulation scenario was timed and took 20 minutes to complete. Virtual patients in each of the five scenarios were stable initially but deteriorated slowly as the simulation progressed. Actions possible within the scenarios include taking patient history, measuring vital signs, performing physical assessment, administering medications, escalating the care, interpreting laboratory test results, and communicating with the patient. For safety reasons, participants remained seated during data collection, VR equipment was wiped between uses, and emesis bags were available in case participants experienced motion sickness. As a token of appreciation, each participant received a low-value voucher covering the cost of a meal.
Instruments
Immediately after completing the scenario of choice, each participant completed a 21-item electronic survey comprising a sociodemographic questionnaire,35,36 the SUS,24,25 a satisfaction questionnaire,37 and open-ended questions.35,36 See Supplemental Digital Content 1, https://links.lww.com/CIN/A242, for all data collection instruments.
Sociodemographic Questionnaire
Participants completed a six-item sociodemographic questionnaire with questions on age, gender, program of study, year of study, previous experience with VR, and the chosen VR simulation scenario.35,36
System Usability Scale
The 10-item SUS is widely used to evaluate the usability of a technology.24,25 To minimize response bias, items 1, 3, 5, 7, and 9 of the SUS are worded positively (ie, favoring VR) and items 2, 4, 6, 8, and 10 are worded negatively (ie, not favoring VR). Each item is scored on a 5-point Likert scale ranging from “strongly disagree” (score of 1) to “strongly agree” (score of 5).
Following data collection, negatively worded items are reverse coded and 1 point is subtracted from the original score for each item. This gives a score of 0 to 4 per item. The scores for all 10 items are then added and then multiplied by 2.5 to get an overall score of 0 to 100, with higher scores indicating greater usability.24,25 System Usability Scale scores are then converted to percentiles (0-100) and graded (F to A+). Grade A indicates “superior performance,” grade C indicates “average performance,” and grade F indicates “failing performance” (Table 2).38 Of note, grades F to A+ refer to the performance of the system (ie, VR) as opposed to the performance of the user. The average SUS score, at the 50th percentile, is 68.39 In this study, Cronbach's α for the SUS was 0.77, indicating high reliability.40
Table 2 -
Interpretation of the System Usability Scale Scores
38
Score |
Percentile |
Grade |
84.1-100 |
96-100 |
A+ |
80.8-84.0 |
90-95 |
A |
78.9-80.7 |
85-89 |
A− |
77.2-78.8 |
80-84 |
B+ |
74.1-77.1 |
70-79 |
B |
72.6-74.0 |
65-69 |
B− |
71.1-72.5 |
60-64 |
C+ |
65.0-71.0 |
41-59 |
C |
62.7-64.9 |
35-40 |
C− |
51.7-62.6 |
15-34 |
D |
25.1-51.6 |
2-14 |
F |
0-25 |
0-1.9 |
F |
Satisfaction Questionnaire
Participants' level of satisfaction with VR simulation and with their overall experience of participating in the study was measured using two items assessed on a 5-point Likert scale ranging from “extremely dissatisfied” to “extremely satisfied.” Scores ranged between 1 and 5, with higher scores indicating greater satisfaction.37
Open-Ended Questions
Participants typed their responses to three open-ended questions that explored the elements of VR simulation that they enjoyed the most, the elements of VR simulation that they enjoyed the least, and recommendations to integrate VR simulation in nursing and midwifery education in the future.35,36
Data Analysis
Quantitative data were analyzed in RStudio (R version 4.04). Data from the sociodemographic questionnaire, satisfaction questionnaire, and the SUS were analyzed using descriptive statistics (ie, frequencies, mean, SD, median, and interquartile range). Data from the open-ended questions were analyzed in NVivo using deductive content analysis.41 Participant excerpts were coded, and similar codes were grouped to form generic categories. Generic categories were then collated under the following three main categories that were pre-determined based on the questions asked: positive aspects of VR simulation, challenges to engaging with VR simulation, and recommendations for integrating VR simulation into nursing and midwifery education. Qualitative data analysis was conducted by one author and cross-checked by a second author to minimize errors and improve data credibility and confirmability.42
RESULTS
Sociodemographic Questionnaire
Fifty students registered their interest in participating. Of those, 43 participated in this study (86% participation rate). Participants' age ranged between 18 and 51 years. Almost half of the participants were general nursing students (n = 21, 48.84%). Most participants were female (n = 39, 90.7%), either in their first year (n = 15, 34.88%) or second year (n = 16, 37.21) of study, and reported having never used VR previously (n = 24, 55.81%). In terms of the chosen VR scenarios, most participants chose either acute severe asthma (n = 16, 37.21%) or alcohol withdrawal with suicidal ideation (n = 11, 25.58%). The full sample characteristics are presented in Table 3.
Table 3 -
Participant Characteristics (N = 43)
Characteristic |
n (%) |
Age, y |
|
Range |
18-51 |
Mean (SD) |
26 (9.15) |
Median (interquartile range) |
21 (10) |
Sex |
|
Female |
39 (90.7) |
Male |
4 (9.3) |
Program of study |
|
General nursing |
21 (48.84) |
Intellectual disability nursing |
10 (23.26) |
Mental health nursing |
7 (16.28) |
Children's and general integrated nursing |
3 (6.7) |
Midwifery |
2 (4.65) |
Year of study |
|
Year 1 |
15 (34.88) |
Year 2 |
16 (37.21) |
Year 3 |
6 (13.95) |
Year 4 |
6 (13.95) |
Previous experience with VR |
|
None |
24 (55.81) |
Once |
15 (34.88) |
Several times |
3 (6.7) |
A lot |
1 (2.33) |
Chosen VR simulation scenario |
|
Acute severe asthma |
16 (37.21) |
Alcohol withdrawal with suicidal ideation |
11 (25.58) |
Diabetic ketoacidosis |
8 (18.6) |
COPD and community acquired pneumonia |
6 (13.95) |
Urosepsis and delirium |
2 (4.65) |
Abbreviation: COPD, chronic obstructive pulmonary disease.
System Usability Scale
With a maximum possible score of 100,24,25 the mean (SD) SUS score was 75.87 (13.7) and the median score was 75 (interquartile range, 21.25). These scores lie between the 70th and 79th percentile, indicating that VR simulation was acceptable and was graded as B.38,39
Satisfaction Questionnaire
Almost all participants were either “extremely satisfied” (n = 29, 67.44%) or “somewhat satisfied” (n = 13, 30.23%) with the statement “using VR simulation to learn about nursing/midwifery clinical scenarios.” Similarly, most participants were either “extremely satisfied” (n = 32, 74.42%) or “somewhat satisfied” (n = 10, 23.26%) with the statement “my overall experience of participating in this study.” Of note, none of the participants were somewhat/extremely dissatisfied with either of the above two statements.
Open-Ended Questions
The main categories, generic categories, and codes are presented in Table 4. The full coding sheet with participant excerpts is included in Supplemental Digital Content 2, https://links.lww.com/CIN/A243, to enhance transparency and maintain an audit trail.
Table 4 -
Main Categories, Generic Categories, and Codes Identified From the Open-Ended Questions
Main Categories |
Generic Categories |
Codes |
Positive aspects of VR simulation |
Informative, enjoyable, and easy way to learn |
Informative Educational Enjoyable and fun to use Beneficial for students Easy to use and learn Great and accessible opportunity to learn Clear |
Fostering self-directed learning without pressure |
Taking control/being independent Self-directed learning Independent decision-making Less panicked Engaging without pressure Not limited by lack of experience Engaging with critical thinking |
Safe environment to experiment without causing harm |
No fear of causing harm Experimenting instead of dealing with a patient Safe way to practice nursing skills Helping without being in the hospital |
Closeness to real-life situations |
Similar to placement Real, lifelike, and accurate |
Interactive and engaging technology |
Very interactive Being able to practice without moving Engaging with the client Feedback on what went well and what can be improved Prompts to run tests Getting advice when confused Instructions to check with the patient |
Various skills available in VR simulation |
Patient assessment Treatment process Accessible patient records Different options to support decision-making |
Challenges to engaging with VR simulation |
Getting used to navigating the VR environment |
Confusing at first Takes a while to get used to VR Hard to understand at first Not knowing where to find things at first Figuring out how to properly use VR Getting used to navigating the room |
Technological issues |
Blurry screen/graphics Ill-fitting and uncomfortable headset Difficult on the eyes Confusing controls Moving the head while seated |
Not as interactive as real-life situations |
Not being able to react in real situations Need to physically do the skills in real life Some options are only available in real life Not as interactive as the clinical setting/classroom |
Difficulties performing certain tasks |
Escalation of care Giving medication Struggling to do things Being alone in the room |
Recommendations for integrating VR simulation into nursing and midwifery education |
Using VR to practice and prepare for clinical placement |
VR as an instrument to practice clinical skills Using VR to prepare for clinical placement |
Incorporating VR into teaching, learning, and assessment |
Using case studies Using different scenarios Teaching about rare situations Incorporating VR into modules Useful resource for exams More beneficial than manikin |
Making VR accessible |
Equipment available in designated room VR as optional Access at own time |
Positive Aspects of Virtual Reality Simulation
Participants “gained valuable knowledge” (Participant [P]1) and described VR simulation as “informative” (P28, P31), “educational” (P18), and a “brilliant learning opportunity” (P8). The VR technology was perceived as a “very pleasant” (P14), “fun to use” (P8), and “enjoyable way to learn and assess skills” (P7). The accessibility of VR simulation was highlighted as an added benefit: “I think I learned a lot from simulation therefore, VR makes simulation more accessible for me…” (P15).
Virtual reality simulation was perceived to promote “independence and clinical decision making” (P6), “facilitate learning by self-directive” (P25), and enable students to “engage with critical thinking” (P3). As a result, participants reported feeling “less panicked than doing in person simulations” (P21) and reported “not being limited by lack of familiarity with medical equipment” (P40).
The positive effect of VR simulation on patient safety featured strongly in participants' written responses whereby VR was described as “a very safe way to practice nursing skills” (P34) “without the fears of harming someone” (P1). Another participant wrote:
I enjoyed the idea of being able to experiment instead of dealing with a real patient. I feel that it could definitely benefit nurses and other allied health professionals. (P26)
Overall, VR simulation was perceived to “bring things to life” (P25) and “very lifelike” (P7, P18) since “patients presenting have real life scenarios” (P28). One participant highlighted “the realistic interaction” in terms of being “able to talk to the patient and the other nurse as if you were in a clinical situation” (P2). Participants often described VR as “very interactive” (P8, P9, P15) and engaging, particularly in the presence of a built-in feedback system: “The feedback part allowing me to know what I did well and what I can improve on” (P27), “prompts to run certain tests” (P40), “the ability to get advice if I was confused” (P27), and “enough instructions on how to check with patients” (P36). Another positive element of VR simulation was the availability of “lots of options with assessing the patient” (P20) and “options to choose from which helped you decide what to do” (P38) including “the process of examination” (P43), “respiratory and cardiac assessments” (P6), and “the treatment process” (P43), among others.
Challenges to Engaging With Virtual Reality Simulation
Engaging with VR simulation was not without challenges. For many, VR “was confusing at first” (P30), as it “took a bit of getting used to at the start” (P4). However, participants eventually “figured out how to use it [VR]” (P20) as evidenced in the following excerpts:
On initial session I felt like I was trying so hard to get used to the VR that the systematic nursing assessment was poor but I think this would improve as I got used to the VR. (P40)
I think it took me few mins before getting the hang of it...which I have expected. (P14)
At first it was hard to understand but when I found my way it was amazing. (P2)
It's a bit hard to navigate at first. In clinical scenario, we'd mostly have everything presented to us already so learning to navigate in VR is a bit difficult at first. (P15)
Technical issues were also reported. These related to the quality of the graphics: “Picture was a bit blurry” (P37) and “I could not read some of the results” (P24); the ill-fitting headset: “Poor fit of headset and needing to still wear glasses so fit can feel a little awkward” (P18); eyestrain: “It was a bit difficult on your eyes” (P6); confusing controllers: “Movement controls were confusing at times” (P39); and having to be seated during the experience: “Moving my head nearly 200 degrees…would be easier standing up” (P43).
Whereas some participants described VR simulation as lifelike, others reported that VR was not as interactive as real-life situations, particularly while performing skills: “When leaning, you need to physically do the skills and the VR simulation doesn't allow you to do actually do the skills” (P2). Moreover, some participants found it difficult to perform certain tasks such as “escalation of care” (P26) and “giving medication” (P9).
Recommendations for Integrating Virtual Reality Simulation Into Nursing and Midwifery Education
Participants made several recommendations to integrate VR simulation into nursing and midwifery education. Many believed that VR can be used “to practice clinical scenarios and develop critical thinking skills” (P3), “in clinical labs and assessments” (P41), and “to help adjust to clinical settings and procedures” (P37). Similarly, they recommended using VR to help students prepare for clinical placement. One participant wrote: “[VR] will prepare nurses and midwives well for when they have to deal with real patients” (P26). Another participant added:
I think it would be especially useful before going on placement for the first time to give us an idea of what to expect. As a first year, I had no idea what to expect so this would give first years a good idea. (P2)
Participants suggested incorporating VR into teaching, learning, and assessment through the use of case studies: “Use these case studies [VR simulation scenarios] and ask the patient questions to come up with a care plan and treatment” (P43); using diverse scenarios: “It [VR] can help bring different scenarios and one can see how they [students] will deal with it” (P16); teaching about rare clinical situations: “A variety of rare situations could be taught about where we wouldn't maybe be able to experience them in real life settings as students” (P17); incorporating VR into pre-existing courses: “Using it [VR] in clinical modules such as anatomy and physiology and pharmacology” (P27); using VR “for exams” (P2); and using VR instead of manikins in some situations: “It's [being] able to visualize bruising that you would be unable to do on a manikin as well as you can communicate to the person unlike manikins” (P39).
Access to VR was recommended by making VR equipment available in a designated location for students to use at their own leisure, “All the equipment would be in the VR room” (P22) and “along with simulations it would be great to have to practice in our own time like a designated area” (P23), as well as having VR as an option that students could avail of: “[VR] as an optional choice” (P18).
DISCUSSION
As a result of VR's ability to provide an immersive medium for learning, educators have and will continue to adopt and develop innovative VR teaching and learning strategies. Although the advantages of VR simulation are evident throughout the literature, limited evidence was available on the usability of incorporating VR simulation in nursing and midwifery education. To obtain initial insights into the potential for using VR technology for simulation training, in this usability study, the potential effectiveness, efficiency, and user satisfaction with VR simulation were explored.
Findings revealed that VR simulation was acceptable, and many participants were satisfied with using VR to learn about clinical scenarios. These findings are not surprising given the ever-increasing numbers of VR users4 and the support within the literature demonstrating the effectiveness of VR simulation training in delivering knowledge to students in an interactive and engaging way when compared with conventional training.10,11,43
Virtual reality is known to promote positive learning experiences, with evidence of improved learning outcomes with higher levels of immersion.44 Virtual reality being viewed as a safe environment to learn was highlighted as a positive aspect of use, a finding that is supported by the literature.11,34 Providing students with a safe environment to learn allows for trial and error, an opportunity not afforded with real-life patients.12,35 Virtual reality's ability to foster self-directed learning without added pressure was also highlighted. Virtual reality simulation with its ability to provide increased access and flexibility,45 in particular, is useful in enabling students to practice for a longer duration and to acquire clinical skills that are not always experienced in the clinical setting.12 Moreover, it is important to note that students who develop fundamental skills in virtual environments are more likely to consider patient safety in clinical practice.46 Additional benefits beyond the development of skills were identified in a recent randomized controlled trial that found that VR promoted knowledge retention, clinical reasoning, self-efficacy, and greater satisfaction with the learning experience among nursing students.47
In a recent systematic review and meta-analysis by Woon et al,10 VR was proposed as being suitable for supplementing conventional teaching and learning methods rather than being used as a standalone approach. This is not without challenges and warrants further exploration. Indeed, challenges to engaging with VR simulation were highlighted by current study participants, particularly in relation to navigating the VR environment and difficulties performing certain tasks. This highlights the need for supervision and continuous feedback to prepare students to use VR, assist them during VR, and debrief them following VR. Using VR in small rather than large classes and allowing students to hire/use VR at their own leisure in a designated area such as the library and computer laboratory were all practical solutions put forward by nursing students in previous research.35 Although not identified by current study participants, the implementation of VR in nursing and midwifery education serves as a learning curve for educators and has workload/teaching load implications—an issue that warrants exploration in future research with educators.35
In addition to pedagogical and technological barriers when seeking to optimize the user experience, there has been thought given to social considerations. Factors that have been described to influence the user experience with VR include age, sex, sensation-seeking tendency, personal innovativeness, and geographical location.48 For example, younger users are more likely to engage with a VR experience than older users,49 whereas older users are more likely to engage with a non-immersive experience.50 Sex also plays a significant role in the design and universality of VR immersive experiences.51 A recent study highlighted how males performed better when the VR avatar was a drone, rendered as a futuristic robot, whereas females fared better when the avatar was a female scientist.52
Recommendations for integrating VR simulation into nursing and midwifery education revolved mainly around using VR in preparation for clinical practice and making VR accessible. In their qualitative study with 26 nursing students, Saab et al35 found that VR technology can potentially facilitate learning, complement current educational approaches, and provide nursing educators with novel and engaging means of content delivery. Key to production and implementation will be action and collaboration by disciplinary and technical experts.51 Additionally, findings from the present study highlight not only the positive potential for VR integration into curricula but also the need for robust support systems and pedagogies to enhance student learning. Virtual reality simulation will deliver a new entry point for learning for students and new horizons of teaching approaches for academics to adopt as part of their practice.52 Keeping pace with healthcare developments will require an equally responsive and progressive approach to VR development to sustain the adoption of this exciting technology in higher education.
Limitations
More than half of the participants were first-time VR users. This could have led to more favorable results and introduced some degree of novelty bias.53 The use of non-probability sampling and asking students to choose the VR scenario increase the risk of self-selection bias and might have affected their opinion of the usability of VR. Although the sample size was large enough to detect usability problems,29 our study was not sufficiently powered to explore confounding relationships between sample characteristics and usability. Moreover, third- and fourth-year nursing and midwifery students, who tend to have more clinical experience, were underrepresented since they were on clinical placement at the time of data collection. This, coupled with the fact that student participation was sought from one university only, impacts negatively on the generalizability and transferability of findings.
CONCLUSION
Despite some technical and logistical challenges, participants were highly satisfied with VR simulation and recommended its implementation in various educational contexts, particularly as a resource to practice clinical skills, prepare for clinical placements, and learn about situations that are seldom experienced in clinical practice. Given the positive results from this usability study, the development and integration of VR simulation in nursing and midwifery education is recommended. However, the space for VR needs to be primed a priori, particularly that the development and deployment of VR simulation are resource heavy. For instance, the undertaking of a pilot study is recommended to help identify the most effective means of leveraging VR simulation and mitigate unforeseen problems. Virtual reality simulation, although novel and engaging, becomes futile if not underpinned by a strong pedagogy and aligned with learning outcomes. Finally, key stakeholders including students and educators need to be trained in VR use prior to implementing VR simulation.
Acknowledgment
The authors thank the nursing and midwifery students who participated in this study, Cathal O'Sullivan for helping with data collection, Rebecca McNamara and the School of Nursing and Midwifery's Executive Assistants for helping with recruitment, Patrick Hayes for designing the study poster, Dr Darren Dahly for providing statistical advice and expertise, and Oxford Medical Simulation for supplying the simulation scenarios.
References
1. Lopreiato JO, Downing D, Gammon W, et al.
The Terminology & Concepts Working Group. Healthcare Simulation Dictionary. 2nd ed. Rockville, MD: Agency for Healthcare Research and Quality; 2020:
https://doi.org/10.23970/simulationv2
2. Meta (2022). Buy Meta Quest 2. Get beat saber included.
https://store.facebook.com/ie/quest/products/quest-2/
5. Singh RP, Javaid M, Kataria R, Tyagi M, Haleem A, Suman R. Significant applications of
virtual reality for COVID-19 pandemic.
Diabetes and Metabolic Syndrome: Clinical Research and Reviews. 2020;14(4): 661–664.
6. Ulenaers D, Grosemans J, Schrooten W, Bergs J. Clinical placement experience of nursing students during the COVID-19 pandemic: a cross-sectional study.
Nurse Education Today. 2021;99: 104746.
7. Zulu BM, du Plessis E, Koen MP. Experiences of nursing students regarding clinical placement and support in primary healthcare clinics: strengthening resilience.
Health SA. 2021;26: 1615.
8. Foronda CL, Alfes CM, Dev P, et al. Virtually nursing: emerging technologies in nursing education.
Nurse Educator. 2017;42(1): 14–17.
9. Morin KH. Nursing education after COVID-19: same or different?
Journal of Clinical Nursing. 2020;9(17–18): 3117–3119.
10. Woon APN, Mok WQ, Chieng YJS, et al. Effectiveness of
virtual reality training in improving knowledge among nursing students: a systematic review, meta-analysis and meta-regression.
Nurse Education Today. 2021;98: 104655.
11. Zackoff MW, Real FJ, Sahay RD, et al. Impact of an immersive
virtual reality curriculum on medical students' clinical assessment of infants with respiratory distress.
Pediatric Critical Care Medicine. 2020;21(5): 477–485.
12. Benham-Hutchins M, Lall MP. Perception of nursing education uses of second life by graduate nursing students.
Computers, Informatics, Nursing. 2015;33(9): 404–409.
13. Chan HY, Chang HC, Huang TW.
Virtual reality teaching in chemotherapy administration: randomised controlled trial.
Journal of Clinical Nursing. 2021;30(13–14): 1874–1883.
14. Chao YC, Hu SH, Chiu HY, Huang PH, Tsai HT, Chuang YH. The effects of an immersive 3d interactive video program on improving student nurses' nursing skill competence: a randomized controlled trial study.
Nurse Education Today. 2021;103: 104979.
15. İsmailoğlu EG, Orkun N, Eşer İ, Zaybak A. Comparison of the effectiveness of the virtual simulator and video-assisted teaching on intravenous catheter insertion skills and self-confidence: a quasi-experimental study.
Nurse Education Today. 2020;95: 104596.
16. Hasyim AR, Syarif S, Ahmad M, Niswar M, Stang, Nasrudin AM. Enhance midwifery student skills about active management third stage labor via learning media.
Gaceta Sanitaria. 2021;35: S284–S287.
17. Lioce L, Lopreiato J, Downing D, et al., Spain AE & the Terminology and Concepts Working Group.
Healthcare Simulation Dictionary (AHRQ Publication No. 20-0019). 2nd ed. Rockville, MD: Agency for Healthcare Research and Quality; 2020:
https://doi.org/10.23970/simulationv2
18. Bracq MS, Michinov E, Jannin P.
Virtual reality simulation in nontechnical skills training for healthcare professionals: a systematic review.
Simulation in Healthcare. 2019;14(3): 188–194.
19. Peddle M, Mckenna L, Bearman M, Nestel D. Development of non-technical skills through virtual patients for undergraduate nursing students: an exploratory study.
Nurse Education Today. 2019;73: 94–101.
20. Fowler C.
Virtual reality and learning: where is the pedagogy?
British Journal of Educational Technology. 2015;46(2): 412–422.
21. Skivington K, Matthews L, Simpson SA, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance.
BMJ. 2021;374: n2061.
23. Tuena C, Pedroli E, Trimarchi PD, et al. Usability issues of clinical and research applications of
virtual reality in older people: a systematic review.
Frontiers in Human Neuroscience. 2020;14: 93.
24. Brooke J. SUS—a quick and dirty usability scale.
Usability Evaluation in Industry. 1996;189(194): 4–7.
25. Brooke J. SUS: a retrospective.
Journal of Usability Studies. 2013;8(2): 29–40.
26. Vermeeren AP, Law EL, Roto V, Obrist M, Hoonhout J, Väänänen-Vainio-Mattila K. User experience evaluation methods: current state and development needs. In:
Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries. 2010: 521–530.
27. Fernandes AS, Feiner SK. Combating VR sickness through subtle dynamic field-of-view modification. In: 2016 IEEE Symposium on 3D User Interfaces (3DUI). Greenville, SC: IEEE; 2016:201–210. doi: 10.1109/3DUI.2016.7460053
28. Alroobaea R, Mayhew PJ. How many participants are really enough for usability studies?. In: 2014 Science and Information Conference. London, UK: IEEE. 2014:48–56. doi: 10.1109/SAI.2014.6918171
29. Cazañas A, de San Miguel A, Parra E. Estimating sample size for usability testing.
Enfoque UTE. 2017;8: 172–185
https://doi.org/10.29019/enfoqueute.v8n1.126
30. Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing.
Behavior Research Methods, Instruments, & Computers. 2003;35(3): 379–383.
31. Nielsen J.
Usability Engineering. San Fransisco, CA: Morgan Kaufmann; 1994.
32. Virzi RA. Refining the test phase of usability evaluation: how many subjects is enough?
Human Factors. 1992;34(4): 457–468.
33. Brown KM, Swoboda SM, Gilbert GE, Horvath C, Sullivan N. Integrating virtual simulation into nursing education: a roadmap.
Clinical Simulation in Nursing. 2022;72: 21–29.
34. Mallik R, Patel M, Atkinson B, Kar P. Exploring the role of
virtual reality to support clinical diabetes training—a pilot study.
Journal of Diabetes Science and Technology. 2022;16(4): 844–851.
35. Saab MM, Hegarty J, Murphy D, Landers M. Incorporating
virtual reality in nurse education: a qualitative study of nursing students' perspectives.
Nurse Education Today. 2021;105: 105045.
36. Saab MM, Landers M, Murphy D, et al. Nursing students' views of using
virtual reality in healthcare: a qualitative study.
Journal of Clinical Nursing. 2022;31(9–10): 1228–1242.
37. Saab MM, Landers M, Cooke E, Murphy D, Hegarty J. Feasibility and usability of a
virtual reality intervention to enhance men's awareness of testicular disorders (E-MAT).
Virtual Reality. 2019;23(2): 169–178.
39. Sauro J (2022). 5 ways to interpret a SUS score—MeasuringU.
https://measuringu.com/interpret-sus-score/
40. Hinton PR, Brownlow C, McMurray I, Cozens B.
SPSS Explained. London, England: Routledge; 2004.
41. Elo S, Kyngäs H. The qualitative content analysis process.
Journal of Advanced Nursing. 2008;62(1): 107–115.
42. Cope DG. Methods and meanings: credibility and trustworthiness of qualitative research.
Oncology Nursing Forum. 2014;41(1): 89–91.
43. Vlachopoulos D, Makri A. The effect of games and simulations on higher education: a systematic literature review.
International Journal of Educational Technology in Higher Education. 2017;14(1): 1–33.
44. Farra SL, Smith SJ, Ulrich DL. The student experience with varying immersion levels of
virtual reality simulation.
Nursing Education Perspectives. 2018;39(2): 99–101.
45. Mendez KJW, Piasecki RJ, Hudson K, et al. Virtual and augmented reality: implications for the future of nursing education.
Nurse Education Today. 2020;93: 104531.
46. Bayram SB, Caliskan N.
The Use of Virtual Reality Simulations in Nursing Education, and Patient Safety. Contemporary Topics in Patient Safety-Volume. London, England: IntechOpen; 2020.
47. Padilha JM, Machado PP, Ribeiro A, Ramos J, Costa P. Clinical virtual simulation in nursing education: randomized controlled trial.
Journal of Medical Internet Research. 2019;21(3): e11529.
48. Suh A, Prophet J. The state of immersive technology research: a literature analysis.
Computers in Human Behavior. 2018;86: 77–90.
49. Coxon M, Kelly N, Page S. Individual differences in
virtual reality: are spatial presence and spatial ability linked?
Virtual Reality. 2016;20(4): 203–212.
50. Plechatá A, Sahula V, Fayette D, Fajnerová I. Age-related differences with immersive and non-immersive
virtual reality in memory assessment.
Frontiers in Psychology. 2019;10: 1330.
51. Reen FJ, Jump O, McSharry BP, et al. The use of
virtual reality in the teaching of challenging concepts in virology, cell culture and molecular biology.
Frontiers in Virtual Reality. 2021;2: 670909.
52. Makransky G, Wismer P, Mayer RE. A gender matching effect in learning with pedagogical agents in an immersive
virtual reality science simulation.
Journal of Computer Assisted Learning. 2019;35(3): 349–358.
53. Acerbi A, Bentley RA. Biases in cultural transmission shape the turnover of popular traits.
Evolution and Human Behavior. 2014;35(3): 228–236.