Secondary Logo

Evaluating the Usability of a Second-Generation Virtual Reality Game for Refreshing Sterile Urinary Catheterization Skills

Kardong-Edgren, Suzan, PhD, RN, ANEF, CHSE, FSSH, FAAN; Breitkreuz, Karen, EdD, MSN, RN, CNE, CHSE; Werb, Michael, BSN, RN; Foreman, Stephen, PhD; Ellertson, Anthony, PhD

doi: 10.1097/NNE.0000000000000570
Feature Articles

Background Virtual reality (VR) allows risk- and anxiety-free practice, mediated by consistent objective feedback.

Purpose This study evaluated the usability of a VR game system for sterile catheterization practice. Participant enjoyment, engagement, likelihood to practice, and comfort using VR are reported.

Methods Thirty-one students and faculty tested a VR game using Oculus Rift devised to allow practice of placing a urinary catheter in a virtual patient. Data were collected via an electronic survey using the System Usability Survey (SUS) and a User Reaction Survey (URS).

Results The SUS score was 64.03. Seventy-five percent of participants rated the game as positive overall on the URS. Left-handed players had more difficulty playing the game. Players with prescription glasses could not comfortably place the Oculus Rift over their glasses to play.

Conclusions The VR game shows promise for refreshing sterile catheterization skills.

Author Affiliations: Professor and RISE Center Director (Dr Kardong-Edgren) and RISE Center Graduate Assistant (Mr Werb), School of Nursing and Health Sciences, and Associate Professor (Dr Foreman), Robert Morris University, Moon Township, Pennsylvania; and CQ Certified, Associate Professor (Dr Breitkreuz), School of Nursing, and Director and Clinical Associate Professor (Dr Ellertson), Games, Interactive Media and Mobile Program, College of Innovation & Design, Boise State University, Idaho.

A.E. and S.K.-E. developed the game and could have a perceived conflict of interest in the study and its results. The other authors declare no conflicts of interest.

Correspondence: Dr Kardong-Edgren, School of Nursing and Health Sciences, Robert Morris University, 6001 University Blvd, Moon Township, PA 15108 (

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (

Accepted for publication: May 12, 2018

Published ahead of print: July 30, 2018

Cite this article as: Kardong-Edgren S, Breitkreuz K, Werb M, Foreman S, Ellertson A. Evaluating the usability of a second-generation virtual reality game for refreshing sterile urinary catheterization skills. Nurse Educ. 2019;44(3):137-141. doi: 10.1097/NNE.0000000000000570

Virtual simulation is now the third most common educational technology used in nursing education.1 Virtual reality (VR) is one form of virtual simulation that is usually characterized by its immersive nature, 3-dimensional characteristics, motion sensors, and haptic devices.2 Nurses are increasingly called upon to evaluate new technologies such as VR for their potential roles in education and competency evaluation.

VR is of particular interest to nurse educators for use in teaching as well as refreshing and validating psychomotor skills. Previous research suggests that although many psychomotor skills are taught and tested, they are not well retained over time.3-5 The time involved with an educator rescheduling, refreshing, and remediating previously taught skills and the costs of supplies required for those refreshing opportunities may be mitigated by a VR skills game. These games are built to provide immediate accurate feedback and can be practiced alone, without supervision. VR technology may be the ideal learning modality for increasingly technologically savvy students and practicing nurses.

Traditional skills teaching has involved a “see one, do one” check-off model that, although superficially efficient, does not produce the necessary learning results.6,7 Adding to this difficulty is the lack of practice opportunities for many high-risk, low-occurrence clinical skills in hospitals, such as sterile urinary catheterization. A VR game could allow practice, mediated by consistent and objective feedback, anywhere, anytime a game is available.8 VR games can be built with a scoring dashboard to allow both learner and instructor to evaluate skills development or deterioration. A game must have a background within the game that simulates the look of a practice setting to allow a player to suspend disbelief.9 Gaming also allows previously unimagined opportunities to exaggerate key components of a skill. Exaggerating key concepts during learning results in that learning to be retained over time.10 For example, although sterile technique and the fact that germs can fall off one’s arms into a sterile field are stressed verbally during lectures, the actual visualization of germs actively falling into a sterile field and piling up while performing a sterile catheterization causes great distress and motivation in a participant. Sterile technique challenges are imprinted.

Back to Top | Article Outline


This article reports findings of a system usability test of a second-generation version of a VR game for sterile catheterization practice. This game version incorporated Oculus Rift (Oculus VR LLC, San Jose, California) with haptics11 that captured arm and hand movements in a VR space. Haptics refers to touch and hand movements to control or interact with VR and is considered essential to immersion in the VR environment. Results of the System Usability Survey (SUS)12 and a User Reaction Survey (URS)13 on participant enjoyment, engagement, likelihood to practice, and comfort using a VR game are reported.

The first version of the game included haptics using special haptic gloves that allowed participants to visualize their own hands in the game.13 This second version eliminated use of the gloves and used Leap motion sensors (Leap Motion, San Francisco, California) to capture the players’ hand and finger movements. Both versions were designed to show the players their own movements and prompt them to complete sterile urinary catheterization on a virtual manikin, in a systematic stepwise fashion. Players were able to pick up the virtual catheter and prepare the virtual task trainer with virtual supplies that look like supplies that they would really use. They also saw germs, which are not visualized in a simulation laboratory or at a patient’s bedside. The authors reported that the game was easy to use and that users were confident in using the system and would use it frequently. Many users thought they would need technical support to use the system. Some users developed vertigo during the game; recent improvements embedded in the headgear could resolve this issue for most participants.

Back to Top | Article Outline

Literature Review

The use of VR in nursing education to date is mostly descriptive in nature.8,14,15 Ulrich et al16 reported the successful use of a VR game to teach decontamination skills. Vottero17 demonstrated that a virtual medication administration machine could be used to mirror real-world medication withdrawal.

One VR game for teaching urinary catheterization skills was found in the literature. Smith and Hamilton18 reported findings from a 2-group study evaluating the use of a VR game developed to promote deliberate practice of Foley catheter insertion after the initial teaching. The screen-based VR game could be played from any location and used a mouse to maneuver through the game. A unique feature of this game was a buzzer that sounded if a step was performed incorrectly. Both groups were initially taught the skill in the traditional manner. However, the experimental group then practiced this skill using the screen-based game with mouse; the control group practiced in the traditional manner. The traditional learners spent approximately a half hour more practicing than the experimental group; however, the experimental group completed more steps in the final check-off procedure. VR gaming holds great promise for engaging participants and lessening the instructor burden of teaching some clinical skills. Additional quantitative studies will help establish this.

Back to Top | Article Outline

Theoretical Framework

Students often practice psychomotor skills only enough to demonstrate mastery for a skills check-off. Schmidt and Lee19 suggest this approach produces only the cognitive level of skills achievement; a participant memorizes the steps to do a skill and can remember them for a short period. Using this approach, participants do not practice enough to reach the associative level of skill learning. At the associative level, participants develop increased speed, dexterity, and a more nuanced skill performance. After enough practice, a skill becomes habituated and can be performed without thought. This is called the autonomous level of skill development.

Recent research suggests that as little as 20 minutes of additional practice, after initial skill mastery, may significantly improve skill retention by changing neurochemical brain processes.20 These chemical changes stabilize skills and prevent learning disruption, even if additional new skills are taught immediately. This extra practice, called overlearning, results in better skill retention. Without it, both the original skill and the learning and retention of additional skills are inhibited. VR gaming may provide an engaging practice method that will encourage overlearning and learning at the associative level or higher. The time needed to reach autonomous skill development is unknown.

Back to Top | Article Outline


Setting and Sample

This study was conducted at a small private university in a Mid-Atlantic state, after institutional review board approval was obtained. The sample included undergraduate juniors and senior nursing students and nursing faculty. All participants had already completed sterile urinary catheter training and had been checked off on the skill at least once already during their nursing program. Participants signed up for a half-hour block of time for study participation. Participants were recruited by word of mouth, class announcements, and snowball sampling. No extra credit or incentives were provided for study participation.

Back to Top | Article Outline


The SUS (α = .92) has been used in more than 1300 previous studies and is freely available for use.12,21 It consists of a 10-item scale with questions about effectiveness, efficiency, and satisfaction with a system such as software, hardware, and applications. Scores range from 0 to 100. Adjectives associated with scores are as follows: an “average” score is 50.9 (SD, 13.8), a “good” score is 71.4 (SD, 11.6), and an “excellent” score is 85.5 (SD, 10.4).12 Scores represent a measure of the perceived usability of a product.

A URS designed by Butt et al13 was used to measure participants’ assessment of the game compared with traditional task trainer or manikin catheter teaching and refresher methods. Validity of the tool was established by 2 subject matter experts who reviewed the tool in the first iteration of the study; reliability data were not calculated on this instrument in its initial use. In the URS, participants were asked to consider their experience with the VR catheter insertion game and respond to the questions using a 5-point scale anchored by strongly agree and strongly disagree. These items addressed issues regarding use and design of the system, engagement with the system, and self-ratings of how much participants would use or learn from and practice in the system. Questions were posed using both positive and negative framing words to reduce framing bias.22 Several items also pertained to the user’s overall confidence and satisfaction with the gaming system.

These 2 instruments were combined into 1 continuous electronic survey. The survey system created unique identifiers for each participant and was completed on an iPad (Apple Inc, Cupertino, CA). Demographic data included age, gender, ethnicity, faculty or semester in the nursing program, and several questions about game-playing behavior and comfort with technology and learning with simulation.

Back to Top | Article Outline


After signing a consent, participants opened a survey link on an iPad to answer demographic and additional questions about previous gaming experience. This link was left open on the device, as participants then moved to the VR game. Participants were fitted with the headgear by the study graduate assistant and positioned at the tip of an imaginary triangle approximately 8 to 10 ft away from 2 Leap motion sensors, forming the triangle base (Photograph, Supplemental Digital Content, The VR headset immediately immersed the participant in a 360-degree view of a hospital room with a female manikin torso draped with a sheet. The participant’s hands appeared as they do in real life if the participant held his/her hands within the Leap motion sensor fields, about waist high.

The game required the participants to refresh their sterile urinary catheterization skills by opening a sterile urinary catheter tray, don sterile gloves, prepare the catheter tray and supplies, and then properly catheterize the torso. The required steps were available in writing in the participant’s left hand, if he/she flipped it palm up facing the participant, at any time. A clock for timing was in view on the back wall of the game, and a scoring system incorporating time on task, number of field contaminations, and the user’s fluidity of movement was built into the game to provide additional motivation and feedback.

A key feature built into the game to emphasize germs was a green cloud with small particles falling from it that appeared when a participant’s hands were not gloved sterilely or when sterile technique was broken. This feature was visible when the participant’s arms were moving about within the game field. This critical feature was exaggerated10 to clearly emphasize what happened with and without sterile technique and why it was so important.

Participants could play the game as many times as they wanted within a half-hour time frame. Upon completion of the game or their time, they completed the usability and experience in the electronic survey. The total practice time and participant score were recorded within the game.

Back to Top | Article Outline


Thirty-one participants completed the study, including 12 men, and 2 were female faculty members. For the SUS, the response ratings for all the participants were averaged. This score was 64.03 and was within the SD of a “good” rating for the survey.12

The raw coefficient α for the URS was .826 in this study.13 The URS results were divided into 4 categories reported in Figure 1-4, Supplemental Digital Content,,,, Figures 1 and 2 report user responses to the system itself. Figures 3 and 4 include user responses regarding their engagement while playing the game and perceptions on use of the game to learn and practice skills.

As seen in Figure 1, Supplemental Digital Content,, 75% of participants rated the game positively overall. Seventy-seven percent said they would like to use the system frequently. Fifty-eight percent thought the system was easy to use, and 80% agreed or strongly agreed they would learn to use the system quickly. In Figure 2, Supplemental Digital Content,, users reported that they would not need to learn more to use the system, and they did not find the system cumbersome, inconsistent, or unnecessarily complex.

In Figure 3, Supplemental Digital Content,, reports learner engagement with the game. Ninety-three percent thought playing the game was fun; 87% felt engaged in their own learning. Supplemental Digital Content, Figure 4,, reports items pertaining to learning and practicing skills in the game. Seventy-seven percent reported working to improve their score and practice time, and 74% reported the game would help them learn to insert a urinary catheter correctly. Ninety percent reported getting feedback when needed; 87% reported the game was challenging.

A Spearman correlation analysis was conducted to explore relationships among demographic variables and questions about use. Respondents answered questions about their professional background in terms of their year in college or degree status. More experienced professionals were more likely to find the VR simulation cumbersome (r = 0.407, P = .023), provided lower overall ratings (r = −0.373, P = .039), and were less likely to experience the exercise as fun (r = −0.374, P = .038). More professional experience correlated negatively with the belief that the game would help them professionally (r = −0.415, P = .020).

Male participants were more likely to spend more free time with computer games (r = −0.355, P = .050), identify themselves as a “gamer” (r = −0.598, P < .001), and report that they are comfortable with technology (r = −0.390, P = .030). Females were more likely to find that elements of the game challenged them (r = 0.407, P = .023) and less likely to conclude that they could easily concentrate on aseptic technique (r = −0.375, P = .037). Females were more likely to rate the simulation as high quality (r = 0.387, P = .032).

Self-reported gamers were more likely to rate VR simulation highly (r = 0.344, P = .058) and to be comfortable with technology (r = 0.560, P = .001). Gamers were less likely to find the simulation demanding (r = −0.371, P = .040). They found that they could easily concentrate on aseptic procedures during the simulation (r = 0.365, P = .043). People who spent more time playing video games were less likely to see the VR simulation as demanding (r = −0.498, P = .004) and more likely to conclude that the simulation provided no challenge (r = 0.369, P = .041). Comfort with technology was positively associated with simulation scores (r = 0.508, P = .004). Respondents were asked to identify their dominant hand. Right-handed respondents had higher scores (r = 0.562, P = .001), learned more quickly (r = 0.380, P = .035), and found it easier to concentrate on aseptic technique (r = −0.402, P = .025).

Back to Top | Article Outline


High-risk, low-occurrence skills such as catheterization are ideal for refreshing in a gaming environment. An interactive game providing consistent accurate feedback could provide “just-in-time” refresher training and could make an impact on patient safety and care. In this study, 75% of positive questions on general usage were answered using agree or strongly agree. Users did not report a great deal of difficulty using the game. This is a significant difference from the first version of this game13 in which users reported thinking they would need technical assistance to use the game. Vertigo was rarely reported in this version of the game; improvements embedded in the headgear seem to have resolved many previous issues. Findings indicate that improvements support the user experience.

Self-reported gamers adapted quickly to the environment and were able to concentrate on sterile technique. The amount of extra learning required to learn to play a new game as well as acquire a new skill must be considered when adopting such a teaching modality. However, the anticipated increased use of games using VR in emerging platforms may mediate the current novelty of VR.

Initial mastery of any skill should be followed by a period of overlearning, described as continued practicing of the skill after initial mastery and performance having plateaued.21 This allows the learner to move from the cognitive to the associative level of learning.20 The challenge of getting a good score and mastery of the skill sequences and sterility requirements may motivate new learners to practice long enough to reach the associate level of learning, as suggested by Schmidt and Lee.19 Recent research suggests that as little as 20 minutes of additional practice, after initial mastery, significantly improves skill retention.20 If the ability to practice without a coach nearby to assist can be accomplished with VR measures, more new skills may be practiced and retained over time. Additional refreshing of skills over time with spaced learning may lead to more skill retention.

Results from this study demonstrated positive usability measures, similar to the findings by Smith and Hamilton,18 who also evaluated a Foley catheter insertion game. They used a computer screen and mouse for skills practice; this game used Leap motion for a more authentic haptic experience. Questions remain for all gaming platforms about the novelty of learning or practicing with a game waning over time. Some participants may already be unengaged when returning for mandatory deliberate practice in the skills laboratories. A gaming component built into practice sessions with a leader board for scoring accuracy might help with engagement. A major question left unanswered thus far is whether the sterile catheterization skill can be taught solely with the game as the educator. A future study will address this.

We experienced 2 unanticipated findings in our study. One was that left-handed players reported more difficulty playing the game. For example, the sterile catheter tray and bedside table currently appear only on the left-hand side of the patient in the bed. The game needs to be able to orient for both right- and left-handed players. A second finding was that players who wore prescription glasses could not comfortably wear the game headgear on their heads while wearing glasses. Minor visual modifications for nearsightedness and farsightedness can be placed into the headgear, but it was not enough to compensate for some prescriptions, limiting usefulness of the game.

Back to Top | Article Outline


This study evaluated a second generation of a VR game using haptics to teach sterile urinary catheterization. Findings indicated that usage of the game was positive and entertaining. A left-handed version of the game and provision for glasses wearers would enhance the usability of the game.

Back to Top | Article Outline


Forneris S, Tiffany J. Future of technology in nursing education part 2: how nursing education programs are currently using educational technology. NLN Center for Innovation in Simulation and Technology. Available at,2017. Accessed April 12, 2018.
Lopreiato JO, Downing D, Gammon W, et al, eds. Healthcare Simulation Dictionary. 2016. Available at Accessed April 12, 2018.
Barsuk JH, Cohen ER, Mikolajczak A, Seburn S, Slade M, Wayne DB. Simulation-based mastery learning improves central line maintenance skills of ICU nurses. J Nurs Adm. 2015;45(10):511–517.
Cason M, Atz T, Horton L. New nurse graduates’ self-efficacy ratings and urinary catheterization skills in a high-fidelity simulation scenario. Clin Sim Nurs. 2017;13(2):71–77.
Missen K, McKenna L, Beauchamp A. Registered nurses’ perceptions of new nursing graduates’ clinical competence: a systematic integrative review. Nurs Health Sci. 2016;18(2):143–153.
Gonzalez L, Sole ML. Urinary catheterization skills: one simulated checkoff is not enough. Clin Sim Nurs. 2014;10(9):455–460.
Kardong-Edgren S, Mulcock P. Angoff Method of setting cut scores for high-stakes testing: Foley catheter checkoff as an exemplar. Nurse Educ. 2016;41(2):80–82.
Jenson CE, Forsyth DM. Virtual reality simulation: using three-dimensional technology to teach nursing students. Comput Inform Nurs. 2012;30(6):312–318; quiz 319-320.
Rizzo AS, Koenig ST. Is clinical virtual reality ready for primetime? Neuropsychology. 2017;31(8):877–899.
Dror I. Technology enhanced learning: the good, the bad, and the ugly. Pragmat Cogn. 2008;16(2):215–223.
Haptics. Available at Accessed April 12, 2018.
Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud. 2009;4(3):114–123.
Butt A, Kardong-Edgren S, Ellertson A. Using game-based virtual reality with haptics for skill acquisition. Clin Sim Nurs. 2018;16:25–32.
Ferguson C, Davidson PM, Scott PJ, Jackson D, Hickman LD. Augmented reality, virtual reality and gaming: an integral part of nursing. Contemp Nurse. 2015;51(1):1–4.
Foronda CL, Alfes CM, Dev P, et al. Virtually nursing: emerging technologies in nursing education. Nurse Educ. 2017;42(1):14–17.
Ulrich D, Farra S, Smith S, Hodgson E. The student experience using virtual reality simulation to teach decontamination. Clin Sim Nurs. 2014;10(11):546–553.
Vottero BA. Proof of concept: virtual reality simulation of a Pyxis machine for medication administration. Clin Sim Nurs. 2014;10(6):325–331.
Smith PA, Hamilton BK. The effects of virtual reality simulation as a teaching strategy for skills preparation in nursing participants. Clin Sim Nurs. 2015;11(1):52–58.
Schmidt RA, Lee TD. Motor Control and Learning: A Behavioral Emphasis. 4th ed. Champaign IL: Human Kinetics; 2015.
Shibata K, Sasaki Y, Bang JW, et al. Overlearning hyperstabilizes a skill by rapidly making neurochemical processing inhibitory-dominant. Nat Neurosci. 2017;20(3):470–475.
System Usability Scale. The Web site. Available at Published 2018. Accessed January 27, 2018.
Framing bias. Available at Published October 2003. Accessed April 11, 2018.

gaming; nursing education; simulation; virtual reality

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved