Secondary Logo

Journal Logo

Measuring Sense of Presence and User Characteristics to Predict Effective Training in an Online Simulated Virtual Environment

De Leo, Gianluca PhD, MBA; Diggs, Leigh A. MPH; Radici, Elena MS; Mastaglio, Thomas W. PhD

Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare: February 2014 - Volume 9 - Issue 1 - p 1–6
doi: 10.1097/SIH.0b013e3182a99dd9
Empirical Investigations
Free

Introduction Virtual-reality solutions have successfully been used to train distributed teams. This study aimed to investigate the correlation between user characteristics and sense of presence in an online virtual-reality environment where distributed teams are trained. A greater sense of presence has the potential to make training in the virtual environment more effective, leading to the formation of teams that perform better in a real environment. Being able to identify, before starting online training, those user characteristics that are predictors of a greater sense of presence can lead to the selection of trainees who would benefit most from the online simulated training.

Methods This is an observational study with a retrospective postsurvey of participants’ user characteristics and degree of sense of presence. Twenty-nine members from 3 Air Force National Guard Medical Service expeditionary medical support teams participated in an online virtual environment training exercise and completed the Independent Television Commission–Sense of Presence Inventory survey, which measures sense of presence and user characteristics. Nonparametric statistics were applied to determine the statistical significance of user characteristics to sense of presence.

Results Comparing user characteristics to the 4 scales of the Independent Television Commission–Sense of Presence Inventory using Kendall τ test gave the following results: the user characteristics “how often you play video games” (τ(26) = −0.458, P < 0.01) and “television/film production knowledge” (τ(27) = −0.516, P < 0.01) were significantly related to negative effects. Negative effects refer to adverse physiologic reactions owing to the virtual environment experience such as dizziness, nausea, headache, and eyestrain. The user characteristic “knowledge of virtual reality” was significantly related to engagement (τ(26) = 0.463, P < 0.01) and negative effects (τ(26) = −0.404, P < 0.05).

Conclusions Individuals who have knowledge about virtual environments and experience with gaming environments report a higher sense of presence that indicates that they will likely benefit more from online virtual training. Future research studies could include a larger population of expeditionary medical support, and the results obtained could be used to create a model that predicts the level of presence based on the user characteristics. To maximize results and minimize costs, only those individuals who, based on their characteristics, are supposed to have a higher sense of presence and less negative effects could be selected for online simulated virtual environment training.

From the Virginia Modeling Analysis and Simulation Center (G.D.L.), and College of Health Sciences (G.D.L., L.A.D., E.R.), Old Dominion University, Norfolk; and MYMIC LLC (T.W.M.), Portsmouth, VA.

Reprints: Gianluca De Leo, PhD, MBA, Virginia Modeling Analysis and Simulation Center, College of Health Sciences, Old Dominion University, Health Sciences Building Suite 2122, Norfolk, VA 23529 (e-mail: gdeleo@odu.edu).

The authors declare no conflict of interest.

This research was sponsored by the Office of the Secretary of Defense and supervised by the U.S. Air Force Research Laboratory under SBIR Topic OSD06-H04, Phase II Contract #8650-07-M-6711.

Simulation-based training is an instructional technique designed to accelerate expertise through skill development, practice, and feedback in settings that replicate real world environments. Simulation offers effective learning through active learner engagement, repetitive practice, and the ability to make the environment more challenging.1 Virtual reality and simulation techniques have been used to successfully train distributed teams. A major reason for using simulation for team training is the ability to reproduce a high-risk environment. Team members can learn how to manage high cognitive loads and how to make good decisions under stress. The US Navy found that teams, whose members knew each other, display higher performance levels, faster reaction times, greater accuracy, and greater mission success when performing military operations compared with teams where members were unfamiliar with each other.2 Human performance is strongly influenced by the interaction among the task, the environment, and the behavior of team members.3

Sense of presence is a state of consciousness, the psychological sense of being in a virtual environment.4 The International Society for Presence Research5 defines presence as the sense of “being there” in a virtual environment. There are 3 different types of presence as follows: personal presence, a measure of the degree to which a person feels like they are part of the virtual environment (VE); social presence, the degree to which other living or synthetic beings also exist in the VE; and environmental presence, the degree the environment itself responds to the person in the VE.4,6 User characteristics, including individual properties such as age, sex, cultural variables, and previous mediated experiences, impact an individual’s sense of presence.6,7 Other dimensions affecting sense of presence include physical space or spatial presence (the sense of physical placement in the mediated environment and the interaction and control over different parts of the environment), engagement (tendency to feel psychologically involved and to enjoy the content), ecologic validity (tendency to perceive the environment as lifelike or real), and negative effects (tendency to have adverse physiologic reactions such as dizziness, nausea, headache, and eyestrain).7,8

The experience of the user is especially important when determining individual levels of presence.9 A greater sense of presence makes the learning experience engaging and relevant.10 The users of a VE experience thoughts, emotions, and behaviors similar to those experienced in real-life situations. Because the user makes mistakes and failures in a protected environment, they can reflect upon them. Sense of presence during the learning experience is also important afterward when users recall the training experience to solve problems and manage situations in a professional context. Higher sense of presence and emotional involvement have been found to lead to higher possibility of recalling the training event through associations.10

This article studies the user characteristics and the sense of presence of 3 Air Force National Guard Medical Service expeditionary medical support (EMEDS) teams that have to perform in a simulated training environment using an online 3-dimensional (3D) virtual reality solution known as GaMeTT [Game Medical Team Training]. The goal of the study was to measure sense of presence and determine which user characteristics are related to a greater sense of presence. A greater sense of presence has the potential to make training in the VE more effective, leading to more efficient and effective EMEDS teams.

Back to Top | Article Outline

METHODS

To achieve our research goal, we designed an observational study with a retrospective postsurvey of user characteristics and measure of sense of presence. Institutional review board approval was obtained for this study.

Back to Top | Article Outline

Participants

An EMEDS is a rapid-response distributed medical team deployed to forward locations during humanitarian relief, wartime contingencies, and disaster response operations. A deployed EMEDS team is composed of highly qualified physicians, nurses, radiologists, surgeons, pharmacists, dentists, laboratory technicians, and communication specialists supported by logistics, administrative, and security staff. Each member of the team is a volunteer. Team members are usually called separately from different locations and then put together as a team with an assigned leader. Distributed teams suffer from geographic separation and difficulties in communication and coordination.11 A typical EMEDS camp is composed of sets of portable tents, equipment, and personnel built in a “modular fashion” to provide increasing medical capabilities for a large number of patients.12 Expeditionary medical support teams are required to be regularly trained. Formal EMEDS training is a 5-day course taught 20 times a year with a class size of 50 to 75 students. After the completion of the formal course, members must participate in another formal training or attend a sustainment training every 24 months.13

Twenty-nine subjects participated in the study. The age of the sample ranged from early 20s to late 50s, with 55% of the sample younger than 40 years. The professions of the sample were nurses (28%), followed by physicians (21%), medics (17%), students (14%), and administrators, x-ray service, medical administrators, medical technicians, and radiology/software professional (3% each). All team members were certified to be part of an EMEDS team. The subjects were predominately male (59% male, 41% female). During the experiment, subjects were given the same role they had in their real EMEDS team. Table 1 shows the user characteristics of the subjects.

TABLE 1

TABLE 1

Back to Top | Article Outline

GaMeTT

GaMeTT is a 3D interactive simulation designed to train universally distributed teams, such as EMEDS, in an immersive, avatar-based, VE (Fig. 1). GaMeTT being built on OLIVE software platform, participants are able to log onto GaMeTT from any secure Internet connection, and they can work to achieve the team training objectives. GaMeTT users are required to have neither high-end computers nor high technology skills. The mouse and arrow buttons on keyboards control the movements of the avatar. A headset and microphone are used for communication.

FIGURE 1

FIGURE 1

Back to Top | Article Outline

Study Design

Three teams of EMEDS composed of 29 volunteers were recruited from a Patriot Exercise training at Volk Field Air National Guard Base located in Juneau County, Wis. Volunteers were assigned to 3 EMEDS teams from Oregon, Indiana, and Canada. Teams I and II were composed of 10 participants each; team III had only 9 participants.

Initially, all the subjects were grouped together in a classroom-type environment with tables positioned in a semicircle. A thirty-minute video highlighted the capabilities of GaMeTT. Then, forty-five minutes were allotted for an introductory training session regarding the basics of navigating the virtual world and the GaMeTT Training System. Subjects learned how to login into GaMeTT, how to move the avatars in the simulated world, how to select objects, how to communicate using the headsets, and how to operate the medical model. A wireless access point was available in the building. After the introductory training session, to simulate the geographic distance between team members as it would be in the ultimate use of the GaMeTT system, the subjects were scattered throughout various rooms in the building.

Subjects were asked to perform their duties as an EMEDS team in a simulated scenario. The scenario lasted for 1 hour 55 minutes. In our scenario, the participants belonged to an EMEDS team that was deployed inside a military base. The participants had the following roles: nurse (3), doctor (1), medical technician (3), administrative officer (1), administrative technician (1), and team leader (1). Team III operated with one less nurse. Team members logged into the GaMeTT training system received an initial briefing from the team leader and waited for the scenario to start. The team reacted to the scenario events using the same supplies and equipment available in real situations. The team also followed the same procedures. Participants performed diagnostic and treatment interventions on virtual patients. Virtual patients responded with appropriate physiologic changes, such as an increase in heart rate, decrease in blood pressure, and increase in respiratory rate. The scenario was created so that the team had to handle military and civilian casualties from 2 events, an explosion and a car accident off base. Table 2 shows the matrix of the events and the flow of patients used in the scenario. The patient cue began with the first virtual patient arriving at the EMEDS site. The virtual patient had a piece of glass in his leg. The virtual patient was stabilized by the team and transported away from the scene by ambulance. The next virtual patient was brought in by helicopter with a lacerated liver. The patient was not stabilized appropriately and died. Real people in Virginia operated the training system. These people controlled the behaviors and movement of the virtual patients, operated the ambulances and helicopter, and generated the 2 events (explosion and accident off base). At the end of the training, an online debriefing session was conducted allowing the team to review and discuss their performances by starting, stopping, and replaying the entire scenario.

TABLE 2

TABLE 2

Back to Top | Article Outline

Outcome Measure

The Independent Television Commission–Sense of Presence Inventory (ITC-SOPI) is a 44 item, self-report questionnaire used to measure sense of presence during and after a media experience, with no reference to objective system parameters. This means that the results collected are subjective to the user, and none of the questions refer to the simulated environment used. The assessment is given after the media experience and is divided into 3 parts. The first part collects demographics and characteristics of the users. The second part has 6 questions, which focus on user impressions and feelings once the media experience is complete. The third part has 38 questions, which focus on user impressions and feelings during the media experience. The questionnaire measures 4 factors as follows: physical space (19 items), engagement (13 items), ecologic validity (5 items), and negative effects (6 items). The 44 items use a 5-point Likert scale, with 1 corresponding to “strongly agree” and 5 corresponding to “strongly disagree.”8,14,15

The ITC-SOPI has proven to be a reliable and valid measure of sense of presence. The ITC-SOPI questions relate to 15 content factors deemed relevant to presence on the basis of theoretical and empirical articles including sense of space, involvement, attention, distraction, control and manipulation, realness, naturalness, perception of time, awareness of behavioral responses, sense of social interaction, personal relevance, arousal, and negative effects (adverse physiologic reactions due to the VE experience such as disorientation, tiredness, dizziness, eyestrain, nausea, and headache). The initial version of the ITC-SOPI was administered to 604 individuals after an experience with one of a range of noninteractive to interactive media including IMAX 3D, IMAX 2D, cinema, video shorts, college film night, or PC game.14 The ITC-SOPI has been used in numerous research studies involving simulated environment and VE.16–19

Back to Top | Article Outline

Statistical Analysis

Survey data were analyzed using SPSS 19.0. Means of the 3 groups were examined against the 4 scales of the ITC-SOPI. An independent samples Kruskal-Wallis test was used to verify if any differences among the 3 groups were found to be statistically significant. No differences were found between the groups. Because no differences were found among the 3 groups, the remaining statistical analyses were run using 1 group of 29 subjects. Because of the small sample size, nonparametric statistics were performed. Kendall τ was used to estimate correlation between items of the user characteristics questionnaire and the ITC-SOPI. To determine any differences between sex and the 4 scales of the ITC-SOPI, an independent samples Mann-Whitney U test was performed.

Back to Top | Article Outline

RESULTS

ITC-SOPI and Group Scores

The means of the 3 groups for the 4 scales of the ITC-SOPI were calculated. The mean scores show that the 3 groups do not differ. The group means of the ITC-SOPI for groups 1, 2, 3, and combined were spatial presence (2.88 ± 0.53, 3.22 ± 0.40, 2.99 ± 0.62, 3.03 ± 0.53), engagement (3.02 ± 0.51, 3.59 ± 0.51, 3.42 ± 0.63, 3.34 ± 0.58), ecologic validity (3.08 ± 0.50, 3.47 ± 0.56, 2.70 ± 1.07, 3.07 ± 0.80), and negative effects (2.58 ± 0.79, 1.78 ± 0.81, 2.68 ± 1.12, 2.37 ± 0.98), respectively. We performed an independent samples Kruskal-Wallis test to verify if any of the differences among the 3 groups were significant. The test showed no significant differences for spatial presence, engagement, ecological validity, or negative effects; no significant differences were expected because the groups were homogenous.

Back to Top | Article Outline

User Characteristics and the ITC-SOPI

Table 3 shows the correlations between background characteristics and the 4 scales of the ITC-SOPI using Kendall τ. Kendall τ, a nonparametric correlation, was used because of the small data set with a large number of tied ranks. Although Spearman statistic is the more popular of the 2 coefficients, research suggests that Kendall statistic is actually a better estimate of the correlation in the population.20 Among the user characteristics “how often you play video games” was significantly related to negative effects, (τ(26) = −0.458, P < 0.01). “Television (TV)/film production knowledge” was significantly related to negative effects (τ(27) = −0.516, P < 0.01) too. “Knowledge of virtual reality” was significantly related to engagement (τ(26) = 0.463, P < 0.01) and negative effects (τ(26) = −0.404, P < 0.05). Independent samples Mann-Whitney U tests were performed to compare differences in sex and the 4 scales of the ITC-SOPI. No significant differences were found for males or females for spatial presence, engagement, or ecologic validity. However, females experienced negative effects more than males (median, 3.09; U = 43.00; z = −2.618; P = 0.009; r = −0.486).

TABLE 3

TABLE 3

Back to Top | Article Outline

DISCUSSION

The medical community has begun to implement team training interventions.21 Team training is especially important in the medical field owing to the high level of personal interaction and owing to the high complexity of the tasks that need to be accomplished. Online training, using simulation solutions, can support the team training objectives and can provide opportunities for team interaction. Participants log into the VE from their own offices or homes and work collectively toward the same goal while optimizing their time. The users characteristics can provide an indication of which person should be selected to participate on the online team training such the one presented in this article.

This study measured user characteristics and sense of presence in EMEDS teams. Determining the amount of immersion and sense of presence felt by users in a VE has been studied extensively. However, few studies have reported user characteristics as an important component when measuring sense of presence.22,23

A previous research study showed more experience with computers increases immersion and sense of presence.24 This study reports that less knowledge about virtual reality and less TV/film production knowledge lead to more negative effects during virtual-reality training. On the other hand, the more knowledge someone has about virtual reality leads to a greater sense of engagement. We believe that to maximize the learning outcome of team training, it is necessary to know which participants should be selected. To make the training more effective, it could also be beneficial to organize introductory courses on virtual reality and gaming for those individuals who have little experience with virtual reality.

A previous research study on a first-person shooter game found no statistically significant difference between men and women when measuring cybersickness.24 Another study, where teams faced a problem-solving activity in a simulated environment, reported that males felt a greater sense of presence and reported more negative effects than women in the VE.22 The differences highlighted by these studies could be associated with the different tasks subjects had to perform in the simulated environment. Our study suggested that females have more negative effects compared with men in the VE.

Researching sense of presence is essential for improving the quality and the effectiveness of online training that adopts simulated environments. Presence plays an important role in the learning process and transfer of skills. A greater sense of presence makes the learning experience engaging and relevant. Edgar Dale’s Cone of Learning25 is a visual metaphor for learning modalities. As we move from the top to the bottom of the cone, we see increased learning as more senses are engaged. According to Dale’s research, practical hands-on experience in real-life contexts allows individuals to remember best what they do and leads to the retention of the largest amount of information. This is in part because the experiences, near the bottom of the cone, closer to real-world experiences, make use of more than one sense. Simulating the real experience is second from the bottom of the cone. High level of sense of presence can make the simulated experience more similar to the real one.

It is indubitable that simulated environments can generate a variety of scenarios with more challenging training opportunities for more experienced EMEDS. Moreover, scenarios can be constructed such as natural disasters and military incidents that cannot be reproduced in the real world. Finally, mistakes and failure can be played back in debriefing sessions allowing users to perform after action reviews.

There are several limitations of this study. We are aware that a single-case study is the weakest of all study designs and that the small sample impacted the power of the study. Another limitation is the use of a self-report measure, which is open to respondents’ recall bias, social desirability bias, errors in self-observation, and acquiescent response. The vague language used in the ITC-SOPI is open to misinterpretation by respondents. However, we felt it was important to not change or rephrase the ITC-SOPI survey for allowing further research to compare results. Another disadvantage of the ITC-SOPI is the fact that the survey is given after immersion. The questionnaire may be influenced by events toward the end of the immersion near the time of the questionnaire administration when participants may become fatigued or bored, thus influencing their responses. Participants may also have been especially tired because they joined the experiment just after the end of the real Patriot training exercises. Another weakness of the study was that participants were situated too close to each other during the introductory training on GaMeTT. Although this weakens any claim on the ability to train remotely, we believe that learning how to use GaMeTT could easily be achieved by watching video tutorials from remote locations.

Future studies should include a stronger study design using a larger population of EMEDS. Data obtained could be used to create a model that identifies which user characteristics lead to a greater sense of presence. Finally, it would be interesting to compare the performance of EMEDS teams trained in the virtual world with EMEDS teams trained in a real world.

Back to Top | Article Outline

ACKNOWLEGMENTS

The authors would like to acknowledge the generous support of the Modeling and Simulation Graduate Research Fellowship provided to Ms. Leigh Ann Diggs by Old Dominion University during academic year 2012.

Back to Top | Article Outline

REFERENCES

1. Weaver SJ, Salas E, Lyons R, et al. Simulation-based team training at the sharp end: a qualitative study of simulation-based team training design, implementation, and evaluation in healthcare. J Emerg Trauma Shock 2010; 3 (4): 369–377.
2. Epsevik R, Johnsen BH, Eid J. Communication and performance in co-located and distributed teams: an issue of shared mental models of team members. Mil Psychol 2011; 23 (6): 616–638.
3. Gaba D, Howard SK, Fish K. Simulation based training in anesthesia crisis resource management: a decade of experience. Simul Gaming 2001; 32: 175–193.
4. Slater M, Wilbur S. A framework for immersive virtual environments. Presence 1997; 6 (6): 603–617.
5. International Society for Presence Research (ISPR). Telepresence. Available at: http://ispr.info. Accessed February 12, 2012.
6. Heeter C. Being there: the subjective experience of presence. Presence 2008; 1 (2): 262–271.
7. Banos RM, Botella C, Alcaniz M, Liano V, Guerrero B, Rey B. Immersion and emotion: their impact on sense of presence. Cyberpsychol Behav 2004; 7 (6): 734–741.
8. Schuemie MJ, Van Der Staaten P, Krijn M, Van Der Mast CAPG. Research on presence in virtual reality: a survey. Cyberpsychol Behav 2001; 4 (2): 183–203.
9. Kaber D, Draper J, Usher J. Influence of Individual Differences on VR Application Design for Individual and Collaborative Immersive Virtual Environments, Virtual Environments Handbook. Stanney K, ed. Mahwah, NJ: Lawrence Erlbaum; 2002.
10. Mantovani F, Castelnuovo G. Sense of presence in virtual training: enhancing skills acquisition and transfer of knowledge through learning experience in virtual environments. In: Riva G, Davide F, IJsselsteijn WA, eds. Being There: Concepts, Effects, and Measurement of User Presence in Synthetic Environments. Amsterdam: IOS Press; 2003.
11. Klein G, Miller TE. Distributed planning teams. J Cogn Ergon 1990; 3 (3): 203–222.
12. National Guard. Expeditionary Medical Support (EMEDS). Available at: http://www.ng.mil/features/HomelandDefense/emeds/factsheet.html. Accessed February 12, 2013.
13. Snyder D, Chan EW, Burks J, Mahyar A, Amouzegar MA, Resnick AC. How Should Air Force Expeditionary Medical Capabilities be Expressed? Pittsburgh, PA: RAND Corporation; 2009: 33–35.
14. Lessiter J, Freeman J, Keogh E, Davidoff J. A cross-media presence questionnaire: the ITC-sense of presence inventory. Presence 2001; 10 (3): 282–297.
15. Riva G. From Communication to Presence: Cognition, Emotions, and Culture Towards the Ultimate Communicative Experience. Amsterdam: IOS Press Inc; 2006.
16. Riva G, Mantovani F, Capideville CS, et al. Affective interactions using virtual reality: the link between presence and emotions. Cyberpsychol Behav 2007; 10 (1): 45–56.
17. Schubert T, Friedmann F, Regenbrecht H. The experience of presence: factor analytic insights. Presence 2001; 10 (3): 266–281.
18. Ravaja N, Saari T, Turpeinen M, Laarni J, Salminen M, Kivikangas M. Spatial presence and emotions during video game playing: does it matter with whom you play? Presence 2006; 15 (4): 381–392.
19. Nakagawa S. A farewell to Bonferroni: the problem of low statistical power and publication bias. Behav Ecol 2004; 15 (6): 1044–1045.
20. Howell DC: Statistical Methods for Psychology. 4th ed. Belmont, CA: Duxbury; 1997: 293.
21. Burke CS, Salas E, Wilson-Donnelly K, Priest H. How to turn a team of experts into an expert medical team: guidance from the aviation and military communities. Qual Saf Health Care 2004; 13 (Suppl 1): i96–i104.
22. De Leo G, Goodman KS, Radici E, Sechrist SR, Mastaglio TW. Level of presence in team building activities: gaming component in virtual environments. IJMA 2011; 3 (2): 1–10.
23. Steuer J. Defining virtual reality: dimensions determining telepresence. J Commun 1992; 42 (4): 73–93.
24. Gamito P, Oliveira J, Morais D, et al. Training presence: the importance of virtual reality experience on the “sense of being there”. Annu Rev Cybertherapy Telemed 2010; 8: 103–106.
25. Dale E. Audiovisual Methods in Teaching. 3rd ed. New York, NY: The Dryden Press; 1969.
Keywords:

Medical training; Sense of presence; Online simulated virtual environment; Serious gaming

© 2014 Society for Simulation in Healthcare