Concomitant with the advances in information, computer and communication technologies, and the incredible rate in the increase of human knowledge creates challenges to learning,1 particularly when learners are in different locations separated from teachers and subject matter experts. Consequently, education is shifting to developing the ability to find information and use it in a timely, effective, and appropriate manner,2 regardless of location. Educational methods are becoming much more learner-centered and interactive to promote learning with understanding, and more importantly, enhance the ability to transfer learning to other settings. These principles apply to the broad arena of healthcare education and training. Advances in technology, including virtual reality (VR), simulation, and collaborative communication networks using the Internet, offer unique opportunities for enhancing learning and sharing knowledge independent of location. Simulation can provide safe, interactive, learning environments for individuals or groups, on-site or over distance and offers a platform for iterative experiences in which students have the opportunity to correct mistakes, test the solutions they devise, as they build understanding of important concepts. Interactive learning allows self, peer, and expert feedback and assessment, honing of skills, and building of confidence. This, in turn, can lead to competent performance, the transfer of knowledge and understanding that can be applied to other related situations, as well as provide a stronger foundation for critical thinking and discovery.
Since 2000, the Universities of New Mexico (UNM) and Hawaii (UH) have been developing methods to enhance the learning process by coupling innovations in medical education with advances in VR simulation and distance learning technologies.3–5 Both UNM and UH have used problem-based learning (PBL) as part of their medical school curriculum,6–8 which has provided a common platform to support the goal of demonstrating the feasibility and acceptability of incorporation of VR simulation and distance interaction over a distributed Internet network. This article reports on the initial observations related to the effects of these additional modalities on learning dynamics commonly measured in a PBL session.
Evaluation Methods and Experimental Design
This study was approved by the institutional research review boards at each institution, and informed consent was obtained from all student subjects participating in the experiments.
Students were volunteers and were paid supplemental compensation for their time and effort related to participation outside of their normal activities. The students were randomly selected for the study groups based on the days they were available. The experimental sessions were held between April and May 2002.
The effects of two different forms of technology were considered in PBL tutorial session comparisons: (1) separation of participants using a multipoint videoconferencing and information sharing capability, in this case the Internet2 Access Grid (AG),9 with inclusion of students across sites and institutions within the tutorial groups, and (2) immersion of one of the students at the local site in a VR simulation environment with observation of the VR scenario by the other participants.
To determine the effects on the learning process with integration of the VR simulation and separation of participants over distance, the evaluation model included analyzing aspects of PBL by comparison of tutorial groups used for the experiments in regard to the following PBL processes: (1) identification of important historical and physical findings relevant to the case and their ability to recall those facts at the end of each tutorial session, (2) comparison of the learning issues generated by the groups, and (3) analysis of tutors’ and students’ experiences in tutorials across experimental groups. In addition, using a checklist, on-site tutorial observers measured content and process behaviors occurring in the tutorials.
Tutorial observers looked for the following tutorial group milestones: (1) establishment of ground rules, (2) reading and reacting to the chief complaint, (3) completion of a problem list and a list of key facts, (4) formation of hypotheses about causes, explanation of mechanisms, and revision of hypotheses based on new information, (5) requests for additional history data, (6) requests for additional physical examination data, (7) requests for additional laboratory data, (8) development of learning issues, and (9) group reflection and assessment. The amount of time spent on each milestone was roughly estimated by counting up the number of 10-minute periods in which each milestone occurred. The milestones for each tutorial were plotted on a set of time lines to allow visual comparison of the relative time and duration of each milestone among tutorials.
For the demonstration, comparisons were made across four separate group formats in a total of six separate sessions, involving second year medical students from the UH John A. Burns School of Medicine (JABSOM) and UNM School of Medicine (UNMSOM). The four experimental formats as shown in Figure 1 consisted of the following:
- Control groups; two groups consisting of six students and a tutor, one at each medical school participating separately in a traditional PBL tutorial session using the written case scenario only without any additional technology, also referred to as the Paper only group
- Distance group; one group of six students, three from JABSOM and three from UNMSOM, with one tutor in a separate location participating over distance with the AG, using the written case scenario only, without VR technologies, also referred to as the AG group
- VR groups; two groups of six students and a tutor, one group at each medical school, participating separately in the PBL tutorial session using VR simulation to augment the paper case scenario, also referred to as the Paper/VR group
- Combination group; one group of six students, three from JABSOM and three from UNMSOM, with one tutor in a separate location participating over the AG, using the VR technologies, also referred to as the AG/VR group.
The Applied Distance and VR Technologies
When separated by distance, the group was assembled over the AG with students and the tutor at geographically separated sites. The AG was developed by the National Computational Science Alliance as a system using Internet2 for similar group-to-group interaction.9 It is designed to facilitate multimedia, multipoint, real-time participatory group interaction and support a variety of applications. AG studio components include multimedia displays, and interactive environments with interfaces to visualization systems. Other multipoint network technologies now available could be used when I2 capability does not exist.
To enhance the interactive experience, a VR patient simulator was developed.3 When using VR simulation in a session, one volunteer student entered the VR environment using a head mounted display (HMD) and a control wand while the other students observed (Fig. 2). The addition of the interactive simulation allows the students to examine and treat a virtual patient, which, in turn, dynamically determines the direction of the case scenario. The simulator has three components: (1) a real-time artificial intelligence simulation engine; (2) a three-dimensional VR environment; and (3) a system for human-computer interaction. The artificial intelligence system reasons with case specific clinical knowledge in the form of rules elicited from a group of subject matter experts. As a result, the clinical course of the scenario will have multiple outcomes depending on the student-generated interventions and provides a unique environment for experiential learning. When wearing a HMD and using a hand-held control wand, the immersed students interacts with the virtual patient during the case through a set of intuitive commands using a variety of objects and tools that can be manipulated by the user. The VR immersive environment, in which the students work, is called Flatland,9 an open source software program developed at UNM.
When using the VR system, the students communicated with each other and with the student immersed in the VR environment. The nonimmersed students can observe on a monitor or projection screen a “first-person view” as seen through the eyes of the immersed student. The case is begun by following a text-based script and then starting the timeline for the virtual patient simulation program as the scenario unfolds. The immersed student works with the group to gather information and initiate interventions. The students and tutor can interact as the case progresses, pause the simulation when appropriate to discuss their observations, hypothesize, and generate learning issues.
The PBL Case
A PBL case was developed following institutional standards for case development, with the intent of creating a test case identical to those used in the current curricula. The case depicts a traumatic head injury scenario subsequent to an automobile crash, with resulting neurologic effects of a progressing epidural hematoma.5 The objective of the case is to provide the tutorial group with an appropriate clinical experience for exploration of underlying basic biomedical science principles. A set of defined learning goals and objectives were established by a group of clinical and basic science knowledge experts thoroughly familiar with standard institutional case development and medical educational process.
A total of thirty-six student volunteers participated; 35 second-year medical students; 18 from JABSOM and 17 from UNMSOM, and one physician assistant student from UNMSOM. No adverse effects were noted among the participants who used the AG or VR technologies.
In the pre-experiment preliminary survey, most student participants reported that they learn well in a variety of settings including small group, case-based discussions, hands-on laboratory sessions, real world case study and lecture-based sessions with hands-on opportunities. There were no significant differences among groups in their preferred learning venue.
Comparison of Experimental Groups and Effects of the VR and AG Technologies
After each tutorial session student participants were asked to rate their tutorial experience. The tutorial experience in all of the groups was positive. Students in all groups attributed the success of their tutorial session to positive attitudes that resulted in positive group dynamics. Index averages on a Likert scale ranged from 3.8 for the distance group tutorial to 4.3 for the control group tutorials on a 5-point scale, with five being the most positive. Analysis of variance among the index values of the four groups indicated that students in the four types of tutorials did not rate their tutorial experience differently (F = 2.09 [df 3,32], P = 0.12). The overall responses to using the AG and VR are summarized in Table 1.
Student ratings of their behavior in their respective tutorials were similar across the four types of tutorials as shown in Table 2. The students noted that at the beginning of the AG and VR sessions, there was more verbal input from the tutors, who gave more explicit directions about the PBL group process, but over time, as the session progressed, the students functioned more independently.
Tutor and Tutorial Observers Findings
The tutorial sessions in all groups was described by the tutors as being very functional for a first meeting. All students were experienced in the PBL tutorial process and some of the students had worked together before in tutorials. The tutors noted that students identified problems and risk factors well in all tutorials. Tutors were satisfied with students’ initial formation of hypotheses as being appropriate to the case. They did not report any differences in the hypotheses formation of any tutorial group. The tutors reported that most groups independently formulated learning issues very well; some groups required prompting or tutor assistance.
The tutorial observers found that tutorial milestones, as previously defined in the methods section, such as completion of a problem list and a list of key facts, formation of hypotheses about causes, explanation of mechanisms, revision of hypotheses based on new information, and development of learning issues, were similar across all groups independent of the applied technologies. The amount of time spent on each milestone was also similar across all groups.
The research team employed PBL as a platform for this demonstration project because both participating institutions use this format in their curriculum and a great number of substantive learning processes occur in PBL tutorials, such as hypothesizing from a set of facts and connecting clinical science to the case. This common educational approach provided a platform for demonstrating the feasibility and acceptability of using these additional VR and distance technologies. Although we demonstrated the feasibility and acceptability of the VR simulation or distance education technology during small group interactions, larger numbers of students and student tutorials would need to be studied to provide the power to assess true effects on learning. As students are often separated at different locations, distance learning technologies may play a significant role and VR simulation provides an additional engaging modality for experiential learning, whether on-site or over distance. Although we have reported on the use of these methods previously,3 this study was an initial step in demonstrating the feasibility of using these approaches without interfering with standard learning processes and group interaction as represented in a PBL session, comparing similar groups of students with or without these additional modalities. It is also possible that these methods may be well suited to other types of learning environments in addition to PBL.
Significant preparation issues and problems surfaced in the development and implementation of these experimental tutorials. These issues included (1) the time to develop the case, the VR, and the distance technologies to a point of usability, (2) the coordination of the case, the VR, and the AG, (3) the orientation of the tutors and students to both the technology and the case, and (4) identifying an adequate number of students to act as experimental subjects. Because of the small number of student subjects, additional studies would be needed to provide sufficient statistical power for analysis of the impact on learning. It is important to recognize that preparation requirements will exist in the development of new cases for education and appropriate evaluation on pedagogy in which these types of technologies are applied. With experience and as the iterative process is refined, we have found that the development of these reusable VR simulation scenarios is less time intensive and becomes less expensive.
The cost of individual VR work stations and access to using advanced communication networks, such as I2 AG, must be considered. However, the equipment costs are also decreasing and other types of multipoint networking strategies can also be employed. We used the AG since it was available at our institutions to demonstrate the effects of interaction over distance. Less expensive collaborative distance technologies can be used for team interaction in a similar fashion without requiring the AG. Furthermore, as demonstrated, several students can observe the simulation without being fully immersed with a HMD, either when interacting on site or at different locations, including remote or rural settings. Because there are less expensive options, this demonstration study was not designed to access cost versus benefit of integrating these specific modalities.
Research into the educational application of virtual environments has been conducted since the early 1990s. Winn has pointed out that the student's ability to interact in an environment experimentally and not just observe it is a critical feature that simulation provides for learning.10,11 According to Winn, the theoretical assumption of learning from simulation is that students construct understanding for themselves by interacting with information and materials, an orientation to learning that has acquired the name “constructivism.”12–16
More recently, we have been evaluating the impact of VR simulation experiences on learning among groups of medical students using a knowledge structure analysis approach.17,18 A group of subject matter experts rates the relatedness of concepts within a given simulation, and Pathfinder is used to derive a single expert knowledge structure.19 This expert knowledge structure is then used as a gold standard against which to compare the students’ knowledge structures. A similarity score, ranging from 0 to 1, is used to compare how close the student's knowledge structures were to the expert's. VR learning would be reflected by higher similarity scores after the VR experience than before.
Perhaps most importantly, these virtual renderings of clinical scenarios allow students to interact and learn in a safe iterative environment in which mistakes can be made and feedback given independent of location. Learning from mistakes offers opportunities for improved understanding, gaining confidence, knowledge transfer, and ultimate better performance. Furthermore, simulation learning environments allow students to make mistakes without risk to patients and safety. In a detailed analysis of the literature and review of military simulation efforts, Champion and Higgins concluded that simulation is an effective and cost-efficient approach to training military personnel, enhancing knowledge transfer, and improving performance.20 Similar conclusions were reported by Satava and Jones21 regarding the potential value of using VR to assess competence. Ziv et al22 also make a compelling argument of the need to develop further simulation-based medical education as an ethical imperative to ensure optimal treatment, patient safety, and well-being.
The evidence from these experiments demonstrated that it is possible to use distance technology and VR in combination or separately to conduct a PBL tutorial clinical case scenario with second-year medical students from two institutions.
In summary, these experiments have demonstrated:
- The feasibility of developing and using an interactive VR patient simulation that could be manipulated by students to portray complex clinical scenarios.
- VR simulation can be integrated into PBL.
- PBL sessions can be conducted over distance with or without VR simulation.
- Distance interactive technologies, such as AG, can be deployed to distant sites, allowing sharing of information as well as supporting VR.
- Institutions can collaborate over distance and share resources to develop pedagogic and technological innovations in medical education.
- VR simulation and distributed sessions do not detract from the learning experience when compared with traditional PBL sessions.
- Students are generally enthusiastic and accepting of the new technologies as used in this project and enjoyed working with similar level students from other institutions.
Bringing people together as virtual teams for interactive experiential learning and collaborative training, independent of distance, provides a platform for distributed “just-in-time” training and education on demand. These developments can decrease the need to have participants travel to a central training facility for those activities, as well as facilitate the rapid deployment of simulations for refreshment and sustainment training or acquisition of new knowledge and skills. Based on this work and reports from members of other research teams,23 the authors suggest that additional research is needed in the following areas: (1) cost versus benefit of utilizing these additional modalities, (2) evaluation of the impact on learning and knowledge transfer, (3) feasibility of integration into the curriculum, (4) applications to distance learning, and (5) use in performance assessment.
The project described was supported partially by grant 2 D1B TM 00003-02 from the Office for the Advancement of Telehealth, Health Resources and Services Administration, Department of Health and Human Services. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the Health Resources and Services Administration.
The following individual provided additional technical and consultative support for the project: James R. Holten IV, MS, School of Engineering, University of New Mexico, Albuquerque, NM 87131 U.S.A.
1. Issenberg, SB, McGaghie WC. Assessing knowledge and skills in the health profession: a continuum of simulation
fidelity. In: Tekian A, McGuire CH, McGaghie WC, et al, eds. Innovative Simulations for Assessing Professional Competence:From Paper-and-Pencil to Virtual Reality
. Chap. 9. Chicago: University of Illinois at Chicago, Department of Medical Education; 1999:125–146.
2. Simon HA. Observations on the Sciences of Science Learning. Paper prepared for the Committee on Developments in the Science of Learning for the Sciences of Science Learning: An Interdisciplinary Discussion. Department of Psychology, Carnegie Mellon University, 1996.
3. Alverson DC, Saiki SM, Jacobs J, et al. Distributed interactive virtual environments for collaborative experiential learning and training independent of distance over Internet2. Stud Health Technol Inform
4. Mowafi MY, Summers KL, Holten J, et al. Distributed interactive virtual environments for collaborative medical education and training: design and characterization. Stud Health Technol Inform
5. Jacobs J, Caudell T, Wilks D, et al. Integration of advanced technologies to enhance problem-based learning over distance: project TOUCH. Anat Rec B New Anat
6. Kaufman A, Mennin S, Waterman R, et al. The New Mexico experiment: educational innovation and institutional change. Acad Med
7. Anderson A. Conversion to problem-based learning in 15 months. In: Boud D, Feletti G, eds. The Challenge of Problem Based Learning
. New York: St. Martin's Press; 1991:72–79.
8. Bereiter C, Scardamalia M. Commentary on Part I: Process and product in problem-based learning (PBL) research. In: Evensen DH, Hmelo CE, eds. Problem-Based Learning: A Research Perspective on Learning Interactions
. Hillsdale, NJ: Lawrence Erlbaum; 2000:185–195.
9. Caudell TP, Summers KL, Holten JIV, et al. A virtual patient simulator for distributed collaborative medical education. Anat Rec B New Anat
10. Winn WD. Learning in virtual environments: a theoretical framework and considerations for design. Educ Media Int
11. Winn WD. Current trends in educational technology research: the study of learning environments. Educ Psychol Rev
12. Moshman D. Exogenous, endogenous and dialectical constructivism. Dev Rev
13. Jonassen DH. Objectivism versus constructivism: do we need a new philosophical paradigm? Educ Technol Res Dev
14. Duffy TM, Jonassen DH. Constructivism: new implications for education technology. In: Duffy T, Jonassen D, eds. Constructivism and the Technology of Instruction: A Conversation
. Hillsdale, NJ: Erlbaum; 1992:1–16.
15. Dede C. The evolution of constructivist learning environments: immersion in distributed, virtual worlds. Educ Technol
16. Windschitl M, Andre T. Using computer simulations to enhance conceptual change: the role of constructivist instruction and student epistemological beliefs. J Res Sci Teach
17. Stevens SM, Goldsmith TE, Summers KL, et al. Virtual reality
training improves students’ knowledge structures of medical concepts. Stud Health Technol Inform
18. Gutierrez F, Pierce J, Vergara, et al. The effect of degree of immersion upon learning performance in virtual reality
, simulations, for medical education. Stud Health Technol Inform
19. Schvaneveldt RW, ed. Pathfinder Associative Networks: Studies in Knowledge Organization
. Norwood, NJ: Ablex; 1990.
20. Champion HR, Higgins GA. Meta-Analysis and Planning of SIMTRAUMA: Medical Simulation
for Combat Trauma Training. USAMRMC TATRC Report No. 00-03;2000.
21. Satava RM, Jones SB. The future is now: virtual reality
technologies. In: Tekian A, McGuire CH, McGaghie WC, et al, ed. Innovative Simulations for Assessing Professional Competence: From Paper-and-Pencil to Virtual Reality
. Chap. 9. Chicago: University of Illinois at Chicago, Department of Medical Education; 1999.
22. Ziv A, Wolpe PR, Small SD, Glick S. Simulation
-based medical education: an ethical imperative. Acad Med
23. McGuire CH, Tekian A. Conclusions and recommendations: a suggested strategy. In: Tekian A, McGuire CH, McGaghie WC, et al, eds. Innovative Simulations for Assessing Professional Competence: From Paper-and-Pencil to Virtual Reality.
Chap. 15. Chicago: University of Illinois at Chicago, Department of Medical Education; 1999:233–240.