Share this article on:

The Evolving Role of Online Virtual Patients in Internal Medicine Clerkship Education Nationally

Lang, Valerie J. MD; Kogan, Jennifer MD; Berman, Norm MD; Torre, Dario MD, PhD

doi: 10.1097/ACM.0b013e3182a7f28f
Research Reports

Purpose Despite the significant resources required to develop and maintain virtual patient (VP) programs, little is known about why this innovation has been adopted and how it is implemented. Understanding needs and implementation strategies is important for effective curriculum planning.

Method In 2009 and 2011, surveys were offered to 110 U.S. internal medicine clerkship directors regarding their goals for adoption of Simulated Internal Medicine Patient Learning Experience VPs. In 2011, respondents were asked how they implemented VPs in their curricula. Results were analyzed using chi-square and Fisher exact test.

Results Responses were obtained from 33 clerkship directors in 2009 and 45 in 2011. Comparing 2009 with 2011, improving students’ knowledge (29/33 [88%] versus 40/45 [91%]), differential diagnoses (27/33 [82%] versus 38/45 [86%]), and ability to identify key findings (26/33 [79%] versus 38/45 [86%]) remained somewhat or very important reasons for adopting VPs. Meeting Liaison Committee on Medical Education ED-2 (31/33 [94%] versus 33/45 [73%], P = .011) and ED-8 requirements (25/33 [76%] versus 25/45 [56%], P = .004) declined in importance. Eight of 38 (21%) replaced a learning activity with VPs, 9/38 (24%) integrated VPs into other learning activities, and 21/38 (55%) simply added VPs onto their curricula.

Conclusions This large, multi-institutional study reports national trends in VP adoption and inte gration. Meeting cognitive learning objectives remained an important reason for adopting VPs, whereas meeting regulatory requirements decreased significantly in importance. Opportunities remain for m ore systematically integrating VPs into clerkship curricula. Clarifying the changing goals may help with this process.

Dr. Lang is associate professor, Hospital Medicine Division, University of Rochester School of Medicine and Dentistry, Rochester, New York.

Dr. Kogan is associate professor, General Medicine Division, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania.

Dr. Berman is professor of pediatrics, Geisel School of Medicine, Dartmouth University, Hanover, New Hampshire.

Dr. Torre is associate professor, Department of Internal Medicine, Drexel University College of Medicine, Philadelphia, Pennsylvania.

Funding/Support: Not applicable.

Other disclosures: None.

Ethical approval: The Case Western Reserve University institutional review board reviewed and approved the protocol.

Previous presentations: An abstract was presented at the Alliance for Academic Internal Medicine Annual Meeting, 2012, in Phoenix, Arizona.

Correspondence should be addressed to Dr. Lang, 601 Elmwood Ave., Box MED-HMD, Rochester, NY 14642; telephone: (585) 275-4912; fax: (585) 276-2212; e-mail:

The dominant learning experience in clinical clerkships has not changed significantly over the past 100 years.1,2 At the heart of the clerkship is a model in which the student, patient, and teacher are the main elements. This clinical experience is supplemented by lectures, case conferences, reading, and other activities. But the advent of online interactive virtual patients (VPs) represents an innovation in clerkship education.

VPs are a form of computer-assisted instruction in which learners work through the steps of diagnosing and managing patients, making clinical decisions without the risk of causing patient harm.3 VPs may be used to meet educational objectives, comply with regulatory requirements, or compensate for the specific needs of individual institutions or clerkship sites. A meta-analysis of VP studies found that they are an effective method for teaching medical knowledge, clinical reasoning, and other skills.4 The Liaison Committee on Medical Education (LCME) specifically notes that VPs may be used to meet ED-2 requirements, which require faculty to define the types of patients and clinical conditions students must encounter and ensure that all students have these experiences.5 The ED-2 requirement overlaps with ED-8, which stipulates that students training at different clinical sites must have similar experiences.5 However, some clinical sites where students rotate during their clerkships vary in their patient populations, including ethnicity, gender, and disease burden. In addition, individual clerkship sites may vary in terms of faculty availability, resident availability, and other resources for teaching.

Online VP programs are currently used in at least 136 medical schools in the United States and other countries.6 Although they are increasingly used in clinical education, VPs require significant resources to develop and maintain.7 Single cases or small collections of VP cases have been implemented in a variety of health professions education programs.4 Technical standards for VPs have been developed,8 and case-authoring tools are available for educators to develop their own cases.9,10

Because of the time and effort required to develop and maintain high-quality VPs, collaborations in Europe11 and the United States7,12–14 have emerged to develop more extensive VP programs. Programs which comprehensively cover national clerkship curricula in pediatrics,7 family medicine,12 and internal medicine13 were developed through collaboration with the respective clerkship directors’ national organizations and are maintained by a nonprofit organization with the support of subscription fees.14 The VPs in this study (SIMPLE: Simulated Internal Medicine Patient Learning Experience) were written using this latter collaborative development model. Learning objectives from the Clerkship Directors in Internal Medicine (CDIM) national curriculum15 were organized into outlines for 36 VP cases. The cases were designed to teach medical knowledge and clinical reasoning. Case authors, solicited from the CDIM membership, used a common pedagogical approach and format. Each case underwent peer review and is updated annually.

VPs have been rapidly adopted by clerkship directors. In 2008–2009, as the SIMPLE VP cases were developed and completed peer review, they were freely available to all clerkship directors for beta-testing. When the VP program was transitioned to a subscription model, 47 schools subscribed during the 2009–2010 academic year, and 77 schools subscribed during the 2010–2011 academic year. Subscriptions were paid for by institutions, rather than individual students, and the decision to subscribe to the program was made by the clerkship directors. Once adopted, VPs may be used to substitute for an existing learning activity (elimination strategy), change or enhance an existing learning activity (integration strategy), or add on to existing learning activities without change or substitution (addition strategy). Multiple calls have been issued to study the implementation of VPs within the clerkship curriculum.16,17

Understanding the goals for adopting VPs and current implementation strategies is the first step to understanding how to optimally integrate them within the larger clerkship curriculum. Kern and colleagues18 describe a six-step model of curriculum development in medical education: identifying a general need, identifying the specific needs of learners and their environment, identifying goals and objectives, choosing educational strategies, implementing the curriculum, and providing evaluation and feedback. The goal of this process is to provide a thoughtfully planned educational experience for medical students. By understanding the needs, goals, and objectives for which VPs are adopted (the first three steps), clerkship directors can make more informed choices about how they implement VPs within the broader clerkship curriculum.

The objectives of this study were thus to understand for what purposes have internal medicine clerkship directors adopted VPs, how have these purposes evolved over time, and what strategies are clerkship directors using to implement VPs in their curricula?

Back to Top | Article Outline


Survey development

Survey items were developed by a group of four clerkship directors (V.L., J.K., and two nonauthors) involved in the development and editing of the VPs in 2009 and eight clerkship directors on the VP editorial board in 2011 (V.L., J.K., and six nonauthors). They were then reviewed by an additional clerkship director with experience in developing VPs (N.B.). The items were based on competencies outlined in the national clerkship curriculum,15 LCME regulations,5 Accreditation Council for Graduate Medical Education compe tencies,19 implementation strategies identified in prior VP studies,20 and logistical issues commonly encountered by clerkship directors. The 2011 survey included items regarding VP implementation strategies by clerkship directors: whether and how many cases were required, whether they identified the required cases or allowed students to choose, whether they cued students to case content, and whether they replaced an existing activity, integrated VPs into an existing activity, or added the VPs onto existing activities. Respondents were asked which activities were replaced or modified, and how successful they perceived their strategies were (response options were “very unsuccessful,” “somewhat unsuccessful,” “neutral,” “somewhat successful,” or “very successful”). The surveys were reviewed for content validity, overall design, and usability by members of the CDIM Research Committee and the CDIM Council, who have knowledge of the subject matter and expertise in survey research methods. After a careful review and discussion of the meaning and relevance of survey items, disagreements were resolved by consensus, and further revisions were made. The surveys were pretested among 14 CDIM Research Committee members and 12 members of the CDIM Council for clarity and appropriateness of content. Survey pilot results were analyzed for nonresponses, missing data, and comments by respondents, which led to additional minor revisions.21 The final survey included 33 items in 2009 and 39 items in 2011. For items about the purposes for using VPs, response options were “not at all important,” “minimally important,” “somewhat important,” or “very important.” Two items in each survey were not included in both years: The items addressing the importance of VPs for saving the cost of textbooks or faculty were only included in 2009, and the items addressing the adoption of VPs because of decreased availability of residents and change in class sizes were only included in 2011. These changes were made because of intercurrent rises in class sizes and resident duty hours restrictions and limitations on survey length. The VP surveys were included in the annual CDIM surveys addressing additional topics in medical student education. The total surveys consisted of 189 items in 2009 and 127 items in 2011.

Back to Top | Article Outline

Survey administration

In 2009 and 2011, the CDIM conducted its annual, voluntary, and confidential e-mail survey of its U.S. and Canadian membership, including 110 institutional members. Each participating school has one CDIM institutional member to whom the survey is sent. Nonresponders were contacted up to two times over three months through e-mail, and one additional time through regular mail, and/or by telephone.

Back to Top | Article Outline

Survey analysis

We performed descriptive statistics on all responses. For items addressing the purposes for using VPs, responses were treated as a dichotomous variable: “Somewhat important” or “very important” were grouped together, and “minimally important” and “not at all important” were grouped together. We asked respondents in 2009 whether they were considering or planning implementation, but comparative analyses were confined to those who were planning implementation. Because implementation required planning to gather the financial resources to subscribe to the program, they were considered a more comparable group to those in 2011 who stated that they had implemented the program. We performed chi-square tests and Fisher exact tests (where n ≤ 5) to compare responses between the 2009 and 2011 surveys. We considered a P value < .05 to be statistically significant. Because of the small numbers of responses in each category regarding implementation strategies, comparative statistics were not performed. We calculated both descriptive and inferential statistics using a standard statistical software program (SPSS, version 19, Armonk, New York).

The Case Western Reserve University institutional review board reviewed and approved the protocol.

Back to Top | Article Outline


We obtained responses from 69 (63%) of the 110 clerkship directors in 2009, and 86 (78%) in 2011. In 2009, 63 (91%) respondents were familiar with SIMPLE VPs, and 40 (58%) had trialed the program. Of the respondents familiar with the program, 33 (52%) were planning implementation during the first subscription year, 18 (29%) were considering implementation, and 12 (19%) were not planning implementation. In 2011, 45 (52%) respondents had implemented SIMPLE VPs, and 41 (48%) were not implementing the program.

In 2009, the reasons most frequently selected as somewhat or very important in choosing to adopt VPs were meeting LCME ED-2 requirements, improving students’ knowledge base, and compen sating for variable diagnoses seen (LCME ED-2 and ED-8) (Table 1). In 2011, the reasons most frequently selected as somewhat or very important were improving students’ knowledge base, ability to develop differential diagnoses, and ability to identify key findings.

Table 1

Table 1

From 2009 to 2011, respondents were significantly less likely to rate LCME requirements as somewhat or very important (Table 1). Meeting ED-2 requirements decreased from 94% (31 of 33) to 73% (33 of 45, P = .011). Compensating for variable patient numbers dropped from 76% (25 of 33) to 40% (18 of 45, P = .001). Compen sating for variable diagnoses seen dropped from 85% (28 of 33) to 60% (27 of 45, P = .004), and compensating for variability in clerkship sites or services (ED-8) dropped from 76% (25 of 33) to 56% (25 of 45, P = .04).

Of the 45 respondents using the VP program in 2011, 36 (80%) required students to complete a mean of 11.5 (median 10; IQR 4, 18) cases, and 9 (20%) made the cases voluntary. Of those who required cases, 26 (74%) specified which cases students were required to complete, and 9 (26%) did not; one respondent did not answer this item. Of those who made the cases voluntary, 2 (22%) gave hints regarding the case content (e.g., indicating that a dyspnea case was cardiac).

In 2011, 38 respondents indicated their implementation strategies: 21 (55%) added the cases to existing clerkship activities without other modifications, 8 (21%) replaced existing activities (most frequently lectures), and 9 (24%) integrated VP cases into an existing activity (most frequently lectures or case-based sessions with faculty) (Table 2). There was a trend toward more frequent perceived success of implementation among those who integrated the cases into existing learning activities (6/9, 67%) than those who replaced learning activities (4/8, 50%) or simply added the cases to existing learning activities (7/21, 33%); the numbers of responses in each category were too infrequent to analyze statistical significance.

Table 2

Table 2

Back to Top | Article Outline


This large, multi-institutional study reports on national trends in VP adoption, use, and integration in internal medicine clerkships and provides insight for curriculum planning. In Kern and colleagues’18 six-step approach to curriculum development in medical education, the first three steps involve identifying a general need, identifying the specific needs of learners and their environment, and identifying goals and objectives for the curriculum. Over the two years of the study, teaching knowledge and clinical reasoning remained the most important reasons for adopting VPs in the clerkship. The effectiveness of VPs in meeting these cognitive objectives is well supported in the medical education literature.4 It is notable that increasing students’ exposure to patients with undifferentiated problems was also important, and this may be related to a need to teach clinical reasoning. Ideally, students would encounter patients at a stage of their illness which is optimal for students’ learning. With the increase in handoffs,22,23 exposing students to “fresh patients” who have not been previously evaluated, diagnosed, and handed off to a student’s team is challenging, but it is associated with improved standardized exam performance.24 VPs presenting with an undifferentiated complaint or finding, such as dyspnea or anemia, provide students with practice in clinically reasoning through these problems before the diagnosis is provided.

Although VPs may be used to meet LCME accreditation requirements,5,25 it is notable that meeting these requirements became a less important reason for adopting VPs. Regulatory requirements and the manner in which the program was marketed did not change substantially over the period of the study. It is possible that clerkship directors who adopted the program later on already had the resources in place to meet regulatory requirements and were, therefore, less motivated to adopt VPs in their clerkships initially. The fourth and fifth steps of curriculum development involve choosing educational strategies and implementing the curriculum.18 It is notable that lectures were the activity most frequently replaced by VPs. This practice supports adult learning by replacing traditionally subject-oriented, scheduled, and passive lectures with problem-oriented, self-directed, and interactive VPs.25,26 Those who integrated VP cases into existing activities also implemented them within classroom activities such as lectures and case-based sessions with faculty.

The sixth step involves program evaluation and feedback.18 In this study, program evaluation was limited to clerkship directors’ perceptions of the success of their implementation strategy. We found a trend toward more successful perceived implementation if they had either replaced or integrated VPs into an existing learning activity, as opposed to adding VPs on top of their existing activities. This finding supports prior studies of a smaller number of institutions examining the impact of VP implementation strategies on students.20,27,28 In a study of VPs in pediatrics clerkships at six medical schools, students perceived that VPs were more effective than traditional instructional tools in teaching them clinical knowledge and clinical reasoning, and the perceived effectiveness was significantly associated with the degree to which VPs were integrated into the clerkship curriculum.20 Simply adding VPs without modifying or removing other parts of the curriculum was considered a low level of integration.20 Similarly, Edelbring and colleagues27 found that students at a single medical school perceived a greater benefit when VPs were more integrated into a preclinical clerkship. In a group where VP cases were simply added onto existing activities, there was lower perceived benefit and lower intensity of case processing.27 In comparison groups where students led a follow-up seminar either presenting the cases or discussing them with a faculty moderator, students perceived a greater learning value and processed the VP cases more intensively.27 In a separate study of rheumatology clerkship students, Edelbring and colleagues28 found that students perceived VP cases in relation to their clinical work, finding that they integrated biomedical and clinical knowledge and provided structure for approaching encounters with real patients; however, they also found that VPs lacked the emotional interactivity found with real patients. On the basis of these findings, we recommend that clerkship directors consider explicitly integrating VPs with students’ clinical and/or conference activities.

There were several limitations to this study. Because the surveys were anonymous, it was not possible to identify which respondents were included in both years or to match data across respondents. The suboptimal survey response rate may have been attributable to survey fatigue, as our study questions were included in larger surveys which addressed multiple additional topics. Although only VP users who were members of the CDIM organization were included in the study, they represented 72% of the institutions using SIMPLE in 2009 and 65% of institutions using SIMPLE in 2011. It is unclear whether the goals and implementation strategies of CDIM members who responded differ from those of nonmembers or those who did not respond. It is possible that those who took the time to respond had considered their reasons for adopting VPs and had made more adjustments to their curricula when they implemented VPs. There may be other purposes for which clerkship directors adopted VPs; a qualitative approach may have revealed additional needs which were not included in our surveys. Also, the 2009 results included only those respondents who indicated a plan to subscribe to the VP program as it shifted from a free model to a subscription model; the 2011 respondents included those who indicated that they had already subscribed. Because of anonymity, it was not possible to verify their subscriptions. Lastly, because of the small number of respondents in each category, we were unable to correlate the different implementation strategies with their perceived success.

Despite these limitations, this is the largest multi-institutional study of trends in VP adoption, and it provides insight for future curriculum planning using SIMPLE VPs specifically and VP programs in general. Most clerkship directors in our study adopted VPs to teach clinical knowledge and clinical reasoning skills, though other VP programs are designed to meet different learning objectives, such as communication skills29 or history-taking skills.30 We recommend that clerkship directors review their curricula to understand which tools they are using to meet each goal. They may consider either integrating VPs into existing activities or identifying activities that are less congruent with adult learning and can be eliminated to avoid overloading students’ time. Clerkship directors in other disciplines may have different goals and logistical challenges than those of internal medicine clerkship directors. For example, pediatrics clerkships must contend with more seasonal fluctuations in the types of infectious diseases that students encounter and an overall lower incidence of important disease entities. However, the principle of reviewing the curriculum to determine which learning activities meet each goal remains the same. Because local conditions and regulatory requirements change over time, we recommend that clerkship directors periodically reevaluate their goals and objectives for VPs and other instructional activities. Future research should explore the optimal approach for designing and implementing VP cases in order to improve knowledge and clinical reasoning, the reasons for changes in clerkship directors’ priorities, and the impact of changes in medical school accreditation standards on curricular design in clerkships. We suggest that future efforts are directed toward assisting clerkship directors who are adopting new educational technology with systematic curriculum planning, identifying their own institutions’ goals, and periodically reevaluating these goals as their needs change over time.

Acknowledgments: The authors thank the SIMPLE (Simulated Internal Medicine Patient Learning Experience) editorial board and the Clerkship Directors in Internal Medicine council for review of survey items.

Back to Top | Article Outline


1. Whitcomb ME. Responsive curriculum reform: Continuing challenges. The Education of Medical Students: Ten Stories of Curriculum Change. 2000 Washington, DC Association of American Medical Colleges
2. Cooke M, Irby DM, O’Brien BC Educating Physicians: A Call for Reform of Medical School and Residency. 2010 San Francisco, Calif Josey-Bass
3. Association of American Medical Colleges, Institute for Improving Medical Education. . Effective use of educational technology in medical education. Accessed May 23, 2013
4. Cook DA, Erwin PJ, Triola MM. Computerized virtual patients in health professions education: A systematic review and meta-analysis. Acad Med. 2010;85:1589–1602
5. Liaison Committee on Medical Education. . Functions and Structure of a Medical School: Standards for Accreditation of Medical Education Programs Leading to the MD Degree. Accessed May 23, 2013
6. Institute for Innovative Technology in Medical Education. Accessed May 23, 2013
7. Fall LH, Berman NB, Smith S, White CB, Woodhead JC, Olson AL. Multi-institutional development and utilization of a computer-assisted learning program for the pediatrics clerkship: The CLIPP Project. Acad Med. 2005;80:847–855
8. Triola MM, Campion N, McGee JB, et al An XML standard for virtual patients: Exchanging case-based simulations in medical education. AMIA Annu Symp Proc. October 11, 2007:741–745
9. Zary N, Johnson G, Boberg J, Fors UG. Development, implementation and pilot evaluation of a Web-based virtual patient case simulation environment—Web-SP. BMC Med Educ. 2006;6:10
10. Begg M. Virtual patients: Practical advice for clinical authors using Labyrinth. Clin Teach. 2010;7:202–205
11. eViP: Electronic virtual patients. Accessed May 23, 2013
12. Leong SL. fmCASES: Collaborative development of online cases to address educational needs. Ann Fam Med. 2009;7:374–375
13. SIMPLE: Simulated Internal Medicine Patient Learning Experience. Lebanon, NH Institute for Innovative Technology in Medical Education Accessed May 23, 2013
14. Berman NB, Fall LH, Chessman AW, et al A collaborative model for developing and maintaining virtual patients for medical education. Med Teach. 2011;33:319–324
15. Alliance for Academic Internal Medicine. . CDIM-SGIM Core Medicine Clerkship Curriculum Guide. Version 3.0. Accessed May 23, 2013
16. Cook DA. The research we still are not doing: An agenda for the study of computer-based learning. Acad Med. 2005;80:541–548
17. Triola MM, Huwendiek S, Levinson AJ, Cook DA. New directions in e-learning research in health professions education: Report of two symposia. Med Teach. 2012;34:e15–e20
18. Kern DE, Thomas PA, Howard DM, Bass EB Curriculum Development for Medical Education: A Six-Step Approach. 1998 Baltimore, Md Johns Hopkins University
19. Accreditation Council for Graduate Medical Education. . Common Program Requirements. Accessed May 23, 2013
20. Berman N, Fall LH, Smith S, et al Integra tion strategies for using virtual patients in clinical clerkships. Acad Med. 2009;84:942–949
21. Survey Research Methods. 20094th ed Thousand Oaks, Calif Sage Publi cations
22. Vidyarthi AR, Arora V, Schnipper JL, Wall SD, Wachter RM. Managing discontinuity in academic medical centers: Strategies for a safe and effective resident sign-out. J Hosp Med. 2006;1:257–266
23. Horwitz LI, Krumholz HM, Green ML, Huot SJ. Transfers of patient care between house staff on internal medicine wards: A national survey. Arch Intern Med. 2006;166:1173–1177
24. Lang VJ, Mooney CJ, O’Connor AB, Bordley DR, Lurie SJ. Association between hand-off patients and subject exam performance in medicine clerkship students. J Gen Intern Med. 2009;24:1018–1022
25. Tworek J, Coderre S, Wright B, McLaughlin K. Virtual patients: ED-2 band-aid or valuable asset in the learning portfolio? Acad Med. 2010;85:155–158
26. Kaufman DM. Applying educational theory in practice. BMJ. 2003;326:213–216
27. Edelbring S, Broström O, Henriksson P, et al Integrating virtual patients into courses: Follow-up seminars and perceived benefit. Med Educ. 2012;46:417–425
28. Edelbring S, Dastmalchi M, Hult H, Lundberg IE, Dahlgren LO. Experiencing virtual patients in clinical learning: A phenomenological study. Adv Health Sci Educ Theory Pract. 2011;16:331–345
29. Bearman M, Cesnik B, Liddell M. Random comparison of “virtual patient” models in the context of teaching clinical communication skills. Med Educ. 2001;35:824–832
30. Deladisma AM, Johnsen K, Raij A, et al Medical student satisfaction using a virtual patient system to learn history-taking communication skills. Stud Health Technol Inform. 2008;132:101–105
© 2013 by the Association of American Medical Colleges