Secondary Logo

The Role for Virtual Patients in the Future of Medical Education

Berman, Norman B. MD; Durning, Steven J. MD, PhD; Fischer, Martin R. MD, MME; Huwendiek, Soren MD, MME; Triola, Marc M. MD

doi: 10.1097/ACM.0000000000001146
Perspectives
Free

The medical education community is working—across disciplines and across the continuum—to address the current challenges facing the medical education system and to implement strategies to improve educational outcomes. Educational technology offers the promise of addressing these important challenges in ways not previously possible. The authors propose a role for virtual patients (VPs), which they define as multimedia, screen-based interactive patient scenarios. They believe VPs offer capabilities and benefits particularly well suited to addressing the challenges facing medical education. Well-designed, interactive VP-based learning activities can promote the deep learning that is needed to handle the rapid growth in medical knowledge. Clinically oriented learning from VPs can capture intrinsic motivation and promote mastery learning. VPs can also enhance trainees’ application of foundational knowledge to promote the development of clinical reasoning, the foundation of medical practice. Although not the entire solution, VPs can support competency-based education. The data created by the use of VPs can serve as the basis for multi-institutional research that will enable the medical education community both to better understand the effectiveness of educational interventions and to measure progress toward an improved system of medical education.

N.B. Berman is professor, Department of Pediatrics, Geisel School of Medicine at Dartmouth, Hanover, New Hampshire.

S.J. Durning is professor of medicine and pathology, Uniformed Services University, Bethesda, Maryland.

M.R. Fischer is professor and chair for medical education, Institut für Didaktik und Ausbildungsforschung in der Medizin, University Hospital, LMU Munich, Germany.

S. Huwendiek is senior lecturer and head of the department, Department of Assessment and Evaluation, Institute of Medical Education, University of Bern, Bern, Switzerland.

M.M. Triola is associate professor and associate dean for educational informatics, New York University School of Medicine, New York, New York.

Funding/Support: None reported.

Other disclosures: None reported.

Ethical approval: Reported as not applicable.

Disclaimer: The ideas and views in this Perspective are those of the authors alone and do not necessarily represent those of the U.S. Department of Defense or the federal government of the United States.

Correspondence should be addressed to Norman B. Berman, One Medical Center Drive, Lebanon, NH 03756; telephone: (603) 653-9888; e-mail: norman.berman@dartmouth.edu.

Much has been written about the challenges facing the current medical education system in the United States.1 Medical knowledge is expanding rapidly, which demands not only more efficient teaching methods but also the teaching of knowledge management, yet lectures and book learning remain primary means of instruction in many medical schools. Much evidence shows the negative impact of the current educational model on student mental health,2 and indirect evidence indicates that improving mental health and capturing intrinsic motivation will have a positive impact on learning.3 Additionally, although clinical reasoning is a cornerstone of medical practice, the continued problem of diagnostic error4 suggests that medical education should focus more on the application of foundational knowledge in diverse contexts to foster both the development of diagnostic expertise and the acknowledgment of one’s own limits. Further, legitimate concerns about medical error5 call into question not only the quality of the health care system but also whether the contemporary U.S. system of medical education needs to better assess the competence of its graduates. The medical education community is working—across disciplines and across the continuum—to identify and implement strategies to improve educational outcomes6; however, the current challenges will remain difficult to solve without a better understanding of the effectiveness of these new educational strategies.

Just as a modern health care system is compelled to translate advances in the basic and clinical sciences into medical practice, a modern medical education system must translate advances in fields such as cognitive and educational psychology, education, the learning sciences, and educational technology into educational practice. The incorporation of technology into education offers the promise of addressing educational challenges in new ways.7,8 Often, modern technologies offer more hope than actual solutions, and there is the potential for this to occur in the use of educational technology in medical education. Our aim with this Perspective is to suggest roles for a specific form of technology-enhanced education—virtual patients (VPs)—in addressing specific challenges facing medical education. We will do this by, first, describing what VPs are and their current roles in medical education and, then, proposing specific educational strategies for the use of VPs and the educational outcomes we believe VPs can facilitate.

Back to Top | Article Outline

What Are VPs?

Medical educators and others have defined “virtual patient” as “an interactive computer simulation of real-life clinical scenarios for the purpose of healthcare and medical training, education, or assessment”9 or “a specific type of computer program that simulates real-life clinical scenarios [through which] learners emulate the roles of health care providers to obtain a history, conduct a physical exam, and make diagnostic and therapeutic decisions.”10 We believe, however, that these definitions are insufficient in characterizing the technologies or features that might be incorporated into a VP. Huwendiek and colleagues11 presented an empirically derived typology including 19 different factors for classifying VPs; factors include, for example, whether or not the scenario has branch points and the use of interactivity and feedback. Kononowicz and colleagues12 adapted a VP classification initially developed by Talbot et al13 to include the underlying technology and the competency being addressed. For the purposes of this Perspective, we are considering VPs to be multimedia, screen-based interactive patient scenarios; this definition excludes other teaching methods that might be considered VPs in Kononowicz and colleagues’ classification such as simple case presentations, VP games, high-fidelity software simulation, mannequin-based simulators, and virtual standardized patients. Our definition also excludes other forms of computer-based education such as digital slide presentations and educational videos.

Back to Top | Article Outline

Current Role of VPs

Medical educators use VPs to achieve widely varied instructional goals including not only teaching core knowledge,14 clinical reasoning,15 and communication skills16 but also assessing learners’ progress.17 Lehmann and colleagues18 demonstrated the value of VPs, when blended with simulation, in supporting the teaching of clinical skills, and Fall and colleagues19 reported on VP development based on comprehensive coverage of nationally accepted curricula. Finally, Berman and colleagues20 reported on collaborative development of VPs across multiple institutions; their work shows that collaboration makes the task of covering broad curricular objectives more manageable while also taking advantage of the ability to deliver VPs at scale, as suggested by Ellaway et al.8

VP use in medical education is substantial. In the United States, VP use is most common in clinical clerkship education. By 2007 the Computer-assisted Learning In Pediatrics Program (CLIPP), a VP program for pediatrics, was used in more than 70 medical schools.19 A mixed-methods study of VP adoption based on the CLIPP program21 demonstrated that the program’s ability to fill gaps in students’ exposure to core clinical problems, the use of a national curriculum, and the program’s development by clerkship directors were important factors leading to broad adoption of the program. VP use is not limited to pediatrics. Surveys of internal medicine (IM) clerkship directors in 2009 and 2011 included questions on the uses and purposes of VPs in IM training programs in the United States.22 On the basis of these data, Lang and colleagues22 reported that meeting regulatory requirements was an important initial motivator and that improving the quality of learning became more important over time. The NetWoRM case-based e-learning project in occupational medicine serves as another successful example of the use of VPs in medical education; specifically, it demonstrates that a consortium project can lead to multi-institutional, national, and international use of a shared collection of VPs.23 Currently, VPs are in use in more than 130 medical schools in the United States and Canada24 and in many European countries.25

For VPs to be effective, they must be used, and there are strategies for integrating VPs, which will promote their use. Berman and colleagues26 showed that VPs can be effectively integrated into clinical education by coordinating their use with other learning activities (e.g., didactics, clinical experiences) and assessments and by making room in the course through the elimination of some lectures and textbook assignments. Hege and colleagues27 investigated a wide range of scenarios for integrating VPs into the medical curriculum and suggested a voluntary rather than an obligatory approach. Huwendiek and colleagues28 identified learner preferences, suggesting the importance of sequencing and aligning VPs with other activities and assessments.

Back to Top | Article Outline

Future Role of VPs

In our introduction above, we have outlined a series of challenges facing medical education (e.g., rapidly expanding medical knowledge, the ongoing occurrence of diagnostic and other cognitive errors, the evolving understanding of learning preferences, the need for better assessment). Fortunately, advances in the science of cognition and learning give educators a better foundation for designing educational strategies to address some of these challenges. Here we discuss five educational strategies, each of which is intended to address an important challenge in medical education. We propose ways in which VPs can be incorporated into these strategies, and we suggest the educational outcomes that can be improved with these strategies. The challenges, VP-based educational strategies, examples of VP educational activities, and expected educational outcomes are outlined in Table 1.

Table 1

Table 1

Back to Top | Article Outline

Leverage interactive learning activities to promote deep learning

Chi,29 an education researcher, has proposed a conceptual framework and provided empirical evidence supporting a hierarchy of learning activities. This framework suggests that instructional interventions that incorporate overtly active, constructive, and/or interactive activities will promote deeper learning, which emphasizes understanding and the application of knowledge over memorization and recall. Having learners track key findings presented in a VP is an example of an active learning activity. Creating a summary statement from the history and physical exam findings of a VP is an example of a constructive activity. VPs offer the advantage of a standardized case presentation, making reliable and valid assessment of such summary statements more feasible. Further, Smith and colleagues30 recently described a framework for evaluating a summary statement which can be applied to real patients, as well as, importantly, VPs. Finally, responding to multiple-choice questions, long menu questions, or other novel question types supported by VP technology, and receiving rich individual feedback on those responses, is an example of an interactive learning activity. Advances in VP software could provide structured feedback on student answers to free-text questions, an even higher level of interactivity. Kopp and colleagues31 have shown that a learner-centered and VP-driven environment incorporating active, constructive, and interactive learning activities, like the ones mentioned above, can foster gains in diagnostic knowledge.

Ellaway32 proposes that medical educators can develop VP-based activities to achieve various specific objectives. In this context, VPs are well suited to support emerging instructional interventions such as the “flipped classroom.” In the flipped classroom model, learners might practice a particular concept on their own with the VP and, then, attend a seminar or a problem-based learning or team-based learning session. VPs with embedded learning analytics can measure student engagement in these activities or predict learning outcomes.

Back to Top | Article Outline

Capture intrinsic motivation to foster mastery and lifelong learning

Several educational theories suggest the importance of motivation33 and the affective nature of learning.34 Intrinsic motivation, which arises from a desire to learn a topic because it is enjoyable and/or interesting, is closely correlated with a mastery goal orientation. Research from a variety of disciplines has shown that mastery goal orientation (e.g., focusing on mastery of the subject) improves learning more than a performance goal orientation (e.g., focusing on getting an “A” in a course or clerkship).3 Further, achievement emotions (i.e., the feelings learners have toward an academic activity) relate to goal orientation, affect motivation, and impact learning outcomes. Learning activities that foster positive emotions, such as enjoyment and pride, rather than negative emotions, such as boredom or frustration, are preferable. VPs can be designed to capitalize on intrinsic motivation, mastery goal orientation, and achievement emotions to improve learning.35

To illustrate, novice learners using VPs can engage with content that is authentic but designed to avoid cognitive overload. Medical educators can design VP courses to match learning or cognitive demands with student capabilities—an ideal that is difficult to achieve with real patients. Further, educators can develop VPs at different levels of difficulty to complement several levels of learner performance. An adaptive VP system could recommend additional activities for learners based on their prior performance, resulting in better matching of learner ability and demands. Learners can receive highly individualized and timely feedback via self-assessment dashboards that show strengths and where improvements are needed. Learners can also repeat VPs or complete additional VPs to improve their performance. This approach can give learners a choice of instructional activities, allowing them to determine their own pace as they progress through the activity.

Back to Top | Article Outline

Apply knowledge to support the development of clinical reasoning expertise

Extensive research shows clearly that clinical reasoning expertise cannot exist without content knowledge,36 yet students can have difficulty applying knowledge of foundational concepts when solving or explaining clinical problems.37 Norman’s review of the educational psychology literature presents a number of strategies to facilitate transfer of conceptual knowledge to the clinical setting; these strategies include embedding the concept in a problem context and incorporating active problem solving at the time of the initial learning.37 Further, both mixed practice (through which problems illustrating different concepts are presented together) and distributed practice (in which experiences are dispersed over time) can result in large and significant learning gains.37 VPs can support each of these approaches. Basic science and other foundational concepts such as statistical analysis and population health can be incorporated into VPs, and conversely VPs can be integrated into basic science education.

Cook and Triola38 have proposed—on the basis of learning theory and a review of the literature—VPs as an ideal instructional method to prepare learners for clinical reasoning in real patients. In a review of educational strategies to promote clinical reasoning, Bowen39 emphasized the following techniques: asking open-ended questions; providing single-sentence summaries of patient problems in abstract terms; asking for discriminating features of a set of diagnostic hypotheses; probing early for differential diagnoses; prioritizing diagnoses; comparing and contrasting diagnostic hypotheses based on real clinical data; demonstrating typical presentations of different diagnostic hypotheses; and presenting the relative probabilities of different diagnoses. Educators can design VPs to incorporate these techniques. In a focus group study of medical students, Huwendiek and colleagues40 found that students perceive many of these techniques as helpful for fostering clinical reasoning when learning with VPs.

Back to Top | Article Outline

Assess learner competence to reduce medical errors

Educators in both undergraduate and graduate medical education are rapidly adopting competency frameworks in an effort to move away from a purely time-based progression through training to, instead, a progression that is also informed by milestones of achieved mastery of a skill. Initiatives such as the Physician Competency Reference Set41 and the Next Accreditation System42 have defined sets of common learning goals that graduating medical students and residents must meet at varying levels of training. VPs can play a key role as medical education transitions to these competency-based assessment systems.43

Although the medical education community knows that medical error is often the result of problems in the health care delivery system,5 the community also knows that it is better for learners to make mistakes on virtual rather than real patients. Given their nature as a screen-based intervention, VPs can be readily and flexibly integrated into assessment activities in almost any setting, can be delivered at any time, and have been used with learners at every level and in multiple disciplines in health professions education and practice.44–47 This asynchronous capability is a natural fit for assessing learners who may vary in the timing of their progression through training. The key feature exam format, which focuses specifically on testing for decision making, has been successfully incorporated into VP-based assessment.48,49 Two advantages of VP-based competency assessment strategies are that medical educators can align them with VP learning cases and that they can be truly standardized from one learner to the next.50

Conversely, VPs could dynamically adapt to the performance level of an individual learner, highlight particular patient factors, and teach the effects of the determinants of health. Medical students, for example, could learn through a progressive pattern of increasingly challenging VPs, the timing of which is generated from their actual patient encounters as recorded in clinical experience logs and the electronic medical record (EMR). These linkages between VPs and experiences, when coupled with established standards for authoring and exchanging VPs,51 could enable a learning health care system in which learners are taught and assessed by VP cases—perhaps even cases that are machine generated directly from deidentified comprehensive patient records in the EMR. This progressive model also suggests natural integrations with other patient simulation modalities such as mannequin-based simulators and standardized patients.

Back to Top | Article Outline

Analyze educational data to develop a better understanding of educational outcomes

VPs, if implemented broadly, can create large amounts of educational data. These types of data, unique to the use of VPs, are relatively new to medical educators, and their use, though not yet well established, has great potential. Learning analytics refers to the use of educational data to assess current performance and predict future performance. Medical educators can apply learning analytic and educational-data-mining techniques across a large number of students and institutions to assess educational gains, and they can use VP-generated learning data to predict success or failure in specific domains.52 To illustrate, clinical instructors could assess the development of a learner’s clinical reasoning and clinical decision making by applying analytics to actions the student makes in a VP. Applied analytics could reflect the student’s understanding of key clinical findings and show how the student’s actions affect the differential diagnosis. Another domain that educators could potentially assess using VP-generated data is learning strategy. Analytic techniques may be able to identify a mastery learning orientation, a marker of a deeper learning strategy.

An area of potential future development is the integration of VP datasets with large educational and clinical databases.53 Combining such large amounts of data from two different arenas could facilitate much more powerful multi-institutional research. Such research and other applications of analytics will be more feasible when standardized VPs are common or shared across multiple institutions and when resources can be pooled to co-create high-quality teaching content and assessment instruments.

Back to Top | Article Outline

Challenges With VPs

Despite all the promise of VPs, their impact on medical education to date remains limited. In 2008, Berman and colleagues54 pointed out several barriers to broad implementation of computer-assisted instruction programs in medical education, and the same challenges exist for VPs today. There is often a disconnect between available VP programs and the needs of the educators who might incorporate them into their teaching or courses. A lack of clarity among educators and learners regarding the educational role of VPs leads to difficulties in effectively integrating VPs in clinical education. A widely accepted system for ongoing financial and technical support of VPs does not yet exist, and significant efforts to support dissemination and adoption of VPs8,9 have not yet resulted in widespread sharing or repurposing. Finally, a lack of sufficient evidence for the features of VPs that create effective learning remains a significant barrier for those skeptical educators who may have seen educational fads come and go in the past. Schifferdecker and colleagues21 identified factors leading to broad VP use (e.g., the ability to fill gaps in clinical exposure, the use of a national curriculum and development by educators) which are consistent with existing models of adoption of innovation, so we believe that with attention to these challenges the impact of VPs will grow.

Back to Top | Article Outline

In Sum

Case-based and patient-based learning will always be at the heart of medical education, and technology will play an increasingly important role in education in the future. VPs are fundamentally a patient-based means of learning enabled by technology. VPs, if incorporated into medical education more broadly, can be an efficient and effective method to achieve the goal of creating a medical education system that better educates the next generation of providers to serve in a transformed health care system that better suits the needs of patients and society.

Well-designed and interactive VP-based learning activities can be used to promote the deep learning necessary in an era of rapid growth in medical knowledge. Clinically oriented learning from VPs can capture intrinsic motivation and promote mastery learning. VPs can help enhance the integration of the foundational sciences and clinical education to promote the development of clinical reasoning skills. We believe that VPs have the potential to be an important component of medical education reform that incorporates critical interdisciplinary topics, interprofessional education, and competency-based learning.

We believe that VPs have the potential to make a significant impact on medical education and that their use will likely grow. The data generated by the use of VPs can facilitate multi-institutional research that will enable the medical education community both to better understand the effectiveness of educational interventions and to measure progress. There is strong conceptual support for the role that VPs can play in the transformation of medical education, but much more can and should be done to take advantage of the benefits they offer.

Acknowledgments: The authors wish to thank Elyse Payson for her support in organizing the authoring of this Perspective and to Sarah Berman for her grammatical editing and many helpful suggestions toward improving the Perspective.

Back to Top | Article Outline

References

1. Irby DM, Cooke M, O’Brien BC. Calls for reform of medical education by the Carnegie Foundation for the Advancement of Teaching: 1910 and 2010. Acad Med. 2010;85:220–227.
2. Dyrbye LN, Thomas MR, Shanafelt TD. Systematic review of depression, anxiety, and other indicators of psychological distress among U.S. and Canadian medical students. Acad Med. 2006;81:354–373.
3. Artino AR Jr, Holmboe ES, Durning SJ. Control-value theory: Using achievement emotions to improve understanding of motivation, learning, and performance in medical education: AMEE guide no. 64. Med Teach. 2012;34:e148–e160.
4. Graber ML, Franklin N, Gordon R. Diagnostic error in internal medicine. Arch Intern Med. 2005;165:1493–1499.
5. Kohn LT, Corrigan J, Donaldson MS. To Err Is Human: Building a Safer Health System. 2000.Washington, DC: National Academy Press.
6. Cooke M, Irby DM, Sullivan W, Ludmerer KM. American medical education 100 years after the Flexner report. N Engl J Med. 2006;355:1339–1344.
7. Sandars J, Haythornthwaite C. New horizons for e-learning in medical education: Ecological and Web 2.0 perspectives. Med Teach. 2007;29:307–310.
8. Ellaway R, Poulton T, Fors U, McGee JB, Albright S. Building a virtual patient commons. Med Teach. 2008;30:170–174.
9. Ellaway R, Candler C, Greene P, Smothers V. An Architectural Model for MedBiquitous Virtual Patients. 2006. Baltimore, MD: MedBiquitous; http://groups.medbiq.org/medbiq/display/VPWG/MedBiquitous+Virtual+Patient+Architecture. Accessed February 23, 2016.
10. AAMC Institute for Improving Medical Education. Effective use of educational technology in medical education: Colloquium on educational technology: Recommendations and guidelines for medical educators. 2007. Washington, DC: Association of American Medical Colleges; https://members.aamc.org/eweb/upload/Effective%20Use%20of%20Educational.pdf. Accessed February 1, 2016.
11. Huwendiek S, De leng BA, Zary N, Fischer MR, Ruiz JG, Ellaway R. Towards a typology of virtual patients. Med Teach. 2009;31:743–748.
12. Kononowicz AA, Zary N, Edelbring S, Corral J, Hege I. Virtual patients—What are we talking about? A framework to classify the meanings of the term in healthcare education. BMC Med Educ. 2015;15:11.
13. Talbot TB, Sagae K, Bruce J, Rizzo AA. Sorting out the virtual patient: How to exploit artifical intelligence, game technology and sound education practices to create engaging role-playing simulations. Int J Gaming Comput Mediat Simul. 2012;4:1–19.
14. Sanders CL, Kleinert HL, Free T, et al. Caring for children with intellectual and developmental disabilities: Virtual patient instruction improves students’ knowledge and comfort level. J Pediatr Nurs. 2007;22:457–466.
15. Garrett BM, Callear D. The value of intelligent multimedia simulation for teaching clinical decision-making skills. Nurse Educ Today. 2001;21:382–390.
16. Sijstermans R, Jaspers MW, Bloemendaal PM, Schoonderwaldt EM. Training inter-physician communication using the dynamic patient simulator. Int J Med Inform. 2007;76:336–343.
17. Round J, Conradi E, Poulton T. Improving assessment with virtual patients. Med Teach. 2009;31:759–763.
18. Lehmann R, Bosse HM, Simon A, Nikendei C, Huwendiek S. An innovative blended learning approach using virtual patients as preparation for skills laboratory training: Perceptions of students and tutors. BMC Med Educ. 2013;13:23.
19. Fall LH, Berman NB, Smith S, White CB, Woodhead JC, Olson AL. Multi-institutional development and utilization of a computer-assisted learning program for the pediatrics clerkship: The CLIPP project. Acad Med. 2005;80:847–855.
20. Berman NB, Fall LH, Chessman AW, et al. A collaborative model for developing and maintaining virtual patients for medical education. Med Teach. 2011;33:319–324.
21. Schifferdecker KE, Berman NB, Fall LH, Fischer MR. Adoption of computer-assisted learning in medical education: The educators’ perspective. Med Educ. 2012;46:1063–1073.
22. Lang VJ, Kogan J, Berman N, Torre D. The evolving role of online virtual patients in internal medicine clerkship education nationally. Acad Med. 2013;88:1713–1718.
23. Kolb S, Wengenroth L, Hege I, et al; NetWoRM Group. Case based e-learning in occupational medicine—A European approach. J Occup Environ Med. 2009;51:647–653.
24. MedU. Homepage. 2013. www.med-u.org. Accessed January 27, 2016.
25. Smothers V, Ellaway R, Balasubramaniam C. eViP: Sharing virtual patients across Europe. AMIA Annu Symp Proc. November 6, 2008:1140.
26. Berman N, Fall LH, Smith S, et al. Integration strategies for using virtual patients in clinical clerkships. Acad Med. 2009;84:942–949.
27. Hege I, Ropp V, Adler M, et al. Experiences with different integration strategies of case-based e-learning. Med Teach. 2007;29:791–797.
28. Huwendiek S, Duncker C, Reichert F, et al. Learner preferences regarding integrating, sequencing and aligning virtual patients with other activities in the undergraduate medical curriculum: A focus group study. Med Teach. 2013;35:920–929.
29. Chi MT. Active–constructive–interactive: A conceptual framework for differentiating learning activities. Top Cogn Sci. 2009;1:73–105.
30. Smith S, Kogan JR, Berman NB, Dell MS, Brock DM, Robins LS. The development and preliminary validation of a rubric to assess medical students’ written summary statements in virtual patient cases. Acad Med. 2016;91:94–100.
31. Kopp V, Stark R, Fischer MR. Fostering diagnostic knowledge through computer-supported, case-based worked examples: Effects of erroneous examples and feedback. Med Educ. 2008;42:823–829.
32. Ellaway RH. Virtual patients as activities: Exploring the research implications of an activity theoretical stance. Perspect Med Educ. 2014;3:266–277.
33. Kusurkar RA, Ten Cate TJ, van Asperen M, Croiset G. Motivation as an independent and a dependent variable in medical education: A review of the literature. Med Teach. 2011;33:e242–e262.
34. Artino AR Jr, Holmboe ES, Durning SJ. Can achievement emotions be used to better understand motivation, learning, and performance in medical education? Med Teach. 2012;34:240–244.
35. Yavner SD, Pusic MV, Kalet AL, et al. Twelve tips for improving the effectiveness of Web-based multimedia instruction for clinical learners. Med Teach. 2015;37:239–244.
36. Eva KW. What every teacher needs to know about clinical reasoning. Med Educ. 2005;39:98–106.
37. Norman G. Teaching basic science to optimize transfer. Med Teach. 2009;31:807–811.
38. Cook DA, Triola MM. Virtual patients: A critical literature review and proposed next steps. Med Educ. 2009;43:303–311.
39. Bowen JL. Educational strategies to promote clinical diagnostic reasoning. N Engl J Med. 2006;355:2217–2225.
40. Huwendiek S, Reichert F, Bosse HM, et al. Design principles for virtual patients: A focus group study among students. Med Educ. 2009;43:580–588.
41. Association of American Medical Colleges. Physician competency reference set. August 2013. https://www.aamc.org/initiatives/cir/about/348808/aboutpcrs.html. Accessed January 27, 2016.
42. Accreditation Council on Graduate Medical Education. Next accreditation system. https://www.acgme.org/acgmeweb/tabid/435/ProgramandInstitutionalAccreditation/NextAccreditationSystem.aspx. Accessed January 27, 2016.
43. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: Theory to practice. Med Teach. 2010;32:638–645.
44. Lowdermilk DL, Fishel AH. Computer simulations as a measure of nursing students’ decision-making skills. J Nurs Educ. 1991;30:34–39.
45. Schittek Janda M, Mattheos N, Nattestad A, et al. Simulation of patient encounters using a virtual patient in periodontology instruction of dental students: Design, usability, and learning effect in history-taking skills. Eur J Dent Educ. 2004;8:111–119.
46. Gesundheit N, Brutlag P, Youngblood P, Gunning WT, Zary N, Fors U. The use of virtual patients to assess the clinical skills and reasoning of medical students: Initial insights on student acceptance. Med Teach. 2009;31:739–742.
47. Waldmann UM, Gulich MS, Zeitler HP. Virtual patients for assessing medical students—Important aspects when considering the introduction of a new assessment format. Med Teach. 2008;30:17–24.
48. Fischer MR, Kopp V, Holzer M, Ruderich F, Jünger J. A modified electronic key feature examination for undergraduate medical students: Validation threats and opportunities. Med Teach. 2005;27:450–455.
49. Bronander KA, Lang VJ, Nixon LJ, et al. How we developed and piloted an electronic key features examination for the internal medicine clerkship based on a US national curriculum. Med Teach. 2015;37:807–812.
50. Stevens A, Hernandez J, Johnsen K, et al. The use of virtual patients to teach medical students history taking and communication skills. Am J Surg. 2006;191:806–811.
51. Triola MM, Campion N, Mcgee JB, et al. An XML standard for virtual patients: Exchanging case-based simulations in medical education. AMIA Ann Symp Proc. October 11, 2007:741–745.
52. Ellaway RH, Pusic MV, Galbraith RM, Cameron T. Developing the role of big data and analytics in health professional education. Med Teach. 2014;36:216–222.
53. Cook DA, Andriole DA, Durning SJ, Roberts NK, Triola MM. Longitudinal research databases in medical education: Facilitating the study of educational outcomes over time and across institutions. Acad Med. 2010;85:1340–1346.
54. Berman NB, Fall LH, Maloney CG, Levine DA. Computer-assisted instruction in clinical education: A roadmap to increasing CAI implementation. Adv Health Sci Educ Theory Pract. 2008;13:373–383.
© 2016 by the Association of American Medical Colleges