Secondary Logo

Journal Logo

A Systematic Review of Serious Games in Training Health Care Professionals

Wang, Ryan BA; DeMaria, Samuel Jr MD; Goldberg, Andrew MD; Katz, Daniel MD

Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare: February 2016 - Volume 11 - Issue 1 - p 41–51
doi: 10.1097/SIH.0000000000000118
Review Article
Free
SDC

Summary Statement Serious games are computer-based games designed for training purposes. They are poised to expand their role in medical education. This systematic review, conducted in accordance with PRISMA guidelines, aimed to synthesize current serious gaming trends in health care training, especially those pertaining to developmental methodologies and game evaluation. PubMed, EMBASE, and Cochrane databases were queried for relevant documents published through December 2014. Of the 3737 publications identified, 48 of them, covering 42 serious games, were included. From 2007 to 2014, they demonstrate a growth from 2 games and 2 genres to 42 games and 8 genres. Overall, study design was heterogeneous and methodological quality by MERQSI score averaged 10.5/18, which is modest. Seventy-nine percent of serious games were evaluated for training outcomes. As the number of serious games for health care training continues to grow, having schemas that organize how educators approach their development and evaluation is essential for their success.

Supplemental digital content is available in the text.

From the Icahn School of Medicine at Mount Sinai, New York, NY.

Reprints: Ryan Wang, c/o Daniel Katz, KCC 8th Floor, One Gustave L Levy Place, Box 1010, New York, NY 10029 (e-mail: ryan.wang@mssm.edu).

The authors of this study have no conflicts of interest to disclose. Drs DeMaria, Goldberg, and Katz are employed by the Mount Sinai Hospital System.

This work should be attributed to the Department of Anesthesiology of the Icahn School of Medicine at Mount Sinai.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.simulationinhealthcare.com).

The motivating factors behind the incorporation of simulation into medical education have been well described.1–3 Features of simulation that lead to effective learning, such as providing standardized, repeated practice as well as specific feedback, were identified and analyzed in systematic reviews by Issenberg et al.4 and Cook et al.5 The use of simulation in medical education was associated with positive results in the acquisition of knowledge and skills and in patient outcomes by separate meta-analyses.6,7 Although these benefits have been well documented and significant, the expensive human resources required to deliver mannequin or standardized patient-based simulation and their availability with respect to busy trainee schedules8 have led some to wonder what the next steps should be regarding what “simulation” is taken to mean in medical education and whether such resource-intensive modalities are needed.9,10

For this reason, serious gaming, which falls under the umbrella of simulation, is poised to take on a greater role in health care training. Bergeron11 defines serious games as “interactive computer applications, with or without significant hardware components,” created for the purpose of imparting knowledge or skills, and which incorporate an element of scoring as well as challenging goals and engaging design. From conveying the sociopolitical forces underlying the American Revolution to helicopter pilot training, serious games have been used effectively as educational tools in a wide range of disciplines.12,13

Graafland et al14 have published 2 instructive studies on serious gaming in medical training. One develops and communicates a framework for assessing medical serious games, those directed at either providers or recipients of health care, from an end-user perspective. This work presents a tool to address an important challenge to the use of serious games, namely, the lack of a structured understanding of a serious game’s purpose and effectiveness as well as attendant risks and benefits. The evaluation tool builds on their earlier systematic review, which surveys serious games in medical education and organizes them by adherence to medical education assessment criteria.15 The utility of an updated systematic review is supported by a number of factors. First, the intervening years allow us to quantify the assertion that serious gaming in medical education is a growing field.14,16 Second, a survey of the current array of training goals, game genres used, and developmental approaches can guide the production of additional games. Third, serious games have been evaluated according to a range of outcomes; holistic quality assessment of study methodologies facilitates the identification of high-quality studies across the different outcomes. Thus, the objectives of this systematic review were to aggregate the available articles on serious games designed for health care professionals, investigate the developmental processes implemented, identify a number of effective games, and assess the evaluation methodologies used.

Back to Top | Article Outline

METHODS

This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.17 To be eligible for inclusion, studies were required to describe or assess serious games created for the purpose of training health care professionals. Our definition of “health care professional” encompassed physicians, nurses, paramedics, physical therapists, and others involved with patient care, at all levels of training (ie, students to advanced level practitioners). A serious game was defined using the criteria provided by Bergeron.11 The stipulations that serious games should be computer applications and that they incorporate an element of scoring were enforced. All types of publications, such as articles, editorials, and conference abstracts, reported in English and published online by December 31, 2014, were included. No lower limit for the publication date was applied. Past reviews were not included; however, their eligible references were.

PubMed, EMBASE, and Cochrane were queried using the following search terms: (serious gam*) OR (videogam*) OR (video gam*) OR (gaming) AND ((educat*) OR (train*)). These terms were selected using an iterative process guided by consultation with a research librarian, faculty from the simulation research center, and past systematic reviews.15,18 The last search date was May 27, 2015. The returned studies were combined with the references of 2 relevant reviews,15,18 and duplicates were removed. Studies were then screened independently and in duplicate by title and abstract (R.W., D.K.) and sorted into the following categories: not a training/education tool, intended for patients or the lay public, not computer based, non–serious game simulation, relevant articles, and serious games for health care professionals. These classes were initially piloted across the first 500 results from PubMed. Any disagreements were settled by a third reviewer (S.D.). A data extraction form was adapted from one used by Issenberg et al4 (2005) for assessing medical simulations (see Text Document, Supplemental Digital Content 1, http://links.lww.com/SIH/A248, which is the complete data extraction form). It was then piloted over the entirety of the included studies. Data extraction was performed independently and in duplicate (R.W., D.K.), with any disagreements decided by the third reviewer (S.D.). Methodological quality was also assessed independently and in duplicate using the medical education research study quality instrument (MERQSI) described by Reed et al.19 If a study used multiple assessment tools (eg, a questionnaire and a multiple-choice examination), a MERQSI score was calculated for each one, with the highest score representing the study in subsequent analysis. Evaluation of studies on the “validity of evaluation instrument” domain was guided by the descriptions of Cook et al.20

Data extraction was performed first using the publication. If additional information was necessary, we attempted to procure a copy or a video of the serious game first by following any links given in the publication; then by an Internet search of the game name, publication title, and author in any combination; and finally by e-mailing the corresponding author for access to the game or for further information. If a determination still could not be made, the field was marked “not specified.” Game genres were categorized according to the nomenclature described by Wolf21 (Table 1). Serious games could be assigned more than 1 genre as some overlap is unavoidable; genres are constantly in flux as new works are being developed.21 Research designs were organized according to the terminology provided by Campbell and Stanley.23 Study outcomes were graded according to their abilities to demonstrate achievement of one of Kirkpatrick’s 4 levels of evaluation: trainee satisfaction with the training tool, gain of skills or knowledge, transfer of learning to the workplace, and finally change in organizational practices (ie, improved patient outcomes).24

TABLE 1

TABLE 1

Back to Top | Article Outline

Statistical Analysis

Cohen κ was used to assess interrater agreement for publication categorization, data extraction, and methodological quality assessment. Data extraction and methodological quality assessment involved numerous domains, such that κ values were reported as an average of the qualitative domains’ κ values. Literature search sensitivity was estimated first by assessing all of the references in relevant past reviews, those of Graafland et al15 or de Wit-Zuurendonk and Oei,18 for satisfaction of the inclusion criteria. The search sensitivity is the fraction of these included studies that were also separately identified by the database search strategy used in this review. Data were organized using Microsoft Excel (Microsoft Corporation, Redmond, WA). Calculations were performed on SPSS 20 (IBM, Endicott, NY).

Back to Top | Article Outline

RESULTS

Literature Review

The literature review identified 48 publications, discussing 42 unique serious games, for inclusion (Fig. 1). Interrater agreement calculated by Cohen κ was 0.97 for categorizing publications, 0.95 for data extraction, and 0.97 for MERQSI items. Literature search sensitivity was 100%. The compiled list of serious games is presented in Table 2, organized by medical field. Eight games were discussed only in nonarticle publications and are presented at the end of Table 2. Of note, the requirement that serious games should be computer applications excluded tools that could be recognized as such by other definitions, such as educational board games or classroom quiz games. The stipulation that serious games contain a scoring element excluded some virtual patient applications. Those with gaming elements were examined on a case-by-case basis. Rationalizations for these decisions are presented in the discussion.

TABLE 2

TABLE 2

FIGURE 1

FIGURE 1

A broad collection of fields was represented in the serious games, from surgery and obstetrics to radiology and primary care. The number of serious games published grew from 4 in 2007 to 42 in 2014, and the number of genres grew from 2 to 8 (see Graph, Supplemental Digital Content 2, http://links.lww.com/SIH/A249, which illustrates the growth of published serious games).

Back to Top | Article Outline

Developmental Data

An assortment of game genres was compatible with a range of learning goals. The development team members and technical resources used were heterogeneous (Table 2). There were too many serious games to describe every one individually; however, we will discuss an effective representative from each genre.

The pediatric board game is an example of the board game genre. Gameplay occurs in the setting of a game board with a 100-tile path. The first player to reach the end wins. Players answer pediatric knowledge questions to advance. Additional game elements include special tiles that act as shortcuts or award “consult cards,” which allow players to skip questions they do not know.49

GeriatriX demonstrates the management simulation and training simulation genres. Players diagnose and treat a number of geriatric patients in a primary care clinic. Each component of the workup has an associated cost. Only 1 diagnostic test can be performed at a time, with scaled in-game times required for the completion of each test. Both time and money must be appropriately allocated to achieve good patient care and high scores. On-screen meters dynamically display patient satisfaction, expenses, and quality of medical care. After each patient has been treated, players receive feedback on the cost-consciousness and effectiveness of their patient care.89

An example of the adaptation and quiz genres is Bronx Jeopardy, an adaptation of the TV game show but for pediatrics.22 Players compete by earning points for correctly answering questions on the psychosocial aspects of pediatric care. Points are lost for incorrect answers. Similar PowerPoint (Microsoft Corporation) templates are available online.83

Underground is an example of the puzzle and platform genres. The player interacts with the game using Nintendo Wii (Nintendo Co, Ltd, Kyoto, Japan) controllers that have been repurposed as laparoscopic tools. The player’s objective is to assist robots in escaping a mine, represented as a series of platforms, by reshaping the environment to allow their egress. The player must build and place elements such as elevators and bridges in appropriate positions for the robots to move toward the exit. To do so, the player must perform a number of actions, which replicate laparoscopic actions in the operating room, such as grasping and cutting.68,69

Elderquest is an adventure game set in a medieval fantasy world, through which the player can navigate freely from a first person perspective. The game narrative asks the player to complete a series of quests, all of which require the player to practice various Association of American Medical Colleges (AAMC) geriatric competencies. For instance, to advance in the game, the player must care for an apothecary who becomes delirious, an exercise in the behavioral and cognitive disorders competency.76,77

Back to Top | Article Outline

Evaluation Data

Of the 42 serious games, 33 (79%) included a study design for evaluating the serious game as a teaching intervention (Table 3). The majority of the studies that did not were developmental or pilot studies. Studies were heterogeneous in many aspects, including the type and number of study participants, the methods of data collection, and the study design. Of the 19 studies that attempted to evaluate their games for improving skill or knowledge gains, only 2 (11%) did not find significant differences between the intervention and comparison groups upon assessment or significant improvement after serious game use in 1 group pretest-posttest studies.49,76 Scores in MERQSI for methodology quality ranged from 6 to 16, with an average of 10.5. Again, because it would be impractical to detail every study, representative studies directed at different Kirkpatrick outcome levels will be described.

TABLE 3

TABLE 3

eMedOffice, a medical practice management training tool, demonstrated trainee satisfaction (Kirkpatrick level 1). It was evaluated by a usability survey completed by medical students after exposure to the serious game. It was rated as having high overall usability. Self-reported knowledge surveys were completed before and after use of the serious game. Self-reported knowledge significantly increased after the intervention.25

The Blood Pressure (BP) Management Game was associated with increased trainee knowledge (Kirkpatrick level 2) and improved patient outcomes (Kirkpatrick level 4). Attending physicians were randomized to 1 of 2 groups: the intervention arm received the serious game, whereas the control group received an online posting of the same educational content. Multiple-choice pretests and posttests were administered to both groups. Both groups scored similarly in the pretest, whereas the intervention group scored significantly higher in the posttest. Patient outcome data, gathered from the electronic medical record, was the time to target BP (<140/90 mmHg) of a hypertensive episode (predefined by the authors) during the study period. A multivariate-adjusted analysis of patients already taking antihypertensive medications showed a significantly reduced time to target BP in patients treated by the intervention cohort.32

No level 3 Kirkpatrick outcomes (ie, transfer of learning to the workplace) were assessed by the included studies.

Back to Top | Article Outline

DISCUSSION

Serious games have the potential to be a disruptive innovation, one that alters the existing market for training modalities,90 because they enjoy many of the same advantages as other forms of simulation2,91 (eg, enhances patient safety, adapts to specific learning objectives, standardizes training) while allowing for reduced operating costs and wider accessibility.92 Before any such potential is realized, an understanding of the present state of serious games is necessary. To that end, this systematic review describes several findings. First, 42 unique serious games for medical education were identified in the literature. Second, serious games were used for training by many medical fields, to facilitate a wide range of learning objectives. As such, this study identified growth in the number of games and genres. Third, the methodological quality of the included studies was heterogeneous, as were the associated study designs. Overall, the findings depict serious gaming in medical education as a modality that continues to grow and establish itself; best practices for its development, evaluation, and use are still being defined. For this reason, educators can be guided by reviews such as this and others15,18 as well as development and assessment frameworks, either those specific to serious gaming14,15 or those addressing simulation at large.93–95

One possible limitation of this review is that the guidelines of Graafland et al14 for the systematic assessment of serious games include a number of parameters omitted from this study. The majority of these omissions occurred because of a difference in scope. Some areas of the framework of Graafland et al are more pertinent to published works available to the public or commercially. Although how a serious game generates income, who manages its content, and who owns any data it produces are important items to consider, this review examines games that are still in development or under evaluation, that is, not yet ready to address such issues. Another limitation of this review on the level of the individual studies is their modest methodological quality scores. The average MERQSI score was 10.5, near the average score of 9.95 calculated by Reed et al,19 who applied the scale to 210 medical education research studies. An additional limitation is our exclusion of non–computer-based training games. We appreciate the similarities between these games and those included in this review as well as the likelihood that such physical games were the predecessors of serious games as defined by Bergeron.11 The focus on computer applications is primarily attributable to their rapid growth and penetration in the field of education. Again, we do not deny the usefulness of non–computer-based serious games; however, the majority of game development in its current form is focused on computer-based games. As such, to maximize the utility of the review for guiding future development, only computer-based games were included. Another limitation is that our study excluded virtual patient applications, whereas past reviews did not.15,18 This is indicative of a larger debate, namely, at what point does a simulation become a game. Virtual patient encounters are not necessarily games by default simply because they use a computer platform. In addition, there is currently no framework for deciding where this line should be drawn. For this reason, we believe that the exclusion is justified. Virtual patient studies with gaming elements were included on a case-by-case basis, as a spectrum exists between simulation and game.64 Analysis of the gamification of simulators offers examples of the elements that draw a pure simulation into the realm of games; these include competition, scoring systems, and rewards for achievements.96,97 The impact of such gamification strategies on training efficacy has yet to be thoroughly elucidated and requires its own study. As such, it will not be explored further here. However, we argue that without the addition of game elements, the transfer of mannequin simulation scenarios to digital applications does not transmute them from simulations to games.

The results of this review agree with those of Graafland et al15 and de Wit-Zuurendonk et al18; all 3 systematic reviews identify serious games in medical education as a growing field that requires continued evaluation and the establishment of best practices. Only 1 of the 42 studies in this review assessed outcomes beyond gain in trainee knowledge or skills, which is in line with the assessment of Graafland et al that serious games as a whole have yet to prove enhanced task performance in reality (ie, improved patient care or outcomes).15 In addition, this study endeavors to build on past work by engaging the developmental aspect of serious games in an attempt to aggregate the resources that have been used and to understand the games themselves. This may serve educators contemplating building their own serious games, as the development of games is a major component of their use (unlike mannequin-based simulation, which generally uses already manufactured devices). Whereas Graafland et al provide a useful theoretical structure for evaluating serious games for training health care professionals, our study describes the practical side of serious game evaluation by compiling the parameters of past study methodologies and organizing them by a holistic methodological quality score.

This review demonstrates that the concerns of serious games literature outside of health care apply to health care gaming as well. Both are interested in improving serious games by examining commercial ones. We organized serious games by genre definitions created for conventional video games in hopes of guiding future development and improving taxonomy. A review by Kirriemuir and Mcfarlane98 identifies distinguishing features between games for learning and those for entertainment to characterize elements specific to serious games that prevent them from enjoying the success of their commercial counterparts. Michael and Chen99 recall how early failed attempts at educational gaming damaged the credibility of serious games. They see rigorous assessment of serious games as the key to swaying skeptics. We hope to have clarified strategies for serious games evaluation, while demonstrating that there is still much work to be done in this area.

In conclusion, serious gaming is a growing health care training platform that serves a range of learning objectives and specialties via numerous game genres. The assessment of serious games as training tools is an essential but currently heterogeneous process, with varying degrees of methodological quality, which overall, are in need of improvement. For serious games to continue its growth in training health care professionals, work must be done to build and empirically verify organizational frameworks for their development, evaluation, and distribution.

Back to Top | Article Outline

REFERENCES

1. Bokken L, Rethans JJ, Scherpbier AJ, van der Vleuten CP. Strengths and weaknesses of simulated and real patients in the teaching of skills to medical students: a review. Simul Healthc 2008; 3(3): 161–169.
2. Okuda Y, Bryson EO, DeMaria S Jr, et al. The utility of simulation in medical education: what is the evidence? Mt Sinai J Med 2009; 76: 330–343.
3. McGaghie WC, Issenberg SB, Barsuk JH, Wayne DB. A critical review of simulation-based mastery learning with translational outcomes. Med Educ 2014; 48(4): 375–385.
4. Issenberg S, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach 2005; 27(1): 10–28.
5. Cook DA, Hamstra SJ, Brydges R, et al. Comparative effectiveness of instructional design features in simulation-based education: systematic review and meta-analysis. Med Teach 2013; 35(1): e867–e898.
6. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 2011; 306(9): 978–988.
7. McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med 2011; 86(6): 706–711.
8. Buttussi F, Pellis T, Cabas Vidani A, Pausler D, Carchietti E, Chiattaro L. Evaluation of a 3D serious game for advanced life support retraining. Int J Med Inform 2013; 82(9): 798–809.
9. Gallagher CJ, Issenberg SB. Simulation in Anesthesia. Philadelphia, PA: Elsevier, Inc; 2007: 72–79.
10. Norman G. Editorial: simulation—savior or Satan? Adv Health Sci Educ Theory Pract 2003; 8(1): 1–3.
11. Bergeron BP. Developing Serious Games. Hingham, MA: Charles River Media; 2006: xvii.
12. Squire K, Jenkins H. Harnessing the power of games in education. Insight 2003; 3: 5–33.
13. Proctor MD, Bauer M, Lucario T. Helicopter flight training through serious aviation gaming. J Def Model Simul 2007; 4(3): 277–294.
14. Graafland M, Dankbaar M, Mert A, et al. How to systematically assess serious games applied to health care. JMIR Serious Games 2014; 2(2): e11.
15. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills training. Br J Surg 2012; 99(10): 1322–1330.
16. Adams SA. Use of “serious health games” in health care: a review. Stud Health Technol Inform 2010; 157: 160–166.
17. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. BMJ 2009; 339: b2535.
18. de Wit-Zuurendonk LD, Oei SG. Serious gaming in women’s health care. BJOG 2011; 118(Suppl 3): 17–21.
19. Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA 2007; 298(9): 1002–1009.
20. Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract 2014; 19(2): 233–250.
21. Wolf MJ. Genre and the video game. Available at: http://http://www.robinlionheart.com/gamedev/genres.xhtml. Accessed June 13, 2015.
22. Jirasevijinda T, Brown LC. Jeopardy!: an innovative approach to teach psychosocial aspects of pediatrics. Patient Educ Couns 2010; 80(3): 333–336.
23. Campbell DT, Stanley J. Experimental and Quasi-Experimental Designs for Research. Boston, MA: Houghton Mifflin Company; 1963: 6–13.
24. Kirkpatrick DL, Kirkpatrick JL. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc; 2006: 26–33.
25. Hannig A, Kuth N, Ozman M, Jonas S, Spreckelsen C. eMedOffice: a Web-based collaborative serious game for teaching optimal design of a medical practice. BMC Med Educ 2012; 12: 104.
26. Borro-Escribano B, Martínez-Alpuente I, Blanco AD, Torrente J, Fernández-Manjón B, Matesanz R. Application of game-like simulations in the Spanish Transplant National Organization. Transplant Proc 2013; 45(10): 3564–3565.
    27. Blanca Borro Escribano. EvaluacionDonante—YouTube. Available at: https://http://www.youtube.com/watch?v=Zn_xqyAgqG4. Accessed June 13, 2015.
      28. UMC St. Radboud. Radboud. Available at: http://radboud.littlechicken.nl/. Accessed June 13, 2015.
        29. Duque G, Fung S, Mallet L, Posel N, Fleiszer D. Learning while having fun: the use of video gaming to teach geriatric house calls to medical students. J Am Geriatr Soc 2008; 56(7): 1328–1332.
          30. Duque G, Demontiero O, Whereat S, et al. Evaluation of a blended learning model in geriatric medicine: a successful learning experience for medical students. Australas J Ageing 2013; 32(2): 103–109.
            31. Duque G. Riskdom. Available at: http://http://www.riskdom.com/. Accessed June 13, 2015.
              32. Kerfoot BP, Turchin A, Breydo E, Gagnon D, Conlin PR. An online spaced-education game among clinicians improves their patients’ time to blood pressure control: a randomized controlled trial. Circ Cardiovasc Qual Outcomes 2014; 7(3): 468–474.
              33. Akl EA, Mustafa R, Slomka T, Alawneh A, Vedavalli A, Schünemann HJ. An educational game for teaching clinical practice guidelines to internal medicine residents: development, feasibility and acceptability. BMC Med Educ 2008; 8: 50.
                34. Diehl LA, Souza RM, Alves JB, et al. InsuOnline, a serious game to teach insulin therapy to primary care physicians: design of the game and a randomized controlled trial for educational validation. JMIR Res Protoc 2013; 2(1): e5.
                  35. InsuOnline demonstração. Available at: https://http://www.youtube.com/watch?v=d6_cleSoZm8. Accessed June 13, 2015.
                    36. Rondon S, Sassi FC. Furquim de Andrade CR: computer game-based and traditional learning method: a comparison regarding students’ knowledge retention. BMC Med Educ 2013; 13: 30.
                      37. Thompson M, Ford R, Webster A. Effectiveness of interactive, online games in learning neuroscience and students’ perception of the games as learning tools. A pre-experimental study. J Allied Health 2011; 40(3): 150–155.
                        38. Cranial nerves. Available at: http://http://www.purposegames.com/game/178. Accessed June 13, 2015.
                          39. Mooney GA, Bligh JG. CyberIST. A virtual game for medical education. Med Teach 1998; 20(3): 212–216.
                            40. Schlegel EF, Selfridge NJ. Fun, collaboration and formative assessment: skinquizition, a class wide gaming competition in a medical school with a large class. Med Teach 2014; 36(5): 447–449.
                              41. Blake J, Goodman J. Computer-based learning: games as an instructional strategy. ABNF J 1999; 10(2): 43–46.
                                42. Johnston B, Boyle L, MacArthur E, Manion B. The role of technology and digital gaming in nurse education. Nurs Stand 2013; 27(28): 35–38.
                                  43. CHERMUG LLP Project Public Site | CHERMUG Project Public Site. Available at: http://http://www.chermug.eu/index.html. Accessed June 13, 2015.
                                    44. Hahn JE, Bartel C. Teaching gaming with technology in the classroom: so you want to be an educator? Nurs Educ Perspect 2014; 35(3): 197–198.
                                      45. Kanthan R, Senger JL. The impact of specially designed digital games-based learning in undergraduate pathology and medical education. Arch Pathol Lab Med 2011; 135(1): 135–142.
                                        46. Kanthan R. Digital games designed for pathology education. Lab Invest 2009; 89: 113A–115A.
                                          47. Pathology games. Available at: http://http://www.emap.usask.ca/resources/path-games/index.php. Accessed June 13, 2015.
                                            48. Fonseca LM, Dias DM, Góes Fdos S, et al. Development of the e-Baby serious game with regard to the evaluation of oxygenation in preterm babies: contributions of the emotional design. Comput Inform Nurs 2014; 32(9): 428–436.
                                              49. Sward KA, Richardson S, Kendrick J, Maloney C. Use of a Web-based game to teach pediatric content to medical students. Ambul Pediatr 2008; 8(6): 354–359.
                                              50. Kendrick J, Maloney C. Medical Boardgame Web site. Available at: http://http://www.medgame.org/. Accessed June 13, 2015.
                                                51. Khan MN, Telmesani A, Alkhotani A, Elzouki A, Edrees B, Alsulimani MH. Comparison of jeopardy game format versus traditional lecture format as a teaching methodology in medical education. Saudi Med J 2011; 32(11): 1172–1176.
                                                  52. Chan WY, Qin J, Chui YP, Heng PA. A serious game for learning ultrasound-guided needle placement skills. IEEE Trans Inf Technol Biomed 2012; 16(6): 1032–1042.
                                                    53. Kurenov S, Cance W, Noel B, Mozingo D. Game-based mass casualty burn training. Stud Health Technol Inform 2009; 142: 142–144.
                                                      54. Delasobera BE, Goodwin TL, Strehlow M, et al. Evaluating the efficacy of simulators and multimedia for refreshing ACLS skills in India. Resuscitation 2010; 81(2): 217–223.
                                                        55. Cardiac Arrest!—ACLS Simulator, Mad Scientist Software. Available at: http://madsci.com/products/product_info.php/products_id/31. Accessed June 13, 2015.
                                                          56. Cook NF, McAloon T, O’Neill P, Beggs R. Impact of a Web based interactive simulation game (PULSE) on nursing students’ experience and performance in life support training—a pilot study. Nurse Educ Today 2012; 32(6): 714–720.
                                                            57. Meterissian S, Liberman M, McLeod P. Games as teaching tools in a surgical residency. Med Teach 2007; 29(9): e258–e260.
                                                              58. Graafland M, Bemelman WA, Schijven MP. Prospective cohort study on surgeons’ response to equipment failure in the laparoscopic environment. Surg Endosc 2014; 28(9): 2695–2701.
                                                                59. Graafland M, Bemelman WA, Schijven MP. A serious game to improve situation awareness in laparoscopic surgery. Surg Endosc 2014; 28(1 suppl): S42.
                                                                  60. Dr. Game: Surgeon Trouble. Available at: http://weirdbeard.nl/lapchol/. Accessed June 13, 2015.
                                                                    61. Graafland M, Vollebergh MF, Lagarde SM, van Haperen M, Bemelman WA, Schijven MP. A serious game can be a valid method to train clinical decision-making in surgery. World J Surg 2014; 38(12): 3056.
                                                                      62. Medialist on the App Store on iTunes. Available at: https://itunes.apple.com/us/app/medialis/id968345677?mt=8. Accessed June 13, 2015.
                                                                        63. Cowan B, Sabri H, Kapralos B, Moussa F, Cristancho S, Dubrowki A. A serious game for off-pump coronary artery bypass surgery procedure training. Stud Health Technol Inform 2011; 163: 147–149.
                                                                          64. Qin J, Chui YP, Pang WM, Choi KS, Heng PA. Learning blood management in orthopedic surgery through gameplay. IEEE Comput Graph Appl 2010; 30(2): 45–57.
                                                                          65. Lin DT, Park J, Liebert CA, Lau JN. Validity evidence for surgical improvement of clinical knowledge ops: a novel gaming platform to assess surgical decision making. Am J Surg 2015; 209(1): 79–85.
                                                                            66. SICKO—Standford University School of Medicine. Available at: http://med.stanford.edu/sm/archive/cme/sicko/game/SICKO.html. Accessed June 13, 2015.
                                                                              67. Cowan B, Sabri H, Kapralos B, et al. A serious game for total knee arthroplasty procedure, education and training. J Cyber Ther Rehabil 2010; 3(3): 285–298.
                                                                                68. Jalink MB, Goris J, Heineman E, Pierie JP, ten Cate Hoedemaker HO. Construct and concurrent validity of a Nintendo Wii video game made for training basic laparoscopic skills. Surg Endosc 2014; 28(2): 537–542.
                                                                                69. Goris J, Jalink MB, Ten Cate Hoedemaker HO. Training basic laparoscopic skills using a custom-made video game. Perspect Med Educ 2014; 3(4): 314–318.
                                                                                70. Inspired by laparoscopy. Available at: http://http://www.undergroundthegame.com/. Accessed June 13, 2015.
                                                                                  71. SeriousGames@Google: Playing Surgery—A Laparoscopy Game for Surgeons on the Nintendo Wii—YouTube. Available at: https://youtu.be/rpSvDvYvJGk?t=1110. Accessed June 13, 2015.
                                                                                    72. Shewaga R, Knox A, Ng G, Kapralos B, Dubrowski A. Z-DOC: a serious game for Z-plasty procedure training. Stud Health Technol Inform 2013; 184: 404–406.
                                                                                      73. Boeker M, Andel P, Vach W, Frankenschmidt A. Game-based e-learning is more effective than a conventional instructional method: a randomized controlled trial with third-year medical students. PLoS One 2013; 8(12): e82328.
                                                                                        74. Kerfoot BP, Baker H. An online spaced-education game to teach and assess residents: a multi-institutional prospective trial. J Am Coll Surg 2012; 214(3): 367–373.
                                                                                          75. Kerfoot BP, Baker H. An online spaced-education game for global continuing medical education: a randomized trial. Ann Surg 2012; 256(1): 33–38.
                                                                                            76. Pomidor A, Brummel-Smith K, Baker S. Elderquest: enhancing learning with video games. J Am Geriatr Soc 2012; 60: S154.
                                                                                            77. Pomidor A, Pomidor B, Granville L, Brummel-Smith K, Baker S. Elderquest. Video game fun with the AAMC competencies. J Am Geriatr Soc 2011; 59: S79.
                                                                                            78. ElderQuest—Adventures in AAMC Geriatrics. Available at: http://http://www.elderquest.org/. Accessed June 13, 2015.
                                                                                              79. Hamrick I, Vian L, Kandukuri S. Virtual reality home visit-teaching home safety. J Am Geriatr Soc 2013; 61(Suppl 1): S41.
                                                                                                80. Haig AJ, Loar S, Maslowski E. Backquack. An on-line computer game that teaches doctors and patients to mess up. PM&R 2012; 4(10): S242.
                                                                                                  81. Haig et al. Consulting—What We Do—Health System Quality Improvement—BackQuack. Available at: http://http://www.backquack.net. Accessed June 13, 2015.
                                                                                                    82. Latessa R, Harman JH Jr, Hardee S, Scmidt-Dalton T. Teaching medicine using interactive games: development of the “stumpers” quiz show game. Fam Med 2004; 36(9): 616.
                                                                                                      83. Jeopardy.ppt. Available at: http://www.rrginc.com/xferfiles/jeopardy.ppt. Accessed June 9, 2015.
                                                                                                      84. Tehrani AS, Omron R, Duval-Arnould J, et al. Low-cost diagnostic gaming to measure symptom-specific diagnostic reasoning skills. Neurology 2012; 78(Meeting Abstracts 1): P07.235.
                                                                                                        85. Rouse C, Bertozzi E, Villafuerte A, Hernández DG, Walker D. Emergency birth!: piloting video game technology as a tool for training critical practices for maternal and neonatal survival among traditional birth attendants. Am J Obstet Gynecol 2014; 210(suppl 1): S236–S237.
                                                                                                          86. Ardea Arts Work. Available at: http://http://www.ardeaarts.org/pages/emergencybirth.html. Accessed June 13, 2015.
                                                                                                            87. Timothy A, Marije H, Theo P, van Heijst A. Serious gaming in ECMO simulation. Int J Artif Organs 2013; 36(4): 272.
                                                                                                              88. Ishtiaq S, Rayyali R, Nabhani S, Dudzinski M, Greenhill D, Caton H. Evaluation of the perceptions and impact of an educational game designed for MPharm students. Int J Pharm Pract 2013; 21(S2): 92–93.
                                                                                                                89. Lagro J, van de Pol MH, Laan A, Huijbregts-Verheyden FJ, Fluit LC, Olde Rikkert MG. A randomized controlled trial on teaching geriatric medical decision making and cost consciousness with the serious game GeriatriX. J Am Med Dir Assoc 2014; 15(12): 957.
                                                                                                                90. Bower JL, Christensen CM. Disruptive technologies: catching the wave. Harvard Bus Rev 1995; 73(1): 43–53.
                                                                                                                91. Gaba DM. The future vision of simulation in health care. Qual Saf Health Care 2004; 13(suppl 1): i2–i10.
                                                                                                                92. Creutzfeldt J, Hedman L, Felländer-Tsai L. Effects of pre-training using serious game technology on CPR performance—an exploratory quasi-experimental transfer study. Scand J Trauma Resusc Emerg Med 2012; 20: 79.
                                                                                                                93. Cook DA, Levinson AJ, Garside S. Method and reporting quality in health professions education research: a systematic review. Med Educ 2011; 45(3): 227–238.
                                                                                                                94. Vankipuram A, Khanal P, Ashby A, et al. Design and development of a virtual reality simulator for advanced cardiac life support training. IEEE J Biomed Health Inform 2014; 18(4): 1478–1484.
                                                                                                                95. Rutherford-Hemming T, Lioce L, Durham CF. Implementing the standards of best practice for simulation. Nurse Educ 2015; 40(2): 96–100.
                                                                                                                96. Kerfoot BP, Kissane N. The use of gamification to boost residents’ engagement in simulation training. JAMA Surg 2014; 149(11): 1208–1209.
                                                                                                                97. Nevin CR, Westfall AO, Rodriguez JM, et al. Gamification as a tool for enhancing graduate medical education. Postgrad Med J 2014; 90(1070): 685–693.
                                                                                                                98. Kirriemuir J, Mcfarlane A. Literature review in games and learning. A NESTA Futurelab Research Report 2004; report 8: 19–21.
                                                                                                                99. Michael DR, Chen SL. Serious Games: Games That Educate, Train, and Inform. Boston, MA: Thomson Course Technology PTR; 2006: 40.
                                                                                                                Keywords:

                                                                                                                Medical education; Health care training; Serious game; Simulation; Review

                                                                                                                Supplemental Digital Content

                                                                                                                Back to Top | Article Outline
                                                                                                                © 2016 Society for Simulation in Healthcare