Secondary Logo

Institutional members access full text with Ovid®

Validity Evidence for a Serious Game to Assess Performance on Critical Pediatric Emergency Medicine Scenarios

Gerard, James, M., MD; Scalzo, Anthony, J., MD; Borgman, Matthew, A., MD, CHSE; Watson, Christopher, M., MD; Byrnes, Chelsie, E., MD; Chang, Todd, P., MD, MAcM; Auerbach, Marc, MD, MSci; Kessler, David, O., MD, MSci; Feldman, Brian, L., MD; Payne, Brian, S., MD; Nibras, Sohail, MD; Chokshi, Riti, K., BS; Lopreiato, Joseph, O., MD, MPH, CHSE

Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare: June 2018 - Volume 13 - Issue 3 - p 168–180
doi: 10.1097/SIH.0000000000000283
Empirical Investigations

Introduction We developed a first-person serious game, PediatricSim, to teach and assess performances on seven critical pediatric scenarios (anaphylaxis, bronchiolitis, diabetic ketoacidosis, respiratory failure, seizure, septic shock, and supraventricular tachycardia). In the game, players are placed in the role of a code leader and direct patient management by selecting from various assessment and treatment options. The objective of this study was to obtain supportive validity evidence for the PediatricSim game scores.

Methods Game content was developed by 11 subject matter experts and followed the American Heart Association's 2011 Pediatric Advanced Life Support Provider Manual and other authoritative references. Sixty subjects with three different levels of experience were enrolled to play the game. Before game play, subjects completed a 40-item written pretest of knowledge. Game scores were compared between subject groups using scoring rubrics developed for the scenarios. Validity evidence was established and interpreted according to Messick's framework.

Results Content validity was supported by a game development process that involved expert experience, focused literature review, and pilot testing. Subjects rated the game favorably for engagement, realism, and educational value. Interrater agreement on game scoring was excellent (intraclass correlation coefficient = 0.91, 95% confidence interval = 0.89–0.9). Game scores were higher for attendings followed by residents then medical students (P c < 0.01) with large effect sizes (1.6–4.4) for each comparison. There was a very strong, positive correlation between game and written test scores (r = 0.84, P < 0.01).

Conclusions These findings contribute validity evidence for PediatricSim game scores to assess knowledge of pediatric emergency medicine resuscitation.

From the Saint Louis University School of Medicine (J.M.G., A.J.S., S.N., R.K.C.), St. Louis, MO; Brooke Army Medical Center (M.A.B.), Fort Sam Houston, TX; Uniformed Services University of the Health Sciences (M.A.B., B.L.F.), Bethesda, MD; Medical College of Georgia (C.M.W.), Augusta University, Augusta, GA; Naval Medical Center San Diego (C.E.B.), San Diego, CA; Uniformed Services University of the Health Sciences (C.E.B.), Bethesda, MD; Keck School of Medicine of University of Southern California (T.P.C.), Los Angeles, CA; Yale University School of Medicine (M.A.), New Haven, CT; Columbia University College of Physicians and Surgeons (D.O.K.), New York, NY; Naval Medical Center Portsmouth (B.L.F.), Portsmouth, VA; Texas Tech University Health Sciences Center (B.S.P.), Lubbock, TX; Department of Psychiatry (S.N.), Saint Louis University School of Medicine, St. Louis, MO; and Val G. Hemming Simulation Center (J.O.L.), Uniformed Services University of the Health Sciences, Bethesda, MD.

Reprints: James M. Gerard, MD, Saint Louis University School of Medicine & SSM Cardinal Glennon Children's Hospital, Room G644, 1465 South Grand Blvd, St. Louis, MO 63104 (e-mail:

The authors declare no conflict of interest.

Supported by the Office of Naval Research.

This work should be attributed to the Department of Pediatrics, Saint Louis University School of Medicine, St. Louis, MO.

The view(s) expressed herein are those of the author(s) and do not reflect the official policy or position of Brooke Army Medical Center, the US Army Medical Department, the US Army Office of the Surgeon General, the Department of the Navy, the Department of the Army, or the Department of Defense, or the US Government.

© 2018 Society for Simulation in Healthcare