Secondary Logo

Journal Logo

Validity Evidence for a Serious Game to Assess Performance on Critical Pediatric Emergency Medicine Scenarios

Gerard, James, M., MD; Scalzo, Anthony, J., MD; Borgman, Matthew, A., MD, CHSE; Watson, Christopher, M., MD; Byrnes, Chelsie, E., MD; Chang, Todd, P., MD, MAcM; Auerbach, Marc, MD, MSci; Kessler, David, O., MD, MSci; Feldman, Brian, L., MD; Payne, Brian, S., MD; Nibras, Sohail, MD; Chokshi, Riti, K., BS; Lopreiato, Joseph, O., MD, MPH, CHSE

Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare: June 2018 - Volume 13 - Issue 3 - p 168–180
doi: 10.1097/SIH.0000000000000283
Empirical Investigations
Free
SDC

Introduction We developed a first-person serious game, PediatricSim, to teach and assess performances on seven critical pediatric scenarios (anaphylaxis, bronchiolitis, diabetic ketoacidosis, respiratory failure, seizure, septic shock, and supraventricular tachycardia). In the game, players are placed in the role of a code leader and direct patient management by selecting from various assessment and treatment options. The objective of this study was to obtain supportive validity evidence for the PediatricSim game scores.

Methods Game content was developed by 11 subject matter experts and followed the American Heart Association's 2011 Pediatric Advanced Life Support Provider Manual and other authoritative references. Sixty subjects with three different levels of experience were enrolled to play the game. Before game play, subjects completed a 40-item written pretest of knowledge. Game scores were compared between subject groups using scoring rubrics developed for the scenarios. Validity evidence was established and interpreted according to Messick's framework.

Results Content validity was supported by a game development process that involved expert experience, focused literature review, and pilot testing. Subjects rated the game favorably for engagement, realism, and educational value. Interrater agreement on game scoring was excellent (intraclass correlation coefficient = 0.91, 95% confidence interval = 0.89–0.9). Game scores were higher for attendings followed by residents then medical students (Pc < 0.01) with large effect sizes (1.6–4.4) for each comparison. There was a very strong, positive correlation between game and written test scores (r = 0.84, P < 0.01).

Conclusions These findings contribute validity evidence for PediatricSim game scores to assess knowledge of pediatric emergency medicine resuscitation.

From the Saint Louis University School of Medicine (J.M.G., A.J.S., S.N., R.K.C.), St. Louis, MO; Brooke Army Medical Center (M.A.B.), Fort Sam Houston, TX; Uniformed Services University of the Health Sciences (M.A.B., B.L.F.), Bethesda, MD; Medical College of Georgia (C.M.W.), Augusta University, Augusta, GA; Naval Medical Center San Diego (C.E.B.), San Diego, CA; Uniformed Services University of the Health Sciences (C.E.B.), Bethesda, MD; Keck School of Medicine of University of Southern California (T.P.C.), Los Angeles, CA; Yale University School of Medicine (M.A.), New Haven, CT; Columbia University College of Physicians and Surgeons (D.O.K.), New York, NY; Naval Medical Center Portsmouth (B.L.F.), Portsmouth, VA; Texas Tech University Health Sciences Center (B.S.P.), Lubbock, TX; Department of Psychiatry (S.N.), Saint Louis University School of Medicine, St. Louis, MO; and Val G. Hemming Simulation Center (J.O.L.), Uniformed Services University of the Health Sciences, Bethesda, MD.

Reprints: James M. Gerard, MD, Saint Louis University School of Medicine & SSM Cardinal Glennon Children's Hospital, Room G644, 1465 South Grand Blvd, St. Louis, MO 63104 (e-mail: gerardjm@slu.edu).

The authors declare no conflict of interest.

Supported by the Office of Naval Research.

This work should be attributed to the Department of Pediatrics, Saint Louis University School of Medicine, St. Louis, MO.

The view(s) expressed herein are those of the author(s) and do not reflect the official policy or position of Brooke Army Medical Center, the US Army Medical Department, the US Army Office of the Surgeon General, the Department of the Navy, the Department of the Army, or the Department of Defense, or the US Government.

The nature of caring for critically ill pediatric patients makes traditional bedside training difficult because the acuity of the situation often limits deliberate instruction and immediate debriefing of trainees. These experiences are also too infrequent to guarantee sustained and meaningful exposure to all critical scenarios. As a result, health care workers often do not receive adequate training in these cases on a clinical basis alone. Previous studies have shown that following completion of formal resuscitation courses, such as the American Heart Association's (AHA) Pediatric Advanced Life Support (PALS) course, cognitive and psychomotor skills rapidly decay. This is particularly true for skills that are infrequently used in clinical practice.1–5 To reinforce these cognitive and psychomotor skills, medical personnel are increasingly trained through scenario-based simulation with debriefing in groups. These simulations, however, require a significant commitment of time and coordination among trainers, trainees, and other ancillary staff, which limits the frequency with which they can be conducted. Moreover, with the focus on group learning, individual knowledge gaps or deficiencies can go uncorrected.6

One alternative simulation method is the use of serious gaming. Serious games are screen-based simulations designed for an educational purpose and are not intended to be played primarily for amusement. Serious games are simulations of real-world events or processes designed for the purpose of solving a problem.7 The use of serious gaming and other screen-based simulations for medical training and assessment continues to grow.8–10 The potential advantages of this educational method include standardization of scenario-based training and debriefing, efficient identification of individual learning needs, automated and customized feedback, virtually unlimited access, increased flexibility for training, and decreased instructor time and related costs.11

We developed a serious game, PediatricSim, for assessment and training on seven pediatric emergency medicine (PEM) scenarios. We chose the PEM environment because it offers a wide variety of acutely ill patients and requires healthcare providers to make numerous assessment and treatment decisions. In addition, this setting would be common to most training programs. The objective of this study is to collect validity evidence for PediatricSim game scores to assess knowledge of PEM resuscitation.

Back to Top | Article Outline

METHODS

The following four of the five domains of Messick's framework for validity evidence were examined: content, response process, internal structure, and relations to other variables.12,13 Messick's fifth domain, consequences of testing, was beyond the scope of this initial study and was not assessed.

Back to Top | Article Outline

Game Description

Built on the Unity game engine (Unity Technologies, San Francisco, CA), PediatricSim is an interactive, three-dimensional, single-player, first-person game that can be downloaded and installed on individual computers or can be played through a web browser. The game allows the player to assume the role of a code leader and direct patient management by selecting assessment and treatment options carried out by ancillary staff avatars. The goal of the game is to assess and initially stabilize a variety of pediatric patients in an emergency department setting. The game features seven scenarios depicting critical pediatric medical diseases including anaphylaxis, bronchiolitis, diabetic ketoacidosis, respiratory failure, seizure, septic shock, and supraventricular tachycardia. Patients range from the ages of 3 weeks to 10 years. The game was designed for a primary target audience of residents and medical students. Context, immersion, rules, user interaction, feedback, and a scoring system are all game elements found within PediatricSim. To improve realism, visual cues have been minimized. For example, no icons exist for equipment and procedures. Players must select these options by clicking on different body parts, which in turn display radial menus from which the player chooses. Screen shots of PediatricSim are shown in Figure 1.

FIGURE 1

FIGURE 1

Players can perform airway, breathing, circulatory, and neurologic assessments. Players receive audio feedback (eg, lung and heart sounds) and text feedback (eg, strength of pulses, skin temperature) in response to their assessment actions. Patients can be placed on a cardiac monitor and pulse oximeter. Players can obtain laboratory and radiographic studies, and they can provide a variety of treatments including medication and fluid administration, bag-mask ventilation, endotracheal intubation, intraosseous needle insertion, and synchronized cardioversion. Medication doses, fluid types and volumes, sizes of equipment, and defibrillator doses must be selected by the player.

Scenarios end when the critical steps for each case have been performed. For example, successfully completing the anaphylaxis case requires administration of two epinephrine doses and one intravenous fluid bolus, which improve the patient's breathing and normalizes blood pressure. The game records a case log that shows each action and time to action performed throughout game play. The game can be played in a tutorial mode, which provides real-time audio or text feedback for incorrect actions, and a nontutorial mode, which only provides feedback upon completion of game play.

Back to Top | Article Outline

Content

The branched-chain logic and educational content for the cases were developed by a team of 11 PEM and pediatric critical care specialists and followed the 2011 AHA PALS Provider Manual and other authoritative references.14–18 Written vignettes were developed for each of the scenarios. Subject matter experts reviewed these vignettes, and in an iterative fashion, key learning points and objectives were determined for each scenario. The final content including correct and incorrect actions for the scenarios was reached by consensus. The principal investigator (J.M.G.) worked in an iterative fashion with the gaming company BreakAway, Ltd (Hunt Valley, MD) to develop the game software.

In conjunction with game development, two of the research team members (J.M.G., B.S.P.) worked iteratively to develop scoring rubrics for each of the scenarios. This scoring tool, embedded in the game, awards points for correctly performing the critical assessment and treatment steps for each scenario as described in the reference materials. Points are deducted for performing potentially dangerous actions (eg, medication dosing errors, administration of incorrect types of intravenous fluids). Both awarded and deducted points are based on a weighted system intended to reflect the relative importance of each assessment and treatment decision for the given scenarios. Players receive up to 10 points for correctly performing the primary assessment and for each critical intervention, up to five points for performing other, less critical interventions, and two points for reassessments. Ten points are deducted for performing potentially dangerous actions. Point values for given interventions are the same across the scenarios. For example, correctly giving a fluid bolus is worth ten points in each scenario. Congruous with the general approach to PALS training, our goal was to develop a scoring tool that placed a slightly greater emphasis on performing critical interventions while still awarding significant points for performing assessments and important, but less critical, interventions. This approach resulted in a scoring system, whereby overall, 55% of the points are awarded for critical interventions, 28% for assessment and reassessment actions, and 17% for important, but less critical, interventions. As the complexity and, therefore, number of critical steps differs for each scenario, the total number of points attainable in each scenario differs. The maximum score achievable is 565 points divided among the cases as follows: anaphylaxis, 80; bronchiolitis, 70; diabetic ketoacidosis, 62; respiratory failure, 77; seizure, 64; septic shock, 116; and supraventricular tachycardia, 96. The scoring rubric for the anaphylaxis case is shown in Appendix 1.

Back to Top | Article Outline

Study Design, Subjects, and Setting

The study was approved by the institutional review boards of the Office of Naval Research and each participating site including Saint Louis University and was conducted from September 2015 to October 2015. Sixty volunteer subjects were enrolled into the study purposefully sampling from three groups with different levels of training: 20 attendings, board certified in either PEM or pediatric critical care medicine; 20 categorical pediatric residents (postgraduate years 1–3); and 20 medical students (years 3 or 4). Subjects were enrolled equally from each of four academic study sites.

Before game play, subjects completed a written pretest of knowledge individually and independently without reference materials. The written test was developed by two of the research team members (J.M.G., A.J.S.) in conjunction with game development. Question development followed the key assessment and treatment teaching points as outlined in the authoritative references used to develop PediatricSim. It was pilot tested among a group of medical students and pediatric residents at Saint Louis University. Based on this pilot testing, several questions were revised or deleted. The final version of the written test (Appendix 2) contained 40 multiple-choice questions comprised as follows: anaphylaxis, 3; bronchiolitis, 2; diabetic ketoacidosis, 6; respiratory failure, 7; seizure, 5; septic shock, 6; supraventricular tachycardia, 6, and general knowledge, 5. The percentages of scoring points contributed by each scenario for both the written test and the game score are shown in Table 1. The investigators sought to match the written test to the game with respect to both content and the relative contribution of each scenario to the overall score.

TABLE 1

TABLE 1

Immediately after completing the written test, each subject viewed a brief video that demonstrated the functional features of the game. After viewing the video, subjects then played through each of the seven scenarios in a fixed order using the nontutorial game mode. Upon completion of data collection, the scoring rubrics were used to score each subject's case logs generated from all of the scenarios. A total game score (the score for all 7 scenarios combined) was recorded as a percentage for each subject by one of the study investigators (J.M.G.).

Back to Top | Article Outline

Response Process

Subjects played the game individually and independently without reference materials throughout game play. A member of the research team proctored each subject throughout the written test and game play sessions, neither of which was time limited. Upon completion of game play, subjects completed a structured demographic, gaming experience, and perceptions survey (Appendix 3). The perceptions portion of the survey addressed the following three main themes: (1) engagement with the game, (2) educational value of the game, and (3) realism of the game.

Back to Top | Article Outline

Internal Structure

The internal consistency of the game was assessed by calculating the overall correlation of scores between different scenarios and by assessing the correlation between individual scenario scores and total game scores. Because the written test was used as evidence to support the validity of the game scores when used to assess knowledge of PEM resuscitation, the internal consistency of this test was also examined. To assess interrater reliability of the scoring rubrics, 84 case logs (20% of all logs) were selected using a random number generator. A blinded scorer independently scored these logs using the scoring rubrics. The blinded scorer, a board-certified PEM physician, was not a member of the research team, had no involvement in designing or conducting the study, and had no knowledge of the experience level of the subjects. The agreement on how each individual item on the scoring rubrics was scored by the study investigator and the blinded reviewer was assessed.

Back to Top | Article Outline

Relations to Other Variables

Within this domain, the study was designed to compare three distinct groups of subjects with different levels of clinical experience and previous PALS training. By enrolling subjects with different levels of experience and PALS training, the study was designed to evaluate whether higher game scores and higher written test scores would be achieved by those with increasing levels of expertise. In addition, the written test was used to correlate game performance with cognitive knowledge of the PEM cases depicted in PediatricSim.

Back to Top | Article Outline

Statistical Analyses

χ2 or Fisher exact test, when appropriate, were used to analyze proportional data, which are reported as number (percent). Continuous data are reported as mean (SD). Cronbach α was used to assess the internal consistency of the written test and game scenarios. An intraclass correlation coefficient was calculated to assess the interrater agreement on game scores between the study investigator and blinded scorer. One-way analysis of variance with post hoc Tukey tests was used to compare groups by level of training for the total game scores and written test scores. Effect sizes for differences between groups were calculated using Cohen d method. Spearman rank correlation coefficient was calculated to correlate total game scores and written test scores for all subjects combined. It was also used to correlate individual scenario scores with total game scores. A two-sided α < 0.05 was considered statistically significant. Analyses were performed using SPSS Version 24.0 software (IBM Corporation, Armonk, NY).

Back to Top | Article Outline

RESULTS

Subject Characteristics

A total of 60 subjects were enrolled into the study. Subject characteristics are shown in Table 2. All of the attendings and residents reported having taken a PALS course compared with only one medical student (P < 0.001). All 20 (100%) of the attendings reported that they “often” or “very often” cared for or observed others caring for seriously ill pediatric patients compared with eight (40%) of the residents and three (15%) of the medical students (P < 0.001). The groups were similar with respect to sex, ethnicity, self-reported computer experience, and gaming habits.

TABLE 2

TABLE 2

Back to Top | Article Outline

Response Process Evidence

The distributions of total game scores and scores for each scenario are shown in Figure 2. Scores tended to cluster by subject level. We found only one extreme outlying score, a medical student who scored 2.4 SDs above the mean for the student group. We generally observed that attendings completed all of the scenarios in 1.5–2 hours, residents in 2.5–3 hours, and students in 3.5–4 hours.

FIGURE 2

FIGURE 2

A summary of the perceptions survey data is shown in Table 3. Most subjects “strongly agreed” or “agreed” that the game was engaging, realistic, easy to use, and had educational value. Overall, 41 subjects (68%) “strongly agreed” or “agreed” that training on the game would increase their confidence to care for pediatric patients with emergency medical conditions. Among residents and students, the target audience to train on PediatricSim, 36 (90%) “strongly agreed” or “agreed” with this statement.

TABLE 3

TABLE 3

Back to Top | Article Outline

Internal Structure Evidence

Internal consistency for the game scenarios was 0.82 (Cronbach α). When comparing individual scenario scores with overall game scores, the following correlation coefficients were found: anaphylaxis, 0.75, bronchiolitis, 0.53, diabetic ketoacidosis, 0.71, respiratory failure, 0.70, seizure, 0.73, septic shock, 0.70, and supraventricular tachycardia, 0.73. The intraclass correlation coefficient for agreement on game scores between the study investigator and blinded scorer was 0.91 (95% confidence interval = 0.89–0.92). Internal consistency for the written test was 0.81 (Cronbach α).

Back to Top | Article Outline

Relations to Other Variables Evidence

Comparisons between groups for game scores and written test scores are shown in Table 4. On both the game and written test, there was an ordering effect with attendings scoring higher than residents who scored higher than the medical students (Pc < 0.01 for each comparison). Large effect sizes were found for all of the group comparisons. Effect sizes for the game scores were as follows: attendings > residents, 1.6; residents > students, 1.7; and attendings > students, 4.4. Effect sizes for the written test scores were as follows: attendings > residents, 1.6; residents > students, 2.4; and attendings > students, 5.3.

TABLE 4

TABLE 4

Spearman rank correlation coefficient for the game and written scores was 0.84 (P < 0.01).

Back to Top | Article Outline

DISCUSSION

The findings of our study provide evidence to support the validity of PediatricSim game scores when used to assess knowledge of PEM resuscitation. In the present study, we examined four domains of validity evidence. Supporting the domain of content evidence, PediatricSim and its scoring rubrics were developed by a team of 11 subject matter experts in the field of pediatric emergency and pediatric critical care medicine and followed the recommendations of the AHA and other authoritative guidelines.14–18 Within the domain of response process, we evaluated PediatricSim following a rigorous study protocol. Subjects played the game independently, in a proctored setting, and seemed to react to the game in the intended manner. As supported by the postgame survey data, the scenarios presented in the game seem to be realistic in content and germane to those training to care for critically ill pediatric patients. Among residents and medical students, the target audience to train on PediatricSim, 90% responded that they believed that training on the game would increase their confidence to care for pediatric patients with emergency medical conditions. Internal structure evidence is provided by high internal consistency for the game scenarios (Cronbach α = 0.82) and high interrater agreement for the scoring rubrics.19 With respect to the domain of relations to other variables, we found a strong ordering effect with subspecialty attendings outperforming residents who outperformed medical students with large effect sizes for all of these differences. In addition, we found a very strong, positive correlation between game scores and scores on a written test covering similar content.20

Somewhat surprisingly, we found that game scores were low across all subject groups. We believe that there are several factors that led to this finding. First, although subjects viewed a brief video demonstrating the functionality of the game, they were not given any practice time before testing on the scenarios. A period of practice before testing might have resulted in subjects better understanding the game's functionality enabling them to achieve higher game scores. However, in further exploring this, we found no consistent pattern of improvement in scores when comparing each subject's first- and last-played scenarios (data not shown). In addition, there are limitations to the game, which may have effected subjects' assessment and treatment decisions. For example, the current version of PediatricSim does not give players the ability to ask the parent or patient any medical history questions. As a result, subjects may not have performed in this simulated setting as they would on a real patient. These potential factors require additional exploration to further guide development and refinement of the game.

For the purpose of this study, we developed a unique written test to assess subjects' knowledge of the specific PediatricSim cases. We intentionally developed a written test that we believed would be difficult. This was done to ensure that we would have a broad range of written test scores, which would improve our ability to correlate these scores to subjects' game scores. Although it was not the goal of our study to collect evidence for the validity of the written test, per se, it was important to do so as we correlated written test scores with game scores to support the validity of the game. In this regard, the written test was developed by subject matter experts and pilot tested before its use. We found good internal consistency and excellent discriminant ability of the written test to discern the three different levels of providers, both of which support its reliability and validity.

In recent years, the number of serious games developed for training health care professionals has grown. A systematic review by Wang et al8 found that from 2007 to 2014, the number of serious games developed for health care training rose from two games in two genres to 42 games in eight genres. Personal computer–based PALS simulators are already in existence. For example, Ventre et al21 have developed and reported on a case-based simulator providing scenarios that cover major PALS algorithms including supraventricular tachycardia, pulseless electrical activity, ventricular tachycardia/ventricular fibrillation, and bradycardia. In addition, HeartCode PALS, developed by the AHA, consists of 12 cases that cover respiratory distress, shock, and cardiac arrhythmias. In comparison, PediatricSim was designed to provide fewer visual cues to the player and thus might offer a more realistic environment for assessment and training.

The Accreditation Council for Graduate Medical Education's Milestone Project emphasizes competency-based education and assessment of residents. A recent study by Mallory et al22 found that 61% of pediatric residency program directors reported using simulation for learner assessment. Of programs that reported using simulation for assessment, 54% used some form of computer modeling. PediatricSim has the potential to be a useful tool for program directors to assess residents' progress and clinical competencies on cases infrequently encountered in clinical training.

This study has several limitations. It is a convenience sample of a relatively small number of voluntary subjects. It is unknown how subjects, if required to train on the game, would perceive its educational value or how they would perform on the game. The primary goal of this study was to evaluate the game as an assessment tool. The effect of training on the game was not directly assessed. Although arguably PediatricSim may not add significant value over traditional methods of assessment (eg, a written test with feedback), the authors believe that demonstrating reliable and valid scores when using the game as an assessment tool is an important first step in considering the potential utility of the game as a training tool. We were also unable to assess how subject's performance in the game translated to actual patient care. Such information would be helpful in establishing what decisions the game could assist educators in making regarding their learner's preparedness for certain clinical environments. Investigations of this correlation should be considered as a next step. Finally, per our study design, we did not impose a time limit on subjects during testing. We generally observed that attendings completed all of the scenarios in 1.5 to 2 hours, residents in 2.5 to 3 hours, and students in 3.5 to 4 hours. However, given that subjects were aware that there was no time limit, we cannot meaningfully comment on how this factor might be used as a metric in assessing players' performances.

Despite these limitations, the present study provides some initial evidence to support the validity of PediatricSim game scores when used to assess knowledge of PEM resuscitation. In its current form, the authors believe that PediatricSim could be used for formative assessment of pediatric residents before rotations where critical patients are likely to be encountered (eg, emergency department, critical care units). Through PediatricSim game play, medical educators could identify and remediate residents' specific knowledge deficiencies before them encountering these cases in a real setting. Future studies are needed to determine whether standalone training on the game will help learners achieve and sustain competency and improve care and outcomes on actual patients. These studies should be designed to determine training frequency and the optimal strategies for incorporating its use within existing training curricula. Exploring Messick's validity domain of consequences was beyond the scope of this initial study. As PediatricSim is further developed and used, however, it will be important to determine whether the game could someday be used for high-stakes assessments. Finally, the investigators believe that the game is ideally suited for medical student and resident trainees; however, future studies looking at training other groups of providers (eg, non-PEM physicians, hospitalists) might help expand PediatricSim's target audience.

Back to Top | Article Outline

REFERENCES

1. Grant EC, Marczinski CA, Menon K. Using pediatric advanced life support in pediatric residency training: does the curriculum need resuscitation? Pediatr Crit Care Med 2007;8:433–439.
2. Su E, Schmidt TA, Mann NC, Zechnich AD. A randomized controlled trial to assess decay in acquired knowledge among paramedics completing a pediatric resuscitation course. Acad Emerg Med 2000;7:779–786.
3. Shilkofski NA, Nelson KL, Hunt EA. Recognition and treatment of unstable supraventricular tachycardia by pediatric residents in a simulation scenario. Simul Healthc 2008;3:4–9.
4. Hunt EA, Walker AR, Shaffner DH, Miller MR, Pronovost PJ. Simulation of in-hospital pediatric medical emergencies and cardiopulmonary arrests: highlighting the importance of the first 5 minutes. Pediatrics 2008;121:e34–e43.
5. Braun L, Sawyer T, Smith K, et al. Retention of pediatric resuscitation performance after a simulation-based mastery learning session: a multicenter randomized trial. Pediatr Crit Care Med 2015;16:131–138.
6. Cappelle C, Paul RI. Educating residents: the effects of a mock code program. Resuscitation 1996;31:107–111.
7. Lopreiato JO. Healthcare Simulation Dictionary. Rockville, MD: Agency for Healthcare Research and Quality; October 2016. AHRQ Publication No. 16(17)-0043.
8. Wang R, DeMaria S, Goldberg A, Katz D. A systematic review of serious games in training health care professionals. Simul Healthc 2016;11:41–51.
9. Dillon GF, Clauser BE. Computer-delivered patient simulations in the United States medical licensing examination (USMLE). Simul Healthc 2009;4:30–34.
10. Dillon GF, Clyman SG, Clauser BE, Margolis MJ. The introduction of computer-based case simulations into the United States medical licensing examination. Acad Med 2002;77(Suppl 10):S94–S96.
11. Chang TP, Gerard JM, Pusic MV. Screen-based Simulations, Virtual Reality and Haptic simulators. In: Grant VJ, Cheng A, eds. Comprehensive Healthcare Simulation: Pediatrics. New York, NY: Springer International Publishing; 2016.
12. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006;119:166.e7–166.e16.
13. Messick S. Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. Am Psychol 1995;50:741–749.
14. Pediatric Advanced Life Support Provider Manual, Dallas, TX; American Heart Association; 2011.
15. Lieberman P, Nicklas RA, Oppenheimer J, et al. The diagnosis and management ofanaphylaxis practice parameter: 2010 update. J Allergy Clin Immunol 2010;126:477–480.
16. Dellinger RP, Levy MM, Rhodes A, et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med 2013;39:165–228.
17. Ralston SL, Lieberthal AS, Meissner HC, et al. Clinical Practice Guideline: the diagnosis, management, and prevention of bronchiolitis. Pediatrics 2014;134:e1474–e1502.
18. Rosenbloom AL. The management of diabetic ketoacidosis in children. Diabetes Ther 2010;1:103–120.
19. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–174.
20. Mukaka MM. A guide to appropriate use of correlation coefficient in medical research. Malawi Med J 2012;24:69–71.
21. Ventre KM, Collingridge DS, DeCarlo D. End-user evaluations of a personal computer-based pediatric advanced life support simulator. Simul Healthc 2011;6:134–142.
22. Mallory LA, Calaman S, Lee White M, et al. Targeting simulation-based assessment for the pediatric milestones: a survey of simulation experts and program directors. Acad Pediatr 2016;16:290–297.
Appendix 1

Appendix 1

Back to Top | Article Outline

Appendix 2. Written Test

For each question below, please circle the single best answer. Please make sure that you answer each question.

  • 1) The normal respiratory rate for an 8 year-old-child is:
    1. 24 – 40 breaths per minute
    2. 22 – 34 breaths per minute
    3. 18 – 30 breaths per minute
    4. 12 – 16 breaths per minute
  • 2) A child presents to the emergency department having a seizure. The correct dose of IV lorazepam (Ativan) to administer to the patient is:
    1. 0.1 mg/kg
    2. 0.3 mg/kg
    3. 0.5 mg/kg
    4. 1.0 mg/kg
  • 3) A 6 year-old-child is apneic. He has palpable pulses. The correct rate to perform bag-mask ventilation is:
    1. 8 – 10 breaths per minute
    2. 12 – 20 breaths per minute
    3. 22 – 26 breaths per minute
    4. 28 – 34 breaths per minute
  • 4) You are performing bag-mask ventilation on a 12 year-old-child who weighs 40 kg. The correct bag size to use is:
    1. A neonatal bag
    2. An infant bag
    3. A child bag
    4. An adult bag
  • 5) A 10 year-old-child who weighs 30 kg is having a seizure. The correct dose of diazepam rectal gel (Diastat) to administer is:
    1. 2.5 mg
    2. 5 mg
    3. 10 mg
    4. 15 mg
  • 6) A 10 year-old-child who weighs 35 kg is having an allergic reaction. The correct dose of intramuscular epinephrine is:
    1. 0.15 mg
    2. 0.3 mg
    3. 0.5 mg
    4. 1.0 mg
  • 7) In infants with supraventricular tachycardia, the heart rate is typically:
    1. ≥160 beats per minute
    2. ≥180 beats per minute
    3. ≥200 beats per minute
    4. ≥220 beats per minute
  • 8) An infant presents to the emergency department with a heart rate of 260 beats per minute. An ECG tracing reveals a narrow QRS complex with no beat-to-beat variability. He is awake and alert. His blood pressure is 90/50 mm Hg. Your first step should be to:
    1. Give an IV fluid bolus of normal saline 20 mL/kg
    2. Attempt vagal maneuvers
    3. Perform synchronized cardioversion
    4. Administer adenosine 0.1 mg/kg IV
  • 9) Which of the following would be the appropriate dose and mode for performing an initial cardioversion attempt on a child in supraventricular tachycardia:
    1. 0.5 – 1.0 J/kg, unsynchronized mode
    2. 0.5 – 1.0 J/kg, synchronized mode
    3. 1.0 – 2.0 J/kg, unsynchronized mode
    4. 1.0 – 2.0 J/kg, synchronized mode
  • 10) Which of the following blood pressures is normal for a 2 month-old-infant:
    1. 65/35 mm Hg
    2. 85/55 mm Hg
    3. 95/65 mm Hg
    4. 105/75 mm Hg
  • 11) Which of the following statements regarding adenosine is true:
    1. It should be administered over 30 – 60 seconds IV
    2. It is the drug of choice for treatment of both supraventricular and ventricular tachycardia
    3. The initial dose for treatment of supraventricular tachycardia is 0.2 mg/kg
    4. It may be given by the intraosseous (IO) route
  • 12) A previously healthy 1 month-old-infant is brought to the emergency department with a heart rate of 260 beats per minute. An ECG tracing reveals a narrow QRS complex with no beat-to-beat variability. Other vital signs are as follows: respiratory rate 80/min, blood pressure 60/35 mm Hg, temperature 99° F. On exam, she is pale and mottled. Your first step should be to:
    1. Apply an ice bag to the face
    2. Establish an IV and give a normal saline bolus of 20 mL/kg
    3. Establish an IV and give adenosine 0.1 mg/kg
    4. Perform synchronized cardioversion
  • 13) Which of the following would be the most appropriate initial fluid resuscitation for a child in diabetic ketoacidosis:
    1. 10 – 20 mL/kg NS over 5 – 10 minutes
    2. 10 – 20 mL/kg NS over 1 hour
    3. 20 – 40 mL/kg NS over 5 – 10 minutes
    4. 20 – 40 mL/kg NS over 1 hour
  • 14) You are called to a room to evaluate a 6 month-old-infant who was admitted to your hospital earlier in the day following an episode of supraventricular tachycardia which resolved in the emergency department. The patient has an IV in place and adenosine is available in the room. The patient’s vital signs are as follows: heart rate 240 beats per minute, respiratory rate 70/min, blood pressure 60/35 mm Hg, temperature = 98° F. On exam, he is pale and mottled. Your first step should be to:
    1. Perform a vagal maneuver
    2. Administer adenosine 0.1 mg/kg IV
    3. Perform synchronized cardioversion 0.5 – 1.0 J/kg
    4. None of the above
  • 15) For treatment of diabetic ketoacidosis, the usual starting dose of an insulin drip is:
    1. 0.01 units/kg/hour
    2. 0.1 units/kg/hour
    3. 1.0 unit/kg/hour
    4. 10.0 units/kg/hour
  • 16) Which of the following statements is true regarding the use of sodium bicarbonate for pediatric patients in diabetic ketoacidosis (DKA):
    1. Sodium bicarbonate should be given if the blood pH is less than 7.1
    2. Sodium bicarbonate should be given if the serum glucose is greater than 500
    3. Sodium bicarbonate should be given to patients in DKA with altered mental status
    4. Sodium bicarbonate should not be given
  • 17) Which of the following treatments would be the most appropriate for a patient in diabetic ketoacidosis with a serum glucose > 500.
    1. Administer an insulin bolus of 0.1 units/kg followed by an insulin drip
    2. Administer an insulin bolus of 1.0 unit/kg followed by an insulin drip
    3. Do not administer an insulin bolus, begin an insulin drip
    4. Do not administer an insulin bolus nor an insulin drip, begin subcutaneous insulin
  • 18) Which of the following statements regarding the use of Mannitol in diabetic ketoacidosis (DKA) is true:
    1. It should be routinely given to patients in DKA to prevent cerebral edema
    2. It should be given to patients in DKA who exhibit signs of cerebral edema
    3. The usual dose is 1 mg/kg IV
    4. It is contraindicated in patients in DKA
  • 19) A 6 week-old, full-term infant who weighs 6 kg requires endotracheal intubation. The most appropriate-sized uncuffed endotracheal tube for this patient would be:
    1. 3.0 mm
    2. 3.5 mm
    3. 4.0 mm
    4. 4.5 mm
  • 20) You are intubating a 6-year-old child with a 5.0 cuffed endotracheal tube. The correct insertion length for the tube is:
    1. 12 – 14 cm
    2. 15 – 17 cm
    3. 18 – 20 cm
    4. 21 – 23 cm
  • 21) You are preparing to intubate a 5 year-old-child who weighs 20 kg. The correct dose of succinylcholine for this patient would be:
    1. 10 mg IV
    2. 20 mg IV
    3. 40 mg IV
    4. 60 mg IV
  • 22) The normal respiratory rate for an infant less than 1 year of age is:
    1. 30 – 60 breaths per minute
    2. 24 – 40 breaths per minute
    3. 22 – 34 breaths per minute
    4. 18 – 30 breaths per minute
  • 23) You are evaluating a child in the emergency department. The child is awake and alert. The proper sequence for the primary assessment of this patient is:
    1. Airway, Breathing, Circulation
    2. Circulation, Airway, Breathing
    3. Breathing, Airway, Circulation
    4. Airway, Circulation, Breathing
  • 24) Which of the following blood pressures is normal for a 10 year-old-child:
    1. 78/40 mm Hg
    2. 88/50 mm Hg
    3. 108/70 mm Hg
    4. 118/80 mm Hg
  • 25) You are evaluating a 3 month-old-infant brought in for evaluation of nasal congestion and cough. On exam, the patient appears to be in no distress. Diffuse bilateral expiratory wheezing is heard. You suspect the patient has bronchiolitis. Your next step/s should be to:
    1. Perform nasal suctioning
    2. Administer an albuterol 2.5 mg nebulization
    3. Administer prednisolone (Orapred) 2 mg/kg orally
    4. All of the above
  • 26) You are evaluating a 6 month-old-infant brought to the emergency department for evaluation of wheezing. On exam, the patient appears to be in no distress. Diffuse bilateral expiratory wheezing is heard. You suspect the patient has bronchiolitis. Your next step should be to:
    1. Obtain a nasal swab for respiratory syncytial virus testing
    2. Obtain a chest x-ray
    3. Obtain a complete blood count
    4. None of the above
  • 27) You are performing bag-mask ventilation on a baby who weighs 7 kg. The correct bag size to use is:
    1. A neonatal bag
    2. An infant bag
    3. A child bag
    4. An adult bag
  • 28) You are caring for a patient in diabetic ketoacidosis. The patient is on an insulin drip and maintenance fluids of 0.45% NaCl. The patient’s serum glucose level has fallen from an initial value of 600 mg/dL to 250 mg/dL. His current pH is 7.1 and bicarbonate level is 10 mmol/L. Your next step should be to:
    1. Decrease the insulin drip rate
    2. Stop the insulin drip
    3. Administer D50 1 mL/kg IV
    4. Add D10 ½ NS to his maintenance fluids
  • 29) An 8 year-old who weighs 28 kg requires endotracheal intubation. The most appropriate-sized cuffed endotracheal tube for this patient would be:
    1. 4.5 mm
    2. 5.5 mm
    3. 6.5 mm
    4. 7.5 mm
  • 30) The most appropriate maintenance IV fluid rate for an infant weighing 7 kg is:
    1. 20 mL/hr
    2. 30 mL/hr
    3. 40 mL/hr
    4. 50 mL/hr
  • 31) A 12 year-old-child presents with severe wheezing, facial swelling and urticarial rash 10 minutes after eating peanuts. Vital signs are as follows: respiratory rate 36/min, blood pressure 80/40 mm Hg, heart rate 160/min. What is the most appropriate initial treatment for this child:
    1. Nebulized albuterol
    2. Normal saline IVF bolus
    3. Methylprednisolone IV
    4. Epinephrine IM
  • 32) A 5 year-old-child is brought to the emergency department for evaluation of fever. Vital signs are as follows: heart rate 160/min, respiratory rate 36/min, blood pressure 60/30 mm Hg, temperature 102° F. On exam, the child is difficult to arouse, capillary refill time is 5 seconds. The most appropriate initial intervention is:
    1. Administer 20 mL/kg isotonic crystalloid over 30 – 60 minutes.
    2. Administer 20 mL/kg isotonic crystalloid over 5 – 10 minutes.
    3. Obtain a complete blood count and blood culture
    4. Begin an epinephrine drip at 0.1 mcg/kg/min
  • 33) A child is brought to the emergency department for evaluation of a severe allergic reaction with wheezing. Which of the following medications would be indicated:
    1. Ranitidine
    2. Methylprednisolone
    3. Nebulized albuterol
    4. All of the above
  • 34) What is the best place to check for a pulse in a 6 month-old-infant:
    1. Carotid
    2. Brachial
    3. Radial
    4. Dorsalis pedis
  • 35) Which of the following is an appropriate site for intraosseous needle insertion:
    1. The proximal tibia just below the growth plate
    2. The distal tibia just above the medial malleolus
    3. The distal femur just above the knee
    4. All of the above
  • 36) You are performing bag-mask ventilation on an 18-month-old child. The best position for the child’s head and neck is:
    1. Hyperextension of the head and neck
    2. Flexion of the head and neck
    3. A “sniffing position” of the head and neck without hyperextension
    4. None of the above
  • 37) A 3 year-old-child presents to the emergency department actively seizing. His vital signs are as follows: respiratory rate 6/min, heart rate 120/minute, blood pressure 110/70 mm Hg, temperature 99.6° F, SpO2 = 78%. On physical exam, gurgling respirations are heard. The most appropriate initial step is to:
    1. a.Begin bag-mask ventilation
    2. b.Administer high-flow oxygen
    3. c.Position and suction the airway
    4. d.Establish an IV and administer lorazepam (Ativan)
  • 38) A 5-year-old child is in fluid-refractory cold shock. The most appropriate treatment would be to:
    1. Start dopamine at 10 mcg/kg/min
    2. Start epinephrine at 1.0 mcg/kg/min
    3. Start norepinephrine at 0.1 mcg/kg/min
    4. Start dobutamine at 10 mcg/kg/min
  • 39) Which of the following statements regarding the use of epinephrine in the treatment of pediatric septic shock is true:
    1. It can be administered via a peripheral IV
    2. It can be administered via an intraosseous line
    3. The preferred route of administration is via a central venous line
    4. All of the above
  • 40) A 12-year-old child having a seizure is given midazolam (Versed). The patient stops seizing, however, he quickly becomes apneic. The most appropriate initial step would be to:
    1. Administer flumazenil (Romazicon) 0.01 mg/kg IV
    2. Administer high-flow oxygen
    3. Perform immediate endotracheal intubation
    4. Begin bask-mask ventilation
Back to Top | Article Outline

Appendix 3. Demographic and Game Perceptions Survey

Please answer each of the following questions:

  • 1) What is your current level of training?
  • Medical Student _____ Resident ______ Staff/Attending _____
  • If you are a staff/attending physician, what is your field of practice? __________________________
  • 2) What is your age? _______
  • 3) What is your gender? Male _____ Female _____
  • 4) What is your ethnicity? _____________________
  • 5) What is your level of computer experience?
  • Very low _____ Low _____ Moderate _____ High _____ Very high _____
  • 6) How often do you play computer or video games?
  • Never _____ Less than once a month _____ Once a month _____ Once a week _____ Daily _____
  • 7) How often do you care for or observe others caring for seriously ill pediatric patients?
  • Never _____ Rarely _____ Sometimes _____ Often _____ Very often _____
  • 8) Have you ever attended a pediatric advanced life support course?
  • Yes _____ No _____
Table

Table

Keywords:

Serious game; simulation; assessment; Pediatric Advanced Life Support

© 2018 Society for Simulation in Healthcare