Secondary Logo

Journal Logo

A Validation Argument for a Simulation-Based Training Course Centered on Assessment, Recognition, and Early Management of Pediatric Sepsis

Geis, Gary, L., MD; Wheeler, Derek, S., MD; Bunger, Amy, PhD; Militello, Laura, G., MA; Taylor, Regina, G., CRC; Bauer, Jerome, P., RN; Byczkowski, Terri, L., PhD; Kerrey, Benjamin, T., MD, MS; Patterson, Mary, D., MD, MEd

Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare: February 2018 - Volume 13 - Issue 1 - p 16–26
doi: 10.1097/SIH.0000000000000271
Empirical Investigations
Free
SDC

Introduction Early recognition of sepsis remains one of the greatest challenges in medicine. Novice clinicians are often responsible for the recognition of sepsis and the initiation of urgent management. The aim of this study was to create a validity argument for the use of a simulation-based training course centered on assessment, recognition, and early management of sepsis in a laboratory-based setting.

Methods Five unique simulation scenarios were developed integrating critical sepsis cues identified through qualitative interviewing. Scenarios were piloted with groups of novice, intermediate, and expert pediatric physicians. The primary outcome was physician recognition of sepsis, measured with an adapted situation awareness global assessment tool. Secondary outcomes were physician compliance with pediatric advanced life support (PALS) guidelines and early sepsis management (ESM) recommendations, measured by two internally derived tools. Analysis compared recognition of sepsis by levels of expertise and measured association of sepsis recognition with the secondary outcomes.

Results Eighteen physicians were recruited, six per study group. Each physician completed three sepsis simulations. Sepsis was recognized in 19 (35%) of 54 simulations. The odds that experts recognized sepsis was 2.6 [95% confidence interval (CI) = 0.5–13.8] times greater than novices. Adjusted for severity, for every point increase in the PALS global performance score, the odds that sepsis was recognized increased by 11.3 (95% CI = 3.1–41.4). Similarly, the odds ratio for the PALS checklist score was 1.5 (95% CI = 0.8–2.6). Adjusted for severity and level of expertise, the odds of recognizing sepsis was associated with an increase in the ESM checklist score of 1.8 (95% CI = 0.9–3.6) and an increase in ESM global performance score of 4.1 (95% CI = 1.7–10.0).

Conclusions Although incomplete, evidence from initial testing suggests that the simulations of pediatric sepsis were sufficiently valid to justify their use in training novice pediatric physicians in the assessment, recognition, and management of pediatric sepsis.

From the Cincinnati Children's Hospital Medical Center (G.L.G., D.S.W., R.G.T., J.B., T.B., B.T.K.); University of Cincinnati Medical Center (A.B.); Applied Decision Science (L.G.M.), LLC; and Children's National Medical Center (M.D.P.), Cincinnati, OH.

Reprints: Gary L. Geis, MD, Center for Simulation and Research, Cincinnati Children's Hospital Medical Center, 3333 Burnet Ave, MLC 12000, Cincinnati, OH 45229-3039 (e-mail: gary.geis@cchmc.org).

Supported by the Agency for Healthcare Research and Quality (Grant #1R18HS020455-01).

M.P. performs occasional consulting for SimHealth Group. The rest of the authors declare no conflict of interest.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.simulationinhealthcare.com).

The early recognition of sepsis remains one of the greatest challenges in medicine today.1 The incidence of sepsis has been climbing at a rate of 1.5% annually, with the cost of care approaching US $17 billion annually in the United States.2–4 In a recent international study of critically ill children, the prevalence of severe sepsis was comparable with adults at 8.2%.5 Failure to recognize the signs and symptoms of sepsis and institute timely and appropriate care leads to higher mortality.6–10 Several studies in children with sepsis suggest that early recognition and treatment with antibiotics and fluid resuscitation improve outcomes.11–16

Sepsis is primarily diagnosed clinically based on a constellation of signs and symptoms. In early stages, when treatment is most effective, sepsis is virtually indistinguishable from more benign febrile illnesses. Early recognition of sepsis requires a heightened awareness, increased index of suspicion, and traditionally a substantial amount of clinical experience.17–19 In the academic ward setting, such as our institution, the novice clinician is typically responsible for the recognition of often subtle signs of sepsis and the initiation of urgent management. Failure to recognize clinical deterioration results in unacceptable delays, culminating in the transfer of a moribund patient to the intensive care unit.20,21

Simulation-based training (SBT) seems to help prevent participants from making clinical mistakes by providing a safe setting in which it is permissible to make mistakes and learn from them.22 Simulation, when used for sepsis education, has been shown to improve sepsis- and shock-related knowledge and improved septic shock diagnosis and management in the laboratory setting.22–28 In addition, in a hospital using in situ simulation, a checklist showed increased compliance with first-hour sepsis-related care tasks in comparison with two hospitals where simulation was not used (61.7% vs. 23.1%, P < 0.01).29

In the first phase of a planned three-phase investigation, we used the critical decision method (CDM) interviewing technique and qualitative analysis to identify and classify behaviors that characterized and differentiated the expert (faculty physicians) from the novice (intern physicians) in the recognition of sepsis at the bedside.30 The CDM technique uses cognitive probes in semistructured interviews to elicit information about how experts formulate their decision-making strategies. The objective of the current study (second phase) was to create a validity argument for the use of an SBT course centered on assessment, recognition, and early management of sepsis in a laboratory-based setting. The planned third stage will be to explore and assess the training course in a larger cohort of novice physicians.

Back to Top | Article Outline

METHODS

This study was reviewed and approved by our institutional review board. To create a validity argument for simulation as a strategy, we needed to create a series of processes (scenario development and testing, outcome measure development and implementation, etc.) to make judgments. To make these judgments, we needed to understand the strengths and limitations of the processes upon which decisions were based. Stated differently, we required evidence to support the validity of our decisions.31 We chose to use a validation theory, articulated by Kane, which prioritizes evidence by highlighting key phases or inferences in planning. As described by Cook et al,31 Kane's validity argument guides the collection and interpretation of validity evidence. First, an educator must consider the decision at hand and a proposed interpretation that would support that decision. Next, with the desired decision in mind, the educator identifies the key claims, assumptions, and inferences associated with this interpretation and use; Kane labels this the “interpretation/use argument.” The educators then develop a plan to test these assumptions and inferences. Finally, guided by this plan, they collect empiric evidence from multiple sources and evaluate this evidence into a validity argument.

Using data from our initial study,30 we developed five pediatric sepsis scenarios incorporating perceptual cues used by expert clinicians. We designed scenarios to be accessible to novices, but also challenging and requiring the detection and interpretation of subtle sepsis cues and clusters of cues. We developed one “garden-path” simulation, two scenarios of compensated sepsis, and two scenarios of uncompensated septic shock. Garden-path scenarios are ones in which the initial presentation suggests a straightforward case. As it unfolds, additional information is presented, which is not consistent with the initial diagnosis. Generally, experts are able to recognize the incongruity and adapt to new information, in the process revising their mental models and plans. Novice clinicians are more constrained in their initial hypothesis and do not recognize or reconcile the atypical elements. A sixth simulation, a supraventricular tachycardia (SVT) scenario unrelated to sepsis, was developed to limit the bias of participant anticipation of sepsis. The six scenarios (see Figure, Supplemental Digital Content 1, which shows sepsis scenario content and flow; http://links.lww.com/SIH/A347) were reviewed by local experts in emergency medicine and critical care medicine and revised based on their feedback.

The simulation facility is a 400-ft2, on-campus simulation laboratory, including two simulation rooms (1 emergency department [ED] setting, 1 inpatient ward setting), a control room separated from the simulation rooms by one-way mirrors, and a debriefing room. Simulation rooms were set up to replicate the intended clinical setting. For example, the ED included an ED stretcher, bedside monitor, IV pole with intravenous (IV) infusion pump, bedside table, and head wall with air, oxygen, and suction capabilities. Noninvasive airway and oxygen equipment were available on a separate wall, arranged as they would be in the ED. Additional resuscitation equipment (eg, IV access supplies, laryngoscopes, and defibrillator) was available in the hall outside the room, as in the ED setting that equipment is not standard for a nontrauma/resuscitation room. As another example, when IV access was achieved and fluids or medications were ordered, those were obtained from outside the room and delivered in real time into a fluid collection system (IV catheter, IV tubing, and Foley catheter bag). Participants could not just verbalize “I have given 20 mL/kg of normal saline” but instead had to instruct the bedside nurse or themselves to deliver the fluid through the IV catheter before any “response” was achieved by the simulated patient.

The simulation laboratory represents an environment in which we are able to place residents in a role in which they need to seek and interpret available cues, rather than rely on a senior, more experienced physician. We included critical cues and cue clusters frequently mentioned in interviews, during our initial study, as important indicators of sepsis.30 It is common in SBT to provide participants with findings from radiographs, electrocardiograms, or distal perfusion by simply relaying the results. In contrast, we presented the learner with raw data to facilitate the development of perceptual skills and pattern recognition in the context of a challenging incident, eg, we presented the physicians with the radiographic image so that they could interpret it firsthand. Classic indicators of sepsis were relatively easy to incorporate by a single value or a trend over time, and the monitors used to measure and display these cues are generally part of simulations. However, some cues required innovative strategies, such as wrapping the hands and feet of the mannequin in ice before training to simulate cold extremities. To improve the fidelity of mental status cues, we used a voice modulator that allowed us to speak through the mannequin in the voice of an infant, a toddler, or school-aged child, depending on the scenario. Videos and screen shots were used to depict changes in skin appearance and distal perfusion, requiring the physicians to interpret the image and independently draw their own conclusions. Another requirement was authentic timing and urgency, because several CDM incidents included situations in which a patient's condition started at an early point in sepsis and slowly deteriorated, making it easier to miss subtle cues. Thus, we designed compensated sepsis scenarios with timelines more representative of early sepsis.

To provide consistency and limit the number of actors needed per session, simulation center staff executed scripted roles and provided relevant cues to participants. The scripting of each role, which included verbatim comments identified in the initial phase of this project, was built into the scenarios to increase authenticity. At minimum, a bedside nurse and a charge nurse were made available to the participant. Participants could consult a respiratory therapist for specific tasks (eg, oxygenation, blood gas sampling), a subspecialist (eg, orthopedic resident, nephrology fellow) if applicable to the scenario, or a parent. Simulation center staff practiced these roles before course implementation and were trained to only provide supporting information/results, when asked, or perform tasks when instructed, by the participant. Staff carried laminated cards (their script) with them to ensure consistency across scenarios. In addition, staff were not allowed to provide clinical insight or make suggestions to the participant.

Each simulation was designed to take 15 minutes to complete. The Situation Awareness Global Assessment Technique (SAGAT) (Appendix 1) was applied by a trained facilitator three times during the 15-minute simulation (at 5–8, 8–11, and 11–14 minutes). The participant was asked to step out of the simulation room, where standardized SAGAT questions were asked.

After completion of each scenario, a two-member facilitation team, aware of the participant experience level, debriefed the participant in a separate room. The lead facilitator was a board-certified pediatric emergency medicine or pediatric critical care faculty physician with at least 10 years of clinical and simulation-based experience. The second facilitator was a nurse or research coordinator, present to ensure that a standardized debriefing took place. All facilitators were trained in the CDM interviewing technique.30 Each debriefing session lasted 20 to 30 minutes. The debriefing format was learner focused, through open-ended questions requiring self-reflection on performance and centered on the participant's assessment of illness severity, identification of potential causes, and early stabilization of critical illness. Table 1 was available as a wall chart in the debriefing room and was used by the facilitator to focus the initial discussion and prompt the participant to classify the rapid cardiopulmonary assessment. The second portion of the debriefing entailed asking the participants to describe and "weigh" the information they used in the scenario. This line of questioning paralleled the SAGAT and forced self-reflection on the assessment data (eg, vital signs, examination findings, laboratory results) and how the participants used the data to prioritize management strategy and interventions. The third portion of the debriefing allowed the participant to ask clinical questions to the lead facilitator regarding assessment and management skills. During the third part of the debriefing, facilitators intentionally did not use the words sepsis, septic, or septic shock and avoided giving the participants “the answer” to the scenario. Upon completion of their fourth scenario, each participant was made aware that three scenarios were sepsis and one was SVT, and a 20-minute sepsis-specific didactic talk was given by the lead facilitator to augment the SBT. Finally, three recent publications on pediatric sepsis were sent by e-mail to each participant.

TABLE 1

TABLE 1

Back to Top | Article Outline

Study Population and Scenario Assignment

A cohort of pediatric residents and faculty (unique from the CDM interviews) were recruited as a convenience sample for the simulations. Novices were defined as first-year residents (interns), intermediate-level physicians were defined as third-year residents, and experts were defined as faculty physicians more than 5 years from fellowship training in critical care or emergency medicine. All physicians participated in the following four scenarios: SVT, the sepsis “garden path,” and two of the other four sepsis scenarios. There were six possible combinations of the two nongarden-path sepsis scenarios; each participant was assigned a two-scenario combination using a randomized block design (Table 2).

TABLE 2

TABLE 2

Back to Top | Article Outline

Outcomes of Interest

The primary outcome was recognition of sepsis by the physician at the end of the scenario. Recognition of sepsis was determined by the response to the question “What is most likely the cause of the patient's illness” on the final (third) SAGAT. All SAGAT interviews were audio recorded and transcribed by a medical transcriptionist. Transcripts were coded by two of four investigators. Three coders were qualitative researchers (A.B., L.M., and R.T.) and one was an ED physician (M.P.). The interviews were randomly assigned to each pair of coders. Using the same themes and categories developed for the SAGAT, the coders independently coded each interview. After two interviews were coded, the coders met in person or by phone to discuss each item and come to consensus. This process resulted in a set of master codes (eg, vital signs or examination findings being used, diagnostic possibilities). Discussions and consensus occurred after every two interviews to prevent coding drift and allow further revision of the coding sheet, if needed. This iterative process continued until each transcribed SAGAT was coded and consensus had been reached. The frequency of each cue type was tabulated across all SAGATs and summarized by participant class (novice, intermediate trainee, and expert). Frequency was calculated based on a single mention of cue rather than the number of times a specific cue was mentioned during the SAGAT.

Secondary outcomes were physician performance in the assessment and early management of simulated patients. Physician performance was assessed by internally derived tools adapted from pediatric advanced life support (PALS) recommendations and early sepsis management (ESM) literature. Each tool contained two parts: (1) a checklist of behaviors to be assessed as performed or not, with 15 behaviors for assessment and 6 behaviors for sepsis management, and (2) a 1 to 7 (poor performance to ideal performance) Likert scale to rate the participant's global (overall) performance in the domains of assessment and sepsis management. The PALS and ESM tools (Appendix 2) were applied during video review of the simulations.

Back to Top | Article Outline

Measures of Outcome

The SAGAT is a method that employs a “freeze technique” to assess levels of situation awareness. A simulation of the system of interest is frozen at random points in time, and the participant(s) quickly responds to questions concerning their perceptions of the situation at that moment. Participants are turned away from the simulation while answering questions on elements related to all three levels of situation awareness.32 This technique reduces recall bias because the participants are queried immediately regarding their detection of simulation elements rather than questioned after the incident. The nature of the inquiries is factual, allowing for objective scoring. Although potentially disruptive, studies have demonstrated that freezes as long as 2 minutes, the threat of freezes, and the number of freezes did not affect performance for pilots and air traffic controllers.33–36 The SAGAT queries are randomly administered and require discrete, well-defined responses.32 Because we were interested in whether participants were noticing appropriate cues, correctly interpreting those cues, and anticipating the participant's future state, our adapted SAGAT used the same four queries at each pause (Appendix 1).

Checklists of correct performance behaviors were developed by a pediatric emergency medicine (G.L.G.) and a pediatric critical care physician (D.S.W.). The checklists were based on PALS recommendations and ESM literature and were reviewed by the entire investigative team. Checklists were constructed to reflect observable behaviors only and were piloted to ensure reliable data abstraction from video recordings of simulations.

Back to Top | Article Outline

Video-Reviewer Training and Scoring

Four members of the simulation center, including one investigator (G.L.G.), were trained in PALS and ESM checklist application. All four reviewers reviewed a pilot simulation and independently applied the checklist; discrepancies were discussed as a group and consensus achieved. This method was repeated for separate videos until all items were scored identically by reviewers. Participant videos were randomized and assigned to two reviewers per video.

When the two reviewers disagreed for checklist items, reviewers met to discuss the scores and achieve consensus. The consensus score was used in analysis. When an item was “not applicable” (eg, patient in compensated sepsis who was never hypotensive for age), it was counted as a “yes” for analysis. When the two reviewers disagreed on a global assessment score, a mean of the two values was entered into the database.

Back to Top | Article Outline

Statistical Analysis

Descriptive statistics were generated for PALS and ESM checklist and global performance scores. Frequency distributions were generated for recognition of sepsis. Additional analyses were conducted using a mixed models approach due to correlated data from two sources. First, all models accounted for repeated measures within participant. Second, each participant was randomized to two of the four scenarios while all were exposed to the fifth scenario. Thus, scenario was included as a random effect. Odds ratios were calculated to compare recognition of sepsis on the first, second, and third SAGAT among levels of expertise. The dependent variable was whether or not sepsis was recognized. The independent variable of interest was the categorical level of experience. Severity (ie, compensated or uncompensated) was included as a covariate because participants (the unit of analysis) were exposed to only three of the five possible scenarios. Models were developed to measure the association between each of the ESM and PALS checklist and global assessment performance scores and level of experience. The dependent variable was the score and the independent variable was the level of experience. Similarly models were developed to measure the association with severity (compensated and uncompensated), but adjusting for level of experience. Finally, four models were developed to determine whether the recognition of sepsis was associated with increased ESM and compliance with PALS. The dependent variable was whether or not sepsis was recognized. The independent variables of interest were the ESM and PALS checklist and global assessment performance. The PALS score models adjusted for severity, whereas the ESM models adjusted for severity and level of experience. SAS version 9.3, including PROC MIXED and PROC GLIMMIX, was used to conduct these analyses.

As internally derived tools, we assessed interrater reliability for the video review. Because two of a possible four reviewers determined the presence or absence of each component, intraclass correlation, assuming a one-way random model, was calculated for each component of the PALS and ESM checklists, as well as the percent agreement. Reviewer agreement on the ordinal PALS and ESM global performance scores was measured using the nonparametric Spearman rank order correlation. IBM SPSS Statistics 24 was used to conduct these analyses.

Back to Top | Article Outline

RESULTS

Eighteen providers were enrolled (6 faculty, 6 third-year residents, and 6 interns). Each participant completed four simulations and data collection was complete in all cases, resulting in 54 sepsis simulations (18 participants for three sepsis simulations/participant) available for analysis. Physicians recognized sepsis in 19 (35%) of 54 sessions. Interns (novices) recognized sepsis in 4 (22%) of 18 scenarios, senior residents (intermediate-level) in 8 (44%), and faculty (expert) physicians in 7 (39%). Table 3 shows recognition of sepsis (at third SAGAT) by experience and scenario. Adjusted for scenario severity, the odds that faculty recognized sepsis at third SAGAT was 2.6 [95% confidence interval (CI) = 0.5–13.8] times that of interns (Table 4). Compared with compensated sepsis simulations (lesser severity of illness), a higher percentage of physicians recognized sepsis in uncompensated septic shock simulations (Table 5). Physicians recognized sepsis in 8 (22%) of 36 compensated simulations compared with 11 (61%) of 18 uncompensated shock simulations (P = 0.005).

TABLE 3

TABLE 3

TABLE 4

TABLE 4

TABLE 5

TABLE 5

Video-based review of the simulations resulted in PALS-based assessment and ESM checklist scores for each of the 54 sepsis simulations. Intraclass correlation and percent agreement were low for initiation of pressor support, correction of electrolyte disturbance, and recognition of hypotension for age. The following exhibited a high (>90%) agreement, but low intraclass correlation: serum glucose obtained, mental status, oxygen saturation, and airway patency assessment (see Table, Supplemental Digital Content 2, which shows reliability of video-based application of PALS and ESM checklist tools; http://links.lww.com/SIH/A348). For the PALS global performance scoring, the correlation coefficient was moderate at 0.534; for the ESM global performance score, the correlation coefficient was strong at 0.622.37

Table 6 shows the unadjusted mean assessment and management scores by experience, scenario, and severity. Overall, the 18 participants achieved a mean (SD) of 12.89 (1.30) for assessment on the 15-item PALS checklist and a mean (SD) of 4.07 (1.16) for management on the six-item ESM checklist. Global performance score results demonstrated means (SD) of 5.23 (1.11) for assessment and 4.76 (1.31) for management. Accounting for correlated measures within subject and scenario and using a P value of less than 0.05, faculty (experts) scored statistically significantly higher than interns (novices) on the PALS global performance score (5.9 vs. 4.3), ESM global performance score (5.5 vs. 4.2), and ESM checklist scores (4.5 vs. 3.7), but not on the PALS assessment checklist. Faculty scored higher than residents on the ESM global performance score (5.5 vs. 4.6). Residents scored higher than interns on the PALS global performance score (5.4 vs. 4.3). There were no significant differences in scores by severity (compensated vs. uncompensated) adjusted for experience.

TABLE 6

TABLE 6

Adjusted for severity, for every point increase in the PALS global performance score, the odds that sepsis was recognized increased by 11.3 (95% CI = 3.1–41.4). Similarly, the odds ratio for the PALS checklist score was 1.5 (95% CI = 0.8–2.6). Adjusted for severity and level of expertise, the odds of recognizing sepsis was associated with an increase in the ESM checklist score of 1.8 (95% CI = 0.9–3.6) and an increase in ESM global performance score of 4.1 (95% CI = 1.7–10.0).

The qualitative analysis of the SAGAT revealed that sepsis was the most commonly mentioned diagnosis for “what's most likely the cause?” across all scenarios and all levels of expertise. Thirty-one different diagnoses were offered, as well as “I don't know.” Frequently mentioned diagnoses included pneumonia, infection, hypovolemia, dehydration, and shock. For “what else are you considering for causes?” 65 different diagnoses were offered, with sepsis, cardiac issue, dehydration, hypovolemia, infection, ingestions/toxins, and pneumonia being the most common. When examining the cues used, there was a high level of agreement across providers and all seemed to focus on the classic indicators of sepsis (oxygenation, heart rate, respiratory status, and blood pressure), as well as mental status and distal extremity perfusion.

Back to Top | Article Outline

DISCUSSION

Multiple examples of effective use of cognitive task analysis, and specifically CDM, to identify novice and expert differences in a variety of domains exist. In healthcare, this method has been used successfully with acute care nurses, neonatal nurses, and residents rotating in an intensive care unit.17,38–40 Despite recognition of differences in their approach to complex clinical problems, there has been little research on cues and cue use strategies for experts and novices in healthcare. There has also not been any organized attempt to exploit these differences to accelerate the development of clinical expertise.41

In our first phase, we identified cues that physicians use to assess and make decisions in the context of real-world incidents involving sepsis.30 The resulting critical cue inventory, detailed examples, and contextual elements provided the foundation for creating SBT scenarios in this second phase aimed at increasing the novice clinician's exposure to sepsis, developed to portray variety in timing and presentation of sepsis, and focused on the critical strategies practiced by experts. Ideally, training scenarios would allow hypothesis testing and force time-dependent decision making and management to accelerate recognition and early management of sepsis.

Thus, we needed to determine whether the combination of our SBT scenarios and outcome measures (SAGAT, PALS assessment tool, and ESM performance tool) created a valid construct for improving our intended population's performance (on sepsis recognition and early management), within our institution, during their first year of pediatric residency training. Creating this validity argument, as framed by Kane, guides the collection and interpretation of validity evidence.31 As described by Cook et al,31 the initial step is to consider the decision at hand and a proposed interpretation that would support that decision. For our study, the decision at hand was whether pediatric physicians benefit from SBT in their recognition of sepsis and early management directed at sepsis. More specifically, does their performance over a series of simulations demonstrate their ability to make correct clinical decisions regarding sepsis, ie, do they “pass” or do they need more training? Our interpretation of those results, or as Kane would state as our (instructors') decision about the learners' abilities based on application of the outcome measures, was that we categorized “correct decisions” as recognition of sepsis by the third SAGAT and achievement of high scores on the ESM checklist and global performance scale. Next, one must identify key claims, assumptions, and inferences associated with this interpretation and use.31 Our hypothesis or interpretation/use argument was development of a five-scenario SBT course using cues embedded from CDM interviews will allow differentiation between pediatric interns (novices) and faculty (experts) in the recognition and early management of sepsis and thus form the basis for future training of novices. Then, one must develop a plan to test these assumptions/inferences.31 This would be our methodology described previously, including convenience sampling of different experience cohorts (interns, third years, and faculty), event-based scenario design augmented by critical cues from CDM interviews, novel strategies to relay these critical cues to participants and scripting of confederate roles, outcome measure development (SAGAT, PALS, ESM) leveraging local expertise and published literature, and standardized debriefing sessions. Finally, one must collect empiric evidence from multiple sources and organize this evidence into a validity argument.31 To test these assumptions, Kane's validity framework “traces an assessment from the scoring of a single observation (scoring), to using the observation score(s) to generate an overall test score representing performance in the test setting (generalization), to drawing an inference regarding what the test score might imply for real-life performance (extrapolation), and then to interpreting this information and making a decision (implications)” [p 563].31

In this second phase of study, we have collected empiric evidence at the scoring and generalization levels. For 54 sessions where single observations were scored dichotomously at the third SAGAT, physicians recognized sepsis in 35%. One would hope that this frequency would be higher, especially among experts. However, this overall test score in the simulation laboratory (generalization) likely represents actual, clinical difficulties in recognizing sepsis in the pediatric population. In our limited cohort, experts were able to identify sepsis more readily than novices, although the difference was not statistically significant and we would feel more secure in this statement if we could have increased sampling. In addition, although comparison of expert-novice groups weakens our argument (see further discussion hereinafter), the differences in performance provide limited evidence toward use of these scenarios in future training.

In addition to sepsis recognition, our other foci were assessment of a critically ill child and application of ESM principles. Single observations of PALS assessment and ESM performance (scoring) demonstrated moderate to high compliance with published recommendations, with mean assessment checklist scores of 12.89 of 15 and mean management checklist scores of 4.07 of 6. Generalizing performance to the test (simulation laboratory) environment, there were significant differences between experts and novices in three of four performance measures, as experts scored higher than novices within management (checklist and global performance) measures but not on the assessment checklist. An explanation may be the interns' participation in PALS at the beginning of their training year where assessment strategies and performance are key foci. In addition, an intern may have greater comfort with assessment compared with higher level management decisions. A major focus of the debriefing sessions was recognition and weighting of information used to make decisions and discussing how and why that information could be used for rapid cardiopulmonary assessment, independent of diagnosis. This may have promoted interns becoming more efficient in their assessment skills. Interestingly, all physicians performed worse in ESM during less severely ill (compensated sepsis) scenarios. An explanation might be that in the undifferentiated ill child, sepsis is only one of many potential diagnoses and physicians are less likely to commit to ESM (aggressive fluid resuscitation and early antibiotics). Importantly, we showed that when PALS assessment fundamentals were used, physicians were more likely to identify sepsis. When this occurred and sepsis was identified, physicians were more likely to adhere to early management recommendations.

However, we are limited at this point of study by our inability to infer implications on novice assessment for, recognition of, and management of pediatric sepsis at the bedside (extrapolation). In addition, the use of expert novice, or known group, weakens our validity argument because association does not imply causation.31 To that end, the simulation scenarios were implemented as a four-session course (third phase) in the following academic year for the cohort of pediatric interns and clinical outcomes for patients were tracked, including need for rapid transfer of septic patients to the intensive care units, need for invasive procedures, volume resuscitation, and vasopressor support. Although the analysis of the training year's (third phase) outcomes is pending, the feedback received from the intern class was overwhelmingly positive. Residency leadership made this course mandatory for all residents, and currently, we offer a weekly two-scenario session to residents using the scenarios and debriefing format developed.

Back to Top | Article Outline

Limitations

The first limitation was the small number of participants and sessions per participant. This likely prohibited us from demonstrating a significant difference between novices and experts in the recognition of sepsis. As noted previously, increased sampling would increase our ability estimate and improve our validity argument. Second, the video reviewers were not blinded to participant experience. This may have influenced their scoring, especially when behaviors, such as mental status, were difficult to assess by video. A computerized manikin cannot be stimulated verbally and physically. We attempted to minimize this limitation by using a voice modulator to allow a verbal response from the “patient” and use the eye opening function of some of the simulators. Third, nonblinding may have influenced the reviewers assignment of global scores, eg, a reviewer could have weighted assessment and management decisions made by “an expert” to be better than ones made by a novice. Fourth, physicians were scheduled into sessions by convenience of their schedules, introducing the potential bias of self-reflection between sessions. Participants may have specifically questioned whether all the scenarios involved sepsis, leading to improved performance in subsequent simulations. We included the SVT scenario for each participant to limit this source of bias. During the debriefing sessions, we intentionally did not use the words sepsis, septic, or septic shock. Instead, we debriefed around assessment and early stabilization of critical illness using the PALS-based debriefing table (Table 1). Finally, the use of internally derived, novel PALS and ESM tools is a limitation. Without previous implementation in a validity argument for SBT in a similar cohort, their use here limits our findings and argument. As noted previously, we do intend to use them in the next phase on a larger cohort of novices. In addition, interrater reliability was not high for all components of the checklists, which needs to be addressed in future work before application of these tools.

This project represents one of the first efforts in healthcare to explicitly embed differences in novice and expert clinicians in cue recognition and use into SBT. Although this approach is initially focused on the recognition of sepsis, it is likely transferable to other critical, time-dependent medical conditions, eg, myocardial infarction. As currently constructed, this training course provides an opportunity for instructors to collect (scoring inference) and interpret evidence surrounding learner assessment, recognition, and management of sepsis in a simulated, or test, setting (generalization inference). To fully accept or reject its argument for use in our population, future study must infer what test performance implies for real-life performance (extrapolation inference) at the bedside in critically ill children and then decide whether this course should be made available, made mandatory or not implemented (implications inference) based on multiple factors, including effectiveness, usefulness, cost, and effort. As Kane notes, “a decision procedure that does not achieve its goals, or does so at too high a cost, is likely to be abandoned even if it is based on perfectly accurate information.”42

Back to Top | Article Outline

CONCLUSIONS

Although incomplete, evidence from initial testing suggests that the simulations of pediatric sepsis were sufficiently valid to justify their use in training novice pediatric physicians in the assessment, recognition, and management of pediatric sepsis. In simulated cases, physicians who adhere closely to PALS guidelines are more likely to recognize sepsis, and recognition is associated with increased compliance with ESM principles.

Back to Top | Article Outline

REFERENCES

1. Wheeler DS, Zingarelli B, Wheeler WJ, et al. Novel pharmacologic approaches to the management of sepsis: targeting the host inflammatory response. Recent Pat Inflamm Allergy Drug Discov 2009;3:96–112.
2. Angus DC, Linde-Zwirble WT, Lidicker J, et al. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med 2001;29:1303–1310.
3. Martin GS, Mannino DM, Eaton S, et al. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med 2003;348:1546–1554.
4. Dombrovskiy VY, Martin AA, Sunderram J, et al. Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003. Crit Care Med 2007;35:1244–1250.
5. Weiss SL, Fitzgerald JC, Pappachan J, et al. Prevalence, Outcomes, and Therapies (SPROUT) Study Investigators and Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network. Global epidemiology of pediatric severe sepsis: the sepsis prevalence, outcomes, and therapies study. Am J Respir Crit Care Med 2015;191:1147–1157.
6. Barrow RE, Jeschke MG, Herndon DN. Early fluid resuscitation improves outcomes in severely burned children. Resuscitation 2000;45:91–96.
7. Booy R, Habibi P, Nadel S, et al. Meningococcal Research Group. Reduction in case fatality rate from meningococcal disease associated with improved healthcare delivery. Arch Dis Child 2001;85:386–390.
8. Kumar A, Roberts D, Wood KE, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit Care Med 2006;34:1589–1596.
9. Pollard AJ, Nadel S, Ninis N, et al. Emergency management of meningococcal disease: eight years on. Arch Dis Child 2007;92:283–286.
10. Odetola FO, Rosenberg AL, Davis MM, et al. Do outcomes vary according to the source of admission to the pediatric intensive care unit? Pediatr Crit Care Med 2008;9:20–25.
11. Han YY, Carcillo JA, Dragotta MA, et al. Early reversal of pediatric-neonatal septic shock by community physicians is associated with improved outcome. Pediatrics 2003;112:793–799.
12. de Oliveira CF, de Oliveira DS, Gottschald AF, et al. ACCM/PALS haemodynamic support guidelines for paediatric septic shock: an outcomes comparison with and without monitoring central venous oxygen saturation. Intensive Care Med 2008;34:1065–1075.
13. Carcillo JA, Kuch BA, Han YY, et al. Mortality and functional morbidity after use of PALS/APLS by community physicians. Pediatrics 2009;124:500–508.
14. Larsen GY, Mecham N, Greenberg R. An emergency department septic shock protocol and care guideline for children initiated at triage. Pediatrics 2011;127:e1585–e1592.
15. Lane RD, Funai T, Reeder R, et al. High reliability pediatric septic shock quality improvement initiative and decreasing mortality. Pediatrics 2016;138:e20154153.
16. Balamuth F, Weiss SL, Fitzgerald JC, et al. Protocolized treatment is associated with decreased organ dysfunction in pediatric severe sepsis. Pediatr Crit Care Med 2016;17:817–822.
17. Crandall B, Getchell-Reiter K. Critical decision method: a technique for eliciting concrete assessment indicators from the intuition of NICU nurses. ANS Adv Nurs Sci 1993;16:42–51.
18. Reischman RR. Critical care cardiovascular nurse expert and novice diagnostic cue utilization. J Adv Nurs 2002;39:24–34.
19. Hoffman KA, Aitken LM, Duffield C. A comparison of novice and expert nurses' cue collection during clinical decision-making: verbal protocol analysis. Int J Nurs Stud 2009;46:1335–1344.
20. Brilli RJ, Gibson R, Luria JW, et al. Implementation of a medical emergency team in a large pediatric teaching hospital prevents respiratory and cardiopulmonary arrests outside the intensive care unit. Pediatr Crit Care Med 2007;8:236–246.
21. Funk D, Sebat F, Kumar A. A systems approach to the early recognition and rapid administration of best practice therapy in sepsis and septic shock. Curr Opin Crit Care 2009;15:301–307.
22. Lewis R, Strachan A, Smith MM. Is high fidelity simulation the most effective method for the development of non-technical skills in nursing? A review of the current evidence. Open Nurs J 2012;6:82–89.
23. Nguyen HB, Daniel-Underwood L, Van Ginkel C, et al. An educational course including medical simulation for early goal-directed therapy and the severe sepsis resuscitation bundle: an evaluation for medical student training. Resuscitation 2009;80:674–679.
24. Hansel M, Winkelmann AM, Hardt F, et al. Impact of simulator training and crew resource management training on final-year medical students' performance in sepsis resuscitation: a randomized trial. Minerva Anestesiol 2012;78:901–909.
25. Nishisaki A, Hales R, Biagas K, et al. A multi-institutional high-fidelity simulation “boot camp” orientation and training program for first year pediatric critical care fellows. Pediatr Crit Care Med 2009;10:157–162.
26. Littlewood KE, Shilling AM, Stemland CJ, et al. High-fidelity simulation is superior to case-based discussion in teaching the management of shock. Med Teach 2013;35:e1003–e1010.
27. Evans KH, Daines W, Tsui J, et al. Septris: a novel, mobile, online, simulation game that improves sepsis recognition and management. Acad Med 2015;90:180–184.
28. Dugan MC, McCracken CE, Hebbar KB. Does simulation improve recognition and management of pediatric septic shock, and if one simulation is good, is more simulation better? Pediatr Crit Care Med 2016;17:605–614.
29. Qian J, Wang Y, Zhang Y, et al. A survey of the first-hour basic care tasks of severe sepsis and septic shock in pediatric patients and an evaluation of medical simulation on improving the compliance of the tasks. J Emerg Med 2016;50:239–245.
30. Patterson MD, Militello LG, Bunger A, et al. Leveraging the critical decision method to develop simulation-based training for early recognition of sepsis. J Cognit Eng Decis Mak 2016;10:36–56.
31. Cook DA, Brydges R, Ginsburg S, et al. A contemporary approach to validity arguments: a practical guide to Kane's framework. Med Educ 2015;49:560–575.
32. Endsley MR. Direct measurement of situation awareness: validity and use of SAGAT. In: Endsley MR, Garland DJ, eds. Situation Awareness Analysis and Measurement. Mahwah, NJ: Lawrence Erlbuam Associates; 2000:1–21.
33. Bolstad CA, Endsley MR. Single Versus Dual Scale Range Display Investigation (NOR DOC 90–90). Hawthorne, CA: Northrop Corporation; 1990.
34. Endsley M. Measurement of situation awareness in dynamic systems. Hum Factors 1995;37:65–84.
35. Endsley MR, Garland DJ. Situation Awareness Analysis and Measurement. New York, NY: CRC Press; 2000.
36. Wright MC, Taekman JM, Endsley MR. Objective measures of situation awareness in a simulated medical environment. Qual Saf Health Care 2004;13(Suppl 1):i65–i71.
37. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–174.
38. Gazarian PK, Henneman EA, Chandler GE. Nurse decision making in the prearrest period. Clin Nurs Res 2010;19:21–37.
39. Fackler JC, Watts C, Grome A, et al. Critical care physician cognitive task analysis: an exploratory study. Crit Care 2009;13:R33.
40. Militello L, Lim L. Patient assessment skills: assessing early cues of necrotizing enterocolitis. J Perinat Neonatal Nurs 1995;9:42–52.
41. Crandall B, Wears RL. Expanding perspectives on misdiagnosis. Am J Med 2008;121:S30–S33.
42. Kane MT. Validation. In: Brennan RL, eds. Educational Measurement. 4th ed. Westport, CT: Praeger; 2006:17–64.
Back to Top | Article Outline

APPENDIX 1. SAGAT

Table

Table

Back to Top | Article Outline

APPENDIX 2. Tools for PALS-Based Assessment and Early Sepsis Management for Pediatric Septic Shock

Table

Table

Table

Table

Keywords:

Expertise; novice; pediatric; resident; sepsis; simulation

Supplemental Digital Content

Back to Top | Article Outline
© 2018 Society for Simulation in Healthcare