The DASH aims to assess those instructor behaviors that evidence and theory indicate facilitate learning and change in experimental contexts. Typically, BARS content is elicited from domain experts.38–40 In the absence of a theoretical or empirical consensus regarding the optimal behaviors for health care simulation debriefings, the DASH was constructed on the premise that research findings and theory from related domains logically transfer to debriefing and could be used to augment BARS content from the traditional approach. Specifically, the DASH synthesizes findings from aviation debriefing; clinical learning and teaching; formative assessment; adult, experiential, and organizational learning; deliberate practice; and the cognitive, emotional, and behavioral bases for mobilizing change in adults.26,31,41–47 For the conventional BARS development approach of eliciting task-related behaviors and categories from domain experts, the developers drew on their domain expertise. Collectively, they have conducted more than 5000 debriefings, and through their simulation instructor training activities, they have observed and provided feedback on more than 2500 debriefings by instructors with a broad range of debriefing styles and skill levels from Asia, Oceania, North America, Europe, and Central and South America.
Using the content outlined previously, the DASH elements were identified and refined in an iterative process known as theory elaboration48–50 in which the test developers worked back and forth between high-level constructs suggested by the literature, their own experience, and semistructured interviews with established debriefing instructor trainers from other simulation centers in North America, Europe, and Australia. The DASH developers thereby identified a set of activities generally accepted as best practices for effective and ineffective debriefing from a broad range of fields and debriefing styles, pertinent to the guiding DASH design principle that it should be applicable to the assessment of a wide variety of universal debriefing behaviors and not linked to any particular debriefing style. Thus, the DASH is designed to assess debriefing quality in a variety of simulation environments, across health care disciplines and educational objectives.
Certain points of the rater training are worth mentioning. The 6 elements were constructed to make them as distinct from one another as possible. Given the nature of describing behavior, it is understandable that raters perceive that the elements have some overlap. They are instructed to ignore this perceived overlap and to rate each element independently of the others. Raters are taught to give element scores, but the scores are not an average of the dimensions. No explicit weighting is given to the dimensions; raters are taught to make those judgments themselves. The dimensions and behavioral examples are intended to provide guidelines and examples but are not intended as checklist items; they are integrated into a global rating at the element level by the rater. This approach is consistent with research supporting the use of global rather than checklist ratings for the evaluation of complex behaviors.51–53
Piloting and User Review
After a working version of the DASH was constructed using psychometric and instructional design methods,54,55 it was reviewed for content and usability. Eight simulation experts from 5 different pediatric tertiary care academic medical centers in the United States and Canada participated in a 2-day in-person intensive review session in October 2008. All 8 experts were from centers participating in the Examining Pediatric Resuscitation Education using Simulation and Scripting (EXPRESS) project,5 a multicenter study examining the impact of scripted debriefing and level of simulation realism on Pediatric Advanced Life Support educational outcomes after simulation sessions. All are practicing physicians in pediatric emergency medicine, critical care, or anesthesia with a minimum of 5 years of experience in simulation and debriefing. For the first round of feedback and revision, each investigator studied the draft rater’s handbook and then discussed each element, posing clarifying questions and suggesting edits to make the language clearer. In the next round, 2 demonstration videos and 2 EXPRESS debriefing videos were reviewed and scored by all 8 experts. Modifications were again made to the DASH based on feedback obtained during this process. The feedback led to a refinement of element titles, reassignment of some dimensions to other elements and the addition of a new dimension regarding the demonstrated content expertise of the debriefer. In addition, this round resulted in more concrete and precise descriptions of behavioral anchors, refinement of the layout of the DASH rater’s handbook for ease of use, and refinement of the layout of the score sheet, condensing it to 2 pages. For the last round, a teleconference format was used, which led to minor final revisions of the language of the elements, dimensions, and behaviors to better reflect terminology familiar to clinician educators.
One hundred fifty-one international health care educators participated in 4.5-hour Web-based interactive DASH rater training sessions. Anonymous IP addresses were used to identify each participant. However, because of shared computer networks at some institutions, not all participants were uniquely identifiable, and therefore, some data were excluded from analysis. Therefore, ratings from a total of 114 rater trainees were analyzed from 2 separate training sessions. The ratings from these sessions were analyzed to assess for reliability and validity. This research was reviewed by the Partners Healthcare Human Research Ethics Committee and determined to be exempt.
The participants included nurses, physicians, other health professionals, and Masters and PhD educators; their work environments ranged from community-based hospitals to academic medical centers. A training session consisted of 4 steps. First, the rater trainees were asked to thoroughly familiarize themselves with the DASH rater’s handbook before the Web-based session; specifically, they were asked to study the 6 elements and develop a working knowledge of the dimensions in each element. At the beginning of the session, the trainers provided a brief didactic summary of each DASH element with highlights of each dimension. Next, the trainers described and illustrated best practices and common pitfalls for rating in general and for the DASH in particular. Finally, in 3 consecutive rounds, the rater trainees watched, rated, and then discussed 3 separate course introductions and subsequent debriefings.
The introductions and debriefings comprised 3 scripted videos that were produced for rater training to exemplify superior, average, and poor debriefings. The debriefings were conducted with 3 different groups of learners who had managed a clinical simulation involving pulseless electrical activity due to pneumothorax. The clinical and behavioral objectives of the case included (1) identification and management of pneumothorax, (2) establishing roles clearly, and (3) team leadership with a focus on stating an action plan. The 3 debriefings used archetypes for superior, average, and poor debriefings suggested by practical experience and the literature related to debriefing, particularly research on the role of psychologic safety, feedback, and reflection in learning. To develop the 3 debriefings, criteria for the quality of feedback conversations described in the organizational behavior, productive conversations, and feedback literature were used: whether the debriefer provides clear and actionable information about the performance of the learners, to what degree the debriefer created a psychologically safe learning environment that allows for specific feedback for key behaviors, and the degree to which the debriefer followed understandable phases of a debriefing.41,46,56–58 The rater trainees had no prior knowledge of any aspect of the debriefings they viewed, including these archetypes.
For each of the 3 rounds of ratings, scores were compiled and posted online in real time. The instructors then led a group discussion on participant ratings to provide reinforcement, corrections, and adjustments. Trainers elicited the participants’ rationales for their ratings and helped calibrate trainees’ assessments to the elements of the DASH. The guiding principle was to rate “through the eyes of the DASH” so as to help participants compare and adjust their ratings to the criteria set forth in the DASH, as the optimal means to obtain rater convergence and interrater reliability.59
Assessment of Reliability and Validity: Statistical Analysis
The following statistical analyses, which use parametric inference, relied on the assumption that the variables of interest were normally distributed. Assumption of normality was considered to be reasonable given the robustness of the employed tests to deviations from normality, visual inspection of the data, and scrutiny of descriptive statistics (ie, skewness, kertosis).
Interrater reliability was assessed for the 114 webinar raters’ scores at the element level and for the overall mean of the 6 elements. Variance component analysis was used to calculate intraclass correlation coefficients (ICCs), which represent the ratio of rater variance to the sum of rater variance and the total variance.
To assess the internal consistency of the tool, Cronbach α was calculated using the same webinar data set for the “average” video. This video was selected because it was considered the most difficult to rate because it did not represent an extreme of performance but blended effective and ineffective behaviors. In addition, this debriefing was rated when the raters had received the most training, at the end of the webinar.
To assess 1 aspect of the validity of the DASH, the mean scores for each of the videos across the 114 webinar rater trainees were calculated and compared by means of a 1-way repeated-measures analysis of variance.
All statistical analyses were performed using STATA version 12.0.
The ICCs for the each element and for the instrument overall are reported in Table 3. Notably, the ICCs are nearly all more than 0.6 for the individual elements. The ICC for element 5 is just less than this at 0.57. The ICC for the DASH overall was 0.74. Cronbach α for the average video was 0.89 across the entire webinar rater data set.
The ratings for the superior, average, and poor videos are shown in Figure 1. The differences among the ratings of the 3 standardized debriefings were statistically significant (F = 486.2, df = 2,226; P < 0.001).
Debriefing has a long and important history of use in the military and aviation industries and has a rapidly expanding role within health care education.23,25,29–31,60,61 Regardless of the specific setting, the goal of debriefing remains the same: to promote reflection and learning and, ultimately, to thereby improve performance. In clinical practice and structured educational encounters, health care providers across the spectrum of training and professional life have many learning opportunities. There is evidence to suggest that simulation accompanied by high-quality debriefings facilitates the transfer of new knowledge, skills, and attitudes to the clinical domain, primarily through the enactment of the reflection stage of experiential learning and by providing the opportunity for the experimentation aspect of adult learning.7,62–65 An assessment tool that helps determine debriefing quality and provides debriefers with valuable feedback can provide crucial support for the educational processes within debriefing.
Data regarding the psychometric properties of the DASH in the context of training raters reveal promising interrater reliability and internal consistency. Although further evidence is required, support for DASH validity is grounded in both its content and the scores arising from its use. For the content, the extensive theoretical background and practice-based experience integrated into the DASH provide support for the content relevance of the DASH. The performance of the DASH scores, specifically the statistically significant difference between the scores for debriefings of varying quality, provides some nascent evidence for the validity of DASH scores. That is, the DASH was designed to measure the quality of debriefing performance, and DASH scores in this study did vary with the described varying debriefing archetypes. However, this evidence is limited by the actor-as-debriefer nature of the videos. More definitive evidence for validity will ideally be sought from the analysis of more complex and larger samples of debriefings. One such study is specifically planned for the data from the EXPRESS study.65,66 Ultimately, the optimal test for a rating tool such as the DASH is whether it predicts learning, not just debriefing quality.
The DASH is distinct from existing health care debriefing tools in 2 ways. Although such other tools exist, they are specifically intended for particular contexts. In addition, none of these debriefing assessment instruments provide behavioral anchors. The DASH and its behavioral anchors rely on extensive and well-defined domains of behavior from activities closely related to debriefing, the debriefing literature, and expert experience. Beyond the psychometric arguments in support of DASH reliability and validity presented here, the 3 levels of granularity of the DASH—elements, dimensions, and behavioral anchors—have the potential both to guide detailed formative assessment and to support rigorous summative assessment. The DASH handbook is a detailed description of the qualities and behaviors that comprise a debriefing. This may help educators use the DASH to provide feedback linked to specific areas of strength and areas for improvement.
Limitations to the present work include those common to all rating instruments and those specific to the DASH. As with all assessment tools, the data presented here speak only to the psychometrics of the DASH data in this particular setting. It is hoped that further studies will examine the properties of the DASH when used by raters from different backgrounds and different simulation settings. Similar to other behavior rating instruments, the DASH is limited in its use to trained users, and thus, rater training is a necessary step to its implementation. Another potential limitation of the DASH concerns its generalizability. Although the foundation of the DASH, through synthesis of relevant theory, empirical data from related fields, and the involvement of multiple experts in its conception, is intended to bridge differences in debriefing styles, how well the DASH is able to assess different debriefing strategies will require further investigation. Because there is no single criterion standard for debriefing quality, the DASH of necessity has judgments embedded within it regarding optimal debriefing behaviors. The DASH development process was aimed at identifying behaviors common to all effective debriefing styles, but the ultimate success of this endeavor will require further empirical evidence.
In conclusion, this study is a first step to our collective understanding of how the DASH performs. The evidence presented here suggests that, in the present setting, the DASH yields reliable data for the assessment of health care simulation debriefings. It is hoped that other studies of the instrument will help it become a useful tool to guide educators in their use of debriefing as a critical educational modality.
The authors thank the EXPRESS investigators; Elizabeth Hunt, MD, MPH, PhD; Monica Kleinman, MD; Vinay Nadkarni, MD, MS; Kristen Nelson McMillan, MD; and Akira Nishisaki, MD, without whom this work could not have been completed, and the authors thank John Boulet, PhD, and Heather L. Corliss, PhD, MPH, for their invaluable statistical guidance. The authors also thank the Simulation in Healthcare reviewers for their feedback that substantially improved this article.
1. Harden RM. Trends and the future of postgraduate medical education
. Emerg Med J
2006; 23: 798–802.
2. Issenberg SB, McGaghie WC, Hart IR, et al.. Simulation
technology for health care professional skills training and assessment
1999; 282: 861–866.
4. Cheng A, Nadkarni V, Hunt E, Qayumi K, EXPRESS Investigators. A multifunctional online research portal for facilitation of simulation
-based research: a report from the EXPRESS pediatric simulation
research collaborative. Simul Healthc
2011; 6: 239–243.
5. Cheng A, Hunt EA, Donoghue A, et al.. EXPRESS—Examining Pediatric Resuscitation Education Using Simulation
and Scripting. The birth of an international pediatric simulation
research collaborative—from concept to reality. Simul Healthc
2011; 6: 34–41.
7. Cook DA, Hatala R, Brydges R, et al.. Technology-enhanced simulation
for health professions education: a systematic review and meta-analysis. JAMA
2011; 306: 978–988.
8. Nishisaki A, Nguyen J, Colborn S, et al.. Evaluation of multidisciplinary simulation
training on clinical performance and team behavior during tracheal intubation procedures in a pediatric intensive care unit. Pediatr Crit Care Med
2011; 12: 406–414.
9. Eppich WJ, Adler MD, McGaghie WC. Emergency and critical care pediatrics: use of medical simulation
for training in acute pediatric emergencies. Curr Opin Pediatr
2006; 18: 266–271.
10. Volk MS, Ward J, Irias N, Navedo A, Pollart J, Weinstock PH. Using medical simulation
to teach crisis resource management and decision-making skills to otolaryngology housestaff. Otolaryngol Head Neck Surg
2011; 145: 35–42.
11. Halamek LP. Teaching versus learning and the role of simulation
-based training in pediatrics. J Pediatr
2007; 151: 329–330.
12. Cheng A, Duff J, Grant E, Kissoon N, Grant VJ. Simulation
in paediatrics: an educational revolution. Paediatr Child Health
2007; 12: 465–468.
13. Adler MD, Trainor JL, Siddall VJ, McGaghie WC. Development and evaluation of high-fidelity simulation
case scenarios for pediatric resident education. Ambul Pediatr
2007; 7: 182–186.
14. Nishisaki A, Hales R, Biagas K, et al.. A multi-institutional high-fidelity simulation
“boot camp” orientation and training program for first year pediatric critical care fellows. Pediatr Crit Care Med
2009; 10: 157–162.
15. Shilkofski NA, Nelson KL, Hunt EA. Recognition and treatment of unstable supraventricular tachycardia by pediatric residents in a simulation
scenario. Simul Healthc
2008; 3: 4–9.
16. Brett-Fleegler MB, Vinci RJ, Weiner DL, Harris SK, Shih MC, Kleinman ME. A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics
2008; 121: e597–e603.
17. Weinstock PH, Kappus LJ, Garden A, Burns JP. Simulation
at the point of care: reduced-cost, in situ training via a mobile cart. Pediatr Crit Care Med
2009; 10: 176–181.
18. Nishisaki A, Donoghue AJ, Colborn S, et al.. Effect of just-in-time simulation
training on tracheal intubation procedure safety in the pediatric intensive care unit. Anesthesiology
2010; 113: 214–223.
19. Cooper JB, Singer SJ, Hayes J, et al.. Design and evaluation of simulation
scenarios for a program introducing patient safety, teamwork, safety leadership, and simulation
to healthcare leaders and managers. Simul Healthc
2011; 6: 231–238.
20. Morgan PJ, Kurrek MM, Bertram S, Leblanc V, Przybyszewski T. Nontechnical skills assessment
-based continuing medical education
. Simul Healthc
2011; 6: 255–259.
21. Hunt EA, Hohenhaus SM, Luo X, Frush KS. Simulation
of pediatric trauma stabilization in 35 North Carolina emergency departments: identification of targets for performance improvement. Pediatrics
2006; 117: 641–648.
22. Dismukes RK, Gaba DM, Howard SK. So many roads: facilitated debriefing
in healthcare. Simul Healthc
2006; 1: 23–25.
23. Fanning RM, Gaba DM. The role of debriefing
-based learning. Simul Healthc
2007; 2: 115–125.
24. Van Heukelom JN, Begaz T, Treat R. Comparison of postsimulation debriefing
versus in-simulation debriefing
in medical simulation
. Simul Healthc
2010; 5: 91–97.
25. Rudolph JW, Simon R, Raemer DB, Eppich W. Debriefing
as formative assessment
: closing performance gaps in medical attention. Acad Emerg Med
2008; 15: 1110–1116.
26. Kolb DA. Experiential Learning: Experience as the Source of Learning and Development
. Englewood Cliffs, NJ: Prentice Hall; 1984.
27. Issenberg SB, McGaghie WC, Petrusa ER, Gordon DL, Scalese RJ. What are the features and uses of high-fidelity medical simulations that lead to most effective learning: a BEME systematic review. Med Teach
2005; 27: 10–12.
28. Schön D. Educating the Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions
. San Francisco, CA: Jossey-Bass; 1987.
29. Rudolph JW, Simon R, Dufresne RL, Raemer DB. There’s no such thing as a “nonjudgmental” debriefing
: a theory and method for debriefing
with good judgment. Simul Healthc
2006; 1: 49–55.
30. Rudolph JW, Simon R, Rivard P, Dufresne RL, Raemer DB. Debriefing
with good judgment: combining rigorous feedback with genuine inquiry. Anesthesiol Clin
2007; 25: 361–376.
31. Dismukes RK, McDonnell LK, Jobe KK. Facilitating LOFT debriefings: instructor techniques and crew participation. Int J Aviat Psychol
2000; 10: 35–57.
32. Bylund C, Brown R, Lubrano di Ciccone B, Diamond C, Eddington J, Kissane DW. Assessing facilitator competence in a comprehensive communication skills training programme. Med Educ
2009; 43: 342–349.
33. Gururaja RP, Yang T, Paige JT, Chauvin SW. Examining the effectiveness of debriefing
at the point of care in simulation
-based operating room team training. In: Henriksen K, Battles JB, Keyes MA, Gary ML, eds. Advances in Patient Safety: New Directions and Alternative Approaches (Vol 3: Performance and Tools)
. Rockville, MD: Agency for Healthcare Research and Quality; 2008.
34. Reed SJ. Debriefing
experience scale: development of a tool to evaluate the student learning experience in debriefing
. Clin Simul Nurs
2012; 8 (6): e211–e217.
35. Arora S, Ahmed M, Paige J, et al.. Objective Structured Assessment
(OSAD): bringing science to the art of debriefing
in surgery. Ann Surg
August 14, 2012 [epub ahead of print].
36. American Educational Research Association, APA, and National Council on Measurement in Education. Standards for Educational and Psychological Testing
. Washington, DC: AERA; 1999.
38. Smith PC, Kendall LM. Retranslation of expectations: an approach to the construction of unambiguous anchors for rating scales. J Appl Psychol
1963; 47: 149–155.
39. Shapira Z, Shirom A. New issues in the use of behaviorally anchored rating scales: level of analysis, the effect of incidence frequency, and external validation. J Appl Psychol
1980; 65: 517–523.
40. Jacobs RAU, Kafry D, Zedeck S. Expectations of behaviorally anchored rating scales. Pers Psychol
1980; 33: 595–640.
41. Argyris C, Putnam R, Smith DM. Action Science: Concepts, Methods and Skills for Research and Intervention
. San Francisco, CA: Jossey-Bass; 1985.
42. Edmondson A. Psychological safety and learning behavior in work teams. Adm Sci Q
1999; 44: 350–383.
43. Knowles MS, Holton EF, Swanson RA. The Adult Learner: The Definitive Classic in Adult Education and Human Resource Development
. 6th ed. Burlington, MA: Elsevier; 2005.
44. Watzlawick P, Weakland JH, Fisch R. Change: Principles of Problem Formation and Problem Resolution
. New York, NY: Horton; 1974.
45. Darling M, Parry C, Moore J. Learning in the thick of it. Harv Bus Rev
2005; 83: 84–92, 192.
46. Ende J. Feedback in clinical medical education
1983; 250: 777–781.
47. Harlen W, James M. Assessment
and learning: differences and relationship between formative and summative assessment
. Assess Educ Princ Pol Pract
1997; 4: 365–377.
48. Vaughan D. Theory elaboration: the heuristics of case analysis. In: Becker H, Ragin C, eds. What Is a Case?
New York, NY: Cambridge University Press; 1992: 173–202.
49. Vaughan D. The Challenger Launch Decision: Risky Technology, Culture and Deviance at NASA
. Chicago, IL: University of Chicago Press; 1996.
50. Davis JP, Eisenhardt KM, Bingham CB. Developing theory through simulation
methods. Acad Manage Rev
2007; 32: 480–499.
51. Martin JA, Regehr G, Reznick R, et al.. Objective structured assessment
of technical skill (OSATS) for surgical residents. Br J Surg
1997; 84: 273–278.
52. Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med
1998; 73: 993–997.
53. Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. OSCE checklists do not capture increasing levels of expertise. Acad Med
1999; 74: 1129–1134.
54. Berk RA. A Guide to Criterion-Referenced Test Construction
. Baltimore, MD: The Johns Hopkins University Press; 1984.
55. Dumas JC, Redish JC. A Practical Guide to Usability Testing
. 2nd ed. Bristol, UK: Intellect; 1999.
56. Weisinger H. The Critical Edge: How to Criticize Up and Down Your Organization and Make It Pay Off
. New York, NY: Little Brown & Co; 1989.
57. Weisinger H. The Power of Positive Criticism
. New York: AMACOM; 2000.
58. Kegan R, Lahey LL. How the Way We Talk Can Change the Way We Work
. San Francisco, CA: Jossey-Bass; 2001.
59. Bernardin HJ, Buckley MR. Strategies in rater training. Acad Manage Rev
1982; 6: 206–212.
60. Lederman LC. Toward a systematic assessment
of theory and practice. Simul Gaming
1992; 23: 145–160.
61. Steinwachs B. How to facilitate a debriefing
. Simul Gaming
1992; 23: 186–195.
62. Savoldelli GL, Naik VN, Park J, Joo HS, Chow R, Hamstra SJ. Value of debriefing
during simulated crisis management: oral versus video-assisted oral feedback. Anesthesiology
2006; 105: 279–285.
63. Draycott TJ, Crofts JF, Ash JP, et al.. Improving neonatal outcome through practical shoulder dystocia training. Obstet Gynecol
2008; 112: 14–20.
64. Crofts JF, Fox R, Ellis D, Winter C, Hinshaw K, Draycott TJ. Observations from 450 shoulder dystocia simulations: lessons for skills training. Obstet Gynecol
2008; 112: 906–912.
65. Siassakos D, Draycott T, O’Brien K, Kenyon C, Bartlett C, Fox R. Exploratory randomized controlled trial of hybrid obstetric simulation
training for undergraduate students. Simul Healthc
2010; 5: 193–198.
66. 66. Donoghue A, Ventre K, Boulet J, Brett-Fleegler M, Nishisaki A, Overly F, Cheng A, Donoghue A. EXPRESS Pediatric Simulation
Research Investigators. Simul Healthc
2011 Apr; 6 (2): 71–77.
Medical education; Health care education; Assessment; Debriefing; Simulation; Psychometrics; Behaviorally anchored rating scale
Supplemental Digital Content
© 2012 Society for Simulation in Healthcare