“I think, therefore I am” - Rene Descartes
“He who knows most, knows best how little he knows.” - Thomas Jefferson
The past decade has seen a considerable increase in interest in understanding the cognitive underpinnings of medical decision making in dynamic acute care arenas. Errors caused by faulty thought processes more than faulty knowledge are called “cognitive errors” and may be caused by bias, heuristic, emotion, and other nonrational cognitive elements.1,2 Although a few groups (including both authors of this article) have been studying cognition and problem solving in dynamic arenas such as emergency medicine, intensive care, or anesthesiology—including some going back more than 20 years3—these concepts are not yet well-known to many clinicians and simulation educators. Moreover, the literature on these topics exists in many different kinds of journals and indices and under a variety of key words, which may make finding salient information more challenging.
Murray et al4 deserve congratulations for their work described in this issue of Simulation in Healthcare that uses simulation to understand more fully such decision making by individuals and teams.4 “Decision Making in Trauma Settings: Simulation to Improve Diagnostic Skills” highlights critical questions about clinical decision making in a dynamic arena. The study design, assumptions, and conclusions can serve as the basis for an important discussion about many facets of this complex process. One issue highlighted in the article of Murray et al is whether people are, in general, prone to certain kinds of foibles in decision making when a situation has some elements consistent with a simple explanation, along the lines of what they expect, but is actually more complex. Another is the interplay between individual decision and group decision making because health care quality and safety are known to be properties of teams rather than just of individuals. At issue as well is the contribution of clinical experience to “expertise” and to “performance.” Are experts and novices prone to different kinds of decision error in their application of knowledge (even when their “book knowledge” is equal)? We must also consider how accurately inferences can be made about thought process from observing surface behaviors. If making accurate inferences is difficult, how else might we better assess thought processes? Ultimately, one of the key goals in answering such questions is to determine how best to train new clinicians in these arenas of complex decision making to implement optimal decision-making strategies.
HOW CAN WE DEFINE “SIMPLE” OR “COMPLEX”? DOES A “COMPLEX” PROBLEM NECESSARILY RESULT IN ANALYSIS OR A “SIMPLE” ONE IN USE OF HEURISTICS?
In medicine, any case could appear to be a simple one if the available data seem to fit well within a recognized “illness script” or pattern. An illness script is a specific type of memory schema that represents generalized patient-oriented clinical events as a unit, in a precompiled knowledge structure. These scripts are stored in long-term memory and are the result of repeated direct or vicarious experiences.5 Use of illness scripts and pattern recognition are examples of intuitive decision strategy shortcuts, also called heuristics. Heuristics and other cognitive biases are widely discussed in the literature as potential sources of compromise to decision making.1,6–10 Heuristics may be applied implicitly, as when we believe that we recognize a pattern and therefore do not engage in any formal reasoning, or deliberately, as when we consciously abandon detailed abstract analysis of the available data and “go with our gut” (sometimes described as “analysis paralysis”). In dynamic settings, some degree of heuristic decision making is inevitable—thinking through in detail every single action would be impossible. Importantly, it is always possible to abandon heuristic or pattern-matching strategies in favor of analysis even when faced with a simple problem; likewise, analysis may be abandoned in a complex situation to go with an intuitively appealing solution or when the need to act becomes pressing.
No strategy works perfectly all the time. As well, problems in highly dynamic areas of medicine are often ill-defined (ie, there is no clear single correct or best solution, and the quality of decision making is subject to judgment that often varies considerably even among content experts in the same domain). Unfortunately, patients do not come with a little flag that says what is really wrong with them. Failure to rapidly identify and treat problems can be just as deleterious as premature closure on the wrong, but early, identification of the problem. Either with classic situations (that do actually match a typical patterns) or with situations that demand quick action on the most serious possible etiology (eg, start cardiopulmonary resuscitation on an unresponsive patient without a quickly palpable pulse), waiting for analysis can be a mistake. Clinicians are thus thrust between the horns of a dilemma—whether to act quickly, risking that the early impression could be wrong, or whether to wait for certainty. Indeed, in many clinical emergencies, “correct” actions that are not implemented quickly become ineffective, and the decision to delay action in favor of analysis is itself an incorrect decision (eg, delayed defibrillation while causes are considered increases mortality11). However, it has also been posited that acting too rashly accounts for more patient harm than a few moments of planning and deliberation.12
Many models of decision making posit a “dual process” of heuristic (or “precompiled”) reasoning along with deliberate analysis (“abstract reasoning”). Croskerry13 reviews the clinical application of the dual-process model of reasoning in emergency medicine (“Type I vs. Type II”). This categorization may be traced to early work from Wason in196014 and decades of research since, recently popularized by Kahneman's book Thinking, Fast and Slow.15 A parallel set of approaches comes from a school of thought termed naturalistic decision making,16,17 in which one well-known model is called recognition-primed decision making (RPD).18,19 In RPD, the key decision maker assembles information to recognize a typical pattern, which then calls up typical solution strategies (this is quite consistent with the “illness script” model mentioned earlier). According to Klein,16 the first solution entertained by experienced decision makers is often satisfactory—that is, the decider quickly selects the first workable solution rather than the best possible solution (“satisficing”), and this is an advantage when faster decisions are better decisions. However, a key analytic feature of RPD is that decision maker then undertakes a brief “mental simulation” of how the solution would work, enabling her or him to see pros, cons, and barriers to the use of that solution for the particular problem (or to question whether the situation does in fact conform to the imagined pattern). Klein points out that this kind of analysis is not comparing potential solutions against each other but simply comparing the favored solution against the context at hand to assess likelihood of success. Also implicit in RPD and explicit in other models (including our own model of decision making in anesthesiology and other dynamic medical domains20–22) is the need for a process of constant reevaluation that can shift from one type of reasoning to another or adjust diagnoses and treatments to match an evolving situation.
All of these models are based on the well-established fact that the capacity for conscious thought is very limited compared with subconscious or automatic and that that vast majority of our waking time is characterized by relative inattention, automaticity, and routine.23 Because such intuitive decision making draws from a repertoire of scripts and patterns that have been experienced either personally or vicariously, we can predict that clinicians with more experience will have a larger repertoire. However, the specific scripts and patterns in that repertoire will by necessity vary among individuals because of at least 2 distinct features: the content of experiences and the availability8 of the experiences. Availability refers to the degree of “memorableness”—the ease with which an experience comes consciously to mind. Availability may be influenced by recent events, emotionally charged events, novelty, and other factors that are distinct from the actual probability of that event occurring. As well, perception that a situation is recognized depends on the data streams that are present, and within those streams, which data are emphasized. In other words, “recognizing” a pattern is not an objective process. Instead, it is a subjective experience, influenced by allocation of attention, availability/memorableness, previous experience, strength of data signals within simultaneous streams, distraction, and so on. Hence, “recognition” is a key step, but it can be faulty itself. If one is initially incorrect in making such a recognition, the potential value of RPD or of using heuristics can be compromised.
Thus, when a complex situation has presenting elements that fit recognizable scripts or patterns, particularly when the pattern or script is expected in the context (eg, hypotension on induction of anesthesia), lack of deliberate analysis may result in a rapid but not especially thoughtful temporizing maneuver, or it may result in the decision to take no action at all, expecting a self-limited course. Because “common things are common,” such lack of deliberate analysis is often quick and correct, thus reinforcing the automatic behavior. Only when the perturbation exceeds the expected duration or proves refractory to initial measures will it become apparent that the situation is actually more complex. In such cases, there may be a delayed or missed diagnosis or treatment of the unexpected etiology. Although such errors are sometimes described as an “overreliance” on heuristics,7 we would argue that it is impossible to make such judgments because (a) intuitive, heuristic, decision making is largely conducted subconsciously and (b) the decision of whether a decision was “good” is highly influenced by hindsight.
Consequently, for a variety of reasons, we have some skepticism of whether the cases used in the study of Murray et al can reliably be categorized dichotomously in advance as “standard/heuristic” versus “complex/analytic.” Data that might readily match a script for one clinician might require analysis for another, even at the same level of training or experience. Moreover, the “application of knowledge”—whether via intuitive or analytic means—cannot be separated completely from knowledge itself. How can we be sure that the failure to diagnose a situation is really caused by heuristic thinking versus analytic approaches as opposed to other causes such as just plain lack of knowledge7 or the failure to perceive the data that suggest a complex etiology?
WHAT IS THE ROLE OF EXPERIENCE IN DECISION ERROR? DO EXPERTS MAKE THE SAME MISTAKES AS NOVICES?
It is known that experts have sufficient attentional resources to perform tasks and monitor information concurrently without hindering performance,24 and cognitive load theory and “chunking” explain how experts handle large volumes of information concurrently.25 However, less is known about exactly how medical decision making differs between more and less experienced clinicians. We largely agree with the core ideas Murray et al proposed to explain why juniors performed better in cases marked as “analytic” (less overconfidence by juniors, more use of pattern-matching strategies by seniors). Indeed, many of the cognitive processes discussed earlier (illness scripts, pattern matching, chunking data, and heuristics that minimize cognitive load) form through hard-won experience and are integral to the efficiency and accuracy often demonstrated by experts. This gain comes at the tradeoff price of a (generally subconscious) vulnerability to a kind of error that may not be seen with analytic thinking. In other words, when one finds a problem to be complex or challenging at the outset, it is hard to be overconfident and easy to be careful in determining what is going on. However, it is impossible to simultaneously perceive a problem to be easy or obvious, and also have insight into error of that perception.
The literature on illness scripts suggests that seemingly small variations in the presentation or perception of a situation can radically change things. According to cognitive load theory,26 experience allows experts to “chunk” data into schema, thereby allowing them to monitor a larger volume of data. The compiling of elements and chunking of data with associated contextual cues play a significant role in the accuracy of experts. In contrast, when situation contravenes the script structure, for example, if information is presented in a random temporal order, rather than in the “script” order, experts' performances are affected much more than novices'. This is true even to the point where experts no longer benefit from their expertise and do not outperform novices.5 It is generally thought that experts solve many problems primarily via intuitive thinking, in contrast to deliberate analysis by nonexperts. Although such intuitive thinking is often highly effective and efficient, it is possible that it is also perhaps more susceptible to bias-induced error.27–30 The conclusions of Murray et al that “unexpectedly, the teams led by more junior residents received higher scores in the complex scenarios” is then not unexpected at all but just bolsters a view already in the literature. The explanation of Murray et al that more experienced team leaders are potentially more reliant on heuristic diagnostic approaches and often overconfident is consistent with prevailing theories as discussed earlier. It might also be true that experts can detect and integrate barely-above-threshold stimuli (a phenomenon variously described as a part of “situation awareness,”31 “gestalt,” or “sixth sense”) and that experts are generally better at using these subtle cues to recognize complexity or severity earlier than novices. Some such cues are difficult to simulate, and this is a known limitation of simulation education and research. Nonetheless, although a lack of such cues could potentially confuse those expecting to see them, this alone should not seriously derail expert decision making across the board.
INDIVIDUAL AND TEAM DECISION MAKING
Decision making by teams is extremely complex because of an extensive array of features including attention and information distribution, leadership, communication, personality, and more. A thorough treatment of these issues is beyond the scope of this article, but we will briefly address a few important areas.
In principle, by “speaking up,” other team members can prevent or mitigate individual errors of decision making by nominal leaders. Not only does this happen too rarely, but it has also been found that sometimes groups (i) increase, rather than decrease, reliance on heuristics, (ii) are more prone to overconfidence than individual members, (iii) may escalate their commitment to a failing course of action, and (iv) may be more influenced by framing effects.32 It is not known to what degree this is true in health care because these data come from business and economics and have not been explicitly replicated in medical domains. It has been well established, however, that “speaking up” to correct potential errors is a major safety problem in health care, and therefore, individual biases may not be systematically corrected at the group level. This is thought to be caused by several different factors: (i) informational cascades, (ii) social shirking, and (iii) reputational cascades. In informational cascades, there is deference to the first or early information verbalized or shared by everyone rather than taking into account critical information possessed by only one or a few people. In social shirking, a supposed “double check” may be missed when one or both parties shirk their independent check, assuming the other party is doing it. In reputational cascades, people of “lower” status often silence themselves to avoid the disapproval or other social or professional consequences (no doubt social shirking is also more common when a higher status person is assumed to have done the check). More often than not, an individual's decision to remain silent offers the most to gain for themselves, even when the group (or the patient) suffers.
In health care, personnel may sometimes be rewarded for a “good pickup”—noticing or figuring out something important that others did not. However, we rarely reward individuals—as we should—for voicing their credible concerns even when those concerns turn out to be wrong or not relevant to the situation.
CAN WE INFER THOUGHT PROCESS FROM OBSERVED BEHAVIOR ALONE?
Murray et al selected 10 real trauma events for the basis of their scenarios, 7 of which were designed to follow “well-recognized patterns of injury,” which could be effectively managed by straightforward implementation of Advanced Trauma Life Support (ATLS) algorithms with no adaptation. The other 3 cases were specifically designed to be refractory to initial ATLS therapy because of coexisting pathology or greater-than-usual severity. This study design raises 3 interesting questions.
First, simply because a case was not correctly recognized by the care team does not mean that “heuristics” failed. Other failures include missing7 or “buggy knowledge,”33 poor teamwork, lack of sufficient reevaluation as to why the usual diagnosis and treatment were inadequate, and “premature closure,” “satisficing,” or confirmation bias. The latter 3 are failures of reasoning that can occur even with an analytic approach. In fact, there is no firm evidence to support the idea—and it is a huge underlying assumption of this study—that the 7 “classic pattern” problems would indeed be immediately and accurately diagnosed by the study participants or by clinicians in general. Using simulation scenarios to understand the decision-making process is invaluable. Yet, making inferences about that process based on the differences in score on the 2 types of scenarios—indeed without studying the reasoning process in depth—may rest on too many assumptions about supposedly unique qualities of the scenarios.
In addition to other variables we have commented on earlier, according to the design of Murray et al, if the teams implemented ATLS algorithms appropriate to the presenting vital sign or symptoms, whether they “recognized” the diagnosis correctly, they would “succeed.” This presents a conundrum. From the patient's standpoint doing the “right” thing for the “wrong” reason clearly has a short-term benefit. However, in the broader picture, we would argue that a diagnostic error is still being made, and the fact that the outcome is “good” is lucky. Were the same signs and symptoms to be due to a wholly different etiology, perhaps, the long-term outcome would be seriously bad. As well, a decision to operate on a patient could represent a reevaluation of diagnostic possibilities (as is the assumption of the study protocol), or it could represent commission bias—the “better safe than sorry” approach in which a tendency toward action is favored, even if not explicitly indicated.2,34
So, ultimately, to what degree can soundness of decision making be inferred solely from observed behaviors and actions without detailed attempts to elicit and understand the actual processes of thought?35 Medical decisions may be classified as the (i) right action for the right reason, (ii) right action for the wrong reason, (iii) wrong action for the right reason, or (iv) wrong action for the wrong reason. Without a specific probe into decision making, rationale remains invisible, even though we believe it is important for both simulation research and education. Incorrect rationale coupled with correct action is likely to remain unremediated or even given positive reinforcement. Similarly, incorrect action with a good rationale may be unfairly criticized.1 Despite good interrater reliability in the techniques used by Murray et al, too much may be assumed about thought process without addressing them explicitly.
We agree with Murray et al that more research is needed to explore how individuals and teams come to their decisions. Cognitive task analysis36–38 and “think aloud”39 protocols have been the criterion standard for studying decision making, although these have limitations in team settings as suggested earlier. As well, these protocols have narrow and specific requirements. In “think aloud”, for example, subjects are not asked to provide explanations about their thought processes. Asking for explanations of decisions while clinicians are performing tasks is significantly reactive—that is, the act of explaining one's thoughts changes subsequent thoughts, resulting in altered task performance (sometimes for the better). Asking subjects to merely verbalize their thoughts during a task is less likely to change performance and provide a simpler trace of thought sequence.38 Another difficulty of all such protocols is trying to apply them to time-pressured settings where in situ probing can increase time to completion of tasks38 and where a person is likely to have far more thoughts than can be verbalized concurrently. One approach to this problem might be to periodically pause the simulation and conduct a brief structured interview about the clinicians' understanding of the situation. Another approach is to build into the scenario an “ecologically” natural inquiry into this understanding.40 One way to do this is via the entry of scripted “confederate” peers or supervisors into the scene—the briefing that ensues may offer clues to the primary clinician's thinking.20 Another such approach that we (D.M.G.) have briefly piloted is to use “telementoring” as an explicit window on the thinking of the clinician(s) in real time. In a real case or a simulation, when the bedside clinician must interact dynamically with a remote expert (eg, “tele-ICU”, or similar for the ED [emergency department] or OR [operating room]), one can capture and record the discussion between the parties. This makes their thinking more overt and accessible than it might be if there was no remote mentor or even if they were in the same room.
Some approaches use completely retrospective interviews about decision processes of cases just completed or asking participants to remember one or more cases from the past. Such techniques are relatively easy to implement with a large number of clinicians, but they may be difficult to validate, partially because of recall bias, hindsight bias, and outcome bias. Moreover, whether conducted in real time or retrospective techniques such as “critical decision method” or cognitive task analysis, each require a deliberate and methodical approach for eliciting, explaining, and then representing task cognition in a comprehensive manner.41
Finally, querying clinicians about their thinking can also be used for education rather than for research. Educational use might choose to use the more invasive technique of asking for explanations for thoughts as a means to enhance the potential learning potential value of the exercise.
Research that uses simulation as a tool to study clinical care processes and individual and teams performance is as important as or more important than studying simulation itself.42 There is a need for more research about decision-making processes, the extent to which we can self-monitor and consciously select our thinking (ie, metacognition), and the contexts in which some processes may be superior to others. Other research might focus on strategies to prevent or interrupt various pitfalls of decision making. One thread might be on the effectiveness and impact of different types of educational interventions to teach specific decision-making strategies or how educational training can itself introduce cognitive bias.43 These could be adapted from those of other disciplines ranging from aviation and nuclear power to business or marketing. Another thread might be on achieving one of the “Holy Grails” of decision making in dynamic medical arenas—developing useful decision support systems that can effectively prevent cognitive errors related to various biases (eg, confirmation, availability, anchoring), premature closure, or a host of other factors. Importantly, such decision support systems would need to address 3 key deficiencies in systems that have been studied to date: (1) clinicians are not good at identifying whether and when they need decision support44,45; (2) decisions are often made even before people are consciously aware of them (early impressions may then bias the processing of later information, influencing the consultation with a colleague or interaction with a decision support tool46,47; and (3) the decision support tools may not actually change the decisions and actions taken.48–50 One of us (M.P.S.) is currently investigating the design of such tools via simulation testing.
In summary, Murray et al do a great service to our clinical fields and to the simulation community by continuing the search for understanding of clinical decision making using the unique window of simulation. Because human performance and thought is indeed complex and health care decision making is perhaps even more complex in the setting of team-based management and patient-centered care, there is much to explore as we expand our empirical knowledge on these matters. In health care, the overarching areas for research can be summarized as: (i) to what degree do automatic factors influence our decisions and which such factors are most important; (ii) which thought strategies are superior in which contexts; and (iii) what decision support tools can best support such strategies to prevent decision error. To be sure, it will be a long, but fascinating, intellectual struggle to understand the intricacies of these research questions. Murray et al have made an important stride in bringing these questions into focus.
1. Stiegler MP, Tung A. Cognitive processes in anesthesiology decision making. Anesthesiology
2014; 120: 204–217.
2. Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med
2003; 78: 775–780.
3. Gaba DM, Maxwell M, DeAnda A. Anesthetic mishaps: breaking the chain of accident evolution. Anesthesiology
1987; 66: 670–676.
4. Murray DJ, Freeman BD, Boulet JR, Woodhouse J, Fehr JJ, Klingensmith ME. Decision making in trauma settings: simulation to improve diagnostic skills. Simul Healthc
2015. [Epub ahead of print].
5. Custers EJ. Thirty years of illness scripts: theoretical origins and practical applications. Med Teach
2014; 37: 1–6.
6. Elstein AS. Heuristics and biases: selected errors in clinical reasoning. Acad Med
1999; 74: 791–794.
7. McLaughlin K, Eva KW, Norman GR. Reexamining our bias against heuristics. Adv Health Sci Educ Theory Pract
2014; 19: 457–464.
8. Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science
1974; 185: 1124–1131.
9. Croskerry P. Achieving quality in clinical decision making: cognitive strategies and detection of bias. Acad Emerg Med
2002; 9: 1184–1204.
10. Mamede S, van Gog T, van den Berge K, van Saase JL, Schmidt HG. Why do doctors make mistakes? A study of the role of salient distracting clinical features. Acad Med
2014; 89: 114–120.
11. Chan PS, Krumholz HM, Nichol G, Nallamothu BK; American Heart Association National Registry of Cardiopulmonary Resuscitation Investigators. Delayed time to defibrillation after in-hospital cardiac arrest. N Engl J Med
2008; 358: 9–17.
12. Rall MGR, Flin R. The ‘10-seconds-for-10-minutes principle’—why things go wrong and stopping them getting worse. Bull R Coll Anaesth
13. Croskerry P. Clinical cognition and diagnostic error: applications of a dual process model of reasoning. Adv Health Sci Educ Theory Pract
2009; 14 (Suppl 1): 27–35.
14. Evans JS. Reasoning, biases and dual processes: the lasting impact of Wason (1960). Q J Exp Psychol (Hove)
15. Kahneman D. Thinking, Fast and Slow
. New York, NY: Farrar, Straus and Giroux; 2011.
16. Klein G. Naturalistic decision making. Hum Factors
2008; 50: 456–460.
17. Zsambok C, Klein G. Naturalistic Decision Making
. Mahwah, NJ: Lawrence Erlbaum Associates; 1996.
18. Klein G OJ, Calderwood R, Zsambok C. A recognition-primed decision (RPD) model of rapid decision making. In: Klein G OJ, Calderwood R. Decision Making in Action: Models and Methods
. Norwood, NJ: Ablex Publishing Corp; 1993: 138–147.
19. Klein G. Recognition-primed decisions. Advances in Man-Machine Systems Research
1989; 5: 47–92.
20. Gaba DM, Howard SK, Small SD. Situation awareness in anesthesiology. Hum Factors
1995; 37: 20–31.
21. Gaba D, Fish KJ, Howard SK, Burden A. Crisis Management in Anesthesiology
. 2nd ed. Philadelphia, PA: Saunders an imprint of Elsevier, Inc; 2015.
22. Gaba D, Fish KJ, Howard SK. Crisis Management in Anesthesiology
. New York, NY: Churchill Livingstone; 1994.
23. Dijksterhuis A. Think different: the merits of unconscious thought in preference development and decision making. J Pers Soc Psychol
2004; 87: 586–598.
24. Gray R. Attending to the execution of a complex sensorimotor skill: expertise differences, choking, and slumps. J Exp Psychol Appl
2004; 10: 42–54.
25. Murphy GL, Wright J. Changes in conceptual structure with expertise: differences between real-world experts and novices. J Exp Psychol Learn Mem Cogn
1984; 10: 144–155.
26. Sweller J, Chandler P. Evidence for cognitive load theory. Cogn Instr
1991; 8: 351–362.
27. Mamede S, van Gog T, van den Berge K, et al. Effect of availability bias and reflective reasoning on diagnostic accuracy among internal medicine residents. JAMA
2010; 304: 1198–1203.
28. Schmidt HG, Rikers RM. How expertise develops in medicine: knowledge encapsulation and illness script formation. Med Educ
2007; 41: 1133–1139.
29. McLaughlin K, Rikers RM, Schmidt HG. Is analytic information processing a feature of expertise in medicine? Adv Health Sci Educ Theory Pract
2008; 13: 123–128.
30. Custers EJ, Boshuizen HP, Schmidt HG. The influence of medical expertise, case typicality, and illness script component on case processing and disease probability estimates. Mem Cognit
1996; 24: 384–399.
31. Schulz CM, Endsley MR, Kochs EF, Gelb AW, Wagner KJ. Situation awareness in anesthesia: concept and research. Anesthesiology
2013; 118: 729–742.
32. Sunstein CR, Hastie R. Wiser: Getting Beyond Groupthink to Make Groups Smarter
. Boston, MA: Harvard Business Review Press; 2015.
33. Woods DD, Decker S, Cook R, Johannsen L, Sarter N. Behind Human Error
. 2nd ed. Burlington, VT: Ashgate Publishing Ltd; 2012.
34. Stiegler MP, Neelankavil JP, Canales C, Dhillon A. Cognitive errors detected in anaesthesiology: a literature review and pilot study. Br J Anaesth
2012; 108: 229–235.
35. Eva KW, Regehr G. “I'll never play professional football” and other fallacies of self-assessment. J Contin Educ Health Prof
2008; 28: 14–19.
36. Kahneman D, Klein G. Conditions for intuitive expertise: a failure to disagree. Am Psychol
2009; 64: 515–526.
37. Vicente KJ. Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work
. Mahwah, NJ: Lawrence Erlbaum Associates; 1999.
38. Fox MC, Ericsson KA, Best R. Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychol Bull
2011; 137: 316–344.
39. Ericsson KA, Simon HA. Verbal reports as data. Psychol Rev
1980; 87: 215–251.
40. Sarter NB, Woods DD. Situation awareness: a critical but ill-defined phenomenon. Int J Aviat Psychol
1991; 1: 45–57.
41. Klein G, Militello L. Some guidelines for conducting a cognitive task analysis. In: Advances in Human Performance and Cognitive Engineering Research
. Emerald Group Publishing Limited; 2001: 163–199.
42. Gaba DM. The future's here. We are it. Simul Healthc
2006; 1 Spec no.: 1–2.
43. Park CS, Stojiljkovic L, Milicic B, Lin BF, Dror IE. Training induces cognitive bias: the case of a simulation-based emergency airway curriculum. Simul Healthc
2014; 9: 85–93.
44. Friedman CP, Gatti GG, Franz TM, et al. Do physicians know when their diagnoses are correct? Implications for decision support and error reduction. J Gen Intern Med
2005; 20: 334–339.
45. Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in medicine. Am J Med
2008; 121: S2–S23.
46. Wieck KE, Sutcliffe KM, Obstfeld D. Organizing and the process of sensemaking. Organ Sci
2005; 16: 409–421.
47. Rudolph JW, Morrison JB, Carroll JS. The dynamics of action oriented problem-solving: linking interpretation and choice. Acad Manage Rev
2009; 34: 733–756.
48. Olomu AB, Stommel M, Holmes-Rovner MM, et al. Is quality improvement sustainable? Findings of the American College of Cardiology's Guidelines Applied in Practice. Int J Qual Health Care
2014; 26: 215–222.
49. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med
2003; 348: 2635–2645.
50. Kostopoulou O, Rosen A, Round T, Wright E, Douiri A, Delaney B. Early diagnostic suggestions improve accuracy of GPs: a randomised controlled trial using computer-simulated patients. Br J Gen Pract
2015; 65: e49–e54.