Secondary Logo

Journal Logo

Research Reports

Making Decisions in the Era of the Clinical Decision Rule: How Emergency Physicians Use Clinical Decision Rules

Chan, Teresa M. MD, FRCPC, MHPE; Mercuri, Mathew PhD; Turcotte, Michelle; Gardiner, Emily MD; Sherbino, Jonathan MD, MEd, FRCPC; de Wit, Kerstin MD, MSc, FRCPC

Author Information
doi: 10.1097/ACM.0000000000003098


Diagnostic error has become an important topic in the health care literature. Diagnostic error is reported in approximately 10%–15% of emergency department cases, which is among the error rates highest reported for a particular unit.1,2 The role of cognitive bias as a component of diagnostic error has received significant attention.3,4 Diagnostic reasoning is a complex process, and eliminating diagnostic error is a challenge. Thus far, the approach to decreasing diagnostic error has assumed that physicians can improve the hypotheticodeductive process that underpins diagnostic reasoning.3–8 Psychologists and educators have often sought to improve physicians’ metacognition by guiding them to apply cognitive forcing strategies to prevent diagnostic error. However, the evidence shows that both novice and experienced practitioners have similar processes. Both groups use a combination of the dual processes (System 1, System 2 thinking).9 The facet that most critically affects diagnostic accuracy is, in fact, experience—which may reflect more knowledge around specific cases, rather than improved reasoning or metacognitive skills.10,11

Researchers have sought to improve reasoning by creating clinical decision rules (CDRs) and Bayesian analyses. Generally, CDRs are tools that mimic some of the presumed but unsubstantiated properties of cognitive forcing strategies—that is, if clinicians could be more systematic in their thinking, if they followed defined rules, then they would make fewer mistakes. While these tools go by various names (such as clinical decision aids, decision support tools, or cognitive scaffolds), they have been most commonly described as CDRs.12 Where such aids are supported by high-quality research, their application is thought to improve patient outcomes and optimize scarce health care resources. Increasingly, the use of and adherence to CDRs by clinicians are assumed to be a marker of quality of care.

There is a perception that greater scaffolding and support in decision making may result in better patient care. Evidence-based diagnostic CDRs are ubiquitous in medicine. For example, the Wells pulmonary embolism (PE) rule,13 the Wells deep vein thrombosis (DVT) rule,14 the Ottawa CT (computerized tomography) head rule,15 and the Ottawa ankle rule16 are widely promoted in emergency medicine. Despite access to these rules, physician practice is variable and, in some cases, CDRs do not appear to be followed.16,17 This can be perceived as a flag for suboptimal care.

Many reasons have been proposed for why physicians do not follow CDRs, including a lack of knowledge of guidelines, physician age, and physician experience.18–20 First of all, physicians, like all humans, are subject to their idiosyncratic cognitive processes. These processes can both aid and hurt their decision making. These processes also govern whether a physician may or may not apply or follow a particular guideline. Persell and colleagues have shown that nonadherence to guidelines is, in many cases, warranted because of patient-specific reasons.21 Within the context of emergency medicine, Mercuri and colleagues demonstrated that while physicians follow guidelines in typical cases, unique patient circumstances can cause the experienced physician to deviate from an available CDR.22 While these studies can provide insight into the locus of the deviation (i.e., factors related to the physician or the patient), little is known regarding the thought process of physicians when they deviate from the CDRs, or why some physicians will follow the rule and others not for similar patients (or in some cases, why a physician will follow the rule for some of her patients and not for others).

Researchers who develop CDRs have framed these failures as a knowledge transfer/deficit problem, clinician nonadherence, or an inherent adoption issue.17,23–25 However, the science of diagnostic reasoning is also at play. For instance, attempts at cognitive forcing strategies, which are touted to be a metacognitive solution for diagnostic error,4–8 have not been shown to improve diagnostic accuracy.10,11 One of the key problems, however, is that it is unclear where (or if) these CDRs fit within physician diagnostic processes.

One area of CDR development is venous thromboembolism—where multiple CDRs have been developed to help guide clinicians through their diagnostic processes for DVT and PE.13,14,26,27 While multiple studies have explored how physicians perform using these diagnostic aids in real life,17,28 to our knowledge, few studies have tried to examine how these CDRs integrate with coexisting physician cognitive processes. This is especially the case with diagnosing undifferentiated medical problems where the physician might be coordinating several diagnostic and therapeutic activities for a single patient.

The purpose of this study was to explore physicians’ diagnostic processes and to examine how physicians incorporate CDRs into their reasoning using simulated cases (with chest pain or leg pain). It is our hope that by better understanding how these cognitive scaffolds are used by experienced clinicians, strategies for implementation may be optimized to improve decision making.


In this study, we used a constructivist grounded theory approach to explore how physicians explained their diagnostic reasoning. A videorecorded encounter of a medical student interacting with a patient with chest pain or leg pain problems served as a prompt. Mindful that we would need both insider and outsider perspectives on this topic, we assembled an investigatory team that provided balanced insights into our data. The investigatory team comprised 3 clinician–researchers (T.M.C., J.S., and K.d.W.), 2 students (E.G. and M.T.), and 1 nonphysician–researcher with an interest in practice variations and medical decision making (M.M.). T.M.C., J.S., and K.d.W. are all specialist emergency physicians, while K.d.W. has additional qualifications in thrombosis medicine. J.S. and T.M.C. are both trained and experienced clinician–educators. All investigators had graduate-level training (T.M.C. and J.S. hold a master’s in education, M.M. and K.d.W. hold doctorates in research methods). E.G. and M.T. were research assistants who were both students at the time (E.G. was a second-year medical student, and M.T. was an undergraduate bachelor’s degree student). Before engaging in any analyses, our investigatory team met to discuss our inherent stances and assumptions to assist us in achieving reflexivity. Each member of the analysis team sought to declare their perceptions about CDRs and their role in decision making. We also discussed our prior training (if we had any) and the sensitizing concepts/frameworks brought to the analysis process. Sensitizing concepts included multiple frameworks, which ranged from the work of Kahneman, Slovic, and Tversky29; diagnostic scripts30–32; CDR construction12; and real-life application22 to Bayesian statistics33 and even to design thinking.34,35 We ensured these concepts remained at the forefront during interview coding and theme generation by encouraging discussions amongst ourselves about the degree to which individual clinical/research experience and prior knowledge influenced our interpretations and summaries.

Population and recruitment

We recruited a purposive sample of practicing emergency physicians of varying training backgrounds and experience level within a single city, but across 3 university-affiliated teaching hospitals (St. Joseph’s Healthcare Hamilton, Juravinski Hospital, Hamilton General Hospital) affiliated with McMaster University. Interviews occurred from August 2015 to July 2016.


Our survey was designed collaboratively and iteratively by our research team and piloted on members of the investigatory team before implementation. We sought to elicit our participants’ thinking processes during 2 phases. All interviews were conducted by our 2 research assistants (E.G. and M.T.) after a training period and practice with the senior investigatory team. In general, the interviews followed a structured process with little requirement to deviate from the script. In the first phase, we asked clinicians to recall recent critical incidents, where they decided to work up a patient for possible diagnosis of DVT and PE. We did not restrict them to cases where the final diagnosis was DVT or PE. See Supplemental Digital Appendix 1, at, for the questions and interview prompts for each phase of the study. In the second phase, our participants were asked to watch 2 of 6 possible videos. Supplemental Digital Appendix 2, at, lists all 6 scenarios. All cases were developed to be suggestive of a diagnosis associated with a common CDR, yet still maintain diagnostic uncertainty. In each video, a medical student speaks with a patient, taking a history. We then encouraged the participant to describe their clinical decision making as if they were discussing the case with the medical student. This approach was used to anchor participants to a common supervision task typical of their clinical practice environment. Participants were also asked to articulate their teaching via mind maps, although these visual artifacts were not used in our present analysis.

In both phases, we attempted to cue our participants to focus their diagnostic reasoning around chief complaints of chest pain, breathlessness, leg swelling, or leg pain. Our rationale for restricting our videos to contain these complaints was that these symptoms might be associated with diagnoses that have well-known diagnostic CDRs. For example, physicians often consider diagnoses of myocardial infarction or PE in patients with chest pain complaints, which may lead them to apply the HEART score36 or Wells PE score,13 respectively. Similarly, the complaint of leg pain or leg swelling may lead a physician to think about DVT, which is associated with the Wells DVT CDR.14 Both phases were audiorecorded and transcribed verbatim by a medical transcriptionist.

Ethical approval

Our study received approval from our institutional review board (Hamilton Integrated Research Ethics Board #15-246).


All interviews were conducted by a trained research assistant (M.T. and E.G.) to ensure that the investigators were not privy to identities of the interviewees (the clinician–investigators were colleagues of many of the participants). We analyzed the data using a constant comparative method to identify themes, initially starting with an open coding process for all of the transcripts and then moving from axial coding to group relevant themes and subthemes. Our analysis team (T.M.C., M.M., K.d.W.) met after independently reviewing a single transcript, which generated an initial list of codes. This coding scheme was then independently applied to a set of 3 transcripts. During each subsequent analysis meeting, we discussed, added, or collapsed our newly emergent codes into the coding schema. Disagreements were resolved through a consensus building process. We eventually came to consensus on all themes throughout all phases of analysis. We repeated this refinement cycle 5 more times (with a total of 7 meetings) to generate a final list of themes and subthemes that seemed to represent the data. With each additional round of analysis, we sought divergent or discrepant themes or patterns in the data. Adjustments to the interview guides were not made since we found that our initial prompts were sufficient to provide insights into the participants’ diagnostic reasoning.


A total of 16 participants were recruited from the 3 teaching hospitals associated with McMaster University during the study period. In total, there were 9 physicians with Royal College of Physicians and Surgeons of Canada training and 8 physicians with an enhanced skills certification in Emergency Medicine via the Canadian College of Family Physicians. Participants had practiced a median of 16 years (interquartile range: 6–24). During the final phase of the interview, all participants endorsed their familiarity with and use of CDRs in the clinical teaching environment. See Table 1 for a full listing of the participants’ demographic characteristics. We halted our recruitment when our analyses showed that our list of codes, themes, and subthemes reached a level of theoretical sufficiency aligned with Charmaz’s constructivist grounded theory approach.37,38 Interviews lasted between 24 and 70 minutes. Below, participants are identified by randomly assigned numbers (P1, P2, etc.) to differentiate yet preserve anonymity.

Table 1
Table 1:
Participant Demographic Characteristics, From a Study of Emergency Physician Decision Making and Clinical Decision Rules, McMaster University, 2015–2016

Diagnostic framework for patients with chest complaints

Overall, the process for thinking about undifferentiated chest pain patients incorporated many of the elements that are to be expected from experienced diagnosticians evaluating such a patient with chest pain. The 5 major themes in their explicit diagnostic process were: refinement of the differential diagnosis, ordering a hierarchy of risk, the decision to test, choosing the tests, and interpreting the test results. Figure 1 depicts how experienced clinicians incorporate CDRs into their diagnostic reasoning.

Figure 1
Figure 1:
Conceptual framework of clinician decision making with clinical decision rules, from a study of emergency physician decision making and clinical decision rules, McMaster University, 2015–2016. Abbreviations: DDx, differential diagnosis; Hx, history; Px, physical exam; CDR, clinical decision rule; CXR, chest X-ray; CT, computerized tomography.

Above all else, the clinical stability of the patient was the first and most critical juncture point for decision making, as evidenced by the following quote:

So my first thought would be is stable or unstable. It seems to me … that he is stable right now; however, I don’t have any vital signs. I would like some vital signs. I would have looked at vital signs. If he was unstable, I would move him to a monitored bed, put some oxygen on him if he required it, if he was hypoxic. (P1)

Next, participants would generate an initial differential diagnosis list, assigning more weight to deadly diagnoses. For instance, one participant stated:

I don’t really, to be honest with you, worry too much about MSK [musculoskeletal] and to a certain extent GI [gastrointestinal] just because those aren’t the things that are going to kill you. (P2)

Our clinicians constantly cycled between elaborative and divergent thinking to consider a wide list of possible diagnoses and the case-specific information from the history and examination. The physicians continued to exclude diagnoses from their differential until they had a clear diagnostic plan. Physicians thought synchronously about diagnosis and testing for the diagnosis. Consideration of acute coronary syndrome was often verbalized by the choice of ordering an electrocardiogram (ECG) and troponin test:

I would rule out ACS [acute coronary syndrome] on him as well. So, that would involve a workup that would be, that would involve troponins and ECGs. But I would send this patient once I had the creatinine, back for a CTPE [sic] because he’s high risk. So CTPE [sic], being the most important thing. (P3)

Most physicians assessed the pretest probability of disease before ordering tests; however, this was usually verbalized as a gestalt estimate (encompassing the clinical information available to them) rather than by application of a structured CDR. The differential diagnosis list was ordered into a hierarchy of risk, meaning that conditions which were most serious (albeit less likely) and those possible diagnoses with a higher perceived likelihood appeared toward the top of the list and conditions which were highly unlikely or lower in clinical severity appeared at the bottom.

We also noted at times that clinicians would engage in multiple rounds of testing, going from the easiest to the most invasive or inconvenient. Our participants might describe ordering bedside tests that quickly augment their knowledge (e.g., an ECG or chest X-ray) and then wait for those results. They would refine their differential diagnoses using these data, before engaging in further, more invasive or less convenient testing. Most importantly, we noted for our clinicians that decision making was not a static end point to them, but a dynamic process, which iteratively considers newly collected information.

Diagnostic framework for patients with leg complaints

The process for considering cases with undifferentiated leg complaints consisted of elements similar to those found in Figure 1. There was similar cyclical thinking noted for evaluating patients with leg symptoms. If the participants used a CDR, they tended to apply it once they had ordered their list of differential diagnoses. The following quote depicts some of that iterative thinking where one participant considered multiple diagnoses at once:

So, given the fact that this person presents with unilateral swollen red angry leg with a history of cancer he’s, everything about this patient sounds like a DVT again…. So, I think you need to rule out whether there’s any injuries just to be sure that wasn’t the situation identified. And take a look at the leg to see, to suggest cellulitis, although they can look exactly the same as DVT. And then I would go on more blood work investigations to rule out DVT. (P5)

Gestalt and diagnostic reasoning

The participants frequently invoked and incorporated the concept of gestalt into their diagnostic reasoning. At times, it simply contributed to their reevaluation of the differential diagnosis list, while at other times, it was the primary driver behind their decision to engage in diagnostic testing. Gestalt was used in place of a CDR by most participants, where it tended to be an estimate that the patient was at high risk of the condition. Sometimes, their gestalt impression was powerful enough to instigate initial management. One participant stated plainly:

So … if it walks like a duck, it’s likely a duck. So, you should probably … treat her, ultrasound, D-dimer her, and send her to thrombo [sic]. (P6)

Not all participants used the term gestalt but rather described their general intuition about a case. One participant stated:

I would say everything about this patient’s presentation tells me that this person is a high likelihood of having a recurrent pulmonary embolism and needs to be worked up for it…. So, everything about it says PE to me. (P5)

The physicians’ overall or gestalt impressions of the simulated cases weighed greatly on their decision making. They constantly recalibrated these impressions as new information was gathered (be it history, physical exam findings, CDR score, or diagnostic test results), reconsidering their differential diagnoses and reprioritizing them according to available information. The clinician’s gestalt seemed to be a fairly important driver for subsequent actions; that is, these overall impressions certainly intersected with decisions to order diagnostic tests, begin presumptive management, or apply a CDR or tool.

Application of CDRs

There were a number of potential CDRs that could be used by our participants in their diagnostic thinking (see Supplemental Digital Appendix 3 at As depicted in Figure 1, CDRs were often relegated to the corners of participant’s minds, optionally applied only after the diagnosis was identified as important and the decision to test had been made. If a certain diagnosis was not considered to be high in their hierarchy of risk, they would not apply a CDR.

Some physicians were so experienced with a particular rule that they intuitively knew the rule outcome. They skipped its application since they knew it would not help to rule in or rule out the diagnosis. In our study, there were several examples where the physicians correctly applied the CDR, calculated the result, and moved onto the next diagnostic step in a fluid manner. A statement by one participant highlights how this occurred for one particular rule, the PE Rule-out Criteria (PERC):

Thinking about … the scores we normally use for the, the PERC score. He already fails because he has a history of DVT, PE and you need to have all 9 criteria ruled out. So you cannot use the PERC. (P1)

Another example of this was the following: “So … she is not ‘PERC’-able so we cannot use PERC to rule her out” (P8).

Another participant stated about the PERC: I can’t use PERC because she’s on the pill, which is one of those nasty little criteria that makes it not superuseful for a lot of young women. (P7)

Meanwhile, other CDRs were sometimes viewed as cumbersome to apply. As such, the construct of the rule and the ability to recall the data required were paramount for their application of the CDR. One participant stated:

Wells score for PE, I would have to look at it again, but he has no clinical signs or symptoms of a DVT, previous DVT/PE so he’s got a score of 1.5 and doesn’t have any of the other things, like heart rate. (P4)

Many participants felt that CDRs held a prominent place for junior trainees with less experience. For example, one participant stated:

I use it because I think it’s a useful tool in teaching and getting people to address the various items that they need to have in hand to be able to make the diagnosis. You don’t want them missing anything. (P9)

Context matters: The dynamic interplay between the clinical context and decision

It was unsurprising that patient-specific factors drove participants’ decision making. Radiation exposure was important if the patient was young, female, or possibly pregnant:

He’s old enough that radiation is not going to be an issue for [his] lifetime risk, otherwise I would have considered a VQ scan … so I would probably send him for a CTPE [sic]. (P1)

Interestingly, since DVT diagnosis was perceived to involve less risk, when the diagnosis was leading in their minds, participants were less to likely attempt to determine the immediate diagnosis. Instead they would presumptively diagnose and arrange for rapid diagnostic testing in the future (e.g., providing overnight anticoagulation and arranging an ultrasound on the next day).

Our cases brought out considerations about the interplay between the clinician’s diagnostic processes and the resources available to them. With the leg pain cases, clinicians frequently mentioned how their decision making was driven by their practice context:

So, I always do the Wells score and then decide whether I’m doing a [D-]dimer or not. Particularly at the urgent care because ultrasound shuts down at a certain time of day. In this particular case, I sent off the [D-]dimer at the same time as I ordered the ultrasound. Because if the ultrasound comes back negative and the [D-]dimer is positive, then I can organize the repeat and kind of go from there. But normally, if I’m not in that sort of time crunch at the end of the day, then I would just wait for the [D-]dimer to come back. (P14)

I’m undecided on the D-dimer or not. It depends a lot on things like the time of day. It depends on how busy the department is because … I’m trying to think what that would change for me, I probably would just send him to the scanner [sic]. (P4)

Depending on which type of hospital they were working in (teaching versus nonteaching), the protocol for investigations might vary based on the trust and respect of consulting radiologists.

If I was working at [Hospital 1], I would just get the CT and they’d say “that’s fine, thank you” because I’m dealing more with people who are experienced to understand how the world works as opposed to more junior people who feel like you have to go through it in this algorithmic way. (P3)


We have described emergency physicians’ diagnostic processing and how this interacts with CDRs. The diagnostic decision-making process is iterative, oscillating between potential diagnoses and case-specific information, which is consistent with prior conceptual literature.39,40 CDRs were not uniformly used, played no role in refining or ordering the differential diagnosis, and seldom factored prominently into the decision to test for a condition. Finally, contextual factors weighed heavily on their considerations for diagnostic testing.

Physician diagnostic reasoning and CDRs

Using a qualitative approach, we developed a model that describes the diagnostic reasoning process of an emergency physician. Within this model, we identified the following critical steps: refinement of the differential diagnosis list; ordering of the differential list into a hierarchy of risk; and finally, a decision to test for a specific set of conditions. Iterative oscillation of thought persisted throughout history taking, examination, and rapid testing, culminating with a short list of diagnoses for which to test. This list was unlikely to include all potential causes for the symptoms, but uniformly included rare causes that were considered deadly. There was an emphasis on serious diagnoses with very low pretest probabilities, implying that value is placed on testing for conditions that are highly unlikely to exist when the disease could cause death, or lead to significant morbidity. This is an important finding as evidence-based medicine often leans on Bayesian reasoning (i.e., inattention to very low probability diagnoses) to reach a final diagnosis. The complexity of handling multiple concurrent differential diagnoses at once can result in highly complicated mental calculus.

We observed a certain tension between clinician reasoning and CDRs, which is not surprising because CDRs are developed to report the presence or absence of one specific disease.41,42 Many emergency department patients have undifferentiated problems, where multiple diagnoses are competing for the physician’s consideration at once. CDRs do not aid these physicians to compare the pretest probabilities of multiple diseases at once. For example, one CDR may provide guidance as to whether you should arrange a CT scan to rule out a PE, but the same CDR cannot assist with determining if you need to order a CT scan to rule out an aortic dissection or a troponin assay to diagnose acute coronary syndrome.

The concept of a clinician’s gestalt was featured prominently within the transcripts, more so than the use of structured decision rules. Additionally, our participants tended to consider both the most likely diagnosis and excluding other dangerous diagnoses in tandem. In many ways, the participants’ transcripts resembled the processes seen in expert decision makers such as firefighters and military commanders known as recognition-primed decision making (RPDM).43 Previously studied by Klein, RPDM is a form of expert decision making that links certain scenarios to previously rehearsed responses (i.e., a certain cue or diagnosis would elicit a specific, previously learned response or management plan). For instance, encountering a 50-year-old patient with chest pain will trigger a diagnostic pathway (e.g., history, physical examination, laboratory and other diagnostic tests) in a different way from a 15-year old with a similar chief complaint. CDR application was triggered in a similar way; physicians responded to specific (and sometimes idiosyncratic) cues to prompt if a CDR was used.

Many of our participants championed the educational merit of CDRs. They suggested that CDRs may help novice clinicians to prioritize key historical or physical exam data points for a particular diagnosis and help to scaffold early diagnostic recognition of exemplars. However, CDRs had lower perceived value for experienced physicians. For our participants, CDRs seem to provide little extra beyond the numerous exemplars, past experiences, and pattern recognition process that are hallmarks of experienced reasoning or gestalt.

Contextual factors and their effect on reasoning

Our findings suggest that emergency physician diagnostic reasoning and use of CDRs in that reasoning is influenced by factors other than pre- and post-test probability. This could explain the divergence between evidence and practice. For instance, the participants in our study often applied several CDRs and investigatory tests to a single case, all at varying stages of their diagnostic process. However, since they anticipated that different tests results would appear in a staggered fashion, the participants iteratively ordered diagnostic tests based on the ease of data acquisition. For example, they would order a chest X-ray to first rule out a pneumonia and pneumothorax because it is easy to acquire, could be done promptly, and could rule out multiple diagnoses. Contrast that with a D-dimer test, which requires a nurse to draw and send the blood to the laboratory, where technicians analyze the blood, and then the lab result might only tell you that you need to order a CT. The D-dimer results in a more drawn-out process. Multiple participants voiced the desire to skip simply to a CT, which could provide more information and potentially and a definite answer to their clinical question.

We encountered several themes about modifying agents. Contextual factors such as the patient’s medical history and environmental idiosyncrasies (e.g., hospital-specific resources) also affected their diagnostic decisions. This concept is not new, and there is evidence to suggest that the gestalt of experienced clinicians may lead to similar diagnostic outcomes when compared with the use of a structured tool.44

Challenges with teaching the application of CDRs in the real world

We hope that by outlining the tensions between the way physicians think about multiple diagnoses at once and the dynamic nature of their decision-making processes, we might facilitate improved methods to help with clinical decision support. Our findings have important implications for how we can represent our diagnostic thinking to trainees, as well as for knowledge translation on new decision-aid tools. Researchers have sometimes framed failure to use a CDR as a knowledge deficit, a knowledge transfer problem, or an adoption issue.17,23–25 However, the act of applying a CDR at the bedside may be much more complex. For example, research suggests that patient-specific factors (such as if the patient is a professional footballer or there is potential legal action) can shift clinicians away from adherence to CDRs.22 Similarly, contextual variations such as increased patient volumes have been shown to affect decision making.45

Independently, neither cognitive debiasing strategies nor CDRs have appeared to achieve their stated aims of reducing error. In practice, neither needs to function independently; but to date, how physicians reconcile these 2 approaches has not been well explored. While it is possible that individual physicians choose to adopt one strategy over another, it is equally possible some attempt to bridge between them. Understanding this is essential if we hope to further reduce diagnostic error—whether that be by creating easier tools to adopt and use or by teaching trainees and physicians to avoid error via metacognition.


The nonclinical setting of our observations may constitute a limitation of our findings. While it is recognized that individuals who are thinking aloud cannot represent their unconscious or nonanalytical reasoning, the “teach aloud” technique is adapted from a well-accepted approach (“think aloud” protocols) for understanding cognition.46 In teaching hospitals, it is common practice to break down and articulate the diagnostic process for trainees, especially medical students. As such, we believe our findings may approximate clinical practice. Another key limitation is that our particular center is an avid research environment with some of the leading experts in thrombosis research, which may have affected participant knowledge and local culture. The Wells score was derived in our city, 2 decades ago. Interestingly, one would assume that since we conducted this study in a center known for its allegiance to evidence-based medicine and CDR, we could expect optimal conditions with regard to uptake of these rules. Thus, if CDRs are not well incorporated here, then one might expect even more resistance to CDRs in other centers where knowledge translation is not as strong.

Our current sampling strategy did not enable us to explore differences between subgroups of clinicians at various levels; there may be variations in practice based on the timing at which the clinicians learned decision support tools (e.g., CDRs, other cognitive scaffolds). Our interviews did not include asking the participating physician to write their medical records, and we did not explore how CDRs are documented. Finally, we designed the cases so participating physicians would test all patients for venous thrombosis. We did not include contentious cases where some physicians might test and others not, so we cannot draw conclusions about the role of CDRs in such cases.

Future directions

Better understanding of the cognitive processes underlying physician’s thinking may allow the creation of improved tools to reduce diagnostic error, and in particular, those tools that are amenable to complex, multipatient care environments (e.g., the emergency department). Based on our exploratory work, new clinical decision support techniques that could help clinicians consider multiple, concurrent diagnoses by applying multiple algorithms or CDRs at once may be more useful than static algorithms that practitioners use only fleetingly during the diagnostic process.


CDRs assume a physician’s static, linear model of clinical decision making. Yet, our findings suggest a mismatch between the purpose of CDRs and the diagnostic processes of experienced physicians. Experienced emergency physicians engage in dynamic, iterative, decision-making processes. Physicians tended to use 3 key steps: refinement of the differential diagnosis list; ordering of the differential list into a hierarchy of risk; and finally, a decision to test for a specific set of conditions. CDRs seldom factored into the decision to test for a diagnosis. Future research should focus on incorporating the iterative process that physicians use when considering multiple diagnoses relevant to a presenting complaint.


The authors would like to thank all participants for generously donating their time and expertise in this study.


1. Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in medicine. Am J Med. 2008;121(suppl 5):S2–23.
2. Graber ML, Kissam S, Payne VL, et al. Cognitive interventions to reduce diagnostic error: A narrative review. BMJ Qual Saf. 2012;21:535–557.
3. Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med. 2003;78:775–780.
4. Croskerry P. Cognitive forcing strategies in clinical decisionmaking. Ann Emerg Med. 2003;41:110–120.
5. Croskerry P. Clinical cognition and diagnostic error: Applications of a dual process model of reasoning. Adv Health Sci Educ Theory Pract. 2009;14(suppl 1):S27–S35.
6. Croskerry P. The cognitive imperative: Thinking about how we think. Acad Emerg Med. 2000;7:1223–1231.
7. Graber M. Metacognitive training to reduce diagnostic errors: Ready for prime time? Acad Med. 2003;78:781.
8. Croskerry P. From mindless to mindful practice—Cognitive bias and clinical decision making. N Engl J Med. 2013;368:2445–2448.
9. Kahneman D. Thinking, Fast and Slow. 2013.New York, NY: Farrar, Strous and Giroux;
10. Monteiro SD, Sherbino JD, Ilgen JS, et al. Disrupting diagnostic reasoning: Do interruptions, instructions, and experience affect the diagnostic accuracy and response time of residents and emergency physicians? Acad Med. 2015;90:511–517.
11. Sherbino J, Yip S, Dore KL, Siu E, Norman GR. The effectiveness of cognitive forcing strategies to decrease diagnostic error: An exploratory study. Teach Learn Med. 2011;23:78–84.
12. McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Users’ guides to the medical literature: XXII: How to use articles about clinical decision rules. Evidence-based medicine working group. JAMA. 2000;284:79–84.
13. Wells PS, Anderson DR, Rodger M, et al. Excluding pulmonary embolism at the bedside without diagnostic imaging: Management of patients with suspected pulmonary embolism presenting to the emergency department by using a simple clinical model and d-dimer. Ann Intern Med. 2001;135:98–107.
14. Wells PS, Anderson DR, Rodger M, et al. Evaluation of D-dimer in the diagnosis of suspected deep-vein thrombosis. N Engl J Med. 2003;349:1227–1235.
15. Stiell IG, Wells GA, Vandemheen K, et al. The Canadian CT Head Rule for patients with minor head injury. Lancet. 2001;357:1391–1396.
16. Stiell I, Wells G, Laupacis A, et al. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries. Multicentre ankle rule study group. BMJ. 1995;311:594–597.
17. Stiell IG, Bennett C. Implementation of clinical decision rules in the emergency department. Acad Emerg Med. 2007;14:955–959.
18. Cabana MD, Rand CS, Becher OJ, Rubin HR. Reasons for pediatrician nonadherence to asthma guidelines. Arch Pediatr Adolesc Med. 2001;155:1057–1062.
19. Cochrane LJ, Olson CA, Murray S, Dupuis M, Tooman T, Hayes S. Gaps between knowing and doing: Understanding and assessing the barriers to optimal health care. J Contin Educ Health Prof. 2007;27:94–102.
20. Choudhry NK, Fletcher RH, Soumerai SB. Improving patient care systematic review: The relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142:260–273.
21. Persell SD, Dolan NC, Friesema EM, Thompson JA, Kaiser D, Baker DW. Frequency of inappropriate medical exceptions to quality measures. Ann Intern Med. 2010;152:225–231.
22. Mercuri M, Sherbino J, Sedran RJ, Frank JR, Gafni A, Norman G. When guidelines don’t guide: The effect of patient context on management decisions based on clinical practice guidelines. Acad Med. 2015;90:191–196.
23. Pathman DE, Konrad TR, Freed GL, Freeman VA, Koch GG. The awareness-to-adherence model of the steps to clinical guideline compliance. The case of pediatric vaccine recommendations. Med Care. 1996;34:873–889.
24. Gaddis GM, Greenwald P, Huckson S. Toward improved implementation of evidence-based clinical algorithms: Clinical practice guidelines, clinical decision rules, and clinical pathways. Acad Emerg Med. 2007;14:1015–1022.
25. Reilly BM, Evans AT. Translating clinical research into clinical practice: Impact of using prediction rules to make decisions. Ann Intern Med. 2006;144:201–209.
26. Kline JA, Courtney DM, Kabrhel C, et al. Prospective multicenter evaluation of the pulmonary embolism rule-out criteria. J Thromb Haemost. 2008;6:772–780.
27. Righini M, Van Es J, Den Exter PL, et al. Age-adjusted D-dimer cutoff levels to rule out pulmonary embolism: The ADJUST-PE study. JAMA. 2014;311:1117–1124.
28. Ong CW, Malipatil V, Lavercombe M, et al. Implementation of a clinical prediction tool for pulmonary embolism diagnosis in a tertiary teaching hospital reduces the number of computed tomography pulmonary angiograms performed. Intern Med J. 2013;43:169–174.
29. Kahneman D, Slovic P, Tversky A. Judgment Under Uncertainty: Heuristics and Biases. 1982.Cambridge, UK: Cambridge University Press;
30. Charlin B, Tardif J, Boshuizen HP. Scripts and medical diagnostic knowledge: Theory and applications for clinical reasoning instruction and research. Acad Med. 2000;75:182–190.
31. Schmidt HG, Norman GR, Boshuizen HP. A cognitive perspective on medical expertise: Theory and implication. Acad Med. 1990;65:611–621.
32. Charlin B, Boshuizen HP, Custers EJ, Feltovich PJ. Scripts and clinical reasoning. Med Educ. 2007;41:1178–1184.
33. Lindgaard G, Pyper C, Frize M, Walker R. Does Bayes have it? Decision support systems in diagnostic medicine. Int J Ind Ergon. 2009;39:524–532.
34. Brown T. Design thinking. Harv Bus Rev. 2008;86:84–92, 141.
35. Gottlieb M, Wagner E, Wagner A, Chan T. Applying design thinking principles to curricular development in medical education. AEM Educ Train. 2017;1:21–26.
36. Backus BE, Six AJ, Kelder JC, et al. Chest pain in the emergency room: A multicenter validation of the HEART Score. Crit Pathw Cardiol. 2010;9:164–169.
37. Charmaz K. Denzin NK, Lincoln YS. Grounded theory in the 21st century: Applications for advancing social justice studies. In: The SAGE Handbook of Qualitative Research, 2005:3rd ed. Thousand Oaks, CA: Sage Publications; 507–536.
38. Bryant A, Charmaz K. The SAGE Handbook of Grounded Theory. 2007:Thousand Oaks, CA: Sage Publications; 229–243.
39. Renouf T, Whalen D, Pollard M, Dubrowski A. Emergency physicians think in spirals. Cureus. 2015;7:e381.
40. Roland D, Snelson E. ‘So why didn’t you think this baby was ill?’ Decision-making in acute paediatrics. Arch Dis Child Educ Pract Ed. 2019;104:43–48.
41. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA. 1997;277:488–494.
42. Stiell IG, Wells GA. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med. 1999;33:437–447.
43. Klein GA. Klein GA, Orasanu J, Calderwood R, Zsambok CE. A recognition-primed decision (RPD) model of rapid decision making. In: Decision Making in Action: Models and Methods. 1993:Norwood, NJ: Ablex; 138–147.
44. Schriger DL, Elder JW, Cooper RJ. Structured clinical decision aids are seldom compared with subjective physician judgment, and are seldom superior. Ann Emerg Med. 2017;70:338–344.e3.
45. Gorski JK, Batt RJ, Otles E, Shah MN, Hamedani AG, Patterson BW. The impact of emergency department census on the decision to admit. Acad Emerg Med. 2017;24:13–21.
46. Ericsson KA, Simon HA. Protocol Analysis: Verbal Reports as Data. 1993.Revised ed. Cambridge, MA: The MIT Press;

Supplemental Digital Content

Copyright © 2019 by the Association of American Medical Colleges