Secondary Logo

Journal Logo

Teaching Evidence-Based Medicine: Should We Be Teaching Information Management Instead?

Slawson, David C. MD; Shaughnessy, Allen F. PharmD


To encourage high-quality patient care guided by the best evidence, many medical schools and residencies are teaching techniques for critically evaluating the medical literature. While a large step forward, these skills of evidence-based medicine are necessary but not sufficient for the practice of contemporary medicine. Incorporating the best evidence into the real world of busy clinical practice requires the applied science of information management. Clinicians must learn the techniques and skills to focus on finding, evaluating, and using information at the point of care. This information must be both relevant to themselves and their patients as well as being valid. The authors discuss the need to teach the applied science of information management along with, or perhaps even instead of, teaching the basic science of evidence-based medicine. All students, residents, and practicing physicians need three skills to practice the best medicine: the ability to select foraging—“keeping up”—tools that filter information for relevance and validity, the skill to select and use a hunting—“just in time”—information tool that presents prefiltered information easily and in a quickly accessible form at the point of care, and the ability to make decisions by combining the best patient-oriented evidence with patient-centered care, placing the evidence in perspective with the needs and desires of the patient. This teaching of information management skills will prepare students and residents for a practice of medicine that requires lifelong learning.

Dr. Slawson is the B. Lewis Barnett, Jr. Professor of Family Medicine, University of Virginia School of Medicine, Charlottesville, Virginia.

Dr. Shaughnessy is at Tufts University Family Medicine Residency, and is adjunct professor of public health and family medicine at Tufts University School of Medicine, Boston, Massachusetts.

Correspondence should be addressed to Dr. Slawson, Department of Family Medicine, Box 800729, University of Virginia Health Sciences Center, Charlottesville, VA 22908; telephone: (434) 924-1146; e-mail: 〈〉.

The professionally sponsored literature for medical practitioners acts as though each practitioner in each American community were supposed to be his own scholarly and scientific institute, screening, sifting, evaluating, assessing, and translating into practical terms the output of medical research that is reported in the periodical literature …. The practitioner, of course, is quite unable to live up to this myth. For that reason, he is likely to have recourse … to those sources that are willing to offer him the digested and preselected information that meets the needs.

—Herbert Menzel, 1966

In the past ten years, two major changes have occurred in the processing of information in medicine: the widespread and easy availability of the medical research literature to both clinicians and their patients, and a push to move away from expert-led medicine to practice directed by patient-oriented, outcomes-based research. Evidence-based medicine (EBM) has become the approach developed to help clinicians manage this information, and many schools and residencies are using EBM to help their learners achieve the practice-based learning and improvement competency mandated by the Accreditation Council for Graduate Medical Education.1 However, teaching clinicians EBM techniques and then expecting them to be expert information managers may set them up for failure by providing skills that are not relevant to their day-to-day practice. This expectation, expressed 40 years ago in the opening quote,2 is likely to be just as unrealistic now.

In this article we explore current approaches to EBM and suggest that a framework of information management, built on the needs of patients rather than on the availability of evidence, offers the knowledge and skills necessary for clinicians to practice medicine that is, as called for by the Institute of Medicine, safe, effective, efficient, and, most important, patient centered.3 This change in orientation will require a new approach to teaching medical students, residents, and practicing physicians.

Back to Top | Article Outline

Evidence-Based Medicine as an Information Framework

The classic EBM approach consists of a five-step process of developing a question using the populations-intervention-comparison-outcome (PICO) format, finding research that may answer the question, evaluating the research for validity, impact, and applicability, applying the information to clinical decision making, and periodically evaluating one’s effectiveness at performing the previous four steps.4

However, a number of individuals, including the developers of the EBM approach, have outlined its problems.4–7 The more common method of information management has been called “satisficing,” whereby busy clinicians will be satisfied with the information they have at hand, sacrificing quality for convenience.8–10 For example, internal medicine residents pursued only 30% of their questions during a typical office session7 and only pursued 70% when specifically given time during their office hours to answer the questions they developed.11 Instead of striving to find the most rigorous evidence, most full-time clinicians report they do very little critical appraisal, instead relying on summaries and practice guidelines, regardless of whether these are evidence based, for information.12–14

In a clinical trial evaluating the application of classic EBM techniques, the only way practicing clinicians were able to answer their questions at the point of care in the real world was to rely on predigested information, such as that provided by abstracting services, or previously self-developed evidence-based summaries.15 Indeed, in the absence of guidelines, the cost of simply obtaining the original research articles can be significant, estimated to be, in one typical example, 432.00 Irish pounds (approximately 791 U.S. dollars) in interlibrary loan costs to answer just one question.16

Since basic EBM skills cannot be applied at the bedside, the use of critical appraisal has been recommended only for select questions encountered in one’s practice, based on frequency.4 For common problems, the EBM approach suggests that each individual clinician, working independently or, at best with a small group, should set aside one or two hours per week to critically appraise all the existing primary literature and/or systematic reviews and come up with his or her own conclusions regarding its validity and relevance.17

Continuing in this approach, clinicians should rely on synthesized information sources for a predigested, evidence-based answer to their questions for diagnoses or issues infrequently encountered in their practices. For problems that occur very infrequently, clinicians should “‘blindly’ seek, accept, and apply the recommendations we receive from authorities in the relevant practice of medicine.”4

In this frequency-based approach to information-seeking, critical appraisal skills are only used in a somewhat intermittent manner to answer common questions in practice and are not used during the care of patients. This approach presupposes and requires that each individual clinician has sufficient critical appraisal skills and the ability to choose the right article(s) to evaluate, the confidence in these skills to rely on the generated answer, the tools or ability to recall the information when needed, and, perhaps most important, the courage to base practice on this self-constructed foundation.

This is not a patient-centered approach. To the patient, the frequency of the problem they currently have is 100% and thus of paramount importance and deserving of the greatest amount of intellectual rigor his or her clinician can supply. There is also a logical inconsistency in this approach: by relying on recommendations of experts for unusual problems in medicine, if the disease is uncommon in the clinician’s practice, it is less likely that the clinician will receive updates, making it even more likely that he or she will have incorrect information.

This method also requires clinicians in practice to stay current, not only with clinical content but also with changes in critical appraisal techniques. For example, the value of concealed allocation when conducting prospective research has only recently entered the EBM literature.18,19 For the evaluation of guidelines or review articles, the Strength of Recommendation Taxonomy (SORT), which includes relevance as well as validity assessment, has only just been introduced.20 Clinicians unaware of these evaluation criteria would be using out-of-date methods while trying to keep up to date!

Back to Top | Article Outline

Patient-Centered Information Management

Our focus on information management grew out of frustration with the limited ability of EBM to meet the needs of clinicians in active practice. Information management focuses on the usefulness of information to patient care, defined as the following:21

Usefulness of any information source =

Relevance of information is defined in terms of its direct applicability to the care of patients and focuses on three qualifications:

  • Does the information focus on outcomes patients care about? That is, will the information help clinicians assist patients to live longer lives, better lives, or both?
  • Is the intervention or practice feasible and is the problem addressed common in one’s clinical practice?
  • Would the information, if true, require a change in one’s clinical practice?

The information should show that an intervention helps patients live longer and better, is feasible to implement, and would require a change in a clinician’s practice. We have coined the term “Patient-Oriented Evidence that Matters” (POEM) to characterize new research findings that meet these criteria.5

Validity is the technical rigor that is the focus of EBM, and work can be defined in terms of the time, money, or effort required to obtain an answer to a clinical question.

It is not the frequency of the clinical problem that decides the approach to the literature, it is several other factors: the clinician’s awareness that new information is available, the feeling that current options are unacceptable or didn’t work, the availability of familiar sources, the perceived likelihood that an answer can be found, the fear of liability if the correct approach is not followed, and the time available to search for the answer.22–24 In a study of 103 physicians with a total of 1,101 questions during clinical practice, the physicians pursued answers to only 36% of their questions. In attempting to answer their questions, half of the participants spent less than 60 seconds before either finding their answer or quitting their search.23

Information management focuses on using currently available information tools to remain up to date with new valid information that is relevant to the care of patients and is accessible while taking care of patients. These information tools can be divided into “foraging tools” that clinicians can used to be alerted to new, relevant, and valid information, and “hunting tools” that allow clinicians to find that information again when they need it.25

Both tools are required for effective practice. The best of these tools provide information that is filtered for relevance to clinical practice, is critically appraised for validity using EBM techniques, and is presented in a style that is easily grasped by busy clinicians, which greatly reduces the amount of work expended to obtain the best information.24List 1 outlines criteria for foraging and hunting tools.

List 1 Criteria for High-Quality Hunting and Foraging Tools

List 1 Criteria for High-Quality Hunting and Foraging Tools

Foraging tools, or current awareness services such as Daily POEMs,26 Journal Watch,27 and others, are available to alert clinicians to new information that should influence their care of patients. However, information obtained in this way rarely results in the clinician’s learning more than simply that the actual information exists (life would be so much easier if we could read something once, reflect on it, and then remember it flawlessly when it is needed). Thus, a hunting tool is needed to retrieve relevant and valid information quickly when it is required in the care of patients. A number of hunting tools exist, but most do not provide an answer in less than a minute, and only a few such as Dynamed28 and InfoRetriever26 have both specific patient-oriented evidence criteria and validity ratings (levels of evidence) for included information.

Back to Top | Article Outline

New Skills Needed

With this focus on information management, rather than simply the basic tenets of critical appraisal, clinicians can focus on understanding, interpreting, and applying the information in their own clinical situation. Good information tools will provide a variety of levels of information, where available, to suit the individual needs of the clinician. Several investigators have had success teaching information management using point-of-care tools that provide prefiltered, prevalidated information to medical students.29–31

Not all sources provide the same degree of relevant and valid information. Information in newsletters often is not complete,32 current awareness services vary widely in their presentation of patient-oriented information33 and review articles often selectively omit or skew crucial information that should affect the care of patients.34 Clinicians need to be armed with the skills to identify sources of accurate and relevant information at the point of care. These are not information evaluation skills but information management skills.

We have created and tested a curriculum to teach the principles of information management.35 The curriculum has three levels:

▪ Level 1 is for clinicians who can use the concepts to make better patient-care decisions;

▪ Level 2 is for teachers and writers who teach clinicians the curriculum and provide evidence-based reviews of original research; and

▪ Level 3 is for researchers who are adept at conducting decision analysis, meta-analysis, and other techniques of synthesizing raw research information into useable clinical information.

List 2 presents the modules in this curriculum, for each level.

List 2 Modules in the Three Levels of an Information Mastery Curriculum Developed by the Authors

List 2 Modules in the Three Levels of an Information Mastery Curriculum Developed by the Authors

At Level 1, the goal of the curriculum is to elevate all clinicians to a level of information management proficiency whereby they can recognize, obtain, and use the highest-quality information available for everyday clinical decision making. In contrast, only a fraction of clinicians from each specialty need detailed training in critical analysis of the original literature, and even fewer need training in Level 3 activities.

The specifics of the curriculum are outlined in print and online resources,35–37 and workshops are regularly presented.38 The curriculum introduces students to the concepts, prepares residents to meet the practice-based learning and improvement competency,1 and equips practicing clinicians for aspects of the maintenance of certification39 process. It also prepares physicians to take advantage of the newly developed point-of-care continuing medical education credit

Back to Top | Article Outline

Skill #1: Select tools for “keeping up”

These are tools that provide new information filtered for relevance and validity using explicit criteria. There are various tools that forage through new research information—newsletters, e-mail sources, and others—though these information sources vary in relevance and validity of the information they present. Clinicians must be able to distinguish “news” (which is simply anything they didn’t know yesterday) from valid patient-oriented evidence that matters to them and to their patients. Foraging tools should be tailored to the specialty and practice of each clinician.25 They should be comprehensive, have a process that is transparent and reproducible to evaluate relevance and validity, and be coordinated with a high-quality hunting tool.

Back to Top | Article Outline

Skill #2: Select the appropriate hunting tool

The clinician should have the skill to select a hunting tool, accessible at the point of care, that presents information prefiltered for relevance, preappraised for validity using explicit criteria, marked with a level of evidence, and placed in the clinical context of the user.40 “Just-in-time”41 hunting tools are available that reduce work by using decision support tools, calculators, and other means to make information more accessible and easier to use.

Back to Top | Article Outline

Skill #3: Develop patient-centered, not evidence-centered, decision making

The goals of medicine should be to relieve or prevent suffering,42,43 to maintain or provide hope, and to prevent, treat, or cure disease. EBM can address well only the third goal. Thus, evidence is there only to factor into the clinical decision; it is not the decision itself. The patient-centered model of care includes consideration of the patient as a person along with seeing the patient in the standard biomedical model as a person with a disease.44 The clinical decision is based on combining the best patient-oriented evidence with patient-centered care in the mind of the physician, who puts the evidence in perspective with the needs and desires of the patient.

Back to Top | Article Outline

Teaching Applied Information Management

Several researchers have shown that clinicians infrequently approach the medical literature for answers, instead relying on their own background knowledge or intuition (i.e., guessing).11,23,45,46 To keep pressing the standard EBM approach is like asking everyone who wishes to make a cake to buy the wheat to grind their own flour, refine their own sugar, and extract their own flavorings before ever starting the cake. Cakes aren’t made this way.

It is not realistic to expect clinicians to develop a “searchable” question, find the information, evaluate it, and make a decision based on it, all at the point of care with an office or hospital ward full of patients. This “high, hard ground” of theory and technique contrasts sharply with the “swampy lowland” of typical clinical practice, where corners have to be cut, an answer—any answer—must be found, and relevance is valued over rigor.33,47 We have the technology and the tools available to allow clinicians to search for the type of information they choose, knowing that the source has filtered out everything already except information that is both relevant and valid. EBM needs to be seen as a necessary but not sufficient basic form of knowledge upon which information management can be practiced. Information mastery, in contrast, is the applied science that allows clinicians to harness resources in the information age.

Back to Top | Article Outline


1ACGME Outcomes Project. General competencies 〈〉. Accessed 9 April 2005.
2Menzel H. Sociological perspectives on the information-gathering practices of the scientific investigator and the medical practitioner. In: McCord D (ed). Bibliotheca Medica: Physician for Tomorrow. Boston: Harvard Medical School, 1966:127–28.
3Committee on Quality Health Care in America, Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st century. Washington, DC: National Academy Press, 2001:5–6.
4Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-Based Medicine. How to Practice and Teach EBM. New York: Churchill Livingstone, 2000, 3–6.
5Shaughnessy AF, Slawson DC, Bennett JH. Becoming an information master: a guidebook to the medical information jungle. J Fam Pract. 1994;39:489–99.
6Grad R, Macaulay AC, Warner M. Teaching evidence-based medical care: description and evaluation. Fam Med. 2001;33:602–6.
7Schilling LM, Steiner JF, Lundahl K, Anderson RJ. Residents’ patient-specific clinical questions: opportunities for evidence-based learning. Acad Med. 2005;80:51–56.
8Simon HA. Models of Man, Social and Rational: Mathematical Essays on Rational Human Behavior in a Social Setting. New York: Wiley, 1957.
9Carter BS, Leuthner SR. Decision making in the NICU—strategies, statistics, and “satisficing”. Bioethics Forum. 2002;18:7–15.
10Gigernenzer G, Todd PM. Simple Heuristics that Make Us Smart. New York: Oxford University Press, 1999.
11Green ML, Ciampi MA, Ellis PJ. Residents’ medical information needs in clinic: are they being met? Am J Med. 2000; 109:218–23.
12Riordan FAI, Boyle EM, Phillips B. Best paediatric evidence: is it accessible and used on-call? Arch Dis Child. 2004;89:469–71.
13McColl A, Smith H, White P, Field J. General practitioners’ perceptions of the route to evidence based medicine: a questionnaire survey. BMJ. 1998;316:361–65.
14Putman W, Twohig PL, Burge FI, Jackson LA, Cox JL. A qualitative study of evidence in primary care: what the practitioners are saying. CMAJ. 2002;166:1525–30.
15Sackett DL, Straus SE. Finding and applying evidence during clinical rounds: the “evidence cart”. JAMA. 1998;280:1336–38.
16Sloane PA, Brazier H, Murphy AW, Collins T. Evidence based medicine in clinical practice: how to advise patients on the influence of age on the outcome of surgical anterior cruciate ligament reconstruction: a review of the literature. Br J Sports Med. 2002;36:200–3.
17Oxman AD, Sackett DL, Guyatt GH. Users’ guides to the medical literature. I. How to get started. JAMA. 1993;270:2093–95.
18Schulz KF, Chalmers I, Hayes RJ, et al. Subverting randomization in controlled trials. JAMA. 1995;273:408–12.
19Schulz KF, Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet. 2002;359:614–18.
20Ebell MH, Siwek J, Weiss BD, et al. Strength of recommendation taxonomy (SORT): a patient-centered approach to grading evidence in the medical literature. Am Fam Physician. 2004;69:548–56.
21Shaughnessy AF, Slawson DC, Bennett JH. Becoming an information master: a guidebook to the medical information jungle. J Fam Pract. 1994;39:489–99.
22Green ML, Ciampi MA, Ellis PJ. Residents’ medical information needs in clinic: are they being met? Am J Med. 2000; 109:218–23.
23Ely JW, Osheroff JA, Ebell MH, et al. Analysis of questions asked by family doctors regarding patient care. BMJ. 1999;319:358–61.
24Ebell MH, Shaughnessy AF. Information mastery: integrating continuing medical education with the information needs of clinicians. J Cont Ed Health Prof. 2003;23:S53–62.
25Shaughnessy AF, Slawson DC. Are we providing doctors with the training and tools for lifelong learning? BMJ 1999;13:1280 〈〉. Accessed 9 April 2005.
26InfoPOEMs, Inc. home page 〈〉. Accessed 9 April 2005.
27JournalWatch Online 〈〉. Accessed 9 April 2005.
28Dynamed home page 〈〉. Accessed 9 April 2005.
29Helwig AL, Flynn D. Using palmtop computers to improve students’ evidence-based decision making. Acad Med. 1998;73:603–4.
30Leung GM, Johnston JM, Tin KYK, et al. A cluster randomised trial of clinical decision support tools to improve evidence-based medicine learning in medical students. BMJ. 2003;327:1090.
31Johnston JM, Leung GM, Tin KYK, et al. Evaluation of a handheld clinical decision support tool for evidence-based learning and practice in medical undergraduates. Med Educ. 2004;38:628–37.
32Shaughnessy AF, Schlicht JR, Vanscoy GJ, Merenstein JH. Survey and evaluation of newsletters marketed to family physicians. J Am Board Fam Pract. 1992;5:573–79.
33Slawson DC, Shaughnessy AF, Barry H. Which should come first: rigor or relevance? J Fam Pract. 2001;50:209–10.
34Shaughnessy AF, Slawson DC. What happened to the valid POEMs? A survey of review articles on the treatment of type 2 diabetes. BMJ. 2003;327:266–69.
35Slawson DC, Shaughnessy AF. Teaching information mastery: creating informed consumers of medical information. J Am Board Fam Pract. 1999;12:444–49.
36Rosser WW, Slawson DC, Shaughnessy AF. Information Mastery: Evidence-Based Family Medicine. 2nd ed. Hamilton, Ontario, BC: Decker Inc, 2004.
37Center for Information Mastery, University of Virginia 〈〉. Accessed 9 April 2005.
38InfoPOEMs, Inc. Events. 〈〉. Accessed April 9, 2005.
39American Board of Medical Specialties Maintenance of Certification 〈〉. Accessed 9 April 2005.
40Smith R. What information do doctors need? BMJ. 1996;313:1062–67.
41Chueh H, Barnett GO. “Just-in-time” clinical information. Acad Med. 1997;72:512–17.
42Cassel EJ. The nature of suffering and the goals of medicine. N Engl J Med. 1982;306:639–45.
43Cassell EJ. Diagnosing suffering: a perspective. Ann Intern Med. 1999;131:531–34.
44Steward M, Brown JB, Weston WW. et al. Patient-Centered Medicine. Transforming the Clinical Method. Thousand Oaks, California: Sage Publications, 1995.
45Curley SP, Yates JF, Young MJ. Seeking and applying diagnostic information in a health care setting. Acta Psychol (Amst). 1990;73:211–23.
46Curley SP, Connelly DP, Rich ED. Physicians’ use of medical knowledge resources: preliminary theoretical framework and findings. Med Decis Making. 1990;10:231–41.
47Schön DA. The Reflective Practitioner. How Professionals Think in Action. New York: Basic Books, 1983:42.
Back to Top | Article Outline

Did You Know?

The first human gene therapy trial for cystic fibrosis was conducted in 1993 by researchers at New York Presbyterian Hospital and Joan & Sanford I. Weill Medical College of Cornell University.

For other important milestones in medical knowledge and practice credited to academic medical centers, visit the AAMC’s “Discoveries and Innovations in Patient Care and Research Database” at 〈〉.

© 2005 Association of American Medical Colleges