Secondary Logo

Journal Logo

Searching for Answers to Clinical Questions Using Google Versus Evidence-Based Summary Resources

A Randomized Controlled Crossover Study

Kim, Sarang, MD; Noveck, Helaine, MPH; Galt, James, EdM; Hogshire, Lauren, MD; Willett, Laura, MD; O’Rourke, Kerry, MLS

doi: 10.1097/ACM.0000000000000244
Research Reports
Free

Purpose To compare the speed and accuracy of answering clinical questions using Google versus summary resources.

Method In 2011 and 2012, 48 internal medicine interns from two classes at Rutgers University Robert Wood Johnson Medical School, who had been trained to use three evidence-based summary resources, performed four-minute computer searches to answer 10 clinical questions. Half were randomized to initiate searches for answers to questions 1 to 5 using Google; the other half initiated searches using a summary resource. They then crossed over and used the other resource for questions 6 to 10. They documented the time spent searching and the resource where the answer was found. Time to correct response and percentage of correct responses were compared between groups using t test and general estimating equations.

Results Of 480 questions administered, interns found answers for 393 (82%). Interns initiating searches in Google used a wider variety of resources than those starting with summary resources. No significant difference was found in mean time to correct response (138.5 seconds for Google versus 136.1 seconds for summary resource; P = .72). Mean correct response rate was 58.4% for Google versus 61.5% for summary resource (mean difference −3.1%; 95% CI −10.3% to 4.2%; P = .40).

Conclusions The authors found no significant differences in speed or accuracy between searches initiated using Google versus summary resources. Although summary resources are considered to provide the highest quality of evidence, improvements to allow for better speed and accuracy are needed.

Dr. Kim is clinical associate professor, Department of Medicine, Rutgers University Robert Wood Johnson Medical School, New Brunswick, New Jersey.

Ms. Noveck is research teaching specialist, Department of Medicine, Rutgers University Robert Wood Johnson Medical School, New Brunswick, New Jersey.

Mr. Galt is curriculum development/instructional design specialist, Robert Wood Johnson Library of the Health Sciences, Rutgers University, New Brunswick, New Jersey.

Dr. Hogshire is clinical instructor, Department of Medicine, Rutgers University Robert Wood Johnson Medical School, New Brunswick, New Jersey.

Dr. Willett is associate professor, Department of Medicine, Rutgers University Robert Wood Johnson Medical School, New Brunswick, New Jersey.

Ms. O’Rourke is library director, Robert Wood Johnson Library of the Health Sciences, Rutgers University, New Brunswick, New Jersey.

Funding/Support: None reported.

Other disclosures: None reported.

Ethical approval: This study was approved by the Rutgers University institutional review board on March 11, 2013.

Previous presentations: This study was presented as a short communication at the Northeast Group on Educational Affairs annual meeting on April 13, 2013, in New York, New York.

Correspondence should be addressed to Dr. Kim, CAB Suite 2300, 125 Paterson St., New Brunswick, NJ 08901; telephone: (732) 235-7253; fax: (732) 235-7144; e-mail: sarang.kim@rutgers.edu.

One of the core competencies residents must demonstrate on completion of residency training, according to the Accreditation Council for Graduate Medical Education, is the ability to locate, appraise, and assimilate evidence from scientific studies related to patients’ health problems.1 To address this competency, most residency training programs teach evidence-based medicine.2,3 In actual practice and especially at the point of care, however, locating, appraising, and assimilating evidence to answer clinical questions is limited by lack of time and difficulty of access. Prior studies suggest that clinicians never pursue answers for the majority of questions and, when they do, they generally spend less than two minutes searching for answers.4,5

To address the challenges in accessing and appraising the medical literature, publishing houses such as Elsevier, Wiley, and EBSCO Industries have developed evidence-based summary resources to provide clinicians with accurate and timely information. These summary resources, which systematically identify, evaluate, and integrate the best available evidence to provide a comprehensive overview of a given problem, are considered the best resources for answering clinical questions at the point of care. They contain summaries for a range of topics, which can be viewed alphabetically or by entering a search term, and teachers of evidence-based medicine (EBM) have increasingly emphasized their use.6 Despite this, many health care professionals commonly use nonmedical search engines such as Google.7–9

Because Google links to a wide variety of resources, including those that are not peer reviewed, without filtering by level of evidence, concerns have been raised over clinicians using that information to make decisions about patient care. Studies comparing the accuracy of Google and other resources have produced conflicting results, but these studies have had methodological flaws, such as nonrandomized design or inappropriate comparisons (most commonly to PubMed, a large bibliographic database not intended for use at the point of care).9–13

The objectives of our study were to compare the speed and accuracy of searches done through Google versus the summary resources, and to describe where answers were ultimately found when searches began with Google versus a summary resource. We hypothesized that use of summary resources would result in a higher percentage of correct answers with a faster time to correct response.

Back to Top | Article Outline

Method

For this randomized controlled crossover study, we enrolled two classes of interns (2011–2012 and 2012–2013) at the Rutgers University Robert Wood Johnson Medical School internal medicine residency program. All interns take a two-week EBM course that consists of four 2-hour workshops on critical appraisal of original studies and a 60-minute interactive session with a medical librarian using three summary resources available through the university’s library: EBSCO’s DynaMed, Wiley’s Essential Evidence Plus, and Elsevier’s FirstConsult.

At the end of the EBM course, the interns in our study took a multiple-choice test in a computer lab using desktop computers. The test presented 10 clinical vignettes, and we assessed the questions (selected from the Medical Knowledge Self-Assessment Program for Students,14 a collection of boards-style questions) for appropriateness of content for medicine interns and to ensure that answers could be found using both the basic Google search engine (not Google Scholar) and the summary resources.

We randomly assigned interns to answer questions 1 to 5 using Google or their choice of summary resources, then instructed them to cross over to the other group for questions 6 to 10. If interns could not immediately find an answer using the assigned resource, they could choose to use any online information resource. Our intervention specified the initial search strategy; we did not dictate where answers should ultimately be found.

To simulate real, time-limited clinical scenarios, we activated a timer allotting a maximum of four minutes per question. If interns could not find an answer using any resource in the allotted time, we instructed them to leave the questions unanswered; they were not to answer based on prior knowledge. Given our goal of comparing search strategies, we wanted to minimize the impact of the participants’ baseline knowledge on the outcomes. The interns documented both the time they spent and the resource where they found the answer.

Our primary outcome was the percentage of correct responses found through searches started with Google versus those started at the summary resources. Secondary outcomes included time to correct response and resources ultimately used by the Google versus the summary resource groups. We compared differences between groups using t test, chi-square, or general estimating equation. We used descriptive statistics to assess the resources used by the two groups. Statistical analyses were performed using SAS software version 9.2 (SAS Institute Inc., Cary, North Carolina).

The university’s institutional review board approved this study. All interns who participated in the EBM course were invited to take the end-of-course test. The participants did not receive incentives or compensation.

Back to Top | Article Outline

Results

All 48 interns in the two years (24 in each year) completed the EBM course and participated in the end-of-course test. In each year, 21 of the 24 interns were categorical, and 3 were preliminary-year interns. All were U.S. medical school graduates. In the 2011–2012 class, 6 of 24 (25%) interns were male, and 7 had other advanced degrees (6 master’s degrees and 1 PhD). In the 2012–2013 class, 8 of 24 (33%) were male, and 2 had advanced degrees (MPH and MS). The median age of the participants was 28 (range 24–32).

The interns found answers for 393 of the 480 (82%) questions administered. Overall, those instructed to start with Google found answers in a wider variety of resources than did those starting with summary resources (Table 1). Resources where the Google group most frequently found answers were commercial medical portals such as Medscape or eMedicine (49 of 191 responses; 25.7%), hospital Web sites (24; 12.6%), Wikipedia (23; 12.0%), and government Web sites such as CDC.gov (18; 9.4%), among others. In contrast, the summary resource group found answers in fewer types of resources, with the majority of answers being found in a summary resource (188 of 202 responses; 93%).

Table 1

Table 1

The mean correct response rate, our primary outcome, was 58.4% for the Google group versus 61.5% for the summary resource group (mean difference −3.1%; 95% CI −10.3% to 4.2%; P = .40). Figure 1 shows time to correct response by assigned group. Mean time to correct response was similar between groups (138.5 seconds for Google group versus 136.1 seconds for summary resource group; mean difference 2.4 seconds; 95% CI −10.8% to 15.5%; P = .72). Figure 2 shows mean percentages of correct, incorrect, and no-response rates by group. There was no difference between groups in proportion of mean correct, incorrect, or no-response rates (P = .35).

Figure 1

Figure 1

Figure 2

Figure 2

Back to Top | Article Outline

Discussion

In this randomized controlled crossover study, we found no significant differences in the speed or accuracy of searches initiated by Google or selected summary resources to answer simulated clinical questions. Although summary resources provide systematically selected and critically appraised information, their use did not increase scores on a test of clinical vignettes compared with Google-based searches that led to a wider variety of resources. Our results do not exclude small differences in accuracy and speed in favor of summary resources, although the magnitude of differences observed is unlikely to be clinically meaningful.

There are several potential explanations for why searches initiated with Google performed similarly, and not worse, compared with summary resources. First, although Google does not filter data according to level of evidence, our study participants were trained in critical appraisal, which may have allowed them to recognize and sort low- versus high-quality evidence. Although Google does not provide access to fee-based summary resources such as those included in our study, it does retrieve practice guidelines and scholarly journal articles, among other types of resources. Another explanation is that answers can be found in many different places, and summary resources have limitations. In fact, several recent studies of summary resources have shown that no single resource can find the best answer to all clinical questions, and that each resource has its limitations.15–17 Notable limitations, especially in comparison with Google, are inefficient search strategies and the relatively narrow content areas covered by summary resources. Unlike Google, where users can enter a specific search question and retrieve a list of potential answers, summary resources require users to search for answers by browsing a disease or symptom topic; if that topic is not covered by the resource, the user has to select another resource, enter another search term, and start the process again. Finally, the participants were likely much more familiar with Google than with the three newly introduced summary resources.

Although summary resources are considered to be based on the best evidence, they lag in terms of ease of navigation, speed, and content coverage. A Google-like engine that efficiently searches among all summary resources could improve the speed and accuracy of summary resource-based searches. Recent advances in information technology may address this issue. Web-scale discovery services can search quickly and seamlessly across a vast range of local and remote content, providing relevancy-ranked results in an intuitive interface, as expected by today’s information seekers.18 Currently, however, this type of service is not widely available.

Our study has several limitations. First, we had a limited number of questions and participants. However, the confidence intervals around our point estimates were small and excluded meaningful differences between the groups in speed or accuracy. In addition, our crossover design limits the prognostic factor imbalance between study groups that hinders small studies. Another limitation of our study is the use of select summary resources licensed by our university’s library. But given the limitations of individual resources as described above, it is unclear that the use of other or more summary resources would have led to different results. In addition, time to response and resources where answers were ultimately found were based on participants’ self-report. Finally, this was a single-institution study with internal medicine interns; our findings may not be generalizable to those from other institutions or disciplines.

In summary, we did not detect any significant differences in speed or accuracy between Google versus selected summary resource-based searches in answering simulated boards-style clinical questions. Although improvements to provide efficient, Google-like searching among all available summary resources may improve speed or accuracy of summary resource-based searches, no convincing evidence currently exists that medical educators ought to discourage the use of Google. Because no single resource can answer all clinical questions, teaching of information literacy and EBM should focus on familiarizing learners with a variety of information resources and teaching them how to critically assess information wherever it resides.

Acknowledgments: The authors thank the internal medicine residents of Rutgers University Robert Wood Johnson Medical School for their participation in this study.

Back to Top | Article Outline

References

1. Accreditation Council on Graduate Medical Education. http://www.acgme.org. Accessed February 11, 2014
2. Sidorov J. How are internal medicine residency journal clubs organized, and what makes them successful? Arch Intern Med. 1995;155:1193–1197
3. Green ML. Evidence-based medicine training in internal medicine residency programs a national survey. J Gen Intern Med. 2000;15:129–133
4. Ely JW, Osheroff JA, Ebell MH, et al. Analysis of questions asked by family doctors regarding patient care. BMJ. 1999;319:358–361
5. Green ML, Ciampi MA, Ellis PJ. Residents’ medical information needs in clinic: Are they being met? Am J Med. 2000;109:218–223
6. DiCenso A, Bayley L, Haynes RB. Accessing preappraised evidence: Fine-tuning the 5S model into a 6S model. Ann Intern Med. 2009;151:JC3-2–JC3-3
7. Duran-Nelson A, Gladding S, Beattie J, Nixon LJ. Should we Google it? Resource use by internal medicine residents for point-of-care clinical decision making. Acad Med. 2013;88:788–794
8. Kim S, Willett LR, Murphy DJ, O’Rourke K, Sharma R, Shea JA. Impact of an evidence-based medicine curriculum on resident use of electronic resources: A randomized controlled study. J Gen Intern Med. 2008;23:1804–1808
9. McKibbon KA, Fridsma DB. Effectiveness of clinician-selected electronic information resources for answering primary care physicians’ information needs. J Am Med Inform Assoc. 2006;13:653–659
10. Greenwald R. And a diagnostic test was performed. N Engl J Med. 2005;353:2089–2090
11. Tang H, Ng JH. Googling for a diagnosis—use of Google as a diagnostic aid: Internet based study. BMJ. 2006;333:1143–1145
12. Thiele RH, Poiro NC, Scalzo DC, Nemergut EC. Speed, accuracy, and confidence in Google, Ovid, PubMed, and UpToDate: Results of a randomised trial. Postgrad Med J. 2010;86:459–465
13. Kingsley K, Galbraith GM, Herring M, Stowers E, Stewart T, Kingsley KV. Why not just Google it? An assessment of information literacy skills in a biomedical science curriculum. BMC Med Educ. 2011;11:17
14. Medical Knowledge Self-Assessment Program for Students 4. 2008 Philadelphia, Pa American College of Physicians
15. Prorok JC, Iserman EC, Wilczynski NL, Haynes RB. The quality, breadth, and timeliness of content updating vary substantially for 10 online medical texts: An analytic survey. J Clin Epidemiol. 2012;65:1289–1295
16. Banzi R, Liberati A, Moschetti I, Tagliabue L, Moja L. A review of online evidence-based practice point-of-care information summary providers. J Med Internet Res. 2010;12:e26
17. Jeffery R, Navarro T, Lokker C, Haynes RB, Wilczynski NL, Farjou G. How current are leading evidence-based medical textbooks? An analytic survey of four online textbooks. J Med Internet Res. 2012;14:e175
18. Vaughan J. Web-scale discovery. Am Libr.. 2011;24:32
© 2014 by the Association of American Medical Colleges