Resources used at the point-of-care (POC) for medical decision making have evolved with changes in technology. One of the first articles published on resource use during POC decision making in a teaching hospital described taking an “evidence cart” on rounds, containing medical textbooks and journals loaded onto a CD-ROM.1 Contrast that with the technology available to residents today: smartphone and tablet technology loaded with medical applications and Internet connectivity, allowing the user to access countless electronic resources at the bedside.
There are two main reasons that we should pay attention to the information resources used by our learners at the POC. First, information resources have the possibility to affect clinical decisions; therefore, they must be of high quality and accuracy. Second, the majority of resident learning is directed by patient encounters within the clinical environment.2 Before the advent of technology for accessing information at the bedside, journals were the preferred resource for answering clinical questions.3 However, one study4 suggests that by 2006, residents were more likely to turn to online electronic resources. Ninety-eight percent of the residents in that study used UpToDate (an evidence-based, physician-authored clinical knowledge database), 44% performed online literature searches, and 35% used Web-based search engines such as Google.
In a more recent study, 80% of junior physicians reported using Google at least once over the course of a week.5 Additionally, young physicians are increasing their use of Web 2.0 community resources, including those available through Google, Facebook (online social network), and Wikipedia (online encyclopedia that can be edited and updated by users).5,6 Web 2.0 resources are broadly defined as those where multiple users continually update and remix data.7 Whereas educators generally consider journals a trusted resource, online resources such as Wikipedia are considered more questionable in the information they provide.6
Many residency programs address this concern by having curricula in place to teach critical appraisal of the literature and how to search for quality evidence.8 The impact of these curricula on resident usage patterns has been mixed.9,10 In one study, 95% of residents participating in an evidence-based medicine (EBM) workshop accessed the primary literature through Ovid (private vendor for accessing MEDLINE database) or PubMed (free database for accessing primarily MEDLINE) compared with only 58% in the control group. In both groups the most frequently accessed resources were summary resources such as UpToDate, MD Consult (Web-based medical resource providing synthesis of medical information), and E-medicine (online medical reference), and 56% of residents used nonmedical search engines (including Google, MSN, and Yahoo) at least once.10 A separate study demonstrated that residents were more likely to approach authoritative sources (faculty) 44% of the time to answer clinical questions at the POC, despite having prior EBM training.11
Barriers to using PubMed and Ovid MEDLINE have included a sense among residents that this is an inefficient use of their time and that they have inadequate knowledge and skill to efficiently employ these resources.12,13 Google Scholar, which searches peer-reviewed articles, theses, books, and abstracts, may be an easier first-step search of the medical literature. Although the Google Scholar search is challenged by the absence of the medical subject headings found in PubMed, the interface is more user-friendly and may lead to information in a fraction of the time.14 It is not clear how residents use Google versus Google Scholar in answering clinical questions.
Although prior studies have looked at learning and reading behaviors of internal medicine residents,2,4 little is known about the drivers of resource selection at the POC and the role and effectiveness of Google and Google Scholar for answering clinical questions. We conducted a multi-institutional survey of internal medicine residents to address three questions. First, what are the resources that internal medicine residents use for POC medical decision making? Second, what are the drivers for selection of these resources? Third, how are residents using Google and Google Scholar to answer clinical questions at the POC?
Survey design and administration
In January 2012, an electronic 14-question survey on resources used for real-time clinical decision making was sent to 299 residents from three internal medicine residency programs: 77 residents at the University of Minnesota Medical School, 51 residents from Oregon Health & Sciences University School of Medicine, and 26 residents from Hennepin County Medical Center in Minneapolis, Minnesota. A power calculation was not performed to determine sample size because the desire was to measure parameters rather than to test a hypothesis. Two of these programs are university-based training programs, and the third is within a county hospital setting. The survey was sent out by the separate program directors (one of whom was A.D.-N.) to their respective programs’ residents on two separate occasions one week apart.
All program directors involved indicated that they had an EBM curriculum in place in their programs that provided training on searching and critically reviewing the medical literature. None of the programs provided smartphones or tablets for their residents.
The survey included questions regarding frequency of use of resources available to all of the residents through their respective institutions for POC medical decision making, factors influencing the decision to use a resource, use of Google and/or Google Scholar in medical decision making, ownership of mobile devices, presence of an evidence-based curriculum in their residency program, and demographics. The survey instrument was reviewed and modified by our Department of Medicine Educational Scholarship Committee. It was piloted on eight chief residents at that school before sending it to study participants. Feedback was obtained from the pilot group to ascertain clarity and understanding of the survey questions. All questions from the pilot study were included in the final survey.
Participation in the survey was voluntary and anonymous to investigators. On completion of the survey, residents could choose to contact an administrative assistant to receive a five-dollar gift certificate as an incentive for their participation. The institutional review boards at all three institutions approved the study.
Statistical methods and data analysis
Frequency of use of each resource and factors influencing the choice of resource are reported using descriptive statistics. Binary logistic regression analysis was performed for each resource to determine whether there was a relationship between the independent variables (i.e., the factors that influence the choice of a resource, shown on the survey as speed, portability, trust quality of information, linked in electronic medical record (EMR), have on my smartphone, have on my tablet, no access to resource, distrust quality of information, too much information, insufficient information, and inconvenient to use) and the dependent variable (i.e., frequency of resource use). Because all levels of the ordinal dependent variable were not well populated across all resources, we dichotomized the dependent variable into use resource at least weekly and use monthly or less, as these were well populated for most resources, represented a meaningful distinction in frequency of use, and allowed for a consistent statistical approach across most resources.15 The regression analysis for the resources “consultation with faculty” and “Google general search engine” used a modified dependent variable of at least daily and weekly or less to have sufficient variability in the dependent variable. We did not perform regression analysis for UpToDate, ACP journals, TRIP database, Cochrane, and Guidelines.gov because of insufficient variability in the dependent variable. Only independent variables with a minimum of 20 responses were included in the model.16 Data analysis was performed using SPSS 19.0 software (IBM Corporation, Somers, New York). Factors were statistically significant whenP < .05.
Of the 299 residents surveyed, 167 responded (response rate 56%). Seventy-seven respondents (45%) were from the University of Minnesota Medical School, 51 (29%) from Oregon Health & Sciences University School of Medicine, and 26 (16%) from the Hennepin County Medical Center. Fifty-three respondents (34%) were in their first year of residency, 45 (30%) were in their second year, 47 (30%) were in their third year, and 9 (6%) were “other” (fourth year or higher). Eighty-nine respondents (58%) were men, and 65 (42%) were women. The ages of respondents ranged from 25 to 41 years. A total of 145 respondents (87%) owned smartphones, with one respondent indicating that the residency program provided them. Forty-three respondents (26%) owned tablet devices, with none indicating that their programs provided them (see Table 1).
Frequency of use of resources
The frequency of use of resources ranged widely (see Figure 1). UpToDate was the most frequently used resource, with 142 respondents (85%) indicating they used it at least daily. Other frequently used resources included consultation with faculty (125; 77%) at least daily and Google general search engine (104; 63%) at least daily. The least frequently used resources were Cochrane (102; 62%) less than once per month, Guidelines.gov (114; 69%) less than once per month, and TRIP database (151, 93%) less than once per month.
Factors that influenced the choice of resource
Respondents indicated that they consider multiple factors when deciding whether or not to choose a particular resource at the POC (see Figure 2). Across all resources, the most frequently cited factors were speed (940/3,852; 24% of all responses), trust in the quality of information (821/3,852; 21%), and portability (509/3,852; 13%). The least frequently cited factors were lack of access to resource (77/3,852; 2%), provides insufficient information (75/3,852; 2%), and have on my tablet (26/3,852; 1%).
The results of the regression analysis (see Table 2) show that the overall model was significant in all cases. The Nagelkerke R 2 varied from 0.1 (consultation with faculty) to 0.59 (Epocrates), indicating that the relationship between the factors that influenced the choice of resource (independent variables) and the frequency of use of the resource (dependent variable) varies from weak to moderately strong. See Table 2 for a listing of all factors and their positive or negative association with frequency of use.
Use of Google in medical decision making
When asked whether they are more likely to use the Google general search engine or use Google Scholar for POC decision making, 106 respondents (68%) indicated they use the Google general search engine, with 41 (30%) indicating Google Scholar. When asked how often their Google search results in an answer to their POC question, 66 respondents (43%) answered “often,” 62 (40%) answered “sometimes,” and 3 (2%) answered “always” or “never.”
When asked to compare the kinds of questions they think the Google general search engine and Google Scholar are effective in answering, many respondents indicated they find the Google general search engine effective for “searching for a trusted Web site I have used before” (114 respondents; 75%) and “searching for general information about a topic/disease from any resource” (109 respondents; 71%). They indicated they used Google Scholar more frequently than the general search engine for “searching for diagnostic strategies in a journal” (66; 44%), “searching for the most current treatment in a journal” (64; 42%), and “search for a specific paper I have seen before” (58; 38%). Several residents reported that they were unfamiliar with Google Scholar as a resource.
Speed, trust in the quality of information, and portability were the biggest drivers of resource selection for the participants in our study. These results are consistent with the Usefulness Equation, U = RxV/W, which describes that a physician’s goal is to find the most trusted and relevant information in the shortest time.17 That is, for information to be useful (U) it should be relevant (Rx), correct (V, for validity), and require little effort to obtain (W, for work). In our study, the relevance is implied because residents are seeking information related to their patients at the POC. Thus, the factors left are work (speed) and validity (trust).
Prior studies have demonstrated that time is a significant factor for physicians when searching for information,12 and this seems to become an even more significant variable for residents. Whereas practicing physicians may spend between 5 and 10 minutes18 searching for information, residents may spend only 2 minutes (maximum of 15 minutes).19 We found that resources that allow residents to quickly access information were more likely to be used with greater frequency. These resources included UpToDate and Google search engine, which were used at least daily by 85% and 63% of residents, respectively, with 91% and 84% of residents, respectively, indicating that speed influenced their decision. Conversely, only 13.3% of residents used PubMed/MEDLINE at least daily. And although speed was also a positive factor in greater use, only 31% of residents indicated that it influenced their decisions. Additionally, “inconvenient to use” was a significant factor negatively associated with the frequency of use, suggesting that although PubMed/MEDLINE may be quick to access, residents may find it more difficult to find answers quickly using PubMed/MEDLINE as compared with UpToDate. PubMed/MEDLINE requires that residents search and synthesize findings from multiple studies, whereas UpToDate provides a concise synthesis of information.
Although 65% of residents listed trust as a factor leading them to select PubMed/MEDLINE as a resource, and trust was a significant positive factor, our results suggest that time factors may lead residents to choose with greater frequency resources that are both trusted and also can be accessed more conveniently and quickly. Residents indicated that they used UpToDate the most frequently of all the resources, and the greatest percentage of residents indicated that they selected it because of speed (91%) and trust of quality of information (78%) compared with other resources. This suggests that speed may be a more important consideration than quality in some instances. Distrust of the quality of information is a significant factor negatively associated with the frequency of use for both Wikipedia and Google search engine, yet they were used with greater frequency than resources for which trust in the quality of information was significant, such as PubMed/MEDLINE.
Google search engine, which, similar to Wikipedia, was listed as fast but less trusted, was used mostly in the discovery phase, where residents are searching for general information about a disease or for a trusted Web site. As one resident noted, “Honestly, I just use Google for speed, for vague or unknown things, and after Google or Scholar gives me a direction to turn, I go to a more trusted source for clinical decision making.”
Google’s diagnostic effectiveness has been described previously,20 and 42% of our respondents reported using Google as a tool for inputting signs/symptoms of a disease to make a diagnosis. Residents were more likely to use Google Scholar for treatment decisions and diagnostic strategies, or to locate a specific article known to them because of the speed of the search. Google Scholar searches have been compared with PubMed searches in the literature, with Google Scholar demonstrating similar recall (sensitivity) but poorer precision than PubMed.21 Newer studies have demonstrated that Google and UpToDate answered evidence-based questions more quickly on major clinical topics, and more correctly within a five-minute time frame, than did PubMed and Ovid.14,22 This suggests that if time is of the essence, Google and UpToDate are effective alternatives to a more detailed and time-consuming literature search for answers to clinical questions at the POC.
The “6S” model may provide a framework for how we can teach residents to find “useful” information.23 The model describes a pyramid with individual studies at the foundation and computerized decision support systems at the peak, with synopses of studies, syntheses, synopses of syntheses, and summaries in between.24 Because resident physicians may be overwhelmed by the volume of information available to them and may lack necessary information management skills, they can be trained to begin searching for evidence toward the top of the pyramid (“summaries” if “systems” are not in place), working their way down the pyramid (increasing the work) as needed to answer the clinical question (see Figure 3). This strategy can increase the speed with which answers informing clinical decision making are arrived at, as well as assuring confidence in the validity of the evidence found. In the era of Web 2.0, many residents are using search engines such as Google/Google Scholar and sources like Wikipedia in the initial discovery phase as tools to take them directly to a different level of evidence in the pyramid—for example, a clinical practice guideline or Cochrane Review.
Additionally, search engines such as Google can be used somewhat as a “system” to answer a particular question (e.g., How do I calculate a water deficit for a patient with hypernatremia?). Navigating this system is dependent on knowing when to delve deeper into the evidence pyramid, and realizing the time (work) investment that will be necessary. Many residents may lack the information management skills necessary to navigate the evidence and may be overwhelmed by the volume of information available.
Our study had a large sample size and an acceptable response rate, with 56% of the residents responding to our electronic survey. Additionally, the study was conducted across multiple sites and within programs of different settings and patient populations (university versus county hospitals). However, there are some limitations to our study. Because it involved only three internal medicine residency programs, it may not be generalizable to all of graduate medical education. Additionally, our data collection is based on self-report of information-seeking behaviors. As a result, it may not accurately reflect real-time behaviors of the residents in this study. The dichotomization of the dependent variable may have caused a loss of information through the collapsing of response levels, resulting in a loss of statistical power and possible failure to detect additional statistically significant factors. Finally, although we defined speed as the time and work required to navigate to a particular resource, this was not defined for study participants, and it is possible that they attached a different meaning to the term “speed.”
Our study findings suggest that residents prefer electronic resources for answering questions at the POC, with UpToDate and Google being the most commonly used resources for medical decision making. It is clear that the responding residents favored speed, trust, and portability in their resources used at the POC. Time and information overload (related to work spent) appear to be the biggest barriers to resource use, supporting the usefulness equation.17 Residents frequently used Google and Google Scholar, but they may have benefited from further training in information management. We should be training our resident physicians not only about what quality information is available (the resources themselves) but also about when to delve deeper into the information pyramid and go directly to the studies themselves, as well as how to navigate to resources, manage, and be good stewards all of the information that is available.
Acknowledgments: The authors would like to acknowledge Dr. Sima Desai from Oregon Health & Sciences University School of Medicine, and Dr. Anne Periera from Hennepin County Medical Center, for allowing the survey of their residents for this report. The authors would also like to acknowledge Ms. Alicia Jackson for her role in administration of the survey, and Mrs. Leah Duncan in her assistance in preparation of the final manuscript.
Funding/Support: Funding for providing gift cards for study participants was obtained through the Applied Clinical Research Program at the University of Minnesota Medical School, Department of Medicine.
Other disclosures: None.
Ethical approval: The institutional review boards at the University of Minnesota Medical School, Oregon Health & Sciences University School of Medicine, and Hennepin County Medical Center approved the study.
1. Sackett DL, Straus SE. Finding and applying evidence during clinical rounds: The “evidence cart.” JAMA. 1998;280:1336–1338
2. Edson RS, Beckman TJ, West CP, et al. A multi-institutional survey of internal medicine residents’ learning habits. Med Teach. 2010;32:773–775
3. Cohen SJ, Weinberger M, Mazzuca SA, McDonald CJ. Perceived influence of different information sources on the decision-making of internal medicine house staff and faculty. Soc Sci Med. 1982;16:1361–1364
4. Lai CJ, Aagaard E, Brandenburg S, Nadkarni M, Wei HG, Baron R. Brief report: Multiprogram evaluation of reading habits of primary care internal medicine residents on ambulatory rotations. J Gen Intern Med. 2006;21:486–489
5. Hughes B, Joshi I, Lemonde H, Wareham J. Junior physician’s use of Web 2.0 for information seeking and medical education: A qualitative study. Int J Med Inform. 2009;78:645–655
6. Metcalfe D, Powell J. Should doctors spurn Wikipedia? J R Soc Med. 2011;104:488–489
8. Green ML. Evidence-based medicine training in internal medicine residency programs a national survey. J Gen Intern Med. 2000;15:129–133
9. Feldstein DA, Maenner MJ, Srisurichan R, Roach MA, Vogelman BS. Evidence-based medicine training during residency: A randomized controlled trial of efficacy. BMC Med Educ. 2010;10:59
10. Kim S, Willett LR, Murphy DJ, O’Rourke K, Sharma R, Shea JA. Impact of an evidence-based medicine curriculum on resident use of electronic resources: A randomized controlled study. J Gen Intern Med. 2008;23:1804–1808
11. McCord G, Smucker WD, Selius BA, et al. Answering questions at the point of care: Do residents practice EBM or manage information sources? Acad Med. 2007;82:298–303
12. Green ML, Ruff TR. Why do residents fail to answer their clinical questions? A qualitative study of barriers to practicing evidence-based medicine. Acad Med. 2005;80:176–182
13. van Dijk N, Hooft L, Wieringa-de Waard M. What are the barriers to residents’ practicing evidence-based medicine? A systematic review. Acad Med. 2010;85:1163–1170
14. Thiele RH, Poiro NC, Scalzo DC, Nemergut EC. Speed, accuracy, and confidence in Google, Ovid, PubMed, and UpToDate: Results of a randomised trial. Postgrad Med J. 2010;86:459–465
15. Preisser JS, Phillips C, Perin J, Schwartz TA. Regression models for patient-reported measures having ordered categories recorded on multiple occasions. Community Dent Oral Epidemiol. 2011;39:154–163
16. Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996;49:1373–1379
17. Slawson DC, Shaughnessy AF. Obtaining useful information from expert based sources. BMJ. 1997;314:947–949
18. Schwartz K, Northrup J, Israel N, Crowell K, Lauder N, Neale AV. Use of on-line evidence-based resources at the point of care. Fam Med. 2003;35:251–256
19. Ramos K, Linscheid R, Schafer S. Real-time information-seeking behavior of residency physicians. Fam Med. 2003;35:257–260
20. Tang H, Ng JH. Googling for a diagnosis—Use of Google as a diagnostic aid: Internet based study. BMJ. 2006;333:1143–1145
21. Anders ME, Evans DP. Comparison of PubMed and Google Scholar literature searches. Respir Care. 2010;55:578–583
22. Hoogendam A, Stalenhoef AF, Robbé PF, Overbeke AJ. Answers to questions posed during daily patient care are more likely to be answered by UpToDate than PubMed. J Med Internet Res. 2008;10:e29
23. DiCenso A, Bayley L, Haynes RB. Accessing preappraised evidence: Fine-tuning the 5S model into a 6S model. ACP J Club. 2009;151:3
24. Haynes RB. Of studies, syntheses, synopses, summaries, and systems: The “5S” evolution of information services for evidence-based healthcare decisions. Evid Based Med. 2006;11:162–164