Medical Students’ Use of Information Resources: Is the Digital Age Dawning?

Abstract

Purpose 

One of the many challenges clinicians face is applying growing medical knowledge to specific patients; however, there is an information gap between information needs and delivery. Digital information resources could potentially bridge this gap. Because most medical students are exposed to personal computers throughout their education, this study postulated that students may be more comfortable using computer-based information resources within clinical interactions.

Method 

In 2001, the authors monitored second-year medical students’ use of a unique digital textbook, UpToDate, as they transitioned from preclinical to clinical years at the University of Iowa Roy J. and Lucille A. Carver College of Medicine. In 2002, at the end of their third year, students were surveyed about their preferred clinical information resources.

Results 

Medical students rapidly adopted UpToDate as a clinical resource during their clinical clerkship as evidenced by a rapid growth in the electronic textbook’s use. One hundred sixteen of a possible 154 students (75%) responded to the survey. More than 85% of respondents identified electronic sources as their primary resource (UpToDate 53%, MDConsult 33%; p < .001 when compared to paper resources). They also reported using the information resources on a daily basis and requiring less than 15 minutes to answer most of their clinical questions.

Conclusions 

This study clearly demonstrates that medical students embrace and use electronic information resources much more than has been reported among practicing clinicians. The current generation of students may be the leaders in a medical culture shift from paper to electronic resources.

One of the many challenges clinicians face is applying growing medical knowledge to specific patients. This is not a new challenge because studies over the last 20 years have demonstrated that both academic physicians¹ and practicing physicians^2,3 generate many questions during the course of patient care. Even in a nonteaching medical environment, physicians generate one question for every two or three patient encounters.³ Unfortunately, physicians actively pursue answers to only about 30% of the questions.^2,3 When physicians do pursue an answer, they most frequently either ask a content expert or use a printed resource.^2,3

These data suggest an information gap between physician information needs and information delivery. Traditionally, continuing medical education has sought to fill this gap but appears to have largely failed to meet this need. Traditional continuing medical education fails to alter physician behavior⁴ and traditional educational interventions fare poorly in encouraging physicians to adopt new clinical practice guidelines.⁵ These are not surprising findings when we consider research on teaching methods and retention rates.⁶ When learners are presented new information in a lecture format, the retention rate is only 5% to 10%. However, when learners use the information immediately, the retention rate may be as high as 80% to 90%. These data provide the justification for pursuing learning at the point of care.

Traditional educational approaches fail to meet the information needs in clinical medicine. Digital information resources delivered via computers could potentially bridge the information gap and allow information to be retrieved and used immediately. Computer technology permits information to be updated more frequently than can be done with printed materials and is not limited by the size and weight of printed materials. However, relatively few practicing physicians use computer-based resources to answer their questions,^2,3 a situation probably due to multiple factors: the availability of computers or computer networks for the physician, the physician’s comfort level using computers, the physician’s expectation that an answer is available through online sources, or the time required to retrieve an answer.² The low use does not reflect the potential strengths of computer-based resources, which include the ability to store huge databases for information recall, to deliver information quickly at the point of care where information effectiveness is maximized,⁷ and to build advising systems or decision support systems to deliver focused patient-management recommendations.^7,8 In early studies, computers were found to beneficially alter physician behavior.^9,10 Poor use may be explained by physician comfort and skills using computers, factors that may be influenced by physician age. Both Gorman and Helfand² and Ely et al.³ studied physicians with a mean age of 45.2 and 48 years, respectively.

Because personal computers have become ubiquitous over the last decade and because most American students are now exposed to personal computers throughout their education, we postulated that medical students may be more comfortable using computer-based information resources within clinical interactions. To test this theory, we monitored medical students’ use of a unique digital textbook, UpToDate, as they transitioned from their preclinical to clinical years, and we surveyed the students at the end of their third year of medical school about their preferred clinical information resources.

Method

Curricular Overview for Information Retrieval

Our institution, the University of Iowa Roy J. and Lucile A. Carver College of Medicine, has a four-year curriculum. During the first two years, the students study a basic science curriculum but are also involved in a four-semester course entitled Foundations of Clinical Practice (FCP) I-IV. In addition to lectures and skills labs, students participate in case-based learning exercises during FCP I and II, and inpatient-based learning exercises in FCP III. As part of the curriculum, they are instructed in the principles of searching the medical literature to obtain clinical information and are taught the basics of medical informatics including searching and quality assessment of medical information on the Internet. They are provided access to a traditional medical textbook published online (Harrison’s Online) and to a resource medical library in digital format (MDConsult). In addition, each student receives a free paper subscription to the New England Journal of Medicine. During their case-based learning exercises, students are required to research clinical questions and produce weekly reports that they share with their colleagues. One component of their grade is determined by the diversity of resources they use to research their questions. During the fourth semester of the course (FCP IV), each student interviews and examines two simulated patients and four actual patients. They are required to research clinical questions on these patients and incorporate their findings into a clinical write-up. During the Spring 2001 semester, we provided UpToDate, an electronic medical textbook, to the FCP IV students in addition to the information resources they were using in the first three semesters. The class was provided a unique username and password for accessing UpToDate over the Internet. UpToDate collected and stored all use data on a weekly basis for this username over 12 months as the students transitioned from the preclinical to the clinical years. The students were unaware that these data were being collected. Our Institutional Review Board reviewed the protocol and granted the study an educational exemption and approved the study design, including student blinding to data collection.

Student Survey

During the third year of the medical curriculum, students rotate on clinical clerkships in internal medicine, surgery, pediatrics, and family medicine. In 2002, at the end of their third year, we electronically surveyed the students in the clerkships about their preferred clinical information resources. We contacted each student by e-mail and provided a link to a Web site on which they could complete a questionnaire. Because e-mail was the preferred communication instrument throughout their medical school career, students were all familiar with using it. We included examples of several different resources in the questionnaire, such as paper textbooks or journals and Medline for electronic literature searching. Harrison’s Online was included as an example of an electronic version of a traditional paper textbook, and UpToDate was an example of an exclusively electronic textbook with rapid searching capability and extensive use of hypertext. MDConsult was included as an example of a digital compilation of textbooks, journals, and other information resources. The Cochrane Library, a compilation of high quality literature reviews on focused clinical topics, was included as an example of an evidence-based medicine collection. In addition, we allowed students to choose “other paper-based resources” or “other Web sites.” To gain some insight into the reason for their preferences, we asked them to provide us information about how often they used the resources, how long they spent using them, and how often they were successful at finding an answer.

Statistical Analysis

We generated summary and descriptive statistics to characterize students’ responses regarding time, frequency and duration of use, and perceived success at finding answers. We tested the perceived importance of paper textbooks compared with seven other information resources. Seven matched-paired comparisons tested the null hypothesis that textbooks were of equal perceived importance with other information resources. With alpha set at .05, individual t values significant at .05/7 (.007) or less were regarded as significant. We also calculated the correlation coefficient for the ratings of current and future usefulness to obtain some measure of the reliability of future use projections.

Results

We initially introduced the UpToDate electronic textbook to the second-year medical students in mid-February, 2001. As shown in Figure 1, use varied from 100 to 300 topics per week during the 12-week semester. The approximate three-week cycle correlates with the class schedule in which examinations occurred every three weeks and suggests that our monitoring methods were sensitive to usage trends. Use was lowest during the week in which an examination was given and highest in the week immediately following an examination, which suggested that the students were using UpToDate for their patient activities and not to prepare for didactic examinations. The course concluded the week of May 12 and the students began their clerkships on July 1. During the period from May 12 through July 1, all students were required to take the United States Medical Licensing Examination Step 1. Beginning in July with their clerkships, use grew continuously from approximately 200 topics per week to 1,700 topics per week just before the Thanksgiving holiday break in November. These data suggested to us that students were heavily using electronic textbook resources while seeing patients on their clinical clerkships. However, these summary data do not differentiate between a few students heavily using the resource and many students using the resource. In addition, these data cannot tell us anything about how use of this resource compared with use of other information resources or about when the students were accessing particular resources. To address this question, we surveyed the medical students near the conclusion of their third year.

Figure 1.: In 2001, UpToDate, an electronic medical textbook, was introduced to medical students at the University of Iowa Roy J. and Lucile A. Carver College of Medicine during the fourth semester curriculum. Monitored class use during that semester and into student clerkships demonstrated significant growth in use. Class use went from approximately 200 topics per week to more then 1,700 topics per week. The fall in use at the end of the time period correlated with students’ scheduled holiday break.

One hundred sixteen of a possible 154 students (75%) responded to our survey. Because the questionnaire was anonymous, we could not identify or characterize respondents from nonrespondents. Respondents reported heavy use of electronic clinical information resources. When we asked the students “What information resource do you use most often when you see patients?” 99 of the 116 (85%) respondents identified either UpToDate (53%) or MDConsult (33%) as their preferred resources. Only 14% listed paper textbooks as their first choice. When asked about their second most preferred resource, MDConsult was the most preferred (38%) and paper textbooks were second (27%). Fifteen percent of the students preferred Medline as their second choice. Most students preferred paper textbooks over electronic resources only as their third choice. These data thus support the conclusion that students’ use of UpToDate during the clerkships reflected a broad preference for electronic resources and not heavy use by a few dedicated students.

Computer-based resources have several potential advantages over other resources, including easier accessibility, more rapid access to the required data, and a higher probability of finding an answer, all of which directly affect whether physicians will pursue an answer to their clinical questions. In the questionnaire, we first asked students to identify when they used a particular resource. Learning is most effective and efficient when it is situational, in other words, tied to actual activity. For all three of the primary resource choices, most students reported using the resource immediately after seeing the patient or during the same day that they saw the patient (see Figure 2a). This was true for both electronic resources and paper textbooks. Too few students identified themselves as preferring paper textbooks to draw statistical conclusions but more students reported using the resource immediately. In addition, those students who preferred using electronic resources but chose paper textbooks for a second or third choice also often used the paper resource immediately or on the same day they saw the patients (see Figures 2b and 2c). Thus, it does not appear that the students chose electronic resources because they were more readily available to them.

Figure 2.: Medical students’ choices of a) primary), b) secondary, and c) tertiary information resources and the frequency of their use, University of Iowa Roy J. and Lucile A. Carver College of Medicine, 2002. Students reported using the information resources most often either immediately or the same day that they saw the patient. See text for description of resources.

Like other medical educators, we try to instill an ethic of continuous learning in our students that encourages daily learning. To determine whether access would influence how often students use an information resource, we asked them to report how frequently they used their primary reference. As shown in Figure 3, most students reported using the resource either daily or more often than weekly. They did not differ in their reported use of MDConsult, UpToDate or paper textbooks, but more students reporting a preference for paper textbooks also reported using the textbook daily.

Figure 3.: Medical students’ reported use of their primary information resources, University of Iowa Roy J. and Lucile A. Carver College of Medicine, 2002. Most students reported daily or greater than weekly use. Students who preferred paper textbook more often reported using the resource on a daily basis. See text for description of resources.

In previous studies involving practicing clinicians, two variables appeared to be important in determining whether they pursued a question: an expectation that an answer could be found and the length of time it would take to find an answer.^2,3 To determine whether these two variables affected the students’ choice of resources, we asked them to report how much time they spent using the resource in one sitting and the proportion of times they successfully found answers to their questions. For their primary resource, students reported that they most often spent between six and 15 minutes, with no difference between electronic and paper resources. Students’ answers did differ, however, between the primary- and tertiary-resource choices. Most students who listed Medline or Harrison’s Online as a second or third choice reported spending more than 30 minutes or 15 to 30 minutes, respectively. This is significantly different from the reported use times for the primary resources. In addition, students who reported using MDConsult, paper textbooks, and UpToDate as their third choices also reported spending more time with the resource compared with students who reported using them as their primary resource. Therefore, students perceived that they required less time with their primary resource. This is indirect evidence that the time required to find an answer influences students’ resource choices. By contrast, the probability of successfully locating an answer did not appear to be an important influence. The great majority of students reported a greater than 50% success rate at finding an answer with all resources except Medline. Thirty-eight percent of the students choosing Medline as their third resource reported 26% to 50% success in arriving at a definitive answer.

The final set of questions on the questionnaire asked the students to rate the overall usefulness of the various resources on a five-point scale ranging from 1 = not useful to 5 = extremely useful. We compared the responses for each resource to the responses for paper textbooks. As shown in Table 1, the mean rating for paper textbooks was 4.04 ± 0.80. The mean ratings for two computer-based resources, UpToDate and MDConsult were significantly higher. The ratings for Harrison’s Online, other Web sites, and the Cochrane Library were significantly lower.

Table 1: Medical Students’ Ratings of Overall Usefulness of Information Resources, University of Iowa Roy J. and Lucile A. Carver College of Medicine, 2001–02

Lastly, students responded to the question, “How likely are you to use each of the resources in the future?” using a five-point scale ranging from 1 = not likely to use to 5 = very likely to use. We again compared the ratings for each resource to paper textbooks. When we compared the responses for each of the resources between current usefulness and likeliness to use in the future, the correlation coefficient for current and future usefulness was statistically significant. However, the students rated three resources—paper textbooks, Medline, and other Web sites—more highly for future use than for current usefulness (see Table 2). Students scored the other resources the same for current usefulness and likelihood to use in the future.

Table 2: Correlation Between Medical Students’ Ratings of Current Usefulness and Ratings of Future Usefulness of Information Resources, University of Iowa Roy J. and Lucile A. Carver College of Medicine, 2001–02

Discussion

Medicine is a continuously evolving clinical science. One of the challenges medical educators face is teaching students to remain current with medical science by becoming life-long learners. While continuing education has been recognized as an important component of Western medicine for 500 years,¹¹ recent studies have demonstrated that traditional continuing education is not effective at achieving the goal of lifelong learning.^4,12 In addition, studies have demonstrated that physicians often do not pursue answers to their clinical questions.^2,13 Part of the explanation for this latter finding may be the time required to find an answer. Although computers hold great potential as a timely information resource for clinicians, studies of physicians have shown that they do not often use computer-based resources.^2,3 These studies, however, have most often evaluated physicians older than 40 years who may not have extensive computer skills. Computers and the Internet have become increasingly integrated into education at all levels, and many medical students now have sophisticated computer skills. In our study, we found that the medical students rapidly adapt to new computer-based resources. In addition, as a group they preferred computer-based resources to paper textbooks even though they continued to recognize the important role of paper textbooks.

One reason that students may choose one resource over another is the availability of the resources. All students at our institution have ample access to computers both on the wards and in the clinic but have variable access to printed textbooks in those locations. Our medical library is available to the students but is a short walk from the ambulatory clinics and clinical wards. However, students do not appear to prefer computer-based resources based solely on access or availability because they reported similar patterns of frequency and timing between electronic and paper resources. This would suggest that they chose the electronic resources for reasons other than simple availability. Similarly, it does not appear that they chose computer-based resources over paper textbooks because of perceived differences in the probability of finding an answer. In the study by Gorman and Helfand,² one variable that predicted whether physicians pursued a clinical questions was their perception that they could find an answer.² This did not appear to affect the students because even those students who chose paper textbooks as their third choice still reported a very high success rate at finding an answer.

Another potentially important variable that we looked at was time. Busy clinicians appear to choose their resources, in part, based on the time required to find an answer.³ We hypothesized that medical students may make similar choices even though they have more time to spend using a resource than does a practicing clinician. Our data supported this hypothesis because students reported spending the longest time with the lowest rated resources. In addition, those students who reported using paper textbooks as their third choice reported spending more time finding answers with paper textbooks than they did with their primary and secondary, usually computer-based, choices. This finding is potentially concerning in an educational venue, however. Although busy clinicians may need quick answers to diagnostic and management questions, we would hope that medical students would spend more time in purely educationally enriching pursuits.

Students did not appear to choose computer-based resources solely by the medium. Two computer-based resources, Harrison’s Online and the Cochrane Library, were rarely selected by students and were not rated highly either as a current resource or projected as being a useful resource in the future. It is speculative to explain the reason for these low evaluation scores, but Harrison’s Online does differ in design and searching capability from UpToDate. Harrison’s Online is structured to look much more like a traditional textbook with searches leading to chapters. UpToDate, on the other hand, has a navigation system that allows for quickly jumping to and through topics to a specific answer to a clinical questions. The Cochrane Library is a database of evidence-based reports on diseases and disease categories. While it is a complete and scrupulously researched resource, its size, limited search capabilities, and frequently limited conclusions may dissuade students from using it extensively. With both Harrison’s Online and the Cochrane Library, the issue of time needed to find an answer may significantly affect their use. Interestingly, students rated Medline equal to paper textbooks both in terms of current usefulness and future usefulness. However, previous studies of Medline found that it required extensive time (average 27 minutes per sitting) and retrieved a relevant answer to only a portion of the questions.^1,14 In addition, Hersch et al.¹⁵ recently reported that medical students were only moderately successful at correctly answering clinical questions using Medline. Our study did confirm that Medline required a significant time commitment, but students reported a higher success rate than has been reported in previous studies.

Finally, students tended to gravitate toward a few high quality resources, and did not rate other Web sites as particularly useful either now or in the future. Part of this may be due to the difficulty in finding and identifying high quality resources on the Internet or to a certain comfort level in becoming familiar with a few resources.

One potential weakness of our study is that students self-reported their resource use. However, our objective data measuring their use of UpToDate correlates well with their responses to the questionnaire. In addition, students answered the questionnaire anonymously and received no penalty for any of their answers. Finally, we had a satisfactory response rate, with the respondents representing a significant proportion of the eligible students.

In conclusion, these data clearly demonstrate that medical students embrace and use electronic information resources much more than has been reported among practicing clinicians. The current generation of students may be the leaders in a medical culture shift from paper to electronic resources. We will need to continue to monitor the effectiveness and utility of electronic resources available at the point of care compared to traditional education and paper resources. The academic medical community will need to ensure that electronic medical resources meet the same high standards that paper textbooks and journals have developed.