Results of the student survey conducted prior to matriculation showed that the majority (65%) of students were “very comfortable with computers” or needed to be “just shown the basics” (29%). No students indicated that they were “unfamiliar with the use of computers.” A small minority (five students) indicated they were very skilled with computers. There was a much broader distribution of students' experiences using web browsers. The class breakdown on use of the WWW was as follows: 32% used it “occasionally,” 35% used it “weekly,” and 28% used it “almost daily.” Subsequent regression analyses using survey data and server statistics revealed no significant correlation between individual students' pre-matriculation computer literacy or use of the WWW and their subsequent use of the network (number of times and total amount of time logged into the server).
Specific Use of CAI in the Course
A variety of CAI lessons were developed to supplement specific course objectives. Analyses of server statistics showed that the number of daily hits in the directories containing specific CAI lessons coincided closely with the times specific subjects were taught. For example, as illustrated in Figure 2A, the LUMEN dissector was used extensively during the first eight weeks of the course on those days when laboratory dissections were scheduled. Each student, on average, retrieved between 30 and 130 files on each of the days a laboratory was scheduled. Figure 2B provides a second example: hits to CAI lessons developed for supplementing instruction on the head and neck significantly increased when that subject was covered in class and again right before the examination on that material. Similar correlations were observed for lessons developed for other sections of the course.
Tutorials and practice examinations
A number of tutorials and practice examinations (both practical and written) were developed specifically for the course. Figure 2C shows the pattern of daily hits for one example of a tutorial on cross-sectional anatomy based on the Visible Human data from the National Library of Medicine. In this and every other case, the students' use of these resources coincided closely with the in-course examinations, with the peak number of server requests occuring on the day before each examination. There was a tendency for increased use with each exam, a trend that was also seen in other tutorials and practice exams provided to the students.
The third general area of CAI used in the course was the LUMEN Web Forum, a web-based, multi-media-capable application that was developed for threaded discussions of course content by students and faculty outside of class time. The forum was restricted to students and faculty in the course; they had to log in (with password). Server logs showed that 114 students (89% of the class) accessed the forum at least once during the course; 43 (38%) of those actively participated in the online discussions. There were 263 postings (152 by students; 111 by faculty); an average of approximately four postings per day during the course. The number of daily logins to the forum was generally constant during the course, with peaks occurring prior to the exams and another large peak during the seventh week, when there was a lively discussion of the anatomy of the perineum (Figure 2D).
The user survey we conducted at the end of the course substantiated much of the data collected from the server statistics. For instance, 81% of the class indicated they logged into the Structure of the Human Body web site daily to four times per week. The students' responses on the use of specific applications also were consistent with the server statistics. For example, 87% indicated they used the LUMEN Dissector during laboratory dissections (see Figure 2A). Similarly, 88% used tutorials and practice exams “only for review” (see Figure 2C). There was a significant correlation for performances of individual students across all exams (i.e., students who performed well on the first exam tended to have high scores on the second exam and on the final).
Correlation of Use of Computers with Performance in the Class
Regression analyses of performance in the class with either server statistics or user surveys for individual students failed to show any significant correlation when the class was treated as a whole or when divided according to grades (upper quartile, upper third, etc., versus lower quartile, lower third, etc.). However, when we analyzed grouped data, we found a significant inverse correlation between grades and the average length of time students logged into the server for the first part of the course (Figure 3). As the course progressed, this correlation diminished due to changes in the login habits of students in the lower third of the grade distribution (Figure 3 and Table 1).
Server statistics on the use of the Web Forum revealed that students in the upper 33% of the class (based on grade) tended to log in more frequently than did students in the lower 33% of the class for all three parts of the course (data not shown). Moreover, regression analysis of final grade versus number of logins showed a stronger correlation for the upper 33% compared with the lower 33% of the class (Figure 4).
Three general conclusions can be drawn from this study. First, a medical student's level of computer literacy at matriculation is not a good predictor of the extent to which he or she will use CAI in the curriculum. Second, server statistics, when combined with user surveys, are important for validating students' use of CAI. Third, the degree and manner of students' use of CAI tended to correlate with their performances in class.
The World Wide Web has undergone phenomenal growth in its number of applications and content.6 Consequently, medical schools are increasingly adopting the web for education and administrative tasks.7–9 The technology offers simple but dynamic tools to teachers for enhancing curricular resources; it facilitates educational interactions among faculty and students; it offers the ability to incorporate up-to-date information quickly and efficiently; and through hypertext, it provides for immediate linking to content in an increasing number of databases.10 The present study offers another important advantage: the capacity to track and analyze use of resources in the context of curricular objectives.
Because multimedia, hypertext applications can now be developed and delivered with great ease, more and more faculty are developing CAI who previously would have been discouraged by the technologic demands (e.g., programming expertise and expensive hardware for producing CD-ROMs). As more faculty develop CAI, the assessment and evaluation of that CAI becomes increasingly problematic. Our study shows that server statistics can be easily used to provide an important component of CAI evaluation; specifically, how and when students use the applications. Identifying specific CAI materials that are used by students most frequently can further enhance the development of those tools.8 At the Stritch School of Medicine, additional formative evaluations were solicited from volunteer students, who were asked to fill out questionnaires addressing CAI design (e.g., navigation, image qualities, presentation formats, etc). These evaluations were conducted by the Office of Computers in Education.
Surveys are used extensively to evaluate CAI.7,8,11 On the one hand, our survey substantiated the overall assumption that access of CAI implied its use. On the other hand, individual discrepancies between the survey and the server statistics were sufficiently large (10% to 20%) to call into question the accuracy and validity of user surveys. For example, ten students who indicated on their questionnaires that they “never logged into the Web Forum” actually logged in between one and five times. Three other students who, according to the server statistics, never logged in indicated that they logged in “frequently.” Without any indications of server malfunction, we conclude that the information provided through user surveys is less reliable, for reasons that may be both intentional and unintentional.
Server-based CAI provides the added advantage of allowing correlations between CAI use and comprehension of course content. In the present study, comprehension was measured by performance on in-course examinations. Our study did not directly address the question of effectiveness of the CAI to produce higher grade scores, as reported previously.11–15 Nevertheless, our analyses revealed some important trends that correlated with class performance. Particularly revealing were the data on server access showing that students in the lower third of the class significantly reduced the length of each session on the server as the course progressed compared with the upper third of the class, whose login times were more consistent (Table 1, Figure 3). This difference in behaviors probably reflects either differences in comprehension of the CAI materials or different study habits. Whatever the reason, it is noteworthy that these differences were indistinguishable by the end of the course.
There are a number of factors that may have confounded the collection and subsequent interpretation of the data on server-based CAI use. First, students may have studied in groups, gathering around a computer screen while only one student actually logged in. Since many applications were designed to be used in groups (e.g., LUMEN Dissector, case-base sessions), we provided a generic login ID for those applications. Some students may have used this generic login ID for individual sessions as well. Finally, the CAI materials are available through the Internet, and as the course progressed, an increasing number of students accessed the server from home. In those cases, their login IDs were not recorded (except for the LUMEN Web Forum). The activities of those students who used third-party providers for their Internet access would not even have been recorded as belonging to the “luc.edu” domain. It is noteworthy that trends were detectable in spite of these confounding factors.
To summarize, the present study highlights an important advantage of web-based applications for medical education, which is the collection of server statistics on CAI. These objective data on utilization are critical to the evaluation of CAI resources. At one level, the hypothesis that effective CAI is appropriately used in the context of specific curricular objectives can be easily tested quantitatively. Additionally, opportunities for evaluating CAI at the level of the individual student are enhanced. One aspect that requires further analysis is the relationship between CAI use and comprehension of course content. In this regard, increasingly sophisticated evaluation paradigms will become available as the technology improves. Finally, our study demonstrates how server statistics may be used to evaluate more traditional evaluation tools such as user surveys.
1. Benyon D, Stone D, Woodroffe M. Experience with developing multimedia courseware for the World Wide Web: the need for better tools and clear pedagogy. Int J Human—Computer Studies. 1997;47:197–218.
2. Friedman RB. Top ten reasons the World Wide Web may fail to change medical education. Acad Med. 1996;71:979–81.
3. Lapeyre AC. The World Wide Web is already changing medical education. Acad Med. 1997;72:563.
4. Koschmann T. Medical education and computer literacy: learning about, through, and with computers. Acad Med. 1995;70:818–21.
5. Espiritu BR, Halama J, Sarmiento M, McNulty JA. The Loyola University Medical Education Network. J Med Educ Tech. 1995;1:18–9.
6. Chan BK. The digital revolution of medical education. JAMA. 1997;278:1784.
7. Clark DJ. Course redesign: incorporating an Internet web site into an existing nursing class. Comp Nursing 1998;16:219–22.
8. Davis MJ, Wythe J, Rozum JS, Gore RW. Use of the World Wide Web server and browser software to support a first-year medical physiology course. Am J Physiol. 1997;72:S1–S14.
9. Zucker S, White JA, Fabri PJ, Khonsari LS. Instructional intranets in graduate medical education. Acad Med. 1998;73:1072–5.
10. MacKenzie JD. The World Wide Web: redefining medical education. JAMA. 1997;278:1785–6.
11. Cotter JR. Computer-assisted instruction for the medical histology course at SUNY at Buffalo. Acad Med. 1997;72:S124–S126.
12. Ayoub JL, Vanderboom C, Knight M, Walsh K, Briggs R, Greken K. A study of the effectiveness of an interactive computer classroom. Comp Nursing. 1998;16:333–8.
13. Lilienfiled LS, Broering NC. Computers as teachers: learning from animations. Am J Pathol. 1994;266:S47–S54.
14. MacFadyen JC, Brown JE, Schoenwald R, Feldman RD. The effectiveness of teaching clinical pharmacokinetics by computer. Clin Pharmacol Ther. 1993;53:617–21.
© 2000 Association of American Medical Colleges
15. Trelstad RL, Raskova J. Teaching pathology without lectures through computer-based exercises, small-group discussions and readings. In: Frisse ME (ed). Proceedings of the Sixteenth Annual Symposium on Computer Applications in Medical Care. New York: McGraw-Hill. 1993:781–2.