Evaluation of Web-based Computer-aided Instruction in a Basic Science Course

Abstract

PURPOSE 

To demonstrate the applicability of server statistics, in combination with user surveys, to evaluate utilization of Web-based computer-aided instruction (CAI) in the undergraduate medical curriculum.

METHOD 

Individual user surveys with students' names provided information about computer literacy prior to the course and use of CAI during the course. Utilization of specific web-based CAI developed for the course was recorded by server software and the daily logs correlated with course content. Regression analyses were used to measure correlation of server access logs of individual students versus information from user surveys and performances in the course based on in-course examinations.

RESULTS 

There was no correlation between computer literacy of students at matriculation and their subsequent levels of use of CAI in the curriculum. Utilization of CAI developed for specific course objectives coincided closely with course content, which is an indication of the effectiveness of the applications in achieving their curricular objectives. In contrast, student use of tutorials coincided most closely with in-course examinations. Students' responses to surveys were generally substantiated by server statistics, but discrepancies were sufficiently large (10% to 20%) to call into question the validity of these surveys. Significant differences in CAI utilization correlated with the performances of students in the course.

CONCLUSIONS 

This study demonstrates an important advantage of web-based applications to collect and evaluate CAI utilization efficiently and objectively at both the level of the class and the level of the individual student.

In 1989, 71 articles appeared that described the use of computers in medical education. Nine years later, more than twice that number of articles (149) addressed the subject, reflecting the quick rise of computer-aided instruction (CAI) (data from NCI PubMed). This greater use of computers can be attributed both to technologic advances, especially the development of the World Wide Web,^1–3 and to the increasing incorporation into instruction of problem-based, small-group, and self-directed learning.⁴.

In the early 1990s, Loyola University of Chicago Stritch School of Medicine began an extensive evaluation of its curriculum. One of the recommendations that arose from that evaluation was to enhance small-group and independent learning by expanding the use of CAI. In 1993, just as Loyola began implementing curricular changes, the National Center for Supercomputing Applications (NCSA) at the University of Illinois, Champagne-Urbana, developed the Mosaic browser, beginning the technologic revolution of the World Wide Web (WWW). With these new technologic tools, several faculty and staff developed the Loyola University Medical Education Network (LUMEN; http://www.lumen.luc.edu/lumen), to deliver CAI tailored specifically to the medical curricula.⁵.

One of our first-year basic science courses, Structure of the Human Body, was among the first to implement these Web-based technologies into its curriculum. In the intervening years, the web site for the course has grown to include more than 10,000 files (html, image, and video) comprising more than 530 megabytes of server storage. The computer lessons, designed to meet specific learning objectives, are used by students in all aspects of the course (lectures, labs, small groups, and independent learning). Additionally, CAI in the course falls into three general categories: (1) supplements to course materials (e.g., multimedia laboratory dissector, lectures, case study materials); (2) tutorials and practice examinations; and (3) a web forum for asynchronous interactions between faculty and students.

In this article, we report the results of evaluations on the use of the computer-based educational materials during the course as it was taught in 1998. For researchers, one important advantage of web-based technologies is the ready availability of utilization data from server statistics; data that can supplement survey data. We evaluated three basic areas of interest: (1) Is there a relationship between students' “computer literacy” at matriculation and the extent of their subsequent use of CAI?; (2) How do students use CAI in the context of the course curriculum?; and (3) Is there a correlation for individual students between their use of CAI and their performances on in-course examinations?

METHOD

Surveys

We conducted two surveys of the students in the course. Prior to matriculating, incoming students filled out a questionnaire on their overall computer literacy; more specifically, on their familiarity with computers, prior computer experiences, whether or not they owned computers, etc. The day before the course's final examination, the students completed a second questionnaire, which asked how they had used specific computer applications in the course and the extents to which they had used them. The students included their names on each survey to ensure full compliance and to allow us to correlate their responses with the server statistics we had collected for individual students. We excluded from the analyses the students who either did not turn in the questionnaire or did not answer specific questions.

Server Statistics

All of the workstations in the medical school use the Windows NT 4.0 operating system, which requires students to log in before accessing the Novell file server. Each student's login is in an access log. We used those logs to determine the number of logins and length of each session for each student by date. Data from the entry logs were extracted and converted to Excel format for further analyses.

The activities on the LUMEN web server are recorded by Netscape server software (Netscape Enterprise 3.6). For each request for a file (html, image, or video), the software records the time and date of the request, the file name, the directory, and the IP address of the “client” (the computer from which the request came). Using a program extraction and reporting language script file, we tallied the number of “hits” (requests) originating in the Loyola University domain (luc.edu) and made to file directories that contained CAI material related to the Structure of the Human Body course. We converted those data to a tab-delimited text file and imported that into Excel for further analyses.

Users of the LUMEN Web Forum were required to enter a login ID and a password. Utilization of the Web Forum was based on the number of times each student logged into the server, which compiled an entry log.

Students were also able to access the networks through external modems. In these cases students were not required to log into the Novell network, but their activities were recorded by the web server as the “luc.edu” domain if they were using the university-approved ISP (Internet Service Provider).

Analyses

Of the 130 students in the course, 124 completed the questionnaires. We calculated correlation coefficients between respective variables by simple regression analysis and compared means with the Student t test.

RESULTS

General Use of CAI in the Course

Server statistics showed that, during the course, clients in the Loyola University (luc.edu) domain made three to four times the number of hits to the LUMEN web site as during the rest of the year (Figure 1). During the two months of the course (August 2, 1998, to October 2, 1998), each student logged into the server an average of 95 times (±4.15 SEM; range = 4 to 274) and spent an average total login time of 52.9 hours (±3.49 SEM; range = 0.6 to 274 hours). Based on these figures, the average length of each login was 0.61 hours (±0.05 SEM; range = 0.12 to 3.09 hours). When we compared the utilization statistics from the three periods that corresponded to the three incourse exams, we found that, as the course progressed, the students increased their average number of logins but decreased the average length of each login (Table 1). Further analysis of these data by performance in the class based on grade revealed differences between the upper and lower thirds of the class as the course progressed, especially in the average length of each login period (Table 1).

Figure 1: Monthly counts of the numbers of server requests (“hits”) from the university domain (luc.edu) spanning the two months (August and September) when the Structure of the Human Body course was taught.

Table 1: Average Numbers of Logins, Total Login Times, and Length of Each Login According to the Three Exam Periods of the Course

Results of the student survey conducted prior to matriculation showed that the majority (65%) of students were “very comfortable with computers” or needed to be “just shown the basics” (29%). No students indicated that they were “unfamiliar with the use of computers.” A small minority (five students) indicated they were very skilled with computers. There was a much broader distribution of students' experiences using web browsers. The class breakdown on use of the WWW was as follows: 32% used it “occasionally,” 35% used it “weekly,” and 28% used it “almost daily.” Subsequent regression analyses using survey data and server statistics revealed no significant correlation between individual students' pre-matriculation computer literacy or use of the WWW and their subsequent use of the network (number of times and total amount of time logged into the server).

Specific Use of CAI in the Course

Supplementary materials

A variety of CAI lessons were developed to supplement specific course objectives. Analyses of server statistics showed that the number of daily hits in the directories containing specific CAI lessons coincided closely with the times specific subjects were taught. For example, as illustrated in Figure 2A, the LUMEN dissector was used extensively during the first eight weeks of the course on those days when laboratory dissections were scheduled. Each student, on average, retrieved between 30 and 130 files on each of the days a laboratory was scheduled. Figure 2B provides a second example: hits to CAI lessons developed for supplementing instruction on the head and neck significantly increased when that subject was covered in class and again right before the examination on that material. Similar correlations were observed for lessons developed for other sections of the course.

Figure 2: Figures A through D show daily counts of the numbers of server requests for the LUMEN Dissector (A), head and neck directory (B), and tutorial for cross-sectional anatomy (C), and the daily number of logins to the Web Forum (D) spanning the nine weeks of the course. Black bars indicate each of the nine weeks (Monday to Friday). Arrowheads indicate when the three in-course examinations were given. The three-day period following the second exam (gray bar) indicates the Labor Day weekend. The shaded area in Figure B denotes the time when the head and neck regions were covered in lecture and lab.

Tutorials and practice examinations

A number of tutorials and practice examinations (both practical and written) were developed specifically for the course. Figure 2C shows the pattern of daily hits for one example of a tutorial on cross-sectional anatomy based on the Visible Human data from the National Library of Medicine. In this and every other case, the students' use of these resources coincided closely with the in-course examinations, with the peak number of server requests occuring on the day before each examination. There was a tendency for increased use with each exam, a trend that was also seen in other tutorials and practice exams provided to the students.

Web forum

The third general area of CAI used in the course was the LUMEN Web Forum, a web-based, multi-media-capable application that was developed for threaded discussions of course content by students and faculty outside of class time. The forum was restricted to students and faculty in the course; they had to log in (with password). Server logs showed that 114 students (89% of the class) accessed the forum at least once during the course; 43 (38%) of those actively participated in the online discussions. There were 263 postings (152 by students; 111 by faculty); an average of approximately four postings per day during the course. The number of daily logins to the forum was generally constant during the course, with peaks occurring prior to the exams and another large peak during the seventh week, when there was a lively discussion of the anatomy of the perineum (Figure 2D).

The user survey we conducted at the end of the course substantiated much of the data collected from the server statistics. For instance, 81% of the class indicated they logged into the Structure of the Human Body web site daily to four times per week. The students' responses on the use of specific applications also were consistent with the server statistics. For example, 87% indicated they used the LUMEN Dissector during laboratory dissections (see Figure 2A). Similarly, 88% used tutorials and practice exams “only for review” (see Figure 2C). There was a significant correlation for performances of individual students across all exams (i.e., students who performed well on the first exam tended to have high scores on the second exam and on the final).

Correlation of Use of Computers with Performance in the Class

Regression analyses of performance in the class with either server statistics or user surveys for individual students failed to show any significant correlation when the class was treated as a whole or when divided according to grades (upper quartile, upper third, etc., versus lower quartile, lower third, etc.). However, when we analyzed grouped data, we found a significant inverse correlation between grades and the average length of time students logged into the server for the first part of the course (Figure 3). As the course progressed, this correlation diminished due to changes in the login habits of students in the lower third of the grade distribution (Figure 3 and Table 1).

Figure 3: Plots of mean grade versus average length of each login on the server for the upper, middle, and lower 33% of the class, based on performances on the exams for the first part (A), the second part (B), and the third part (C) of the course. The regression lines and p values for correlation coefficients are provided.

Server statistics on the use of the Web Forum revealed that students in the upper 33% of the class (based on grade) tended to log in more frequently than did students in the lower 33% of the class for all three parts of the course (data not shown). Moreover, regression analysis of final grade versus number of logins showed a stronger correlation for the upper 33% compared with the lower 33% of the class (Figure 4).

Figure 4: Plots and regression lines of grade versus number of Web Forum logins over the entire course for individual students in the upper 33% (A) and the lower 33% (B) of the class, based on final grade. The correlation coefficient for the upper 33% gave a p = .06.

DISCUSSION

Three general conclusions can be drawn from this study. First, a medical student's level of computer literacy at matriculation is not a good predictor of the extent to which he or she will use CAI in the curriculum. Second, server statistics, when combined with user surveys, are important for validating students' use of CAI. Third, the degree and manner of students' use of CAI tended to correlate with their performances in class.

The World Wide Web has undergone phenomenal growth in its number of applications and content.⁶ Consequently, medical schools are increasingly adopting the web for education and administrative tasks.^7–9 The technology offers simple but dynamic tools to teachers for enhancing curricular resources; it facilitates educational interactions among faculty and students; it offers the ability to incorporate up-to-date information quickly and efficiently; and through hypertext, it provides for immediate linking to content in an increasing number of databases.¹⁰ The present study offers another important advantage: the capacity to track and analyze use of resources in the context of curricular objectives.

Because multimedia, hypertext applications can now be developed and delivered with great ease, more and more faculty are developing CAI who previously would have been discouraged by the technologic demands (e.g., programming expertise and expensive hardware for producing CD-ROMs). As more faculty develop CAI, the assessment and evaluation of that CAI becomes increasingly problematic. Our study shows that server statistics can be easily used to provide an important component of CAI evaluation; specifically, how and when students use the applications. Identifying specific CAI materials that are used by students most frequently can further enhance the development of those tools.⁸ At the Stritch School of Medicine, additional formative evaluations were solicited from volunteer students, who were asked to fill out questionnaires addressing CAI design (e.g., navigation, image qualities, presentation formats, etc). These evaluations were conducted by the Office of Computers in Education.

Surveys are used extensively to evaluate CAI.^7,8,11 On the one hand, our survey substantiated the overall assumption that access of CAI implied its use. On the other hand, individual discrepancies between the survey and the server statistics were sufficiently large (10% to 20%) to call into question the accuracy and validity of user surveys. For example, ten students who indicated on their questionnaires that they “never logged into the Web Forum” actually logged in between one and five times. Three other students who, according to the server statistics, never logged in indicated that they logged in “frequently.” Without any indications of server malfunction, we conclude that the information provided through user surveys is less reliable, for reasons that may be both intentional and unintentional.

Server-based CAI provides the added advantage of allowing correlations between CAI use and comprehension of course content. In the present study, comprehension was measured by performance on in-course examinations. Our study did not directly address the question of effectiveness of the CAI to produce higher grade scores, as reported previously.^11–15 Nevertheless, our analyses revealed some important trends that correlated with class performance. Particularly revealing were the data on server access showing that students in the lower third of the class significantly reduced the length of each session on the server as the course progressed compared with the upper third of the class, whose login times were more consistent (Table 1, Figure 3). This difference in behaviors probably reflects either differences in comprehension of the CAI materials or different study habits. Whatever the reason, it is noteworthy that these differences were indistinguishable by the end of the course.

There are a number of factors that may have confounded the collection and subsequent interpretation of the data on server-based CAI use. First, students may have studied in groups, gathering around a computer screen while only one student actually logged in. Since many applications were designed to be used in groups (e.g., LUMEN Dissector, case-base sessions), we provided a generic login ID for those applications. Some students may have used this generic login ID for individual sessions as well. Finally, the CAI materials are available through the Internet, and as the course progressed, an increasing number of students accessed the server from home. In those cases, their login IDs were not recorded (except for the LUMEN Web Forum). The activities of those students who used third-party providers for their Internet access would not even have been recorded as belonging to the “luc.edu” domain. It is noteworthy that trends were detectable in spite of these confounding factors.

To summarize, the present study highlights an important advantage of web-based applications for medical education, which is the collection of server statistics on CAI. These objective data on utilization are critical to the evaluation of CAI resources. At one level, the hypothesis that effective CAI is appropriately used in the context of specific curricular objectives can be easily tested quantitatively. Additionally, opportunities for evaluating CAI at the level of the individual student are enhanced. One aspect that requires further analysis is the relationship between CAI use and comprehension of course content. In this regard, increasingly sophisticated evaluation paradigms will become available as the technology improves. Finally, our study demonstrates how server statistics may be used to evaluate more traditional evaluation tools such as user surveys.