The first reports of Web-based medical education appear in 1992,1 building on 30 years of computer-assisted instruction. Proponents claimed computer-assisted instruction is superior to text-based, lecture, and traditional educational methods for reasons that include control by the learner over content, time, and place of learning2; enhancement of learning, reasoning, and efficiency3,4; and cost savings.5 Many studies that reported advantages contained methodologic flaws6 and reported advantages unrelated to computer-specific features.7
Web-based learning (WBL) represents a further evolution of computer-assisted instruction. Technical advantages of WBL include universal accessibility, ease in updating content, and hyperlink functions that permit cross-referencing to other resources.8 These technical advances, specifically hyperlink and searching capabilities, fit the constructivist learning theory, where learners search out and create their own knowledge bases. However, as was evident with computer-assisted instruction, potential advantages may not translate into significant improvements in educational outcomes. With the widespread adoption of WBL, it is critical that medical educators have evidence regarding its performance as a learning medium. This paper reviews the medical, dental, and nursing WBL evaluation literature to: (1) identify which facets of WBL have been evaluated, (2) describe the evaluation strategies used, (3) synthesize the findings, and (4) discuss educational implications and future research directions.
We searched Medline and ERIC for “Computer-assisted Instruction” and “Internet or World Wide Web” and “Education, continuing or Education, dental, or Education, medical, or Education, nursing” from 1966 to January 2002, limited to English language.
Having selected articles from their abstracts, two authors independently applied the criteria in Figure 1 to classify each article as descriptive or evaluative. If the abstract was unavailable or lacked sufficient information, the article was reviewed. When the two authors disagreed, the third author broke the tie.
We categorized the WBL articles by evaluation domains. Where possible, we divided studies into those evaluating features specific to the Web medium versus those specific to educational content.
We determined percentages of descriptive and evaluative studies, identified evaluation domains, described evaluation strategies, and synthesized the findings. The small number of studies and heterogeneity of study designs and participants precluded combining the data by meta-analysis.
Results and Discussion
We identified 206 WBL papers between 1992 and 2001 in Medline and ERIC. As shown in Figure 1, 76 met our criteria for inclusion, 41 (59%) being descriptive and 35 (46%) evaluative. The two authors agreed on this classification for 71 of 76 papers (kappa = 0.869, p < .001). In the five case of disagreement, the third author classified one study as descriptive and four as evaluative.
From the 35 evaluation papers, we identified four domains: knowledge gains, learner attitudes, learning efficiency, and program cost. We divided studies of learner attitudes into those evaluating Web-specific and content-specific attributes. We could not similarly divide studies measuring knowledge gains due to multiple confounding educational influences in all but two studies.9,10 Table 1 depicts the evaluation domains, keywords used to describe the focus of the investigation, and the number of studies in each category.
Domain 1: Studies Evaluating Knowledge Gains
Twenty studies evaluated knowledge gains with WBL interventions. Eighteen measured changes in multiple-choice test scores; one used a written case analysis; and one, a multiple-choice test plus a standardized patient interview. Research methods included pretest/post-test self-controlled studies, self-selected controlled studies, assigned (non-randomized) controlled studies, assigned crossover trials, and randomized controlled trials. The most common research design was a pretest/posttest self-controlled study, but there were several randomized controlled trials (see Appendix Table A). The pretest/posttest studies provided strong evidence that WBL interventions resulted in knowledge gains in medical students, practicing physicians, and dentists.11,12,13,14,15 However, studies without a control group cannot separate Web-specific from content-specific learning gains.
Studies where a WBL group and a control group received identical educational content allowed identification of learning gains attributable specifically to the Web medium. Studies that met this criterion demonstrated no difference in knowledge gains between groups.16,17,18,19
As the research designs strengthened, results confirmed WBL was comparable but not superior to other educational methods. One assigned crossover study showed no difference in scores between the WBL and slide/tape groups.10 In randomized controlled trials, authors found WBL superior to no educational method20,21 and equivalent to text-based methods.9 Of three studies of traditional courses with and without Web enhancement, one reported enhanced learning and two did not.22,23,24 In summary, WBL improved posttest scores on multiple-choice tests but did not outperform other educational methods.
Domain 2: Learners' Attitudes
We categorized studies of learners' attitudes into Web-specific and content-specific. Web-specific studies were (1) comparisons between WBL and other methods, (2) predictors of satisfaction with technology, (3) patterns of Web use, and (4) Web-specific enhancements of self-directed learning or learning stimulation (see Appendix Table B). Content-specific studies were evaluations of (1) courses, (2) predictors of satisfaction with courses, and (3) self-reported changes in effectiveness of learning and confidence (see Appendix Table C).
- Do learners prefer WBL to other educational methods? Several studies assessed learners' preference for WBL and their desire to use WBL again (see Appendix Table B). Most learners planned to use WBL again12,15,21 and preferred WBL to continuing medical education conferences, lectures, video, audiotapes, journals, or text-books.17,23 One study compared WBL with slide/tape and reported that 71% of students preferred the WBL.10 A randomized controlled trial reported higher satisfaction with WBL compared with print materials, on a scale of 5 to 20 (means: WBL = 17, text = 15, p < .001).9 Many of these studies introduced selection bias by recruiting participants over the Internet. Nonetheless, two strong studies demonstrate preferences for WBL over slide/tape and text-based materials.9,10
- What predicts learners' satisfaction with Web technology? Properties specific to WBL include Web-site accessibility, navigation, and attractiveness. Studies that evaluated these features universally reported high learner satisfaction.10,12,16,21,23,26,29 The main predictor of satisfaction with WBL was download speed. One cross-sectional observational study reported that students who perceived fast download time gave above-average course ratings more often than did students who perceived slow download time (OR 4.25), and concluded that download speed was as important to learners' satisfaction as content.30 Four additional articles reported slow download speed as WBL's major disadvantage and most significant barrier to learning for graduate nursing, dental, radiology, and medical students.10,24,27,31
- How do learners react to asynchronous interactions with faculty and peers? There are positive and negative reports of the asynchronous interactions with instructors and peers typical of WBL.21,32,33 One study of a Web-based nursing course reported that learners were more likely to work together and felt more comfortable disagreeing with the instructor or asking awkward questions.34 However, a qualitative study reported equal numbers of positive and negative comments regarding asynchronous interaction with instructors.21 There is insufficient evidence to determine when asynchronous communication benefits or adversely affects learning.
- What is known about patterns of Web use in WBL? Patterns of Web use were determined from learners' self-reports or from server statistics. Information from server statistics indicated higher use immediately before an examination and up to a 20% discrepancy from self-reported use.35 Self-reported use varied extensively between studies, with little clarification or explanation.27,36,37 For example, in one randomized controlled trial, 72% of the Web-based group reported never using the Web materials, citing lack of time as the major barrier.27
- Does WBL enhance or stimulate learning or encourage learners to look for answers? There are mixed reports on WBL in these areas. In one study, only 50% of learners believed the Web-based experience enhanced their learning.24 Another study reported increased stimulation to learn in a WBL program versus a traditional course. However, this study was limited by a high crossover between groups—70% of WBL students also chose to attend lectures.18 A third study of nursing students reported that 85% of graduate and 28% of undergraduate students would search for answers personally rather than asking an instructor because of how the course used the Web.34 This suggests that WBL may stimulate learning differently depending on educational level. It is unclear whether WBL outperforms other media by enhancing or stimulating learning or encouraging learners to seek out answers.
- How do learners rate WBL courses? Many authors reported positive evaluations of WBL programs,13,26,31 but most studies comparing WBL with other methods found similarly high ratings for both modalities.17,27,32 However, one randomized controlled trial reported higher overall evaluations from students in the WBL group compared with a text-based group.9
- Which design features predict satisfaction with an educational course? In a correlation model, sound instructional technique was the only variable that contributed to the variance in students' satisfaction with their learning in a WBL course.38 Variables without effect included previous experience with WBL, self-rated competence with technology, frequency of communication between classes, learners' age, remote group size, technology, and course management. The authors concluded that sound pedagogy is more important that technology in determining learners' satisfaction.
- Do WBL programs improve learners' confidence? Studies that evaluated learners' self-reported confidence or competence reported improvements after any structured educational intervention. WBL programs achieved changes similar to those attained with other methods.11,13,15,21,28,37 There is strong evidence that well-designed WBL programs can improve learners' confidence, similarly to other educational methods.
Domain 3: Studies Evaluating Efficiency of Learning
Only two studies evaluated changes in learning efficiency. One well-designed randomized controlled trial comparing WBL with text-based learning reported that learners achieved equivalent test scores with shorter study times using WBL materials (27 minutes versus 38.5 minutes). The authors calculated efficiency scores (median improvement in score per hour with 95% confidence interval) of 8.6 (7.1–11.7) for WBL and 6.7 (5.9–8.1) for text-based learning (p = .04).9 However, the second study reported conflicting findings, with only ten of 32 students perceiving that WBL was more efficient than studying a textbook.31 Learning efficiency needs further study in well-designed trials.
Domain 4: Studies Evaluating Costs of WBL Programs
Only one 1995 retrospective study evaluated the cost of WBL. The authors calculated lower direct and indirect (distribution) costs for WBL compared with text. However, they did not consider the educational design costs for either modality and assumed that hardware, software, and other equipment were available, and that the institution had a commitment to technology.39 Based on one paper, cost saving in the printing and distribution of materials is a potential advantage of WBL, but further study is needed to determine whether lower distribution costs offset the costs of technical support, and whether WBL saves or costs money in terms of faculty time.
This study reviewed only medical, dental, and nursing literature, although there are many publications on WBL in other higher education fields. The authors limited the search to these health professions because they share similar undergraduate, postgraduate, and continuing educational methods as well as many content areas that differ from those in other types of higher education. With the trend towards interdisciplinary education, many future WBL programs will target multiple health professions. Nevertheless, this wide variety of learners at varying educational levels in different health professions might possess different motivations or learning biases that could confound the results of our review. Another potential weakness of the review is missed studies. There is no Medline heading for WBL. Instead, the authors searched using the keywords “Internet” or “World Wide Web,” which may have decreased the number of studies located.
Despite the rush to embrace WBL, it does not address all the challenges of medical education. It is a valuable addition to our educational armory, but it does not replace traditional methods such as text, lectures, small-group discussion, or problem-based learning. Educators still must define WBL's unique educational contribution. Evaluation of WBL is in its infancy. Although most learners welcome WBL (provided that download speed is fast), and give high satisfaction ratings, there is no evidence that students learn more from Web-based programs than by traditional methods. Students may learn more efficiently, but there is minimal information about the relative costs of WBL programs. Finally, curriculum development and instructional design are no less important for Web-based educational interventions than for other media. Educators must recognize that poorly designed educational programs or materials are not improved by being presented on a Web page.
Medical educators must use well-designed curricula regardless of the method of delivery. We did not find Web-based programs to be superior to traditional methods in terms of gains in learning or learners' satisfaction. We recommend that educators tailor their teaching media to learners' needs rather than assume that WBL is intrinsically superior. When designing a Web-based educational program, educators should create materials that load quickly from all types of Internet connections.
Directions for Future Research
Research into WBL is in the early stages and many research questions remain unanswered. Which learners benefit most from WBL? Can WBL increase learning efficiency? Are there cost savings compared with other educational methods? Does WBL enhance knowledge using other evaluation methods besides multiple-choice tests? How can interactions between learners and instructors be fostered and preserved with WBL? Does WBL as a second modality add to the educational experience? Finally, investigators may wish to design trials that compare the strengths and weaknesses of educational methods rather than establish the superiority of one medium over another. Such trials would clarify the most appropriate uses of WBL in medical education.
1. Computer communication for international collaboration in education in public health. The TEMPUS Consortium for a New Public Health in Hungary. Ann NY Acad Sci. 1992;670:43–9.
2. Piemme TE. Computer-assisted learning and evaluation in medicine. JAMA. 1988;260:367–72.
3. Clayden GS, Wilson B. Computer-assisted learning in medical education. Med Educ. 1988;22:456–67.
4. Henry JB. Computers in medical education: information and knowledge management, understanding, and learning. Hum Pathol. 1990;21:998–1002.
5. Bolwell C. Using computers as instructional technology in the pressure ulcer field. Decubitus. 1993;6(4):20–5.
6. Clark RE. Reconsidering research on learning from media. Rev Educ Res. 1983;53:445–60.
7. Keane DR, Norman GR, Vickers J. The inadequacy of recent research on computer-assisted instruction. Acad Med. 1991;66:444–8.
8. Haag M, Maylein L, Leven FJ, Tonshoff B, Haux R. Web-based training: a new paradigm in computer-assisted instruction in medicine. Int J Med Inform. 1999;53:79–90.
9. Bell DS, Fonarow GC, Hays RD, Mangione CM. Self-study from Web-based and printed guideline materials. A randomized, controlled trial among resident physicians. Ann Intern Med. 2000;132:938–46.
10. Ludlow JB, Platin E. A comparison of Web page and slide/tape for instruction in periapical and panoramic radiographic anatomy. J Dent Educ. 2000;64:269–75.
11. Engel SS, Crandall J, Basch CE, Zybert P, Wylie-Rosett J. Computer-assisted diabetes nutrition education increases knowledge and self-efficacy of medical students. Diabetes Educator. 1997;23:545–9.
12. Francis B, Mauriello SM, Phillips C, Englebardt S, Grayden SK. Assessment of online continuing dental education in North Carolina. J Cont Educ Health Prof. 2000;20(2):76–84.
13. Harris JM, Salasche SJ, Harris RB. Can Internet-based continuing medical education improve physicians' skin cancer knowledge and skills? J Gen Intern Med. 2001;16:50–6.
14. Kronz JD, Silberman MA, Allsbrook WC, Epstein JI. A Web-based tutorial improves practicing pathologists' Gleason grading of images of prostate carcinoma specimens obtained by needle biopsy: validation of a new medical education paradigm. Cancer. 2000;89:1818–23.
15. Sakowski HA, Rich EC, Turner PD. Web-based case simulations for a primary care clerkship. Acad Med. 2001;76:547.
16. Cho IS, Park HA. Development of a Web-based CAI program for maternity nursing practice. Medinfo. 1998; 9:t-9.
17. Fulkerson PK, Miller A, Lizer S. Using WWW-based instruction modules and e-mail for a remote neurology course. Acad Med. 1999;74:576–7.
18. Woo MA, Kimmick JV. Comparison of Internet versus lecture instructional methods for teaching nursing research. J Prof Nurs. 2000;16:132–9.
19. Yucha C. Insights learned from teaching pathophysiology on the World Wide Web. J Nurs Educ. 2000;39:68–72.
20. Balcezak TJ, Lynch P, Jackson S, Richter J, Jaffe CC, Cadman EC. A Web-based risk management and medical-legal curriculum for graduate medical education. J Biocommunication. 1998;25(4):2–5.
21. Curran VR, Hoekman T, Gulliver W, Landells I, Hatcher L. Web-based continuing medical education. II: Evaluation study of computer-mediated continuing medical education. J Cont Educ Health Prof. 2000;20:106–19.
22. Chan DH, Leclair K, Kaczorowski J. Problem-based small-group learning via the Internet among community family physicians: a randomized controlled trial. MD Comput. 1999;16(3):54–8.
23. Perryer G, Walmsley AD, Barclay CW, Shaw L, Smith AJ. Development and evaluation of a stand-alone Web-based CAL program. A case study. Eur J Dent Educ. 2000;4(3):118–23.
24. Lipman AJ, Sade RM, Glotzbach AL, Lancaster CJ, Marshall MF. The incremental value of Internet-based instruction as an adjunct to classroom instruction: a prospective randomized study. Acad Med. 2001;76:1060–4.
25. Mehta MP, Sinha P, Kanwar K, Inman A, Albanese M, Fahl W. Evaluation of Internet-based oncologic teaching for medical students. J Cancer Educ. 1998;13:197–202.
26. Agius RM, Bagnall G. Development and evaluation of the use of the Internet as an educational tool in occupational and environmental health and medicine. Occup Med. 1998;48:337–43.
27. Baumlin KM, Bessette MJ, Lewis C, Richardson LD. EMCyber School: an evaluation of computer-assisted instruction on the Internet. Acad Emerg Med. 2000;7:959–62.
28. Fleetwood J, Vaught W, Feldman D, Gracely E, Kassutto Z, Novack D. MedEthEx Online: a computer-based learning program in medical ethics and communication skills. Teach Learn Med. 2000;12:96–104.
29. Hashiba M, Inagawa K, Matsuto T, Motonaga A, Yamakawa T, Akazawa K. Application of the Real Audio package to computerized medical lectures. Med Inform Internet Med. 2000; 25:239–45.
30. Sekikawa A, Aaron DJ, Acosta B, Sa E, LaPorte RE. Does the perception of downloading speed influence the evaluation of Web-based lectures? Public Health. 2001;115:152–6.
31. Horsch A, Balbach T, Melnitzki S, Knauth J. Learning tumor diagnostics and medical image processing via the WWW—the case-based radiological textbook ODITEB. Int J Med Inform. 2000;58–59:39–50.
32. Rose MA, Frisby AJ, Hamlin MD, Jones SS. Evaluation of the effectiveness of a Web-based graduate epidemiology course. Comput Nurs. 2000;18:162–7.
33. Thornam CL, Phillips S. Interactivity in online and face-to-face sections of a graduate nursing course. Techtrends. 2001;45(1):34.
34. Thiele JE, Allen C, Stucky M. Effects of Web-based instruction on learning behaviors of undergraduate and graduate students. Nurs Health Care Perspect. 1999;20:199–203.
35. McNulty JA, Halama J, Dauzvardis MF, Espiritu B. Evaluation of Web-based computer-aided instruction in a basic science course. Acad Med. 2000;75:59–65.
36. Letterie GS, Salminen ER, McClure GB. An electronic bulletin board for instruction in reproductive endocrinology in a residency in obstetrics and gynecology. Fertil Steril. 1996;65:883–5.
37. Stillman GA, Alison J, Croker F. Using the World Wide Web to improve medication calculation skills. Innovations Educ Training Int. 1999;36:17–25.
38. DeBourgh GA. Technology is the tool, teaching is the task: student satisfaction in distance learning. In: SITE 99: Society for Information Technology & Teacher Education International Conference, 10th, San Antonio, TX, February 1999.
39. Cimino C, Reichel J, Serrano M. Cost efficient management of educational material. Proceedings—Annual Symposium on Computer Applications in Medical Care. 1995;493–7.