Instructional Design Variations in Internet-Based Learning for Health Professions Education: A Systematic Review and Meta-Analysis

Abstract

Purpose 

A recent systematic review (2008) described the effectiveness of Internet-based learning (IBL) in health professions education. A comprehensive synthesis of research investigating how to improve IBL is needed. This systematic review sought to provide such a synthesis.

Method 

The authors searched MEDLINE, CINAHL, EMBASE, Web of Science, Scopus, ERIC, TimeLit, and the University of Toronto Research and Development Resource Base for articles published from 1990 through November 2008. They included all studies quantifying the effect of IBL compared with another Internet-based or computer-assisted instructional intervention on practicing and student physicians, nurses, pharmacists, dentists, and other health professionals. Reviewers working independently and in duplicate abstracted information, coded study quality, and grouped studies according to inductively identified themes.

Results 

From 2,705 articles, the authors identified 51 eligible studies, including 30 randomized trials. The pooled effect size (ES) for learning outcomes in 15 studies investigating high versus low interactivity was 0.27 (95% confidence interval, 0.08–0.46; P = .006). Also associated with higher learning were practice exercises (ES 0.40 [0.08–0.71; P = .01]; 10 studies), feedback (ES 0.68 [0.01–1.35; P = .047]; 2 studies), and repetition of study material (ES 0.19 [0.09–0.30; P < .001]; 2 studies). The ES was 0.26 (−0.62 to 1.13; P = .57) for three studies examining online discussion. Inconsistency was large (I² ≥89%) in most analyses. Meta-analyses for other themes generally yielded imprecise results.

Conclusions 

Interactivity, practice exercises, repetition, and feedback seem to be associated with improved learning outcomes, although inconsistency across studies tempers conclusions. Evidence for other instructional variations remains inconclusive.

Internet-based medical education has proliferated rapidly since the advent of the World Wide Web in 1991. Potential advantages of Internet-based learning (IBL) over other instructional methods include flexibility in time and location of learning, economies of scale, facilitation of novel instructional methods, and the potential to personalize instruction to individual needs.^1–3 Hundreds of published articles have described and evaluated the use of IBL in health professions education.⁴

Educators need evidence-based guidance on how to develop effective IBL.³ Over 200 studies have reported the results of comparing IBL either with no intervention or with traditional (noncomputer) instructional methods in health professions education.⁴ In a previous report (2008),⁴ we sought to identify salient principles regarding when and how to use IBL. While that meta-analysis supported the effectiveness of IBL, the evidence did not clearly identify principles to guide future implementations. Previous reviews have encountered similar limitations.^5–8

An alternative approach to identifying evidence-based principles involves direct comparisons of one Internet-based intervention against another.^9–13 Evidence from such studies, if properly reviewed and synthesized, could inform decisions on when and how to effectively use IBL. We are not aware of previous systematic reviews addressing these questions.

In the present study, we sought to identify and quantitatively summarize all studies involving health professions learners that compared IBL with another computer-based instructional format. We focused our review on health professions learners because—even if fundamental principles of learning apply broadly—the topics, learning objectives, and learners in health professions education vary from other fields of study.

Method

We planned, conducted, and reported this review in adherence to standards of quality for reporting meta-analyses (QUOROM and MOOSE).^14,15

Question

We sought to answer the following question: “What characteristics of IBL interventions, as compared with other computer-based interventions, are associated with improved outcomes in health professions learners?”

Study eligibility

We included studies published in any language that investigated use of the Internet, in comparison with another computer-based intervention, to teach health professions learners at any stage in training or practice, using the Kirkpatrick outcomes¹⁶ of (1) satisfaction, (2) knowledge or attitudes, (3) skills (in a test setting), and (4) behaviors (in practice) or effects on patients.

Definitions of key variables (e.g., cognitive interactivity) have been detailed previously.⁴ We defined health professions learners as students, postgraduate trainees (i.e., residents or fellows), or practitioners in a profession directly related to human or animal health, including physicians, nurses, pharmacists, dentists, veterinarians, and physical therapists. We defined IBL as computer-assisted instruction³ using the Internet or a local intranet as the means of delivery. This included Internet-based tutorials, virtual patients, discussion boards, e-mail, and Internet-mediated videoconferencing.

We excluded studies if all of the computer interventions investigated resided only on the client computer or CD-ROM or if the use of the Internet was limited to administrative or secretarial purposes. We also excluded studies that did not report outcomes of interest or were published only in abstract form.

Study identification

We described our search strategy (Supplemental Digital Content, Box 1, https://links.lww.com/ACADMED/A14) previously.⁴ Briefly, one of us (P.J.E.), a senior reference librarian with expertise in systematic reviews, designed a strategy to search MEDLINE, CINAHL, EMBASE, Web of Science, Scopus, ERIC, TimeLit, and the University of Toronto Research and Development Resource Base. Search terms included words defining delivery concepts (such as Internet, Web, e-learning, and computer-assisted instruction) and participant characteristics (such as “education, professional,” “students, health occupations,” and “internship and residency”). Because the World Wide Web was first described in 1991, our search included all articles published after January 1, 1990. The last search date was November 17, 2008. We identified additional studies both by scanning authors' files and previous reviews and by hand searching the reference lists of all included articles.

Study selection

Four of us (D.A.C., A.J.L., D.M.D., and S.G.), working independently and in duplicate, screened all titles and abstracts to determine whether we should include an article. In the event of disagreement or insufficient information in the abstract, we reviewed the full text, again independently and in duplicate. We resolved conflicts by consensus. Chance-adjusted interrater agreement for study inclusion, determined using intraclass correlation coefficient¹⁷ (ICC), was 0.73.

Data extraction

We developed a data abstraction form through iterative testing and revision. We abstracted data independently and in duplicate for all variables requiring reviewers' judgment. We determined interrater agreement using ICC, and we resolved conflicts by consensus.

We abstracted the following information:

• number and training level of learners,
• topic,
• study design: presence of pretest (ICC: 0.84), number of groups (ICC: 0.97), and method of group assignment (ICC: 1.0),
• course length (ICC: 0.70),
• level of cognitive interactivity (low, moderate, high; ICC: 0.71),
• quantity of practice exercises (absent, few, many; ICC: 0.84),
• outcome assessment method (subjective/objective; ICC: 0.79), and
• quantitative outcome results (mean and standard deviation [SD], or other information for calculating effect size [ES]).

When articles contained insufficient outcomes data, we requested this information from authors.

Desiring to use a common quality metric for both randomized and observational studies, we abstracted information on methodological quality using an adaptation of the Newcastle–Ottawa scale for grading cohort studies.^4,18 We rated the following:

• representativeness of the intervention group (ICC: 0.47),
• selection of the control group (ICC: 0.93),
• comparability of cohorts: either statistical adjustment for baseline characteristics in nonrandomized studies (ICC: 0.34) or randomization (ICC: 1.0) and allocation concealment for randomized studies (ICC: 0.38),
• blinding of outcome assessment (ICC: 0.54), and
• completeness of follow-up (ICC: 0.71).

Three of us (D.A.C, A.J.L., and S.G.) iteratively reviewed all articles to inductively identify themes among the research questions or research hypotheses and to achieve consensus on definitions (Table 1). Then, working first independently and then by consensus, we (again D.A.C, A.J.L, and S.G.) grouped each study by research theme.

Table 1: Research Themes Addressed by Studies Included in a Systematic Review of Internet-based Instruction

Data synthesis

We abstracted information separately for outcomes of satisfaction, knowledge, skills, and behaviors/patient effects. We converted means and SDs or odds ratios to standardized mean differences (Hedges' g effect sizes).^19–22 When sufficient data were unavailable, we used tests of significance (e.g., P values) to back-calculate ESs using standard formulae.²³ For crossover studies, we used (1) means or exact statistical test results adjusted for repeated measures or, if these were not available, (2) means pooled across each intervention.^22,24 For two-group pretest–posttest studies, we used (1) posttest means or exact statistical test results adjusted for pretest or, if these were not available, (2) differences in change scores standardized using pretest variance. If neither P values nor any measures of variance were reported, we used the average SD from all other included studies.

Because we anticipated large inconsistency (heterogeneity), we used random-effects models to pool weighted ESs across studies within research themes. We used the I² statistic²⁵ to quantify inconsistency across studies. I² estimates the percentage of variability across studies that is not due to chance; values greater than 50% indicate large inconsistency. We conducted meta-analyses to pool study results for all themes addressed by two or more studies, except where reviewers agreed that the implementations of that theme were too dissimilar. In addition to the inductively identified themes, we coded the level of cognitive interactivity and quantity of practice exercises, and we performed meta-analyses pooling the results of studies in which these codes varied between study arms. Some studies appeared in more than one meta-analysis. For each meta-analysis, we evaluated separately outcomes of satisfaction and learning (i.e., knowledge or—for the two studies not reporting knowledge—skills or behaviors). Too few studies reported skills and behaviors to permit meta-analysis using these outcomes alone. To explore the robustness of findings to synthesis assumptions, we conducted subgroup analyses based on method of group assignment and study quality.

We used StatsDirect 2.6.6 (Cheshire, United Kingdom) to pool ESs and SAS 9.1 (Cary, North Carolina) to calculate ICC. We defined statistical significance by a two-sided alpha of .05.

Results

Trial flow

We identified 2,527 citations using our search strategy and an additional 178 articles from scanning author files and reviewing reference lists. From these we identified 369 potentially eligible articles (Figure 1), of which 51 reported comparisons between an Internet-based intervention and other computer-assisted instruction.^26–76 Eight of these articles^{29,31,33,34,38,39,50,64} also reported comparisons with no intervention or with a noncomputer intervention, and these appeared in our previous systematic review.⁴ Two of the studies we included were published online ahead of print.^60,73

Figure 1: Trial flow for a systematic review of Internet-based instruction.

We contacted authors of 17 articles for additional outcomes information and received information from 8. One otherwise eligible study³² contained insufficient data to calculate an ES for any outcome, so we excluded it from our final analysis. Four studies had more than two groups and/or used a factorial design to study more than one research theme.^38,51,59,74 We included these studies no more than once per meta-analysis.

Study characteristics

Table 2 summarizes key study features. The Internet-based courses addressed a wide range of medical topics such as chest pain, geriatric psychiatry, electrocardiogram interpretation, math skills, periodontology, communication skills, and psychotherapy. A total of 8,416 learners participated, including 3,290 medical students, 1,504 postgraduate physician trainees, 865 physicians in practice, 303 nurses in training, 485 practicing nurses, 153 dental students, 151 pharmacists in training, 73 practicing pharmacists, and 1,592 other learners (other allied health or mixed groups). Of the 51 studies we included, most (38 [75%]) reported knowledge outcomes and 29 (57%) reported satisfaction, whereas only 4 (8%) reported behaviors or patient effects and 3 (6%) reported skills (See Table 2 and Supplemental Digital Content, Table 1, https://links.lww.com/ACADMED/A14).

Table 2: Description of Studies Included in a Systematic Review of Internet-Based Instruction (N = 51)

Table 2: (Continued)

Table 2: (Continued)

Study quality

Table 3 summarizes the methodological quality of the included studies. Of the 51 studies, 30 (59%) were randomized. Three of 38 (8%) knowledge assessments, one of three (33%) skills assessments, and two of four (50%) assessments of behavior or patient effects used self-report measures. Two studies compared course completion rates as a measure of satisfaction^51,61; all other studies used self-reported satisfaction. Fifteen (52%) of the 29 studies assessing satisfaction, 12 (32%) of the 38 studies assessing knowledge, 1 (33%) of the 3 studies assessing skills, and 1 (25%) of the 4 studies assessing behaviors and patient effects lost more than 25% of participants from the time of enrollment, or they failed to report follow-up. Quality scores (the number of quality criteria present; 6 points maximum) ranged from 0 to 6, with a mean of 3.3 (SD = 1.6).

Table 3: Quality of Studies Included in a Systematic Review of Internet-Based Instruction (N = 51)

Table 3: (Continued)

Research themes

We identified 22 distinct research themes (Table 1). Figure 2 summarizes meta-analysis results for themes addressed by two or more studies, as well as for studies in which the coded level of interactivity or practice exercises varied between groups. The Supplemental Digital Content (https://links.lww.com/ACADMED/A14, Table 2) has additional analysis details including subgroup analyses and study-specific ESs. Further details of analyses are available from the authors on request. To illustrate how studies varied while still focusing on the same theme, we present in the Supplemental Digital Content (https://links.lww.com/ACADMED/A14, Box 2) an in-depth examination of the theories, conceptual frameworks, and instructional methods for several studies reflecting one theme.

Figure 2: Random-effects meta-analysis of different Internet-based instructional designs. All study groupings reflect inductively identified research themes (Table 1), except those marked as “coded” which include studies for which the coded level of that feature (interactivity or practice exercises) varied between groups; see text for details. Boxes represent the standardized effect size (Hedges' g), and bars represent the 95% confidence interval (95% CI). The vertical line represents no effect. The pooled estimate is calculated using a random-effects model. *I² undefined for analyses with two studies.

We did not identify research themes for all coded elements. For example, although we found differences in the coded quantity of practice exercises, we found no studies designed to compare such differences. Similarly, we found fewer studies hypothesizing differences between levels of interactivity (n = 15) than we found with differences in our coding of this feature (n = 21).

Studies investigating interactivity

Fifteen studies compared different levels of interactivity,^{35,39,40,43,49,50,52,56,57,59,62,68,70,73,74} defined as cognitive (mental) engagement with the course other than online discussion (which we analyzed separately, see below). Course designers enhanced engagement using a range of tasks, as outlined below (See also Table 2, Themes).

Eight of these studies explored the use of questions to enhance interactivity. Two of three randomized trials that compared the use of self-assessment questions versus no questions reported statistically significantly higher knowledge test scores for modules with questions,^49,62 while the third reported no significant difference.³⁹ Another randomized trial reported similar outcomes whether or not learners were required to actively respond to the question.⁴⁰ A fifth randomized trial compared IBL modules with case-based questions and tailored feedback against an online practice guideline without questions and found significant improvement in knowledge test scores for interactive IBL.³⁵ Adding extra questions before⁷³ or after⁷⁰ an Internet-based module that already contained several interactive questions did not significantly alter outcomes in two randomized trials. Finally, a nonrandomized study added case-based questions and learning tasks to an Internet-based course consisting of text, video clips, and interactive models and found a nonsignificant association between lower interactivity (no questions) and higher knowledge test scores, but similar satisfaction ratings.⁵⁰

Two randomized trials evaluated the effect of actively summarizing information. Creating a summary of a patient scenario improved knowledge test scores compared with not creating a summary.⁵⁶ However, creating a summary of a tutorial's didactic information did not improve knowledge test scores in comparison with reviewing an instructor-prepared summary, and preference was neutral.⁵⁹

A small randomized trial found a modest (ES 0.57) but nonsignificant benefit of Internet-based modules with self-evaluation, animations, and video in comparison with static PDFs of the same information.⁵⁷ Studying Internet-based, worked-example cases with intentional errors led to significantly improved learning outcomes (knowledge, skills, or behaviors and patient effects) compared with cases without errors in another randomized trial.⁷⁴

Finally, three studies evaluated Internet-based tutorials with varying levels of interactivity, but the differences were poorly defined. One randomized trial compared a multicomponent module with tailored feedback on clinical practice performance against “flat-text Internet-based CME [continuing medical education] modules,”⁴³ while a crossover study compared “interactive Web-based modules” with “noninteractive narrated slide presentations.”⁵² Both found significant improvements in learning outcomes for the interactive group. A third study with ambiguous design found a significant association between knowledge scores and use of an interactive IBL game versus noninteractive “traditional” computer-assisted instruction.⁶⁸

For the 15 studies reporting learning outcomes (knowledge, skills, or behaviors and patient effects), the pooled ES favoring interactivity was 0.27 (95% confidence interval [CI], 0.08 to 0.46; P = .006), I² = 90%. Although statistically significantly different from 0, this is considered a small effect.⁷⁷ Seven studies investigating interactivity reported satisfaction outcomes, with a pooled ES of 0.39 (95% CI, −0.12 to 0.90; P = .13), I² = 95%. Subgroup analyses showed similar findings for high- and low-quality studies (Supplemental Digital Content, Table 2, https://links.lww.com/ACADMED/A14).

A second meta-analysis pooled data from 21 studies in which the coded level of interactivity varied between study arms.^{28–30,33,35–37,39,40,47,49,50,52,54–58,62,68,74} In this coding, learner tasks such as practice exercises, information syntheses, essay assignments, and group collaborative projects supported higher interactivity levels. In this analysis, the pooled ES for learning was 0.53 (95% CI, 0.33 to 0.73; P < .001), I² = 89%. This is considered a medium-sized effect.⁷⁷ Twelve of these studies evaluated satisfaction,^{29,34,37,39,40,47–50,53,58,62} with a pooled ES of 0.31 (95% CI, −0.13 to 0.74; P = .17), I² = 92%.

Studies investigating practice exercises

Although not identified as a separate research theme, the study protocol specified coding the quantity of practice exercises, and this rating varied between study arms for 10 studies.^{27,29,35,39,43,49,50,55,57,62} Meta-analysis revealed a pooled ES of 0.40 (95% CI, 0.08 to 0.71; P = .01), I² = 92%. Subgroup analyses demonstrated similar findings for high- and low-quality studies (see Supplemental Digital Content, Table 2, https://links.lww.com/ACADMED/A14). For the five studies reporting satisfaction outcomes, the pooled ES was 0.59 (95% CI, −0.10 to 1.27; P = .094), I² = 95%. However, when including only randomized trials, this ES was statistically significantly different from 0 (P = .005; see Supplemental Digital Content, Table 2, https://links.lww.com/ACADMED/A14).

Studies investigating online discussion

Seven studies evaluated the impact of online discussion.^{27,29,32,37,38,48,53} Five of these used written asynchronous text-based discussion (e.g., discussion boards or e-mail). Three randomized trials compared Internet-based tutorials with and without online discussion.^27,32,37 None of these three studies demonstrated a statistically significant effect on learning outcomes, although students in one study noted significantly higher satisfaction with the online discussion format.³⁷ A fourth study added a “virtual clinic” (in which students discussed patient cases online) to an existing IBL activity. A comparison with historical controls observed no association with course ratings but a significant association with higher test scores.²⁹ Finally, one study altered a course to promote greater online collaboration and found greater satisfaction among trainees compared with historical controls using a previous course in which online discussion was less prominent.⁴⁸

Two observational studies evaluated the addition of live audio discussion to Internet-based courses without discussion. One used historical controls,⁵³ whereas the other allowed participants to self-select the presentation format.³⁸ Both studies found an association between live audio discussion and higher satisfaction.

For the three studies of online discussion to report learning outcomes, the pooled ES was 0.26 (95% CI, −0.62 to 1.13; P = .57), I² = 93%. In subgroup analyses, the pooled ES varied by study design (P_interaction <.001), with lower pooled ES for the two randomized trials (−0.14) than for the lone nonrandomized study (1.01; see Supplemental Digital Content, Table 2, https://links.lww.com/ACADMED/A14, for details). For the five studies reporting satisfaction outcomes, the pooled ES was 0.32 (95% CI, 0.14 to 0.51; P < .001) with I² = 4% and similar effects for quality subgroups.

Studies investigating feedback

Two randomized studies^28,74 examined the effects of feedback. One study compared feedback with no feedback,²⁸ whereas the other compared detailed feedback versus providing only the correct answer.⁷⁴ The results of both studies favored the more intensive feedback option, with a pooled learning ES of 0.68 (95% CI, 0.01–1.35; P = .047).

A number of other studies used feedback as part of the instructional design, most often in conjunction with self-assessment questions^{35,39,40,49,59,70} but also regarding clinical practice performance⁴³ and explicit feedback from peers.⁴⁸ Unfortunately, simultaneous changes in other instructional design features (confounding) precluded isolating the effect of feedback in these studies.

Studies investigating repetition and spacing

Several studies addressed the theme of spacing (spreading a fixed amount of instruction over time) and repetition (repeating the same instructional activities multiple times). For spacing, one randomized trial compared spreading 40 short IBL modules across 10 weeks using e-mail, versus having all 40 modules available in the first week.⁵¹ The small benefits (ES 0.12 for both knowledge test scores and course completion rates) were not statistically significant. Another randomized study compared a video clip divided into eight small segments against the intact clip and found statistically significant improvements in knowledge test scores (ES 0.88) for the segmented (spaced) format, but there was no difference in satisfaction or skill ratings.⁴⁷ We did not pool these results because the interventions varied substantially.

Considering repetition, two randomized trials compared delivering study questions and answers via e-mail over several weeks and repeating each question once⁷² or twice,⁶³ against nonspaced/repeated instruction (either a Web-based module on the same topic,⁷² or the same questions/answers delivered together on day 1⁶³). The pooled learning ES was 0.19 (95% CI, 0.09–0.30; P < .001).

Studies investigating strategies to promote learner participation

Four observational studies evaluated ways to enhance learner participation in Internet-based courses, including providing printed course guides and enhanced technical support to course participants,⁴⁴ using a Web server instead of a more complicated learning management system,⁴⁵ modifying the screen appearance to promote participation in online forums,⁴⁸ and making the course a more integral part of the curriculum.⁶¹ All of these interventions were associated with improved outcomes; however, because of the heterogeneity in approaches, we did not pool these results.

Studies investigating audio

Two studies explored the use of audio in Internet-based tutorials. One randomized study compared Internet-accessible PowerPoint presentations using written text or an audio voice-over.⁴¹ Although knowledge test scores did not differ significantly between groups, the audio group reported significantly higher satisfaction. However, the audio format also took significantly longer to complete. The other study used a single-group crossover design in which half the Internet-accessible PowerPoint presentations had supplemental audio information that reinforced the text and encouraged further study. This intervention was associated with statistically greater knowledge test scores and satisfaction.⁶⁶ Pooled analyses of these two studies revealed ESs favoring audio of 1.26 (95% CI, −0.36 to 2.88; P = .13) for learning and 0.76 (95% CI = 0.50 to 1.02; P < .001) for satisfaction.

Another two studies explored the use of audio in Internet-based communication. These crossover studies^34,64 found statistically significantly greater preference for Internet-mediated videoconferences over synchronous, online, text-based chat sessions, with a pooled ES of 1.15 (95% CI, 0.15–2.15; P = .02).

Studies investigating instructor-synthesized information

Three randomized trials compared instructor-synthesized information against existing information available on the Internet. One study found a significant benefit on knowledge test scores from an instructor-synthesized series of dermatologic images and brief text.³⁶ Another study found mixed results, with no statistically significant differences for knowledge, skill, or behavior scores, but a large, positive effect on satisfaction for the instructor-synthesized material.⁴⁶ The third study, comparing IBL modules against an online practice guideline, has already been discussed under Interactivity.³⁵ The pooled learning ES was 1.09 (95% CI, −0.20 to 2.39; P = .10), I² = 96%.

Studies investigating games and simulation

Three observational studies^30,50,68 compared learning inductively from games or simulations versus learning from sequentially presented didactic material, with results favoring didactic,⁵⁰ favoring games,⁶⁸ or showing no difference.³⁰ The pooled ES for learning outcomes was 0.07 (95% CI, −0.55 to 0.68; P = .83), I² = 95%.

Studies investigating adaptive navigation

Two randomized trials evaluated Internet-based interventions that adapted according to learner responses. One compared a “narrative” (adaptive) virtual patient with a “problem-focused” virtual patient³¹ and found evidence suggesting improved communication skills for the adaptive format. The other⁷¹ tested learners' knowledge before presenting information and allowed learners to skip module sections if they answered correctly. Although knowledge test scores did not significantly differ, the adaptive format required significantly less time. Because of the heterogeneity in research themes of these studies, we did not pool these results.

Studies investigating blended learning

Two observational studies compared Internet-only versus blended Internet/face-to-face courses. In one,³³ the combination of Internet-mediated and face-to-face discussion was associated with somewhat lower learning outcomes than an Internet-only approach, but multiple variables including cultural differences and language barriers between the learner groups could have contributed to this finding. In another study, students in an Internet-based course could self-select Internet-based or face-to-face discussion groups.⁶⁹ Those choosing Internet discussion had higher course grades than those selecting face-to-face.

Other research themes

Eight studies^{26,38,42,44,45,55,61,76} compared different computer-based learning configurations, and a number of themes were addressed by one study each. These are summarized in the Supplemental Digital Content, Box 3, https://links.lww.com/ACADMED/A14.

Discussion

This systematic review identified a modest number of studies investigating how to improve IBL by comparing one Internet-based intervention with another computer-based intervention. These studies collectively explored a relatively large number of research themes. However, in most cases only a small number of studies had investigated a given research theme. Moreover, the operational definitions of the interventions (and the differences between interventions) varied widely from study to study even within a given theme.

Pooled ESs for satisfaction and/or learning outcomes (knowledge, skills, or behaviors and patient effects) were positive but small⁷⁷ for associations with nearly all of the themes identified. However, the pooled estimates for satisfaction differed significantly from 0 only for associations with interactivity, online discussion, and use of audio for both tutorials and online discussion, whereas estimates for learning differed significantly only for associations with interactivity, practice exercises, feedback, and repetition. Inconsistency (heterogeneity) between studies was large (≥89%) for all but online discussion and satisfaction. These inconsistencies allow us to draw only weak inferences.

Limitations and strengths

Our review has several limitations. First, only a few studies investigated any given research theme, precluding quantitative synthesis of results in many instances. Second, even within a given theme, the conceptual definitions (e.g., what constitutes “interactivity”?), study and comparison interventions, outcomes, and research methods varied. We emphasized similarities when grouping studies, but we acknowledge that important differences may explain much of the observed inconsistency among studies.⁷⁸ The small number of studies precluded subgroup analyses to explore this heterogeneity. We did not use funnel plots to assess for publication bias because these are misleading in the presence of large heterogeneity.⁷⁹ Third, although some studies reflected high methodological quality, the average quality was relatively poor. However, analyses restricted to more rigorous studies yielded similar findings in most instances. Fourth, many articles failed to report key details of the interventions or outcome measures. We obtained additional outcomes data from some but not all authors. Finally, space limitations do not permit a detailed review of the theories and frameworks that support each of the themes identified. However, in our Supplemental Digital Content we illustrate this foundation for one theme (Box 2, https://links.lww.com/ACADMED/A14).

Our review also has several strengths. The question of how to improve IBL is timely and of great importance to medical educators. To present a comprehensive summary of evidence, we kept our scope broad regarding learners, interventions, outcomes, and study design. The systematic literature search encompassed multiple databases supplemented by hand searches and had few exclusion criteria. We conducted all aspects of the review process in duplicate, with acceptable reproducibility.

Comparison with previous reviews

In comparison with our recent reviews identifying 130 studies comparing IBL with no intervention and 76 studies comparing IBL with non-Internet methods,⁴ the 51 studies identified in this review represent a relatively small body of evidence. This is consistent with a review summarizing only the type of study, which reported that only 1% of reviewed studies on computer-assisted instruction compared alternate, computer-based interventions.⁸⁰

We are not aware of previous systematic reviews of studies comparing one computer-assisted instructional intervention with another in health professions education, although some previous reviews included such studies along with no-intervention and media-comparative studies.^81,82 Outside of medical education, authors have provided nonsystematic summaries^83,84 and broadly focused reviews,^6,85,86 but again we are not aware of comprehensive and methodologically rigorous syntheses focusing on how to improve computer-assisted instruction. However, several reviews^5,6,87 and other authors^9–13 have issued a call for more research of this type, suggesting that the present review fills an unmet need.

Implications

The synthesized evidence suggests that interactivity, practice exercises, repetition, and feedback improve learning outcomes and that interactivity, online discussion, and audio improve satisfaction in IBL for health professionals. Although educators should consider incorporating these features when designing IBL, the strength of these recommendations is limited: We found relatively few studies; existing studies address a diversity of themes; even within themes, the interventions and outcomes vary; study findings are inconsistent; and methodological quality is relatively low. Clear guidance for practice will require additional research. Insights from outside the health professions may also be useful.⁸⁴

To strengthen the evidence base, researchers must first come to agreement on what is being studied. Shared conceptual and theoretical frameworks, consistent definitions for interventions and comparison interventions, and the use of common outcome measures may help. Working from shared frameworks, interventions, and outcomes will permit replication across learner groups and different educational objectives. The present summary and synthesis of evidence, along with the research themes identified, form a foundation for such work, and several of the included studies provide exemplary models to follow.

As has been documented previously,⁸⁸ few studies addressed outcomes of skills, behaviors in practice, or effects on patient care. Such outcomes would be desirable in future research.^89,90 However, investigators should ensure that outcomes align with interventions,⁹¹ and they might consider demonstrating effects on applied knowledge and skills before evaluating effects on higher-order outcomes.⁹²

Finally, although the evidence summarized here begins to inform the question, “How should we use IBL?” it largely fails to address the question, “When should we use IBL?” Authors have argued that these decisions are largely pragmatic, based on relative advantages of the Internet over other instructional delivery systems.² Evidence to guide these decisions will derive from studies, including qualitative analyses, designed to clarify relationships between potential advantages and specific topics, course objectives, and learner characteristics.^13,93

Conclusions

Although existing evidence does not permit strong recommendations for educational practice, this review has highlighted promising areas for future research. Evidence will derive from multiple sources, including randomized and observational quantitative studies and rigorous qualitative research. Clear conceptual frameworks, focused research questions, well-defined interventions, study methods appropriate to the question, and adherence to reporting standards when disseminating results all will help advance the science of IBL.

Acknowledgments:

The authors thank Melanie Lane, Mohamed Elamin, MBBS, and M. Hassan Murad, MD, from the Knowledge and Encounter Research Unit, Mayo Clinic, Rochester, Minnesota, for their assistance with data extraction and meta-analysis planning and execution, and Kathryn Trana, from the Division of General Internal Medicine, Mayo Clinic, for assistance in article acquisition and processing.

Funding/Support:

This work was supported by intramural funds and a Mayo Foundation Education Innovation award.

Other disclosures:

Dr. Levinson is supported in part through the John R. Evans Chair in Health Sciences Education Research.

Ethical approval:

Because no human participants were involved, this study did not require ethical approval.

Previous presentations:

Select portions of these results were presented in symposia at the 2009 meetings of MedBiquitous (April 29, 2009; Baltimore, Maryland), the Association of Medical Education in Europe (August 31, 2009; Malaga, Spain), and the Association of American Medical Colleges (November 9, 2009; Boston, Massachusetts).

Disclaimer:

Dr. Cook had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.