Secondary Logo

Journal Logo

The effectiveness of internet-based e-learning on clinician behavior and patient outcomes: a systematic review protocol

Sinclair, Peter BN(Renal), MPhil; Kable, Ashley Dip Teach Nurse Ed, RN, Grad Dip Health Serv Man, PhD, MACN; Levett-Jones, Tracy PhD, RN, MEd & Work, BN, DippAppSc (Nursing)

JBI Database of Systematic Reviews and Implementation Reports: January 2015 - Volume 13 - Issue 1 - p 52–64
doi: 10.11124/jbisrir-2015-1919

Review question/objective The objective of this systematic review is to identify, appraise and synthesize the best available evidence for the effectiveness of internet-based e-learning programs on health care professional behavior and patient outcomes.

Background Technological innovation has not only impacted social change in recent years but has been the prime driver of educational transformation.1The newest consumers of post-secondary education, the so-called ‘digital natives’, have come to expect education to be delivered in a way that offers increased usability and convenience.2 Health care professionals (HCPs) in the clinical setting, particularly those in rural and remote communities, are no different.3-5 Today's health workforce has a professional responsibility to maintain competency in practice through achieving a minimum number of hours of continuing professional development.6 Consequently, HCPs seeking professional development opportunities are reliant on sourcing these independently according to individual learning needs.7 However, difficulties exist in some health professionals' access to ongoing professional development opportunities, particularly those with limited access face-to-face education8,9due to geographical isolation or for those not enrolled in a formal program of study.10,11These issues challenge traditional methods of teaching delivery; electronic learning (e-learning) is at the nexus of overcoming these challenges.

The term e-learning originated in the mid-1990s as the internet began to gather momentum.1Electronic learning can be broadly defined as any type of educational media that is delivered in an electronic form.12Terms such as computer-assisted learning, online learning, web-based learning and e-learning are often used synonymously but all reflect knowledge transfer via an electronic device. This broad definition allows for a gamut of multimedia to be used for the purpose of constructing and assessing knowledge. Multimedia typically used in e-learning range from the now archaic Compact Disc Read-Only Memory (CD-ROMs), to the simple Microsoft PowerPoint, or the more advanced and complex virtual worlds such a second life. Electronic learning can be delivered in asynchronous or synchronous formats, with the latter (for example interactive online lectures via platforms such as BlackboardCollaborate or WebEx) more commonly used in formal educational settings according to set timetables of study.1

Person-to-person interactivity is an important enabler of knowledge generation and while functionalities such as web 1.0 (discussion board and email) and more recently web 2.0 (Wikis and blogs) allow for this to occur both synchronously and asynchronously, it is usually utilized in formal educational contexts only. However, the economy of formal education does not allow for free access to courses which proves challenging for HCPs seeking quality educational opportunities who choose not to undergo a formal program of study or are just looking to meet a specific learning need. Alternatively, asynchronous e-learning is a more learner-centred approach that affords the opportunity to engage in learning at a time and location that is convenient and enables the learner to balance professional development with personal and work commitments.13These learning opportunities are self-directed and do not require a human to facilitate learning, rather, technology officiates/facilitates the learning process and, in the asynchronous e-learning context, the learner negotiates meaning independently.14

Health-related e-learning research has focused on several domains including media comparative designs15,16, self-efficacy17,18, user satisfaction19,20, instructional design21, knowledge outcomes9,22-28, clinical skills development17,29,30, and facilitators/barriers to its use.31The benefits of e-learning are well documented in terms of increased accessibility to education, efficacy, cost effectiveness, learner flexibility and interactivity.32However, some fundamental methodological and philosophical flaws exist in e-learning research, not least the use of comparative design studies. Comparison between e-learning and traditional teaching methods are illogical and methodologically flawed because comparison groups are heterogeneous, lack uniformity and have multiple confounders that cannot be adjusted for.33,34As early as 1994, researchers34in computer-assisted learning were citing these limitations and called for a fresh research agenda in this area. Cook33,35repeated this call in 2005 and again in 2009 and noted a paucity of research related to patient or clinical practice outcomes. Electronic learning is not an educational panacea and research needs to progress from pre- and post-interventional and comparative designs that evaluate knowledge increases and user satisfaction. It is time to move towards determining whether improved self-efficacy or knowledge gained through e-learning improves patient outcomes or influences clinical behavior change and whether these changes are sustained. In order to develop the empirical evidence base in e-learning, research needs to be guided by established theoretical frameworks and use validated instruments to move from assessing knowledge generation towards improving our understanding of whether e-learning improves HCP behavior and more importantly, patient outcomes.

One suitable framework that is congruent with e-learning research is Kirkpatrick's36four levels of evaluation. Kirkpatrick's model is hierarchically based with level one relating to student reaction and how well the learner is satisfied with the education program. Level two pertains to learning and the evaluation of knowledge, level three expands on this and considers whether the education has influenced behavior. In the context of this review, behavior change is any practice that is intrinsically linked with the outcomes of the e-learning program undertaken. Finally, level four evaluates the impact on outcomes such as cost benefit or quality improvements.36,37The majority of e-learning research has focused on participant experience and knowledge acquisition, outcomes that correspond with the first two levels of Kirkpatrick's model.38To date, few studies have examined the effectiveness of internet-based e-learning programs on HCP behavior, which aligns with Level 3 of Kirkpatrick's model.

Studies exist that use self-reported measures of intention to change behavior,39,40 however self-reported intention to change does not necessarily translate into actual behavior change.41 Studies that have not used self-reported measures of behavior change have used objectively measured evaluation criteria including objective structured assessment of technical skills (OSATS) using various methods including simulation task trainers42 and clinical simulations using standardized patients43 scored by a panel of experts using standardized assessment tools. Carney et al.44 used a national reporting and data system to measure the impact of a single one hour e-learning program undertaken by radiologists (n=31) aimed at reducing unnecessary recall during mammography screening. Carney et al. reported a null effect and attributed this to the complexities of behavior change, suggesting that longer term reinforcement of principles relating to mammography recall was required to effect behavior change. These findings also suggest that a multi-modal intervention may be required in order to reduce excessive recall rates in this area, rather than a single intervention. Contrary to Carney et al., Pape-Koehler et al.42 and Smeekins et al.43 reported positive findings using randomized controlled designs to test the efficacy of e-learning interventions on individual's surgical performance42 and the detection of child abuse43, respectively. Pape-Koehler et al. used a 2x2 factorial design to demonstrate that an e-learning intervention significantly improved novice surgeon (n=70) surgical performance of a laparoscopic cholecystectomy (change between pre-post test OSATS p 0.001) when used in isolation or in combination with a practical training session compared to practical training alone. Smeekins et al. demonstrated that a 2 hour e-learning program improved nurses' (n=25) ability to detect child abuse in an emergency department. The nurses in the intervention (n=13) group demonstrated significantly better (p=0.022) questioning techniques and consequently, higher quality history taking, to determine children at risk of child abuse when compared with the control group who received no training at all.

These three exemplar studies demonstrate the broad range of applications e-learning has in HCP education, as each study used different designs, had different subject areas and target health care professionals. This reflects the conceptual and practical challenges of the area of research that addresses levels three of Kirkpatrick's model. For this reason, the e-learning research agenda in health should focus on whether knowledge generated through e-learning is able to be re-contextualized into clinical practice, and influence sustained clinical behavior change and patient outcomes.

A preliminary search of PubMed, CINAHL, The Cochrane Library, The JBI Database of Systematic Reviews and Implementation Reports, ERIC and PROSPERO was conducted to determine if a systematic review on the topic of interest already existed. This search identified four systematic reviews that specifically reviewed outcome measures of knowledge and skill improvement in the domain of e-learning. Two38,45 examined research conducted in nursing, with the other two46,47 in orthodontics. Lahti et al.38 systematic review examined the impact of e-learning on nurses' and nursing students' knowledge, skills and satisfaction. Lahti et al.38 were unable to demonstrate a statistical difference between cohorts undertaking e-learning compared to conventional teaching methods, findings that were not replicated by Du et al.45 This may be due to the decision by Lahti et al. to include studies that utilized CR-ROM and that, despite being published in 2014, the actual review took place in 2010 and did not capture several significant studies published after this date. The systematic review by Du et al.45 examined the efficacy of online distance education in terms of knowledge acquisition and retention, and skill performance in employed nurses and nursing students. This review identified nine randomized controlled trials (RCTs) of which five studies considered skill performance. Four of these studies demonstrated superior or equivalent improvement in skills compared to control groups. Similar findings were noted in two other systematic reviews conducted by Al-Jewair et al.46,47 who concluded that computer-aided learning was at least as efficacious as conventional teaching methods in improving knowledge in undergraduate and postgraduate orthodontic students and educators.

These reviews included blended and fully online studies of varying instructional design quality. Seven out of the nine studies identified by Du et al. utilized interactive elements to facilitate communication between students and teachers which would confound the results of the review given that interactivity in this manner is known to improve the user experience and knowledge outcomes.48 To date, there are no database-indexed systematic reviews that identify, appraise and synthesize the best available evidence for the effectiveness of internet-based e-learning programs on HCP behavior and patient outcomes using objectively administered evaluation criteria.

1 School of Nursing and Midwifery, Faculty of Health and Medicine, University of Newcastle, New South Wales

2 University of Newcastle Evidence Based Health Care Group, affiliated with the New South Wales Centre for Evidence Based Health Care Australia: a Collaborating Centre of the Joanna Briggs Institute

Corresponding author: Peter Sinclair

Back to Top | Article Outline

Inclusion criteria

Types of participants

This review will consider studies whose participants were HCPs in any health care context.

For the purpose of this review, a HCP will be any individual who requires a degree qualification to practice in their respective field

Back to Top | Article Outline

Types of intervention(s)

This review will consider studies that evaluated an internet-based e-learning program.

For the purposes of this review internet-based e-learning programs are defined as any asynchronous educational intervention that is mediated electronically via the internet.

Back to Top | Article Outline

Types of outcomes

This review will consider studies that include the following outcome measures:

  • Impact on clinician behavior measured using objectively administered evaluation criteria
  • Impact on patient outcomes measured using objectively administered evaluation criteria

For the purposes of this review, impact on clinician behavior is defined as the degree to which the intervention influenced their ability to perform the skill for which the intervention was designed. For the purposes of this review, impact on patient outcomes is defined as the degree to which patients' health care outcomes were affected (either positively or negatively) as a result of the intervention.

Back to Top | Article Outline

Types of studies

This review will consider any RCTs and quasi-RCTs. In the absence of RCTs and quasi-RCTs, other research designs such as non-randomized before and after studies and analytical and descriptive observational studies will be considered.

Back to Top | Article Outline

Search strategy

The search strategy aims to find both published and unpublished studies using a variety of databases. A three-step search strategy will be utilized in this review. An initial limited search of MEDLINE and CINAHL will be undertaken followed by analysis of the text words contained in the title and abstract, and of the index terms used to describe article. A second search using all identified keywords and index terms will then be undertaken across all included databases. Thirdly, the reference lists of all identified reports and articles will then be searched for additional studies. Studies in English published from 2004 to August, 2014 will be considered for inclusion in this review. This time frame was selected on the basis that recent systematic reviews in the domain of e-learning (Du et al.45 and Lahti et al.38) identified suitable papers from the year 2004 onwards.

The databases to be searched include:

  • Cochrane - CENTRAL
  • Embase
  • ERIC
  • Mosby's Index
  • Scopus

Grey literature

A search for unpublished studies using Google Scholar, Mednar and Proquest will be undertaken to locate any relevant policies, government reports, dissertations, theses and conference proceedings.

Initial Search Terms

The following search terms will be used:

  • Internet-based learning, computer based learning, computer assisted learning, web based learning, online learning, e-learning, distance education, internet, educational technology, information communication and technology
  • Clinical assessment, psychomotor skill, behavior
  • Patient outcomes
  • Quantitative
  • Healthcare professional
Back to Top | Article Outline

Assessment of methodological quality

Papers selected for retrieval will be assessed by two independent reviewers for methodological validity prior to inclusion in the review using standardized critical appraisal instruments from the Joanna Briggs Institute Meta-Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI) (Appendix I). Any disagreements that arise between the reviewers will be resolved through discussion, or with a third reviewer.

Back to Top | Article Outline

Data collection

Two reviewers will extract data independently from papers included in the review using the standardized data extraction tool from JBI-MAStARI (Appendix II). The data extracted will include specific details about the interventions, populations, study methods and outcomes of significance to the review question and specific objectives.

Back to Top | Article Outline

Data synthesis

Quantitative data will, where possible be pooled in statistical meta-analysis using JBI-MAStARI. All results will be subject to double data entry. Effect sizes expressed as odds ratio (for categorical data) and weighted mean differences (for continuous data) and their 95% confidence intervals will be calculated for analysis. Heterogeneity will be assessed statistically using the standard Chi-square and also explored using subgroup analyses based on the different quantitative study designs included in this review. Where statistical pooling is not possible the findings will be presented in narrative form including tables and figures to aid in data presentation where appropriate. These results will be combined to arrive at a conclusion from the research

Back to Top | Article Outline

Conflicts of interest

The authors declare no conflicts of interest exist.

Back to Top | Article Outline


This systematic review is being conducted as part of Peter Sinclair's PhD.

Back to Top | Article Outline


1. Garrison DR. E-learning in the 21st century: A framework for research and practice. 2nd ed. New York: Routledge; 2011.
    2. Palfrey J, Gasser U. Born digital: Understanding the first generation of digital natives. New York: Basic Books; 2013.
      3. Maloney S, Chamberlain M, Morrison S, Kotsanas G, Keating JL, Ilic D. Health professional learner attitudes and use of digital learning resources. J Med Internet Res. 2013;15(1):e7.
      4. Sinclair PM, Levett -Jones T. The evolution of the nephrology educators'network. J Ren Care. 2011;37(1):40-6.
      5. Wellard S, Bethune E. Learning issues for nurses in renal satellite centres. Aust J Rural Health. 2000;8(6):322-6.
      6. Sinclair PM, Bowen L, Donkin B. Professional nephrology nursing portfolios: Maintaining competence to practise. Ren Soc Aust J. 2013;9(1):35-40.
        7. Mills J, Field J, Cant R. Rural and remote Australian general practice nurses' sources of evidence for knowledge translation: a cross-sectional survey. Int J Evid Based Healthc. 2011;9(3):246-51.
        8. Lenthall S, Wakerman J, Opie T, Dunn S, MacLeod M, Dollard M, et al. Nursing workforce in very remote Australia, characteristics and key issues. Aust J Rural Health. 2011;19(1):32-7.
        9. Bennett PN, Jaeschke S, Sinclair PM, Kerr PG, Holt S, Schoch M, et al. Increasing home dialysis knowledge through a web-based e-learning program. Nephrology. 2014;19(6):345-51.
        10. Doorenbos AZ, Kundu A, Eaton LH, Demiris G, Haozous EA, Towle C, et al. Enhancing access to cancer education for rural healthcare providers via telehealth. J Cancer Educ. 2011;26(4):682-6.
        11. Curran VR, Fleet L, Kirby F. Factors influencing rural health care professionals' access to continuing professional education. Aust J Rural Health. 2006;14(2):51-5.
        12. Clark RC, Mayer RE. E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning: John Wiley & Sons; 2011.
          13. Sinclair PM, Carstairs M, Shanahan B, Schoch M. The development of a medication calculation competency and quality use of renal medicine e-learning program. Ren Soc Aust J. 2014;10(2):58-60.
            14. Melhuish K, Falloon G. Looking to the future: M-learning with the iPad. Computers in New Zealand Schools: Learn Media Technol,. 2010;22(3).
              15. Ghoncheh R, Koot HM, Kerkhof AJ. Suicide Prevention E-Learning Modules Designed for Gatekeepers. Crisis. 2014;35(3):176-85.
              16. Smits P, de Graaf L, Radon K, de Boer A, Bos N, van Dijk F, et al. Case-based e-learning to improve the attitude of medical students towards occupational health, a randomised controlled trial. Occup Environ Med. 2012;69(4):280-3.
              17. Blackman IR, Mannix T, Sinclair PM. Developing renal nurses' buttonhole cannulation skils using e-learning. J Ren Care. 2014;40(1):55-63.
              18. Shen D, Cho M-H, Tsai C-L, Marra R. Unpacking online learning experiences: Online learning self-efficacy and learning satisfaction. Inter High Educ. 2013;19:10-7.
                19. Liaw S-S. Investigating students' perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the Blackboard system. Comput Educ. 2008;51(2):864-73.
                20. Sun P-C, Tsai RJ, Finger G, Chen Y-Y, Yeh D. What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction. Comput Educ. 2008;50(4):1183-202.
                21. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med. 2010;85(5):909-22.
                22. Alemagno SA, Guten SM, Warthman S, Young E, Mackay DS. Online Learning to Improve Hand Hygiene Knowledge and Compliance Among HealthCare Workers. J Contin Educ Nurs. 2010;41(10):463-71.
                23. Atack L, Luke R. Impact of an online course on infection control and prevention competencies. J Adv Nurs. 2008;63(2):175-80.
                24. Beeckman D, Schoonhoven L, Boucqué H, Van Maele G, Defloor T. Pressure ulcers: e-learning to improve classification by nurses and nursing students. J Clin Nurs. 2008;17(13):1697-707.
                25. Brunero S, Lamont S. The ‘difficult’ nurse-patient relationship: development and evaluation of an e-learning package. Cont Nurs. 2010;35(2):136-46.
                  26. Estrella MM, Sisson SD, Roth J, Choi MJ. Efficacy of an internet-based tool for improving physician knowledge of chronic kidney disease: an observational study. BMC Nephrol. 2012;13:126.
                  27. Larsen R, Zahner SJ. The Impact of Web-delivered Education on Preceptor Role Self-Efficacy and Knowledge in Public Health Nurses. Public Health Nursing. 2011;28(4):349-56.
                  28. Tait M, Tait D, Thornton F, Edwards M. Development and evaluation of a critical care e-learning scenario. Nurse Educ Today. 2008;28(8):970-80.
                  29. Bloomfield JG, Jones A. Using e-learning to support clinical skills acquisition: Exploring the experiences and perceptions of graduate first-year pre-registration nursing students - a mixed method study. Nurse Educ Today. 2013;33(12):1605-11.
                  30. Kelly M, Lyng C, McGrath M, Cannon G. A multi-method study to determine the effectiveness of, and student attitudes to, online instructional videos for teaching clinical nursing skills. Nurse Educ Today. 2009;29(3):292-300.
                  31. Docherty A, Sandhu H. Student-perceived barriers and facilitators to e-learning in continuing professional development in primary care. Educ Prim Care. 2006;17(4):343-53.
                  32. Ehlers U-D, Pawlowski JM, editors. Handbook on Quality and Standardisation in E-learning. Berlin: Springer; 2006.
                    33. Cook DA. The Research We Still Are Not Doing: An Agenda for the Study of Computer-Based Learning. Acad Med. 2005;80(6):541-8.
                    34. Friedman CP. The research we should be doing. Acad Med. 1994;69(6):455-7.
                    35. Cook DA. The failure of e-learning research to inform educational practice, and what we can do about it. Med Teach. 2009;31(2):158-62.
                    36. Kirkpatrick DL. Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler; 1994.
                      37. Galloway DL. Evaluating distance delivery and e-learning is kirkpatrick's model relevant? Perf Impr. 2005;44(4):21-7.
                        38. Lahti M, Hätönen H, Välimäki M. Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis. Int J Nurs Stud. 2014;51(1):136-49.
                        39. Blackman IR, Mannix T, Sinclair PM. Developing renal nurses' buttonhole cannulation skills using e-learning. J Ren Care. 2014;40(1):55-63.
                        40. Schroter S, Jenkins D, Playle R, Walsh K, Probert C, Kellner T, et al. Evaluation of an online Diabetes Needs Assessment Tool (DNAT) for health professionals: a randomised controlled trial. Trials. 2009; 10:63.
                          41. Davis D, O'Brien MAT, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282(9):867-74.
                          42. Pape-Koehler C, Immenroth M, Sauerland S, Lefering R, Lindlohr C, Toaspern J, et al. Multimedia-based training on Internet platforms improves surgical performance: a randomized controlled trial. Surg Endosc. 2013; 27(5):1737-47
                          43. Smeekens AEFN, Broekhuijsen-van Henten DM, Sittig JS, Russel IMB, Ten Cate OTJ, Turner NM, et al. Successful e-learning programme on the detection of child abuse in Emergency Departments: A randomised controlled trial. Arch Dis Child. 2011;96(4):330-4.
                          44. Carney PA, Abraham L, Cook A, Feig SA, Sickles EA, Miglioretti DL, et al. Impact of an educational intervention designed to reduce unnecessary recall during screening mammography. Acad Radiol. 2012; 19(9):1114-20
                          45. Du S, Liu Z, Liu S, Yin H, Xu G, Zhang H, et al. Web-based distance learning for nurse education: a systematic review. Int Nurs Rev. 2013;60(2):167-77.
                          46. Al-Jewair TS, Azarpazhooh A, Suri S, Shah PS. Computer-assisted learning in orthodontic education: a systematic review and meta-analysis. J Dent Educ. 2009 Jun;73(6):730-9.
                            47. Al-Jewair TS, Qutub AF, Malkhassian G, Dempster LJ. A systematic review of computer-assisted learning in endodontics education. J Dent Educ. 2010 Jun;74(6):601-11.
                              48. Liaw S-S, Huang H-M. Perceived satisfaction, perceived usefulness and interactive learning environments as predictors to self-regulation in e-learning environments. Comput Educ. 2013;60(1):14-24.
                              Back to Top | Article Outline

                              Appendix I: Appraisal instruments

                              MAStARI appraisal instrument





                              Back to Top | Article Outline

                              Appendix II: Data extraction instruments

                              MAStARI data extraction instrument






                              E-learning; clinician behavior; patient outcomes

                              © 2015 by Lippincott williams & Wilkins, Inc.