Secondary Logo

Journal Logo


The effectiveness of internet-based e-learning on clinician behavior and patient outcomes: a systematic review protocol

Sinclair, Peter BN(Renal), MPhil; Kable, Ashley Dip Teach Nurse Ed, RN, Grad Dip Health Serv Man, PhD, MACN; Levett-Jones, Tracy PhD, RN, MEd & Work, BN, DippAppSc (Nursing)

Author Information
JBI Database of Systematic Reviews and Implementation Reports: January 2015 - Volume 13 - Issue 1 - p 52-64
doi: 10.11124/jbisrir-2015-1919
  • Free


Inclusion criteria

Types of participants

This review will consider studies whose participants were HCPs in any health care context.

For the purpose of this review, a HCP will be any individual who requires a degree qualification to practice in their respective field

Types of intervention(s)

This review will consider studies that evaluated an internet-based e-learning program.

For the purposes of this review internet-based e-learning programs are defined as any asynchronous educational intervention that is mediated electronically via the internet.

Types of outcomes

This review will consider studies that include the following outcome measures:

  • Impact on clinician behavior measured using objectively administered evaluation criteria
  • Impact on patient outcomes measured using objectively administered evaluation criteria

For the purposes of this review, impact on clinician behavior is defined as the degree to which the intervention influenced their ability to perform the skill for which the intervention was designed. For the purposes of this review, impact on patient outcomes is defined as the degree to which patients' health care outcomes were affected (either positively or negatively) as a result of the intervention.

Types of studies

This review will consider any RCTs and quasi-RCTs. In the absence of RCTs and quasi-RCTs, other research designs such as non-randomized before and after studies and analytical and descriptive observational studies will be considered.

Search strategy

The search strategy aims to find both published and unpublished studies using a variety of databases. A three-step search strategy will be utilized in this review. An initial limited search of MEDLINE and CINAHL will be undertaken followed by analysis of the text words contained in the title and abstract, and of the index terms used to describe article. A second search using all identified keywords and index terms will then be undertaken across all included databases. Thirdly, the reference lists of all identified reports and articles will then be searched for additional studies. Studies in English published from 2004 to August, 2014 will be considered for inclusion in this review. This time frame was selected on the basis that recent systematic reviews in the domain of e-learning (Du et al.45 and Lahti et al.38) identified suitable papers from the year 2004 onwards.

The databases to be searched include:

  • Cochrane - CENTRAL
  • Embase
  • ERIC
  • Mosby's Index
  • Scopus

Grey literature

A search for unpublished studies using Google Scholar, Mednar and Proquest will be undertaken to locate any relevant policies, government reports, dissertations, theses and conference proceedings.

Initial Search Terms

The following search terms will be used:

  • Internet-based learning, computer based learning, computer assisted learning, web based learning, online learning, e-learning, distance education, internet, educational technology, information communication and technology
  • Clinical assessment, psychomotor skill, behavior
  • Patient outcomes
  • Quantitative
  • Healthcare professional

Assessment of methodological quality

Papers selected for retrieval will be assessed by two independent reviewers for methodological validity prior to inclusion in the review using standardized critical appraisal instruments from the Joanna Briggs Institute Meta-Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI) (Appendix I). Any disagreements that arise between the reviewers will be resolved through discussion, or with a third reviewer.

Data collection

Two reviewers will extract data independently from papers included in the review using the standardized data extraction tool from JBI-MAStARI (Appendix II). The data extracted will include specific details about the interventions, populations, study methods and outcomes of significance to the review question and specific objectives.

Data synthesis

Quantitative data will, where possible be pooled in statistical meta-analysis using JBI-MAStARI. All results will be subject to double data entry. Effect sizes expressed as odds ratio (for categorical data) and weighted mean differences (for continuous data) and their 95% confidence intervals will be calculated for analysis. Heterogeneity will be assessed statistically using the standard Chi-square and also explored using subgroup analyses based on the different quantitative study designs included in this review. Where statistical pooling is not possible the findings will be presented in narrative form including tables and figures to aid in data presentation where appropriate. These results will be combined to arrive at a conclusion from the research

Conflicts of interest

The authors declare no conflicts of interest exist.


This systematic review is being conducted as part of Peter Sinclair's PhD.


1. Garrison DR. E-learning in the 21st century: A framework for research and practice. 2nd ed. New York: Routledge; 2011.
    2. Palfrey J, Gasser U. Born digital: Understanding the first generation of digital natives. New York: Basic Books; 2013.
      3. Maloney S, Chamberlain M, Morrison S, Kotsanas G, Keating JL, Ilic D. Health professional learner attitudes and use of digital learning resources. J Med Internet Res. 2013;15(1):e7.
      4. Sinclair PM, Levett -Jones T. The evolution of the nephrology educators'network. J Ren Care. 2011;37(1):40-6.
      5. Wellard S, Bethune E. Learning issues for nurses in renal satellite centres. Aust J Rural Health. 2000;8(6):322-6.
      6. Sinclair PM, Bowen L, Donkin B. Professional nephrology nursing portfolios: Maintaining competence to practise. Ren Soc Aust J. 2013;9(1):35-40.
        7. Mills J, Field J, Cant R. Rural and remote Australian general practice nurses' sources of evidence for knowledge translation: a cross-sectional survey. Int J Evid Based Healthc. 2011;9(3):246-51.
        8. Lenthall S, Wakerman J, Opie T, Dunn S, MacLeod M, Dollard M, et al. Nursing workforce in very remote Australia, characteristics and key issues. Aust J Rural Health. 2011;19(1):32-7.
        9. Bennett PN, Jaeschke S, Sinclair PM, Kerr PG, Holt S, Schoch M, et al. Increasing home dialysis knowledge through a web-based e-learning program. Nephrology. 2014;19(6):345-51.
        10. Doorenbos AZ, Kundu A, Eaton LH, Demiris G, Haozous EA, Towle C, et al. Enhancing access to cancer education for rural healthcare providers via telehealth. J Cancer Educ. 2011;26(4):682-6.
        11. Curran VR, Fleet L, Kirby F. Factors influencing rural health care professionals' access to continuing professional education. Aust J Rural Health. 2006;14(2):51-5.
        12. Clark RC, Mayer RE. E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning: John Wiley & Sons; 2011.
          13. Sinclair PM, Carstairs M, Shanahan B, Schoch M. The development of a medication calculation competency and quality use of renal medicine e-learning program. Ren Soc Aust J. 2014;10(2):58-60.
            14. Melhuish K, Falloon G. Looking to the future: M-learning with the iPad. Computers in New Zealand Schools: Learn Media Technol,. 2010;22(3).
              15. Ghoncheh R, Koot HM, Kerkhof AJ. Suicide Prevention E-Learning Modules Designed for Gatekeepers. Crisis. 2014;35(3):176-85.
              16. Smits P, de Graaf L, Radon K, de Boer A, Bos N, van Dijk F, et al. Case-based e-learning to improve the attitude of medical students towards occupational health, a randomised controlled trial. Occup Environ Med. 2012;69(4):280-3.
              17. Blackman IR, Mannix T, Sinclair PM. Developing renal nurses' buttonhole cannulation skils using e-learning. J Ren Care. 2014;40(1):55-63.
              18. Shen D, Cho M-H, Tsai C-L, Marra R. Unpacking online learning experiences: Online learning self-efficacy and learning satisfaction. Inter High Educ. 2013;19:10-7.
                19. Liaw S-S. Investigating students' perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the Blackboard system. Comput Educ. 2008;51(2):864-73.
                20. Sun P-C, Tsai RJ, Finger G, Chen Y-Y, Yeh D. What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction. Comput Educ. 2008;50(4):1183-202.
                21. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med. 2010;85(5):909-22.
                22. Alemagno SA, Guten SM, Warthman S, Young E, Mackay DS. Online Learning to Improve Hand Hygiene Knowledge and Compliance Among HealthCare Workers. J Contin Educ Nurs. 2010;41(10):463-71.
                23. Atack L, Luke R. Impact of an online course on infection control and prevention competencies. J Adv Nurs. 2008;63(2):175-80.
                24. Beeckman D, Schoonhoven L, Boucqué H, Van Maele G, Defloor T. Pressure ulcers: e-learning to improve classification by nurses and nursing students. J Clin Nurs. 2008;17(13):1697-707.
                25. Brunero S, Lamont S. The ‘difficult’ nurse-patient relationship: development and evaluation of an e-learning package. Cont Nurs. 2010;35(2):136-46.
                  26. Estrella MM, Sisson SD, Roth J, Choi MJ. Efficacy of an internet-based tool for improving physician knowledge of chronic kidney disease: an observational study. BMC Nephrol. 2012;13:126.
                  27. Larsen R, Zahner SJ. The Impact of Web-delivered Education on Preceptor Role Self-Efficacy and Knowledge in Public Health Nurses. Public Health Nursing. 2011;28(4):349-56.
                  28. Tait M, Tait D, Thornton F, Edwards M. Development and evaluation of a critical care e-learning scenario. Nurse Educ Today. 2008;28(8):970-80.
                  29. Bloomfield JG, Jones A. Using e-learning to support clinical skills acquisition: Exploring the experiences and perceptions of graduate first-year pre-registration nursing students - a mixed method study. Nurse Educ Today. 2013;33(12):1605-11.
                  30. Kelly M, Lyng C, McGrath M, Cannon G. A multi-method study to determine the effectiveness of, and student attitudes to, online instructional videos for teaching clinical nursing skills. Nurse Educ Today. 2009;29(3):292-300.
                  31. Docherty A, Sandhu H. Student-perceived barriers and facilitators to e-learning in continuing professional development in primary care. Educ Prim Care. 2006;17(4):343-53.
                  32. Ehlers U-D, Pawlowski JM, editors. Handbook on Quality and Standardisation in E-learning. Berlin: Springer; 2006.
                    33. Cook DA. The Research We Still Are Not Doing: An Agenda for the Study of Computer-Based Learning. Acad Med. 2005;80(6):541-8.
                    34. Friedman CP. The research we should be doing. Acad Med. 1994;69(6):455-7.
                    35. Cook DA. The failure of e-learning research to inform educational practice, and what we can do about it. Med Teach. 2009;31(2):158-62.
                    36. Kirkpatrick DL. Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler; 1994.
                      37. Galloway DL. Evaluating distance delivery and e-learning is kirkpatrick's model relevant? Perf Impr. 2005;44(4):21-7.
                        38. Lahti M, Hätönen H, Välimäki M. Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis. Int J Nurs Stud. 2014;51(1):136-49.
                        39. Blackman IR, Mannix T, Sinclair PM. Developing renal nurses' buttonhole cannulation skills using e-learning. J Ren Care. 2014;40(1):55-63.
                        40. Schroter S, Jenkins D, Playle R, Walsh K, Probert C, Kellner T, et al. Evaluation of an online Diabetes Needs Assessment Tool (DNAT) for health professionals: a randomised controlled trial. Trials. 2009; 10:63.
                          41. Davis D, O'Brien MAT, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282(9):867-74.
                          42. Pape-Koehler C, Immenroth M, Sauerland S, Lefering R, Lindlohr C, Toaspern J, et al. Multimedia-based training on Internet platforms improves surgical performance: a randomized controlled trial. Surg Endosc. 2013; 27(5):1737-47
                          43. Smeekens AEFN, Broekhuijsen-van Henten DM, Sittig JS, Russel IMB, Ten Cate OTJ, Turner NM, et al. Successful e-learning programme on the detection of child abuse in Emergency Departments: A randomised controlled trial. Arch Dis Child. 2011;96(4):330-4.
                          44. Carney PA, Abraham L, Cook A, Feig SA, Sickles EA, Miglioretti DL, et al. Impact of an educational intervention designed to reduce unnecessary recall during screening mammography. Acad Radiol. 2012; 19(9):1114-20
                          45. Du S, Liu Z, Liu S, Yin H, Xu G, Zhang H, et al. Web-based distance learning for nurse education: a systematic review. Int Nurs Rev. 2013;60(2):167-77.
                          46. Al-Jewair TS, Azarpazhooh A, Suri S, Shah PS. Computer-assisted learning in orthodontic education: a systematic review and meta-analysis. J Dent Educ. 2009 Jun;73(6):730-9.
                            47. Al-Jewair TS, Qutub AF, Malkhassian G, Dempster LJ. A systematic review of computer-assisted learning in endodontics education. J Dent Educ. 2010 Jun;74(6):601-11.
                              48. Liaw S-S, Huang H-M. Perceived satisfaction, perceived usefulness and interactive learning environments as predictors to self-regulation in e-learning environments. Comput Educ. 2013;60(1):14-24.

                              Appendix I: Appraisal instruments

                              MAStARI appraisal instrument

                              no caption available.
                              no caption available.

                              Appendix II: Data extraction instruments

                              MAStARI data extraction instrument

                              no caption available.
                              no caption available.

                              E-learning; clinician behavior; patient outcomes

                              © 2015 by Lippincott williams & Wilkins, Inc.