Secondary Logo

Journal Logo

Peer Reviewing E-Learning: Opportunities, Challenges, and Solutions

Ruiz, Jorge G. MD; Candler, Chris MD; Teasdale, Thomas A. DrPH

doi: 10.1097/ACM.0b013e31803ead94
Educational Strategies

Peer review is the foundation of academic publication and a necessary step in the scrutiny of any scholarly work. Simply defined, peer review is the attentive, unbiased assessment of any scholarly work that is submitted for formal scrutiny.

Although medical school faculty increasingly use technology in clinical teaching, e-learning materials are often not subjected to a rigorous peer review process. The authors contrast peer review of e-learning materials with that of print materials, describe peer review issues regarding e-learning materials, propose approaches to address the challenges of peer review of e-learning materials, and outline directions for refinement of the e-learning peer review process. At its core, the peer review of e-learning materials should not differ substantially from that of traditional manuscripts. However, e-learning introduces new demands that impel reviewers to consider aspects that are unique to educational technology, including pedagogy, format, usability, navigation, interactivity, delivery, ease of updating, distribution, and access. Four approaches are offered to ease the burden and improve the quality of e-learning peer review: develop peer review training, embrace multidisciplinary peer review, develop guidelines, and provide incentives and compensation. The authors conclude with suggestions about peer review research.

Dr. Ruiz is assistant professor of clinical medicine, Division of Gerontology and Geriatric Medicine, University of Miami Miller School of Medicine, Miami, Florida, associate director for education/ evaluation, Geriatric Research, Education and Clinical Center (GRECC), VA Medical Center, Miami, Florida, and senior investigator, Stein Gerontological Institute, Miami, Florida.

Dr. Candler is co-director and editor, MedEdPORTAL, and director of educational technology, Association of American Medical Colleges, Washington, DC.

Dr. Teasdale is associate professor of geriatric medicine, The Donald W. Reynolds Department of Geriatric Medicine, University of Oklahoma College of Medicine & Veterans Affairs South Central Health Care Network Mental Illness Research, Education and Clinical Center (MIRECC), Oklahoma City VAMC, Oklahoma City, Oklahoma.

Correspondence should be addressed to Dr. Ruiz, GRECC (11GRC), Miami VAMC, 1201 NW 16th Street, Miami, FL; telephone: (305) 5757-3388; fax: (305) 575-3365; e-mail: (

Peer review is an accepted and essential step in the publication process of print biomedical literature.1 Thoughtful scrutiny by unbiased peers is the hallmark of traditional peer review that enables authors to transform scholarly work into scholarship. In addition, peer review fulfills an essential quality-control requirement, because it ensures that published materials meet accepted standards. Print-based scientific journals have relied on some form of peer review to assess the scholarship of research since the 19th century. However, the process of peer review has not been equally applied to educational scholarship.2

Clinician educators are increasingly using technology to complement and enhance clinical teaching and learning activities. We use e-learning to refer to the use of Internet technologies to deliver a broad array of approaches that enhance learners’ knowledge and performance.3–5 Common synonyms for e-learning are Web-based learning, online learning, distributed learning, computer-assisted instruction, or Internet-based learning. Distance learning and computer-assisted instruction are two well-known e-learning modalities included in our definition. Distance learning takes place at locations remote from the point of instruction. Computer-assisted instruction, on the other hand, uses stand-alone multimedia packages for learning and teaching.6 Examples of e-learning include tutorials, case-based learning, simulations, and game-based learning modules. E-learning is fast becoming an established approach in medical education.

Some years ago, the development of computer-based education materials was not considered evidence of scholarship in U.S. medical schools.7 That stance has eroded over the past decade, but indiscriminate acceptance as scholarship is also not justified. Historically, e-learning materials have not been rigorously subjected to a formal, valid, and reliable evaluation process.8,9 For e-learning to be fully and widely recognized as evidence of scholarship, peer review must become standard practice. Previous approaches to the evaluation of digital materials have focused primarily on content and presentation and not on the scholarly and pedagogical aspects of e-learning.10 Alur and colleagues10 used strict inclusion criteria, looking for the adherence of medical teaching Web sites to instructional methods reflecting learning principles such as critical thinking, independent learning, evidence-based learning, and feedback. They found that the Web sites focused mainly on content, with fewer than 20% of sites using the four teaching methods. Knight et al9 developed and implemented a peer review instrument that focused mainly on the format, presentation, feedback, and evaluation of Web-based curricula. This peer review model is an attempt to identify and differentiate teaching Web sites according to a set of criteria that reflects the learning paradigm and, in this way, to assist medical educators creating e-learning materials to achieve recognition for their work, with the ultimate goal of achieving high-quality e-learning. Based on Boyer’s work on the domains of scholarship, Glassick and colleagues11 proposed six criteria to guide assessment of scholarly work beyond the traditional paradigms of teaching, research, and service. These criteria (clear goals, adequate preparation, appropriate methods, significant results, effective presentation, and reflective critique) can be especially helpful to peer reviewers as they decide whether educational materials, including e-learning materials, meet the accepted standards of scholarship. The Association of American Medical Colleges (AAMC) MedEdPORTAL project12 recently incorporated the Glassick criteria in its peer review form for e-learning materials.

In this paper, we discuss similarities and differences between peer review of e-learning and print materials, focusing specifically on delivery rather than content. We describe peer review concepts, content, and practice with regard to e-learning materials. We also propose four approaches to handling various challenges of peer reviewing e-learning materials.

Back to Top | Article Outline


At its core, the peer review of e-learning materials should not differ substantially from the review of traditional scholarship. To do otherwise is to confer an unequal standing among academic products submitted for educational purposes. For both traditional manuscripts and e-learning materials, the reviewer must examine the submission’s strengths and weaknesses in terms of the veracity, relevance, and effectiveness of content. The information contained in the educational materials must be correct, up to date, and matched to learning outcomes and to the appropriate level of learners.13 However, e-learning requires consideration of additional dimensions. For example, is the appearance conducive to education? Are multimedia elements used effectively? Is it easy to “navigate” through the online material? Is the interactivity appropriate for the level of the learner? Are special computer skills, hardware, or software required? What special considerations are relevant regarding reproducibility? These and other questions place new demands on reviewers. In this section, we suggest the following aspects: pedagogy, format, usability, navigation, interactivity, delivery, and currency.

Back to Top | Article Outline


Mortimore refers to pedagogy as “any conscious action by one person designed to enhance learning in another.”14 E-learning materials introduce a new learning paradigm that allows implementation of instructional strategies that could not readily be accomplished with print. The design of e-learning systems informed by educational theory and research can enhance the learning experience. Examples of successful e-learning materials include multimedia case-based simulations and computer games, which allow medical learners access to a varied clinical case mix free from temporal and spatial restrictions.15 This and other challenges may add to the complexity of the review for faculty not familiar with advances in e-learning methodologies.

Back to Top | Article Outline


Print media is limited to text and static graphical elements such as pictures and figures. E-learning materials, in their most versatile and popular format, go beyond these formats to include hyperlinked text and multimedia elements that are only possible with computers. However, the addition of media elements introduces important challenges to reviewers accustomed to classic print materials. Specifically, the media elements must be evaluated for their appropriateness, educational value, and overall effectiveness. For example, a video depicting the “flash” seen within a syringe confirming successful phlebotomy can be powerful, but a video illustrating various types of needle gauges offers nothing more than what may be illustrated in a simple photograph.

Back to Top | Article Outline


Print materials have been widely used for centuries, so their usability is time tested. The history of e-learning, on the other hand, is comparatively short, and usability factors are continuously evolving. In human–computer interaction and computer science, usability refers to the quality of a user’s experience when interacting with a product or system such as a Web site or software application.16 In contrast with the simple physical format of print, e-learning input devices include keyboards, mice, touch screens, stylus sticks, and verbal commands.

Usability criteria for Web-based materials, which constitute the bulk of e-learning, have been devised17 but are not yet widely used. Although government, industry, and higher education are beginning to adopt these and other guidelines, the typical peer reviewer is not familiar with these concepts and will need to acquire new skills to assess this new genre of educational materials.

Back to Top | Article Outline

Navigation: Linearity versus branching

Subject matter experts who create print media usually present their content in a sequential and predictable fashion, which allows learners to easily follow one prescribed path that the author has set for them in advance. There are abundant examples of e-learning materials that simply deliver an automated linear presentation. These can be highly effective; whether they are depends on the target audience, learning outcomes, and topic. However, e-learning provides new opportunities that can exploit the hypermedia features and nonlinear structure that have made the World Wide Web so popular.

In the context of navigation, moving from the linear presentation of material to a branching paradigm opens a new world of opportunities for medical learners. Such flexibility affords new opportunities to explore content, but it may also overwhelm and confuse learners. Although little is formally known about the educational benefits and pitfalls of branching navigation, the peer review of e-learning materials may nonetheless consider the educational benefits of such navigation. The parallel for print-based peer review is that reviewers often consider the overall organization of content, including the table of contents, embedded figures and tables, and references.

Back to Top | Article Outline


One of the main instructional and pedagogical advantages of e-learning lies in its interactivity.3,15 The concept of interactivity is still controversial, lacking even an accepted definition.18 Yacci18 outlines four criteria defining interactivity: instructional interactivity is a message loop, occurs from the learner’s point of view, has two outputs (content learning and affective benefits), and contains mutually coherent messages. An example will illustrate the concept. The medical student (learner perspective) interacts with a Web-based virtual patient by asking her about the characteristics of her abdominal pain, to which the virtual patient responds (message loop). The virtual patient answers that the pain is “stabbing” in nature (mutually coherent message). The learner acquires knowledge regarding the virtual patient’s symptoms and becomes satisfied by it (content learning and affective outputs). According to these criteria, traditional print materials are low in interactivity. On the other hand, multimedia e-learning offers the potential for higher levels of interactivity. Virtual patients and Web-based simulations are dramatic examples of highly interactive educational materials.

E-learning reviewers must be aware of the current state of the art for interactivity potential in order to provide valid assessments of products under review. Moreover, the reviewer must be sufficiently comfortable with the concepts to recognize when an attempt to push the interactivity envelope results in a less useful educational approach than might be achieved otherwise. The opposite can be true when interactivity is underutilized. A particularly important issue is for reviewers to discriminate between good or effective versus substandard or ineffective interactivity.

Back to Top | Article Outline


Dissemination is a condition of scholarship. Emerging technologies surrounding e-learning represent new categories for the distribution of educational content. E-learning materials can be delivered through a wide range of media, including the Internet, CD-ROMs, DVDs, and PDAs. We do not include videotape among e-learning media because of its linear and noninteractive presentation. Online repositories and digital libraries are particularly well suited for the delivery of e-learning products. The peer reviewer would be expected to recognize the subtle differences in how the content is organized and delivered through varied media to determine whether the materials are potentially effective and efficient. For example, experts in the field recognize that e-learning materials designed for Web-based delivery may not be equally well delivered via a DVD.

Back to Top | Article Outline


Once a book or journal article is published, it is difficult to update its content without considerable cost and time. Creators of print-based educational content face much greater distribution challenges than do their counterparts who create e-learning products. Information technologies allow e-learning materials to be updated much more easily and efficiently than print. Updating e-learning materials may entail costs similar to, or even greater than, print materials, but the relative ease of redistributing updated e-learning content offsets those costs. Consequently, authors should be encouraged to refine and enhance their work using new information from ongoing evaluation, new biomedical discoveries, or changes in the underlying technology. This presents a unique challenge for the traditional model of peer review, which has focused on the evaluation of relatively static documents. Editors of e-learning publications should recommend that their published authors maintain and openly display a revision history that indicates all significant changes to content after the item was published. A publication that undergoes substantial modification could be submitted as a new version for publication, a model similar to progressive book editions. Furthermore, any item that contains time-sensitive information should be reviewed again after a reasonable period of time. A potential challenge for editorial staff and peer reviewers is the likelihood of increased workload resulting from new versions of submitted e-learning materials.

Back to Top | Article Outline

Approaches to Facilitate E-Learning Peer Review

We have described seven domains in which peer reviewer of e-learning materials differs from review of conventional print scholarship. We now suggest four approaches that have the potential to ease the burden and improve the quality of e-learning peer review.

Back to Top | Article Outline

Develop peer review training

The increasing level of complexity that is required to peer review e-learning materials makes it imperative for experts to design and develop training programs for prospective peer reviewers. This training should cover the basics of innovative educational technologies, including e-learning content such as tutorials, simulations, virtual patients, distance learning, asynchronous modalities, and online assessment. Studies by Callaham and colleagues19,20 suggest that conventional face-to-face workshops are ineffective at improving the general quality of peer review efforts. The proposed educational program must include a variety of delivery methods, including e-learning methods, mentoring of junior peer reviewers performing actual reviews, and annotated examples of well-conducted reviews of e-learning. These should all be posted for public access. Abbreviated training approaches may offer busy educators opportunities to acquire a basic set of knowledge and skills needed to conduct a peer review of e-learning materials.

Back to Top | Article Outline

Include multidisciplinary peer review

Many journals already make use of statisticians and methodologists to assist editors in evaluating the methods sections of submitted manuscripts. The rationale behind this practice is the recognition that many health care professionals have insufficient facility with statistical techniques. In support of this notion, a review of the literature revealed that physicians with master degrees in public health and epidemiology were far more effective peer reviewers than were physicians without such additional expertise.21 The process of peer reviewing e-learning materials might also benefit from this multidisciplinary approach. MedEdPORTAL12 is already using a combination of reviewers from varied disciplines, some with a background in e-learning, to assist in their peer review process of selected educational materials. We suggest that a wide array of other professionals be considered to compose the pool of available peer reviewers. When e-learning materials are submitted for review, appropriate members of this pool who have expertise in instructional design, usability issues, software development, educational technology, and Web design should be included, as appropriate, to ensure that peer review parameters specific to e-learning materials are appropriately appraised.

Back to Top | Article Outline

Develop guidelines

The formulation of technical specifications and standards in e-learning for software technologies facilitates the development, deployment, maintenance, and operation of e-learning systems. With regard to peer review, specifications and standards are nonexistent. The World Association of Medical Editors22 and other organizations provide valuable advice and suggestions regarding the responsibilities of editors, the specific role of reviewers, the development of a peer review policy, and editorial decision making. These principles apply to not only established print-based journals but also e-learning publication venues. In particular, the AAMC MedEdPORTAL group has developed guidelines and a standardized peer review instrument using the consensus of experts in the field—an important step to ensure a coherent and rational peer review process.12

Back to Top | Article Outline

Provide incentives and compensation

Discussion of incentives and compensation for peer review activities is common within academic and professional circles. This dialogue is as relevant for reviews of e-learning materials as it is for print media. However, editors of e-learning peer review have to compete with the long-established prestige and tradition of minimal compensation for print peer review. It is conceivable that without an initial push, the establishment of expertise in peer review of e-learning materials may not occur or may fail to gather momentum. Monetary compensation occurs, but it is rare.23 Although the satisfaction and prestige associated with the peer review activity are real, other forms of valued compensation must be developed to ensure that the general pool of qualified reviewers remains large and motivated. For example, training courses in peer review could provide CME/CEU credits. Formal letters of commendation could be generated from journal editors to deans and department chairs as recognition and reinforcement of an individual’s peer review activities for evidence of scholarship. A new metric might be the number of e-learning products (or manuscripts) reviewed annually by a given reviewer.

In addition, editors may find it necessary to devote additional resources for administrative support of associate editors and peer reviewers. For example, investments in the automation of the peer review process through the acquisition of peer-review-management software can improve the efficiency of the process and decrease the burden on editors, reviewers, and clerical staff. New incentives to perform peer review must be established. However, in this era of increasing clinical and educational responsibilities, shrinking reimbursements, and faculty shortages, this issue becomes especially problematic for the editors and administrators of digital repositories.

Back to Top | Article Outline

The Work Ahead

Research in peer review of print materials does not reveal sufficient solid evidence for deciding criteria that can help editors make consistent decisions regarding publication. Jefferson et al24 note that editorial peer review lacks agreement about aims and methods employed. A systematic review of the effects of editorial review also has failed to show any conclusive evidence for the effectiveness of peer review of print materials.24 To better address the challenges of peer reviewing e-learning materials, a consensus must be sought regarding minimum guidelines for peer review, and reviewers must be evaluated on their expertise in those skills before the process will be genuinely valued in academic circles.25 The concealment of author or reviewer identity, use of publication and statistical checklists, effect of training, modes of communication between editors and reviewers, and reviewer bias are issues related to the review of conventional print scholarship that are probably not dissimilar from that of e-learning materials. Researching some of the issues inherent to the efficiency and effectiveness of the peer review process and its impact on users, content producers, and other stakeholders should be an important goal. The results of such research, incorporating quantitative or qualitative data, can improve the quality and outcomes of this process with regard to both e-learning and traditional print materials.

Back to Top | Article Outline


In this paper, we strongly advocate the development and adoption of peer review guidelines for e-learning materials. Certain aspects of e-learning review are comparable to print-based peer review, but several new considerations must be addressed. Although we acknowledge that not all e-learning materials should be subjected to formal peer review, those that are under consideration as evidence of scholarship do warrant a rigorous process. Reviewers should ideally be offered instruction regarding the fair and efficient evaluation of the relevant domains. In addition, the formidable challenges to the establishment of an effective peer review process for e-learning materials requires a comprehensive approach that develops peer review training, embraces multidisciplinary peer review, develops guidelines, addresses incentives and compensation, and encourages peer review research.

Back to Top | Article Outline


Dr. Ruiz would like to thank Dr. Bernard Roos for his mentorship. Dr. Ruiz would also like to acknowledge the support from the D. W. Reynolds Foundation, the Miami VAMC Geriatric Research, Education and Clinical Center, and the State of Florida Agency for Health Care Administration (Florida’s Teaching Nursing Home Program). Dr. Teasdale would like to acknowledge the support from the D. W. Reynolds Foundation and the VA South Central Health Care Network Mental Illness Research, Education and Clinical Center (MIRECC.)

Back to Top | Article Outline


1 Rennie D. Editorial peer review: its development and rationale. In: Godlee F, Jefferson T, eds. Peer Review in Health Sciences. 2nd ed. London, UK: BMJ Publishing Group; 2003:1–13.
2 Fincher RM, Work JA. The scholarship of teaching in health science schools. J Vet Med Educ. 2005;32:1–4.
3 Rosenberg M. E-Learning: Strategies for Delivering Knowledge in the Digital Age. New York, NY: McGraw Hill; 2001.
4 Harden RM, Hart IR. An international virtual medical school (IVIMEDS): the future for medical education? Med Teach. 2002;24: 261–267.
5 Ruiz JG, Mintzer MJ, Leipzig RM. The impact of e-learning in medical education. Acad Med. 2006;81:207–212.
6 Ward JP, Gordon J, Field MJ, Lehmann HP. Communication and information technology in medical education. Lancet. 2001;357: 792–796.
7 Bader SA. Recognition of computer-based materials in the promotion guidelines of U.S. medical schools. Acad Med. 1993; 68(10 suppl):S16–S18.
8 Glenn J. A consumer-oriented model for evaluating computer-assisted instructional materials for medical education. Acad Med. 1996;71:251–255.
9 Knight CL, Sakowski HA, Houghton BL, Laya MB, DeWitt DE. Developing a peer review process for web-based curricula: minting a new coin of the realm. J Gen Intern Med. 2004;19:594–598.
10 Alur P, Fatima K, Joseph R. Medical teaching websites: do they reflect the learning paradigm? Med Teach. 2002;24:422–424.
11 Glassick C, Huber M, Maeroff G. Scholarship Assessed: Evaluation of the Professoriate. San Francisco, CA: Jossey-Bass; 1997.
12 Association of American Medical Colleges. MedEdPORTAL. Available at: ( Accessed January 9, 2007.
13 Berry E, Parker-Jones C, Jones RG, et al. Systematic assessment of World Wide Web materials for medical education: online, cooperative peer review. J Am Med Inform Assoc. 1998;5:382–389.
14 Watkins C, Mortimore P. Pedagogy: what do we know? In: Mortimore P, ed. Understanding Pedagogy and Its Impact on Learning. London, UK: Paul Chapman; 1999:1–19.
15 Clark D. Pedagogy and E-Learning. Brighton, UK: Epic; 2006.
16 U.S. Department of Health and Human Services. Usability Basics. Available at: ( Accessed January 9, 2007.
17 Nielsen J. Designing Web Usability: The Practice of Simplicity. Indianapolis, Ind: New Riders Press; 1999.
18 Yacci M. Interactivity demystified: a structural definition for distance education and intelligent computer-based instruction. Educ Technol. 2000;40:5–16.
19 Callaham ML, Schriger DL. Effect of structured workshop training on subsequent performance of journal peer reviewers. Ann Emerg Med. 2002;40:323–328.
20 Callaham ML, Wears RL, Waeckerle JF. Effect of attendance at a training session on peer reviewer quality and performance. Ann Emerg Med. 1998;32:318–322.
21 Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA. 1998;280:231–233.
22 World Association of Medical Editors. A Syllabus for Prospective and Newly Appointed Editors. Available at: ( Accessed January 16, 2007.
23 Eysenbach G. Peer-review and publication of research protocols and proposals: a role for open access journals. J Med Internet Res. 2004;6:e37.
24 Jefferson T, Alderson P, Wager E, Davidoff F. Effects of editorial peer review: a systematic review. JAMA. 2002;287:2784–2786.
25 Reference data not available.
© 2007 Association of American Medical Colleges