The review process starts when a manuscript is received at the editorial office. The first decision focuses on whether the paper should be sent for external review. Journals vary in their practices, but typically an editor reads the paper and makes an early decision for review or no review based on general factors such as the mission of the journal, the match between the audience for the paper and the audience for the journal, the appearance of obvious fatal flaws, and the balance between the content of the paper and other recent publication decisions. Sometimes redundancy is viewed as a benefit, as when a paper provides confirmation of a new idea or method; sometimes it detracts from the perceived contribution because it is seen as being unnecessary repetition of known information. The central reason that a review decision is made at this stage is that reviewers must be conserved. In almost all cases reviewers are uncompensated for significant amounts of time, ranging from a few hours to “one day” per manuscript.1 Their expertise is an asset given freely—“compensation” often amounts to annual recognition in a published list of reviewers—and editors try hard not to burden them unduly.
If the initial decision is to send the manuscript out for review, reviewers are selected (see the next article in this chapter), and reviews are requested. Journals have different types of review systems. Two basic ones are “board review” (all the reviewers are members of a review board or editorial board, which has the responsibility of reviewing the journal's manuscripts) and “pool review” (a large number of specialists of many types join the journal's pool of potential reviewers).2,3 Most journals use a mixed form that relies heavily on pool review for the initial reviews but also uses board review to add another review or resolve discrepancies. The exact mechanism by which reviewers are selected also varies from journal to journal. In some cases authors are asked to nominate potential reviewers for their articles, and reviewers are sometimes asked to nominate potential reviewers to increase the reviewer pool. The reviewers may be chosen because of either their content or methodologic expertise, and the mix of reviewers for a paper will depend on the type of review the editor has established as standard for the journal.
The pool may very well exist as an electronic list of potential reviewers. In this list, in addition to current contact information and areas of interest and expertise, journals may track other types of data relevant to the review process. For example, it is quite common to track how often a reviewer is asked to review as well as the timeliness and helpfulness of the reviews.
Numerous other elements of the review process vary from journal to journal, depending on the journal's history and resources. Journals have developed their own strategies to assure, as much as possible, that the review process is timely and fair for authors and reviewers alike. For example, some journals contact potential reviewers (via phone or fax) ahead of time to ask them if they are able to review a manuscript in a specified amount of time. A few send the review and associated materials without contact, based on prior agreement between the journal and its reviewers. Journals differ in the amounts of time given to reviewers for doing reviews, the reminders they use when reviews are overdue, and the process for giving up on unresponsive reviewers.
The numbers of reviews solicited also vary. Many journals request reviews from two or three external reviewers, while others use more. One decision the editor must make is whether to request more reviews than are needed, in the hope that at least a minimum number will be completed in a reasonable time. Alternatively, the policy may be to request a small number and then to follow up with additional reviewers when there is disagreement or unanticipated delay.
One element of the review process that is often discussed and that has received research recently is “blinding,” also called “masking.” Blinding has to do with concealing the identity of reviewers, authors, or both. The traditional and most common form is to conceal the reviewer's identity from the author; the less common is the effort to conceal also the author's identity from the reviewer. Conclusions from studies on the topic are mixed. Originally the thought was that masking the author's identity would lead to fairer reviews,4,5 and the process was endorsed by a small but growing minority of journals. However, other work suggests that masking author identity does not improve the quality of reviews.6 It is often felt, especially by reviewers themselves, that blinding is not uniformly effective because reviewers are often still able to identify the author.7,8 This view is not surprising, since reviewers are often experts who are familiar with current work in the field. Although the quality and the quantity of the evidence are not strong, the research to date seems to show that reviewers are not as accurate as they think they are in guessing the author's identity.9 On the more important point, a growing body of research seems to give little evidence that masking the author's identity affects the publication decision. Still, it is a process that continues to be studied, if for no other reason than that authors consider it more fair to themselves. Currently, most journals do not blind the reviewers to the authors' identities.
There is also discussion about the desirability of revealing the reviewers' identities to the authors. Supporters believe that comments will be more fair and balanced when the reviewers publicly identify themselves, and that this openness will help to define the review process as a collegial one from which both the author(s) and reviewer(s) can benefit and enter into professional dialogue.7 One study showed that having reviewers sign their reviews supports the open review of manuscripts.10 Informally, however, it is fair to say that the balance of reviewers' sentiment is against the process. Power imbalances could make it difficult for some reviewers (junior faculty members, for example) to be appropriately critical if the review had to be signed, and it would be too hard to offer critical comments about the work of a colleague or friendly acquaintance. Few if any educational research journals require reviewers to sign their reviews, although some reviewers sign as a matter of principle.
There is another form of masking that is very rarely used but worth nothing—to blind the editor to the identity of the author, the reviewers, or both until a publication decision is made. This process would be based on the same policy considerations that would lead to blinding reviewers to the identity of authors—to limit positive or negative influence on the decision process.
After reviewers are selected, they receive, usually through the mail but now increasingly electronically, a packet that contains the manuscript, review forms, instructions, and sometimes associated materials (e.g., instructions to authors, sample reviews). As noted in the article on review forms later in this chapter, the forms vary in their content and the types of issues that reviewers are asked to address. One commonality is that the reviewer is almost always asked to provide text comments, perhaps separating major from minor issues. Sometimes reviewers are asked to comment sequentially on each major section of the manuscript. Other tasks reviewers may be asked to do include completing rating scales on topics such as clarity, importance, and methodologic soundness. What the reviewer is really doing is assessing whether the different parts of the manuscript meet expectations—this is a measurement task. The measurement task is usually followed by a complementary task—making a recommendation about publication. A recommendation is a judgment based on a reviewer's overall assessment of the worth of the manuscript. (The judgment part of the process is considered more extensively in Chapter 3.) Naturally, this recommendation item will vary depending on the number of categories the journal allows (e.g., accept/reject; accept/provisional accept/revise/reject; accept/revise/reject). Reviewers are sometimes also given the option of writing confidential comments to the editor. Most journals allow reviewers to suggest that a manuscript be referred for a statistical consultation; specification of other types of needed methodologic or technical expertise is rare.
From the reviewer's perspective, several things might be done to make the review process more pleasing after the editorial office's initial efforts to screen out manuscripts that are unlikely to be published and to match the manuscript's content to the reviewer's interests. Giving ample time (e.g., three weeks) is important, and reminders are helpful. Forms should be easy to complete and return. Some journals ask that all materials be returned, while for others it is sufficient to mail or fax the review form and then destroy all other materials. If reviewers must mail back some materials, prestamped and pre-addressed return envelopes should be provided. Some journals are conducting parts or all of the review process electronically. There are no data yet on reviewers' reactions to electronic reviews.
In addition to the process of deciding how (and whether) to solicit reviews for a manuscript, a number of procedures might be considered to support or round out the general review process. For example, the editor and editorial staff must decide whether the reviewers' comments will go to the author, and if they do, whether the editorial staff edit them or send them as received. Two advantages to sending them as they are received is that it reduces the summary and distillation process for the staff, and it can give the author a sense that he or she is getting as much feedback as possible. On the other hand, reviewers' comments often conflict with one another or opinions of the editor, and sometimes reviews contain unhelpful or even inappropriate comments. There is no right or wrong answer—it is simply yet another matter that must be considered. Other issues about which editors (and their editorial boards) make decisions is whether to inform reviewers of the publication decision, either by copying them on the communication to the author or by sending some other kind of summary; whether to involve the same reviewers if revisions are required; and whether to routinely solicit reviewers' opinions about all or parts of the review process. Obviously, there are a multitude of ways these issues can be addressed to create “the review process” at any particular journal. (See Figure 1 for a flow chart of the general review and publication process.)
Once the reviews are available, the editor must make a publication decision. The complex ways in which that decision is reached are discussed in the article on the publication decision later in this chapter.
1. Yankauer A. Who are the peer reviewers and how much do they review? JAMA. 1990;263:1338–40.
2. Weller AC. Editorial peer review in US medical journals. JAMA. 1990;263:1344–7.
3. Hargens LL. Variation in journal peer review systems: possible causes and consequences. JAMA. 1990;263:1348–52.
4. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. JAMA. 1990;263:1371–6.
5. Fisher M, Friedman SB, Strauss, B. The effects of blinding on acceptance of research papers by peer review. JAMA. 1994;272:143–6.
6. Justice AC, Cho MK, Winker MA, et al. Does masking author identity improve peer review quality? A randomized control trial. JAMA. 1998;280:240–2.
7. Lock S. A Difficult Balance: Editorial Peer Review in Medicine. Philadelphia, PA: ISI Press, 1985:122–3.
8. Cho MK, Justice AC, Winker MA, et al. Masking author identity in peer review: what factors influence masking success? JAMA. 1998;280:243–5.
9. Rosenblatt A, Kirk SA. Recognition of authors in blind review of manuscripts. J Soc Service Res. 1980;3:383–94.
10. Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998;280:237–40.
11. Knoll E. The communities of scientists and journal peer review. JAMA. 1990;263:1330–2.
Armstrong JS. Research on scientific journals: implications for editors and authors. J Forecasting. 1982;1:83–104.
Bligh, J. What happens to manuscripts submitted to the journal? Med Educ. 1998;32:567–70.
Chubin DE, Hackett EJ. Peer review and the printed word. In: Chubin DE, Hackett EJ. Peerless Science: Peer Review and U.S. Science Policy. Albany, NY: SUNY Press, 1990.
Freedman DX. Editorial processes, safeguards, and remedies. In: Miller DJ, Hersen M (eds). Research Fraud in the Behavioral and Biomedical Sciences. New York: John Wiley & Sons, 1992.
Hargens LL. Variation in journal peer review systems: possible causes and consequences. JAMA. 1990;263:1348–52.
Knoll E. The communities of scientists and journal peer review. JAMA. 1990;263:1330–2.
National Research Press. Part 4: Responsibilities. In: Publication Policy. 〈http://www.monographs.nrc.ca/cgi-bin/cisti/journals/rp/rp2_cust_e?pubpolicy
〉. Accessed 6/5/01.
Weller AC. Editorial peer review in US medical journals. JAMA. 1990; 263:1344–7.
Review Criteria for Research Manuscripts
Joint Task Force of Academic Medicine and the GEA-RIME Committee