Share this article on:

Selection and Qualities of Reviewers

Caelleigh, Addeane S.; Shea, Judy A.; Penn, Gary

Review Process and Publication Decision: Chapter 1

The quality and usefulness of a journal rest on the quality of research submitted, its reviewers' critiques, and its editor's judgment. The reviewer is the vital heart of the process, and one of the editor's primary responsibilities is to create and maintain a high-quality, productive group of reviewers. To do so, journals are concerned with several areas of reviewers' performances: their knowledge, judgment, constructiveness, ability to write clearly, and willingness to work within the journal's guidelines. In addition, in terms of selecting reviewers for individual papers, editors are responsible for having and enforcing fair policies about how reviewers are selected and for seeing that the natural biases inherent in the review system are countered as much as possible.

Journals recruit and organize their reviewers for assignment to manuscripts. For example, editors take into account reviewers' expertise and availability when asking them to review. Also, they take various approaches to monitoring each reviewer's performance, often by rating each review and sometimes by conducting special studies. Journals identify possible reviewers through contacts at professional meetings, personal acquaintances, editorial boards, literature searches, society membership lists, and manuscript bibliographies.1 Some journals encourage authors to suggest reviewers for their manuscripts. Occasionally, potential reviewers seek out journals with unsolicited offers to help. A few journals still require that reviewers be personally known to the editor, but most now use large reviewer pools, and the editor and reviewer may never have met. A factor contributing to the growth of large pools of reviewers is the availability of computer software that now makes it easy to keep detailed files of information about reviewers' expertise, experience, and special interests. Selecting reviewers for an individual manuscript can begin with matching the manuscript topic to a reviewers' area of expertise, but can go farther and ensure that, for example, technical expertise is augmented by particular professional experiences.

A reviewer's expertise must be complemented by judgment in order to be useful to a journal. Therefore, the journal is concerned about whether the reviewer can make balanced judgments, keep a sense of proportion when assessing the strengths and weaknesses of a research project, apply appropriate standards, and give definite opinions. To give these opinions, the reviewer must be able to express complex ideas clearly—and, if possible, concisely—for both the editor and the author, whose needs are not always the same. The reviewer serves as adviser to the editor but also as teacher to the author, especially to the less experienced. In writing critiques, the reviewer should use the criticisms in the review as opportunities to tutor. In reviewing excellent reports by highly experienced researchers, reviewers serve as peer assessors; in contrast, when reviewing weaker reports by novice researchers, they serve very much as teachers. Further, many journals put considerable emphasis on reviewers' giving constructive criticism to authors and expect the reviewers to give their opinions (even, when necessary, stern judgments) in a collegial spirit accompanied by suggestions for improvement. At a minimum, all journals should require civility in reviewers' comments.

Journals look for different attributes and qualities in reviewers, depending on the nature of the journal, the role of the reviewer, and the editor's (and sometimes the sponsoring society's) policies.2,3 Unfortunately, there is little empirical evidence to guide a journal in establishing a profile of the good reviewer. Studies at clinical journals to assess the characteristics of peer reviewers who produce good reviews have reported conflicting findings. Although these studies use slightly different definitions of quality, the core categories used in assessing reviews and reviewers have been very similar.

Stossel, in the earliest study, found that reviewers with lower academic or professional status produced better critiques of papers than the higher-status reviewers did—and were less likely to refuse to review.4 Several years later, Evans et al. reported that the reviewers who produced the best reviews were less than 40 years old, were known to the editors, and were from highly respected (highly rated) academic institutions.5 A few years later, Black et al. found the only significant factor associated with higher-quality ratings by both editors and authors to be reviewers trained in epidemiology or statistics; reviewers' younger age was an independent predictor of the editors' (but not the authors') ratings, while reviews by members of an editorial board were rated of poorer quality by authors (but not by editors). The ratings increased with the time spent on a review—up to three hours, but not beyond.6 The variety of characteristics, or lack of characteristics, that can be relied on to identify high-quality reviewers lead Fletcher and Fletcher to conclude, in a recent overview,

The results of these studies suggest that editors should not have fixed views of what kinds of reviewers might return good reviews. Because the characteristics of good reviews might vary from one setting to another, it seems editors should continue the common practice of grading their own reviewers but recognize that this is an imperfect predictor of their future performance.7

Previous experience with a reviewer can be the most useful guide. Many journals evaluate the reviews they receive, and they refer to the quality ratings for past reviews when considering a reviewer for a manuscript. They also use the ratings, of course, when deciding whether to retain the reviewer. Many editors use simple rating systems that serve primarily as rough triage systems, but some are trying to develop more sophisticated, even standardized instruments.8–10 Editors study reviewers' effectiveness in specific aspects of review, as well as the overall effectiveness of the review process. Baxt et al. examined how well reviewers at a clinical journal could identify major and minor flaws in a manuscript sent for review.11 Callaham et al. (who are in the same group, at the same journal) then studied the reliability of the editors' subjective ratings of review quality and found them to be moderately reliable and correlated with the reviewers' abilities to report manuscript flaws.12 Journals also are aware of the intrinsic, unconscious biases in the review process, and some research is beginning to be done on this complex topic as well.13–15 The biases may be in terms of intellectual positions, preferences for positive outcomes in research, and personal social convictions, as well as the oft-cited biases having to do with ethnicity, nationality, gender, and status.

Reviewing consumes a great deal of reviewers' valuable time, yet apart from acknowledgements or thanks from journals it is generally uncompensated. Therefore, it is essential that a journal's reviewer pool be regularly replenished. First, journals need to avoid sending too many papers to the best reviewers, so that these reviewers will not burn out or refuse to review because of the workload. Also, new, qualified reviewers must be added as other reviewers retire or need to take leave from reviewing. Finally, if a journal is successful in attracting more papers, it will need more reviewers. Similarly, if it decides to use more reviewers per paper as a way to improve the review process, it will need more reviewers. Reviewers should be encouraged to involve junior colleagues who would make good reviewers, and most journals have procedures for sharing a reviewing assignment. These colleagues could, when they are ready, be recommended to a journal as full reviewers. Some journals are responding to the obvious need for reviewer training and for more and better feedback to reviewers. The Annals of Internal Medicine has a very supportive program, for example, including an annual report to each reviewer showing where that reviewer's performance falls in the spectrum of the journal's entire pool of reviewers.

Back to Top | Article Outline

References

1. Weller AC. Editorial peer review in US medical journals. JAMA. 1990; 263:1344–7.
2. Wilson DG. Ethics for authors and editors. Learned Publishing. 1999; Oct:7.
3. Polak JF. The role of the manuscript reviewer in the peer review process. AJR: Am J Roentgenol. 1995;165:685–8.
4. Stossel TP. Reviewer status and review quality: experience of the Journal of Clinical Investigation. N Engl J Med. 1985;312:658–9.
5. Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of peer reviewers who produce good-quality reviews. J Gen Intern Med. 1993;8:422–8.
6. Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA. 1998;280:231–3.
7. Fletcher RH, Fletcher SW. The effectiveness of editorial peer review. In: Godlee F. Jefferson T (eds). Peer Review in Health Sciences. London, U.K.: BMJ Books, 1999:45–56.
8. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review: a randomized trial. JAMA. 1990;263:1371–6.
9. Feurer ID, Becker GJ, Picus D, Ramirez E, Darcy MD, Hicks ME. Evaluating peer reviews: pilot testing of a grading instrument. JAMA. 1994;272:98–100.
10. van Rooyen S, Black N, Goodlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999;52:625–9.
11. Baxt WG, Waeckerle JF, Berlin JA, Callaham ML. Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Ann Emerg Med. 1998;32(3 Pt 1):310–7.
12. Callaham ML, Baxt WG, Waeckerle JD Wears RL. Reliability of editors' subjective quality ratings of peer reviews of manuscripts. JAMA. 1998;280:229–31.
13. Gilbert JR, Williams ES, Lundberg GD. Is there gender bias in JAMA's peer review process? JAMA. 1994;272:139–42.
14. Ernst E, Resch KL. Reviewer bias: a blinded experimental study. J Lab Clin Med. 1994;124:178–82.
15. Caelleigh AS, Hojat M, Steinecke A, Gonnella JS. Effects of reviewers' gender on assessment of a gender-related standardized manuscript [abstr]. 2001 Peer Review Congress, Barcelona, Spain, to be presented September 19–21, 2001.
Back to Top | Article Outline

RESOURCES

Siegelman SS. Assasins and zealots: variations in peer review. Special report. Radiology. 1991;178:637–42.
    Back to Top | Article Outline

    Section Description

    Review Criteria for Research Manuscripts

    Joint Task Force of Academic Medicine and the GEA-RIME Committee

    © 2001 Association of American Medical Colleges