I Heard Reviewers Are Graded: Is This True?
Sheryl Martin, Executive Administrator, Editor’s Office
The answer to this week’s question is “true.” Reviewers are graded after completing each review, on a scale of 1-5, by the editor handling that particular manuscript. The Editors have graded reviews for several years in an effort to make sure reviewers are well selected, that reviews are helpful to authors, and that they are completed in a timely manner. To give you an idea of the activities of our reviewers, in 2012, 1,231 expert peer reviewers submitted 3,703 reviews (ie, approximately 3 papers each).
The editors use a rating system so that everyone who reviews is treated fairly. Reviews are graded on a 1-5 scale, with 5 being exceptional and 1 being poor.
- A score of 5 is reserved for the exceptional review, the rare outstanding critique that is comprehensive, objective, and insightful. An exceptional review evaluates purpose of the study, study design, scientific validity, and conclusions by numbering questions and constructive suggestions to be addressed by the author. It also Includes comments to the editor about whether this is something new and important and useful to our readers and the review is submitted promptly. Approximately 10% of reviews are considered exceptional and are given a 5.
- A score of 4 indicates a very good review. This is an excellent review that indicates that the paper was carefully evaluated. It includes helpful comments to the author and editor with well-documented reasons for the decision and is returned within the two week deadline. A score of 4 is given to approximately 25% of reviews.
- A score of 3 is given for the average, satisfactory review (approximately 50% of reviews). The analysis not as well organized, documented, or as complete as above but is reasonable with adequate comments for the authors. A score of 4 is given to approximately 50% of reviews.
- A score of 2 is given to a very poor review and is considered below average. This review is very brief and not well evaluated. The reasons for the decision are not explained and comments to the authors are not helpful. This also includes reviews that are received very late. The score of 2 is given to approximately 10% of reviews.
- A score of 1 is given to 5% or less of reviews. They are considered unacceptable. A reviewer who receives a score of 1 may not be considered for further review. Reasons for grade could include evidence of bias; unfair, faulty reasoning; and comments to author either absent, inappropriate, or inadequate to explain how the paper was rated. It also includes reviewers who had to be prodded to get their reviews in.
Those who consistently score “1” or “2” may be dropped as reviewers. In this way, we can ensure that our review process helps our authors to improve their manuscripts, and helps the editors to select the best submissions for publication.