In Chapter 2, many criteria are presented that are meant to be used by reviewers as guideposts or reminders during the evaluation phase of a review. For the most part, for each of the criteria, the manuscript can be assessed either by using a dichotomous approach such as “yes/no” or “has it/does not have it,” or by using a rating scale that incorporates such terms as anchor points. Using the criteria is somewhat akin to scoring a test and marking each item “right” or “wrong” or assigning “partial credit” or “not answered.” Once the measurement phase is done, the task that remains is to use the criteria to make the overall judgment about the suitability of the scores. Reviewers almost always are asked to make recommendations for or against publication of the manuscript. Making the judgment is also similar to a testing situation, only in this task, instead of scoring each item, the reviewer is setting the standard, deciding “how much is enough.” Did the author(s) meet enough of the criteria, or the right criteria to lead to a positive recommendation? Or, stated negatively, did the work contain certain flaws or show an accumulation of shortcomings that mandate a negative decision?
At least two features of this final decision deserve mention: It is a recommendation, and it is a judgment. In “Publication Decision” in Chapter 1, processes used by the editor for making the decision are discussed. The main point is that the final decision belongs to the editor. That person almost always wants recommendations (not votes) from the reviewers—and a recommendation requires a judgment.
Reviewers are invited to review a manuscript on the basis of their expertise and prior work (see the first article in Chapter 1). They are expected to know enough to address the criteria (or to seek outside help for technical issues) and then to meld all of their little, individual decisions together into an overall recommendation. There is no formula for coming up with the recommendation that is the same for all reviewers, or even for one reviewer from one manuscript to the next. It is not as easy as counting the favorable and unfavorable marks on the list of criteria and deciding whether the count (i.e., score) is high enough for a “publish” recommendation. Almost never do, or should, all items count equally. In some cases, problems in one area can be “fatal” (e.g., a seriously flawed study design), and no accumulation of strengths elsewhere can compensate. In other cases, however, strengths in one area can counterbalance or even outweigh weaker elements, depending on the nature of the study. In general, it is a non-compensatory system, that is, a series of strengths cannot offset a fatal flaw or a series of important shortcomings. For example, a new and highly creative way to assess medical students' clinical skills might be expected to have a very strong Methods section so that the reviewer can glean a very good understanding of what was done. Other parts of the manuscript cannot be ignored, but they might not be (either implicitly or explicitly) weighed so heavily. On the other hand, a technical article that focuses on modeling the decision-making processes of residents and novices when faced with new and familiar tasks would likely need very strong introduction and data analysis sections.
How a reviewer puts all of the pieces together and makes a final judgment is a very individual thing. Occasionally a manuscript will have a fatal flaw. When this happens the recommendation is easy. But fatal flaws do not happen often. Usually the answer is not so clear. Although the inconclusiveness can be disconcerting to a reviewer, especially a relative novice, it may help to recognize that the recommendation is just that—a recommendation. The advice given to the editor may or may not be followed. When the editor's decision is contrary to the reviewer's recommendation, it seldom means that the reviewer gave a bad assessment.
Typically the balance of the scores on the criteria should be roughly aligned with the final recommendation. For example, it is not helpful to the editor when the majority of the criteria are marked high and the recommendation is “not to publish.” Many journals give reviewers the option of writing confidential comments to the editor that will not be shared with the author. This would be the place to explain the lack of congruency between criteria scores and the recommendation. The opposite scenario is probably a more common situation: A manuscript is marked down for many criteria, with well-grounded constructive criticism, and then the recommendation is to publish or ask for revisions. It appears to be very difficult for a reviewer to actually mark the “reject” category. This reluctance is understandable from the perspective of the reviewer, who is likely to have had his or her own share of negative publication decisions and is quite familiar with the angst such decision letters cause. But the primary commitment of the reviewer is to help the editor (and, by extension, the reader and the literature). Often the recommendation itself is not passed on to the author. Even if it is, if it is aligned with the individual marks on the criteria the recommendation can be taken in the right context.
Aside from the manuscripts that have readily apparent fatal flaws, or even multiple nonfatal flaws, there are many reasons to make a “reject” recommendation. For example, a manuscript might be very well done but fail to add anything to the existing literature. Or it might be inappropriate (or at least not interesting) to the readership. A rejection is also called for when the reviewer has a nagging “feeling” that something is wrong with the manuscript, for example, there is a misalignment between the study objectives and the study methods, or when the results do not match the conclusion.
An “accept as is” recommendation is relatively rare but it does happen—it is a logical outcome from the criteria review. A reviewer's job is not to be unduly critical, and when an excellent manuscript is spotted, the reviewer should mark it accordingly and point out to the editor the strong features that make it so appealing. Reviewers do not have to—of course, should not—restrict themselves to negative comments!
A middle ground of recommending revisions is certainly appropriate for many manuscripts, but reviewers must use this recommendation cautiously—it should not be used to soften or postpone what will almost certainly be a final rejection decision. A “revise” recommendation is most helpful to the editor and to the author(s) when it is accompanied by explicit comments about what needs to be addressed to improve the manuscript based on the existing methods and data. This means that is usually is not realistic to ask authors to change what has already occurred during the study phase. Manuscripts can be rewritten for clarity, different analyses can be done, more details on methods can be given, conclusions and interpretations can be amplified or reduced, and the authors can be directed to previous literature that was omitted. But in general authors—and reviewers—need to work with what is there. If a reviewer is primarily convinced that major weaknesses cannot be “fixed,” (e.g., having used an uncommon, non-generalizable experimental case), then a “reject” recommendation is in order.
Reviewers need to be clear about which of their comments are passed on to the authors. As noted above, many journals give reviewers the option of writing confidential comments to the editor. It is an option, not a requirement. If the reviewer has summarized the major positive and negative issues when responding to the criteria, there may not be a need to write anything else. Occasionally, however, reviewers will have remaining comments or insights that they do not wish to share with the author. As with other communications to the editor, comments are most helpful when they are clear and explicit. Ideally they would expand on points already made to the author so that the editor is not left with a review in which the criteria assessments do not match the recommendation.
In the end, the recommendation requires a judgment with which the reviewer is comfortable. It is a judgment but it is not capricious. It requires an integration of many evaluations and reflections—intuition and gestalt are a valid part of the formula. It gets easier with experience but it is rarely straightforward. If a reviewer goes into the process with the understanding that the goal is to be helpful to the editor and fair to the authors without being unduly critical, she or he will be poised to do a good job.
Review Criteria for Research Manuscripts
Joint Task Force of Academic Medicine and the GEA-RIME Committee