With experience, each reviewer develops a strategy for reviewing manuscripts. Some look at the title and immediately go to the tables and figures, as they would for a journal article. Others read through in one fell swoop to get an overall feeling for the report before getting their red pencils out for careful and detailed evaluation. In “Review Criteria for Research Manuscripts,” the review criteria and related topics are presented with these various types of reviewers in mind. From a global perspective, the document is divided into three main sections: (1) the review process and publication decision, (2) the review criteria, and (3) the reviewer's recommendations and etiquette. The review criteria per se, presented in the middle section of the document, follow a general strategy that takes the reviewer through four steps that mirror the process likely to be followed in real life with a manuscript in hand, that is, (1) the body of the manuscript: introduction, method, results, discussion, and conclusion, (2) the title, authors (when revealed), and abstract, (3) presentation and documentation, and (4) scientific conduct.
As this order suggests, reviewing a manuscript is not linear. It is necessarily iterative. For example, while the abstract orients the reviewer initially, it can be fully evaluated only after the entire manuscript has been reviewed in detail. Finally, and in the spirit of a practical tool for reviewers, the document ends with three appendices, one containing the complete list of criteria in a checklist format, another containing all the resources listed throughout the document, and one showing examples of manuscript review forms in use at different journals.
All articles in the chapter on review criteria follow the same format. First the list of review criteria is presented. These criteria were derived according to a four-stage process. The task force members began by listing criteria freely using a nominal group process, yielding an initial list of 288 criteria. Then the members selected specific sections and reviewed the literature, supplemented the initial list, and finally pruned it down to key issues of specific concern to reviewers. A complete draft of the criteria by topics was then assembled and sent to two external reviewers (in the spirit of practicing what we preach!) for comments and suggestions. Finally the current set of 77 criteria was settled on.
Certain criteria are applicable depending on the type of research reported in any given manuscript, such as qualitative, quantitative, descriptive, inferential, or evaluative. Thus the list is all-encompassing rather than selective or organized according to research types. The criteria are stated as generally as possible so that reviewers can apply them within the framework of different sciences and along the spectrum of methods. The criteria are not, however, intended to be applied as a formula. They are not of equal weight, in terms of either the research being reported or the presentation in the paper. The reviewer judges whether and how the author has met the criteria, and judges the relative weights to give to different criteria and different sets of criteria. Not all criteria will apply to one paper, depending on the nature of the study, although most will apply to most papers.
The criteria are deliberately short and self-explanatory to the typical reviewer, who is an initiated investigator or practitioner. The criteria are stated in a positive form—for example: “The introduction builds a logical case and context for the problem statement”; “The design is appropriate (optimal) for the design question.” The positive form allows the reviewer to recognize the strengths of the manuscript while pinpointing weaknesses and fatal flaws. Reviewers should indicate to the editor the strengths of a manuscript. However, their main responsibility is to ensure that only sound research gets accepted for publication; thus they should also indicate weaknesses and flaws. The review criteria are formulated to help reviewers focus their attention and provide directly useable statements for writing the review, conserving time and energy. Special attention was paid to limiting the number of criteria in each section by focusing on key, critical issues especially important to reviewers, as opposed to creating long lists of guidelines for planning a study or writing an article. Obviously, although useful to researchers and authors, the criteria are intended foremost for reviewers.
Following the list of criteria is the main body of each article, presented under the subheading “Issues and Examples Related to the Criteria,” which includes referenced background information, a brief description of the criteria or a rationale for their inclusion, and some examples to illustrate the concepts. Each article ends with the cited references and a more detailed list of complementary resources, thus providing a practical tool within a scholarly package. The references and resources are intended for reviewers who want to read more in depth or for educators who want to plan faculty development activities to train reviewers or authors.
The set of review criteria can be viewed as a meeting place of three communities: the community of peer reviewers, the community of author—researchers, and the community of editors and publishers. Every member shares a common goal of maintaining high-quality standards for the field. While reviewers and editors want to ensure top quality by promoting high standards, author—researchers may at times feel mistreated or unhappy about the decisions being made about their manuscripts. The criteria we propose represent a clear and explicit expression of the expectations of the reviewers that can be shared both among peer reviewers and among fellow researchers and authors. Having the criteria available for authors—as journals make some of theirs available if asked—helps to make the process more fair and open. In the end, everyone benefits from an outstanding body of scientific knowledge. Authors write better papers, and reviewers make better evaluations. As a consequence, editors can make more informed publication decisions, and faculty and researchers have better papers on which to base their decisions and pursue new lines of inquiry.
Review Criteria for Research Manuscripts
Joint Task Force of Academic Medicine and the GEA-RIME Committee