Review Form

Steinecke, Ann; Shea, Judy A.

Review Process and Publication Decision: Chapter 1

Areview form is the instrument that a reviewer uses when he or she assesses the quality of a manuscript. A review form should make it easy for the reviewer to supply both the editor and the author with detailed information about the strengths and weaknesses of a manuscript. The editor will use the information to help decide whether or not to publish the article; in some cases, an editor relies on nothing other than the information contained in a manuscript's reviews. The author can use the information from a review form to better understand the decision made by the editor and to learn about the strengths and weaknesses of the manuscript and methods for improvement. In this way, the review form is a tool for reviewers to communicate one set of information that can be used for different purposes by the editor and the author.

No standard review form is used among the journals in any scientific field; further, there are no studies that demonstrate the best format for review forms. The features of review forms vary widely depending on the type of information each editor seeks from reviewers—some request narrative comments alone,1 while others combine checklists, global ratings, and narrative comments.2 Most are composed to solicit the same general categories of information: the importance of the research question, the quality of the research, the justifiability of the conclusions, and the appropriateness of the manuscript for publication in the journal.

In the description and discussion below, the specific examples are taken from research journals in academic medicine and the health professions, but they could have been taken from any research field in the biosciences and social sciences. Appendix 3 gives examples of forms from both fields.

Different response formats (global ratings, checklists, narrative comments, etc.) offer reviewers different advantages, so they are often used in combination. Global rating scores, such as Teaching and Learning in Medicine's four-point scale (“Should this paper: be published in TLM, be published with minor modifications, be reconsidered only after major changes, be rejected”) are efficient and enable the editor to quickly rank a manuscript. However, global rating scores alone do not assist the editor in making a judgment by providing a nuanced discussion of the manuscript's strengths and weaknesses, nor do they provide the author with detailed comments. For these reasons, global ratings are often not shared with the author.

Checklists direct the reviewer to evaluate specific aspects of a study, as in this example from Medical Education.

Back to Top | Article Outline

Methods

Some journals used a dichotomous (yes/no) scale, with or without comments, while the review forms of others use rating scales for every item (e.g., unacceptable, poor, satisfactory, excellent). For reviewers, the benefits of a checklist include the ready identification of the key elements of a manuscript that should be judged for soundness, particularly those having to do with research method. Further, the reviewer's evaluation of these key elements often provides a source of positive feedback for authors on the review form. Whether the “direction” that checklists provide to reviewers in evaluating manuscripts ultimately leads to a more complete evaluation of a manuscript's soundness is open to debate. While one study found that only 25% of reviewers identified methodologic flaws in their narrative comments,3,4 the risk is probably as great that reviewers will not identify methodologic flaws in their checklist responses. However, in combination with an opportunity to discuss a manuscript's strengths and weaknesses in narrative form, checklists can be particularly valuable for identifying significant problems within a manuscript.

Table. No caption av...

Most journals ask reviewers to provide some form of narrative comment, through open-ended questions or a section for free response. Narrative comments allow reviewers to respond with the greatest level of detail. The Academic Medicine review form asks reviewers to answer a series of brief open-ended questions such as

2. Please explain why the research design and analysis of the data is or is not appropriate for the research question.

and

4. Given the topic and research question, who in the academic medicine community would be interested in reading this paper? Who (if anyone) should read this paper?

These questions free reviewers from preformulated responses and allow detailed critique.

Alternatively, some journals simply provide reviewers space to add any comments they wish to make about the manuscript. However, free text comments in either format do present several drawbacks for reviewers. They require more time and effort to complete than do global ratings and checklists. Review forms that ask for narrative comments tend to be longer because they need to leave the reviewer with enough space to record comments. Many reviewers respond to questions with hand-written comments, which, with length, can be increasingly difficult for the editor and author to decipher. Alternatively, when reviewers type their comments on separate sheets of paper the added step may discourage timely completion of the review. The British Medical Journal is one journal that has solved these issues by doing away with the review form altogether. Instead, they send reviewers only a copy of the instructions for review and have reviewers return their comments on stationery.

Reviewers should be aware of two ways that review forms can inadvertently affect their reviews. First, when completing a review using a form that combines response formats, reviewers may inadvertently contradict what they have indicated on the checklist with what they have written in their comments. For example, a reviewer's global ratings for a manuscript may be positive for publication, whereas narrative comments may elucidate a fatal flaw. When narrative comments are requested, reviewers should be careful to provide the editor and the author with more nuanced and detailed information about the manuscript, explain the severity of the problems they have detected in the manuscript, and estimate the probability of correcting the problem with a revision.

The second problem with review forms that reviewers should be mindful of is that studies have revealed reviewers' biases based on the direction, significance, or controversial nature of findings.5,6 Review forms that offer multiple response formats can reduce the risk of a review form's overly directing a reviewer's response. An ill-phrased question can unintentionally lead a reviewer to a biased response. To help diminish bias, reviewers should scrutinize questions for partiality and take advantage of other modes of response to fully explain their responses.

There is ongoing debate about whether or not review forms should be tested for inter-rater reliability. Many studies have demonstrated a lack of agreement between reviewers' recommendations, with correlations averaging.3,7 Editors of clinical specialty journals, especially those who rely most heavily on reviewer's recommendations in decision making, value high inter-rater reliability,8,9 and tested review instruments have been described.10 But what is desirable in reviews of clinical manuscripts may not be desirable for medical education manuscripts. Although review forms for these disciplines share many characteristics and aims, particularly for assessing study design and method, medical education editors may prefer forms that allow competing viewpoints, which some suggest strengthens the quality of the decision-making process.11

Finally, in addition to providing reviewers with enough response formats, including a clear statement of the journal's purpose and instructions with the review form can help reviewers understand their responsibilities, the information the editor is seeking, how that information will be used in decision making, and the kinds of responses that are useful to authors. It has been demonstrated that with mailed questionnaires, the attractiveness of the questionnaire's format, the questionnaire's length, and the ease of filling out and returning the questionnaire can affect response rates.5 To encourage reviewers to respond, editors should consider these issues when designing review forms. One recent example of journals' taking convenience for the reviewer into consideration is the increasing use of e-mail- or Web-based review forms to improve response rates and times. By using a standard review form for its research manuscripts and by giving reviewers a choice of response formats (checklists, global ratings, narrative comments for authors and/or editors) a journal can create a review form that can inform both the editor and the author with sufficient and appropriate responses.

Back to Top | Article Outline

References

1. Bligh J. What happens to manuscripts submitted to the journal? Med Educ. 1998;32:567–70.
2. Nyleena M, Riis P, Karlsson Y. Multiple blinded reviews of the same two manuscripts. JAMA. 1994;272:149–51.
3. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263:1385–9.
4. Bordage G. Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports. Acad Med. 2001;76:889–96.
5. Armstrong JS, Hubbard R. Does the need for agreement among reviewers inhibit the publication of controversial findings? Behav Brain Sci. 1991;14:136–7.
6. Fiske DW, Fogg L. But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. Am Psychologist. 1990;45:591–8.
7. Cullen DJ, Macaulay A. Consistency between peer reviewers for a clinical specialty journal. Acad Med. 1992;67:856–9.
8. Cicchetti DV. The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behav Brain Sci. 1991;14:119–35.
9. Cho MK, Bero LA. Instruments for assessing the quality of drug studies published in the medical literature. JAMA. 1994;272:101–4.
10. Eckberg DL. When nonreliability of reviews indicates solid science. Behav Brain Sci. 1991;14:145–6.
11. Selitz C, Jahoda M, Deutsch M, Cook SW. Research Methods in Social Relations. New York: Holt, Rinehart & Winston, 1959.
Back to Top | Article Outline

RESOURCES

For sample review forms from a variety of scholarly journals, see Appendix 3.
Back to Top | Article Outline

Section Description

Review Criteria for Research Manuscripts

Joint Task Force of Academic Medicine and the GEA-RIME Committee

© 2001 Association of American Medical Colleges