Peer review is central to the scientific process and involves the review of scientific output by a body of peers. Experts review a manuscript and provide feedback on the work, offering a recommendation of whether the work is worth publishing1,2. In this article, we aim to provide guidance on how to peer review a scientific paper.
Publishing a manuscript makes it accessible to the scientific community and is a step toward improving scientific understanding and, in biomedical literature, may also enhance patient care. Peer review is important because it serves to uphold the quality of the literature. Peer reviewers can act as gatekeepers to prevent poor quality research entering the literature, but also have a duty to promote scientific and intellectual freedom3. There are a number of criticisms of the existing peer review process and a number of developments have aimed to address this, these include blinding peer reviewers to the authors of the manuscript they are to review, peer reviewing after publication and various adaptations to the model of how feedback is provided including collaborative and interactive review4,5.
Being asked to peer review a paper suggests the reviewer’s knowledge of the field and/or critical appraisal skills are respected enough to be entrusted with gauging the quality of scientific research. There are great many advantages to peer reviewing, both to the scientific community and the individual themselves. This has been discussed in our earlier article6.
Peer review is rarely taught. Surveys of the science, technology, and medical community have found that 64% of authors that have not yet carried out peer reviewing would like formal training5. In this article we aim to provide a systematic guide on how to peer review a scientific paper.
The steps of peer review
Context
The first thing to consider when reviewing a paper is the context, if you have been invited, you are likely an expert in the field, but it is still worth double checking recent papers or reviews to ensure you are completely up to speed on the topic.
Reporting guidelines
At this point, it is also worth pulling up any reporting guidelines that are relevant to the paper to have alongside you as you work through the paper as these provide very useful pointers as to what should be included and how the information should be presented. Authors should state in their methods that they are reporting in line with a particular reporting guideline and submit a completed guideline checklist, referencing the page numbers particular items have been completed. Journals should mention this in the guide for authors as well.
There is some evidence for the effectiveness of guidelines. A Cochrane systematic review focusing on randomized controlled trials (RCTs) and CONSORT found that 25 of 27 outcomes assessing completeness of reporting, favored CONSORT-endorsing journals over nonendorsers, of which 5 were statistically significant.
Cobo et al7 performed a masked RCT to assess the impact of an additional review based on reporting guidelines like STROBE or CONSORT. In their 92 paper sample, they found an greater improvement from baseline in the additional review group than conventional review alone (43% vs. 20%).
A summary of important guidelines is outlined in Table 1.
;)
Table 1: Important reporting guidelines and their respective study types.
Overview
The next thing to do is flick through the paper, this allows you to get a rough overview of what you are dealing with. For example, has someone submitted their whole 50,000 word PhD thesis with no apparent headings, or is it a well laid out manuscript, clearly headed, and well formatted. While this is not a key part of peer reviewing, it certainly enables you to quickly appreciate the calibre of the work.
Key questions
There are key questions that you must aim to answer in your review:
- What is the research question (or hypothesis) the authors are trying to address? Is it clearly defined?
- Is the research question worth answering?
- Are the methods appropriate to answer the research question? Are they sufficiently robust?
- Have the authors provided answers to the research question(s)?
- Are their conclusions justified by the data and the methods?
- What does this study add to the scientific literature?
These questions are vital to answer explicitly in your review report. Peer reviewers do this in different ways, and certain journals will require specific questions to be asked. When conducting your first peer reviews, it is often then helpful to break down individual comments by section.
Introduction
The introduction is where authors describe the background to the topic and make a persuasive argument for what is currently unknown or uncertain. This should be followed by a single line outlining the central aim or hypothesis of the work, together with the type of study design used. The introduction should be concise, but provide a clear explanation that is appropriate for the level of readership of the journal. The key questions the reviewer should ask are:
- Is the central objective of the present work clear?
- Does this work follow clearly from previous work?
- What is the importance of this new objective?
Methods
The methods are a critical section of the paper. The following components should be addressed:
Protocol. Protocol is there a publicly registered and published protocol for the study which describes the research questions and the methods a priori. This can be useful to see if authors have changed their outcomes post hoc and therefore if selective reporting, misreporting or nonreporting is occurring. This was clearly shown by the COMPARE project, where of 67 trials checked, 354 outcomes were not reported, 357 new outcomes were silently added and on average each trial reported 58.2% of its specified outcomes and silently added 5.3 new outcomes.
Ethical approval. Ethical approval is required prospectively for all research studies involving human participants. The reference number and ethical review board providing approval should be clearly stated, with dates of approval also provided.
Protocol registration. Protocol registration or how the study was planned should also be stated here, if there is an available copy of the protocol, ensure you have a copy to hand to identify if the methods outlined there are the same as in the manuscript. If there are discrepancies between the 2, the reasons for this should be provided in the paper.
Study design. What is the design of the study that the authors have selected to answer the above question? Is it appropriate? Are there any particular reasons the authors outline for deciding to follow this design? For a new surgical or interventional technique, how does this study fit in with the pathway for accrual of evidence, for example have they mentioned the IDEAL framework15.
A very useful framework for the author and peer reviewer to use is the PICOS framework.
Population—define the demographics of the study population. This should include the sample size, age, and any relevant medical co-morbidities/previous interventions of relevance to the study. For example 3000 men that have had previous primary coronary interventions for ST elevation myocardial infarction.
Intervention—define the intervention in the present study. This should include the type of intervention and the length of exposure. If a procedure is carried out, a detailed description is required. For example, the use of statins at a specified dose for 1 year.
Comparison—describe any controls or comparison interventions that are studied. For example, the use of a placebo drug, which looks identical to the interventional drug.
Outcome—describe the outcome measures that will be extracted to assess any effect in the study. For example, mortality rates in 1 year; recurrence rates of ST elevation myocardial infarction in 1 year.
Situation—the context of the study is important. What kind of study area that is hospital, community. For example, outpatients’ cardiology clinic in a developed country.
Participant selection. This should include explicit inclusion and exclusion criteria that were applied in the cohort selection and the dates of the study. If this is a RCT, then the method of randomization should be stated, power calculations should be provided for all prospective studies. Similarly, the location and population the cohort is being drawn from should be described.
Adequate description of technical procedures. Any technical procedures used in the study (eg, how a drug was made up, how a certain surgical procedure was performed) should be referenced if standard, or described in a way such that a peer could repeat the activity. If there are any differences in how 2 groups were treated, this should be made very clear.
Clear definition of standardized outcomes. Outcomes should be defined a priori and should have a standard definition, this is especially important with multisite studies, where the local level for a particular marker of disease may be different from other centres. Complications should use standard scales, such as Clavien-Dindo for postoperative complications or postoperative morbidity surveys16,17. Standardization of outcomes allows clearer comparisons to be drawn, increases accuracy of data and enables future inclusion in meta-analysis. Another example would be in-hospital mortality which may be standardized to “any mortality prior to hospital discharge” where as early mortality may be defined as “death within 30 days of an operative procedure.” These would both include similar patients, but the overall groups would be different.
Data collection. The method of data collection should be described, including if this was done prospectively or retrospectively and if standardized forms were used to collect the data. The status of data collectors (Were they blinded to treatment allocation?) must be made clear. Was any training given to data collectors to boost accuracy and reduce inter-rater variability, what quality control measures were put in place?
Follow-up. The type of follow-up should be reported as this can affect the data collection and introduce specific biases. The length of follow-up is also important as certain outcome measures may be come invalid if follow-up is not sufficiently long.
Accounting for non–follow-up. This is particularly important in large cohort and randomized studies and a plan should be in place for dealing with loss to follow-up. This may include how the data that was already collected would be dealt with statistically, or what level of data was appropriate for inclusion of the patient within the cohort. Figures for loss of follow-up should be presented in results.
Standard definitions or scales. Standard definitions should be provided for all comorbid diseases and for any other important covariates. Standardized, and ideally internationally recognized definitions should be used and referenced. Examples include the definition of Chronic Kidney Disease according to eGFR(1)(2)(2)(2)(3)(3), and anemia according to WHO standard definitions18,19.
Statistical methods. For all outcomes of interest or for all questions, there should be a clearly defined statistical method. This may range from using simple comparisons of means (Student t test, etc.) to logistic regression modelling, where statistical methods must include how variables were selected. Handling of variables that were missing from the dataset must be described in detail. The statistical package used to perform all analyses should also be stated. Statistics is more than just about generating P-values and the importance of other measures like confidence intervals should not be overlooked. Where more than one rater exists for a particular variable, inter-rater reliability (eg, Cohen kappa) should be reported.
Key questions
- Is this study design appropriate for the question posed? Is it methodologically robust enough?
- Could this design be reproduced from the details provided in the methods?
- Are there any potential grey areas that could become confounders or bias?
Results
Results present the findings of the experiment or study. How this is presented will depend upon the study design, but it should broadly include: demographic information to highlight the comparability of any groups, results for each statistical method and for each outcome. Are the data appropriately presented, using tables, charts, and images where necessary (pie charts are rarely ever the appropriate presentation). Comments provided in this area by peer reviewers have been found to improve the quality of data graphs in published papers20. It is worth going through the results alongside the methods, specifically seeing if there any results with omitted methods or any methods where the results are not available? Are statistical measures of uncertainty or variability reported?
Key questions
- Are full data and results provided, is anything missing?
- Is the layout clear and concise, with standard reporting of statistical measures?
- Are there any results with missing methods, have they cherry picked a result with a good P-value?
Discussion
There are a number of ways that discussions are presented, but here are some key subsections in order:
- A summary of the research findings.
- Discussion of the research findings in the context of the literature (ideally including both index papers and recent literature).
- Implications of the work for clinical practice and/or research.
- Future research questions generated or now needing to be answered.
- Strengths, weaknesses and limitations of the work.
Ideally, the discussion opens with a summary of the findings of a paper and this should directly answer the question posed at the end of the introduction. All results discussed should have been reported in the results section, with no new results presented in the discussion section.
This should then be followed by appropriate contextual discussion: have the authors missed a key paper that is crucial to their conclusions? If there is a surprising finding, then efforts should be made to account for this. It is critical to ensure that the authors are making statements that their results can back up, if you read it and think there has been a quantum leap from data to discussion, say so.
Next, limitations and strengths should be discussed. No study is perfect, and all designs come with inherent flaws. The authors should seek to outline these and how they alter their interpretation of the results.
The implications of the study findings on research should be described next, specifically if there are further questions that need to be investigated. It is here that new hypotheses may be discussed. It may be that the study you are reviewing provides the definitive answer to the question and no further study is needed, if so, they should state this.
The impact of the work on patients and clinical practice should be the final part of the discussion. This includes whether and how the study should change practice. This would be rare outside the context of a major high-quality RCT or meta-analysis of high-quality RCTs. It may indicate need for a small change to how we care for patients, and this should be explored here. There should be a logical flow to the discussion and the reader should be taken on a sensible journey. Often further definitive research is called for from nonrandomized studies. It should be noted however, that an RCT is not always practical or ethical in all areas, for example pediatric and emergency populations.
Conclusion
The conclusion should be the takeaway message(s) of the paper. It should not include numbers, unless necessary to summarise a point. It should end with implications and direction of the work for the wider clinical and academic community. They key thing to note is whether the conclusions can be justified by the data and the methods.
Other important points to note
- Funding—who paid for the study? Who provided equipment?
- Authorship—who did what in the study? Criteria for authorship have been outlined by ICMJE21. Some journals also like to identify a guarantor. The guarantor accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish.
- References—are these up to date? Are the citations appropriate to the journal style?
- Conflicts of interest—what are the conflicts of interests of the respective authors?
- Images—is appropriate consent provided for all images if used? Are they of sufficient quality? Print journals will often require images captured at higher resolutions.
- Reporting guidelines—does the manuscript fulfil relevant reporting standards (Table 1).
A checklist for reviewers is provided in Table 2.
Table 2: Peer reviewer checklist.
Decision/providing feedback—structuring your peer review report
Feedback typically consists of an opening few sentences that briefly outline the work and an opinion on whether the paper is of significant interest/quality and should be published (with or without revisions). This is followed by a point by point critique. This is often easiest if laid out in bullet points per each section, rather than as a single block of text. For example, revisions categorized into introduction, methods, results, etc.
These can also be divided into major and minor points. The “key points” in the overview section at the start of this article are a useful framework when structuring a peer review.
Journals differ on the overall decision they can provide, but these may include:
Accept—if you feel the paper is ready for publication in its current form.
Accept with minor revisions—if some small, minor changes are required, for example spelling/grammar/reference issues. The decision implies that if these changes are made, the paper would be accepted.
Minor revision—a small or modest revision is required to improve the paper, for example small additional details needed to any of the major sections of the paper, perhaps certain discussion points not elaborated fully or a recent, relevant paper is not discussed and referenced.
Major revision—major “surgery” of the paper is required, for example significant gaps in the methods or discussion, poorly presented results. This often requires more data to be collected for the study, or additional outcomes to be assessed.
Reject and invite resubmission—the topic or research question is interesting, but the wrong methods or insufficiently robust methods are used and hence the data is not reliable. Authors are being asked to do it again differently. This decision may also be used where a paper requires enormous amounts of changes, or where an author repeatedly fails to respond to calls for revision.
Reject—the paper is currently unacceptable (please ensure you submit comments alongside a reject decision, as these can be very useful for an author to improve their work before their next submission).
Typically, this decision is based on a combination of your point by point feedback, the overall clarity of the paper and the importance of the question being answered. It should be noted that it is not the reviewer’s responsibility to do a proof read and point out all the grammatical, spelling and sentence structure issues if there are many. A reviewer is within their right to state that a paper is simply unintelligible if the English language is so poor that the paper cannot be understood. Authors can then submit a revised version or the editor can reject it on that basis. There are numerous English language services that authors can be pointed to that will perform such a task for a fee. Large international institutions may even such people in-house.
Other resources
Other resources such as online courses exist, provided by journals and external organizations22,23. Individual support and advice should also be provided by your journal editors.
Summary
Peer reviewing is central to the scholarly publication process. Peer reviewers, along with editors are guardians of the scholarly record and have an important role in the advancement of science. We have outlined a detailed a step-wise guide in how to effectively peer review a paper. We have outlined key points that should be addressed in any peer review, which should provide a manual, or how to guide for any peer review.
Ethical approval
Ethical approval was not required for this study.
Sources of funding
There were no external sources of funding for this study.
Author contribution
All authors contributed toward the final paper. A.J.F.: production of content/editing; K.K.: production of content/editing; B.G.: production of content/editing; R.A.A.: production of content/editing.
Conflict of interest disclosure
R.A.A. is the founding and executive editor of the IJS Publishing Group. The remaining authors declare that they have no financial conflict of interest with regard to the content of this report.
Research registration unique identifying number (UIN)
No research registration was required for this study.
Guarantor
The guarantor for this study is author R.A.A.
Acknowledgments
The authors are indebted to all members who helped them complete this study.
References
1. VoYs.
Peer review: the nuts and bolts. London; 2012.
2. House of Commons: Science and Technology Committee.
Peer review in scientific publications. 2011;251. Available at:
https://publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/85602.htm. Accessed June 8, 2011.
3. Ayalew E.
Peer review mechanisms: the bottleneck of academic freedom. JHEA/RESA (Journal of higher education in Africa). 2012;9:91–112.
4. Smith R.
Peer review: a flawed process at the heart of science and journals. J R Soc Med 2006;99:178–82.
5. Taylor & Francis Group.
Peer review: a global view. 2016;1–24. Available at:
https://authorservices.taylorandfrancis.com/peer-review-global-view/. Accessed June 8, 2017.
6. Koshy K, Fowler AJ, Gundogan B, et al.
Peer review in scholarly publishing part A: why do it? IJS Oncol 2018;3:e56.
7. Cobo E, Cortés J, Ribera JM, et al. Effect of using reporting guidelines during
peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial. BMJ 2011;343:d6783.
8. Gagnier JJ, Kienle G, Altman DG, et al. The CARE guidelines: consensus-based clinical case reporting guideline development. J Med Case Rep 2013;7:1–6.
9. Agha RA, Fowler AJ, Saeta A, et al. The SCARE statement: consensus-based surgical case report guidelines. Int J Surg 2016;34:180–6.
10. Agha RA, Fowler AJ, Rajmohan S, et al. Preferred reporting of case series in surgery; the PROCESS guidelines. Int J Surg 2016;36:319–23.
11. von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med 2007;82:251–9.
12. Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869. c869.
13. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339:b2700. b2700.
14. Ogrinc G, Mooney SE, Estrada C, et al. The SQUIRE (Standards for Quality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration. Qual Saf Heal Care 2008;17(suppl 1):i13–32.
15. Hirst A, Agha RA, Rosin D, et al. How can we improve surgical research and innovation?: The IDEAL framework for action. Int J Surg 2013;11:1038–42.
16. Grocott MPW, Browne JP, Van der Meulen J, et al. The Postoperative Morbidity Survey was validated and used to describe morbidity after major surgery. J Clin Epidemiol 2007;60:919–28.
17. Dindo D, Demartines N, Clavien P-A. Classification of surgical complications. Ann Surg 2004;240:205–13.
18. Vitamin and Mineral Nutrition Information System. Haemoglobin Concentrations for the Diagnosis of Anaemia and Assessment of Severity. Geneva: World Health Organisation; 2011.
19. Levey AS, de Jong PE, Coresh J, et al. The definition, classification, and prognosis of chronic kidney disease: a KDIGO Controversies Conference report. Kidney Int 2011;80:17–28.
20. Schriger D, Raffetto B, Drolen C, et al. The effect of
peer review on the quality of data graphs in Annals of Emergency Medicine. Ann Emerg Med 2017;69:444–52.
21. International Commitee of Medical Journal Editors. Defining the role of authors and contributors. ICMJE authorship guidelines. 2017. Available at:
www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html. Accessed November 3, 2017.
22. Hodge C. Free online course on
peer review. Nature Masterclass. 2017. Available at:
https://masterclasses.nature.com/users/4925-claire-hodge/posts/20006-free-online-course-on-peer-review. Accessed August 4, 2017.
