What Is Known/What Is New What Is Known
Peer review is considered a central component of the publication process.
There is no current standard or guideline by which peer review is completed.
What Is New
Peer review typically results in moderate agreement between the reviewers.
Inter-reviewer agreement correlates strongly with the editor's initial decision.
The editor's initial decision strongly correlated with the final disposition of the submission.
Scholarly work is an essential component of an academic physician's career, and yet, it has become progressively more challenging to publish in prestigious journals. Many submissions are rejected without peer review, and only a minority of submissions are accepted for publication. Factors such as the number of reviewers, rounds of review, length of time for review completion, backlog to publications, and/or inability to fill a particular issue, can all affect the acceptance rate (1) peer review process , itself, also alters acceptance rates significantly (1)
Though the manuscript review process (MRP) is described in various scientific journals in the scientific community, no clear consensus or standard regarding the MRP itself has been developed. Despite a long-standing history of peer review in the scientific community, discussion and examination of the components of peer review and the best manner in which to conduct peer review continues (2–5) JPGN is an international pediatric peer reviewed journal with a 2020 impact factor of 2.94 that prioritizes original research manuscripts in the fields of developmental biology and pathogenesis related to intestinal, hepatobiliary, and pancreatic function. It ranks 53 of 88 in Gastroenterology & Hepatology, 45 of 89 in Nutrition & Dietetics, and 17 of 128 in Pediatrics. Given the importance of scholarly work in academic medicine and the need to publish high quality research in prestigious journals, better understanding of the MRP will be useful for authors, reviewers, and editorial boards, regardless of subspecialty or discipline. We aim to describe the MRP at the Journal of Pediatric Gastroenterology and Nutrition (JPGN ), assess the correlation between editor decisions and reviewer recommendations, and provide transparency to the process for prospective authors.
METHODS
We accessed submission data for all manuscripts submitted to JPGN in 2018 from both the North American (NA) and European (EU) editorial offices. Available information included reviewers’ manuscript scores (scale of 1–5 with 1 being the best) and recommendations (accept [A], accept with revisions [AWR], revise no guarantee [RNG], reject [R]), the editor's first decision (A, AWR, RNG, R) following peer review, the editor's rating (scale of 0–100 with a score of >70 considered acceptable) of the reviewers’ reviews on the manuscript, and final decision (A or R). Reviewers were asked to rate manuscripts based on five categories: quality of figures, tables, legends; design, methods; originality of submission; scientific contribution; and overall priority score. The dates of manuscript submissions and revisions, dates of reviewer recommendations, editor's initial decision, and final decisions were also collected. All data was gathered using the JPGN manuscript submission website Editorial Manager.
Statistical methods included chi-square tests for categorial variables and Kruskal-Wallis tests for continuous variables. Dunn test was used to perform the pairwise comparisons followed by Kruskal-Wallis test; the Bonferroni method was used to adjust for multiple comparisons. The intra-class correlation (ICC) coefficient was used to quantify agreement between two reviewers with a 95% confidence interval, and the Spearman correlation coefficient (rho) was used for the correlation between continuous variables. The study was deemed exempt by the Institutional Review Board at the University of Illinois College of Medicine Peoria. This study was approved by JPGN editorial board.
RESULTS
Data Overview
Of the 1023 total submissions, the most common submission types included original articles and case reports, which together comprised 75.7% of all submissions (Table 1 ). Of all 1023 submissions received by JPGN , 407 (39.8%) received an editorial decision to Outright Reject, meaning that these were not subjected to peer review. Of these, about half were original articles (229/407, 56%) and a quarter were case reports (103/407, 25.3%). Submission types with the highest acceptance rates included editorials (5/5, 100%), awards (2/2, 100%), meeting proceedings (1/1, 100%), societal papers (15/16, 93.8%), and letters to the editor (34/47, 72.3%) (Table 1 ). Overall acceptance rate to JPGN was 36.4% (372/1023) (Table 2 ). If case reports are removed from the numerator and denominator, the overall acceptance rate changes minimally to 40.7%. Of the 1023 total submissions, 38 had the editor's initial decision of A (3.7%), 110 to AWR (10.7%), 269 to RNG (26.3%), and 606 to R (59.2%) (Table 2 ); the 606 R included the 407 Outright Reject. Almost all manuscripts with the editor's initial decision to AWR (109/110) or RNG (223/269) were ultimately accepted (Table 2 ). Of note, 16 of 1023 submissions could not proceed to a final decision of A or R due to lack of revised resubmissions.
TABLE 1 -
Distribution of submission categories and number of articles accepted
Submission category: overall
Number of articles (%), N = 1023
Number of articles accepted by submission category (%)
Original articles
632 (61.8)
214 (33.9)
Case reports
142 (13.9)
13 (9.2)
Short communications
72 (7.0)
31 (43)
Image/video of the month
66 (6.5)
33 (50)
Letters to the editor
47 (4.6)
34 (72.3)
Reviews
22 (2.2)
8 (36.4)
Society papers
16 (1.6)
15 (93.8)
Supplements
9 (0.9)
9 (100)
Rapid communications
5 (0.5)
3 (60)
Editorials
5 (0.5)
5 (100)
Topic of the month
4 (0.4)
4 (100)
Awards
2 (0.2)
2 (100)
Meeting proceedings
1 (0.1)
1 (100)
Submission category: North American office
Number of articles (%), N = 443
Number of articles accepted by submission category (%)
Original articles
229 (51.7)
119 (52)
Case reports
54 (12.2)
13 (24.1)
Short communications
44 (9.9)
25 (56.8)
Image/video of the month
60 (13.5)
27 (45)
Letters to the editor
28 (6.3)
20 (71.4)
Reviews
10 (2.3)
4 (40)
Society papers
8 (1.8)
8 (100)
Rapid communications
2 (0.5)
2 (100)
Editorials
2 (0.5)
2 (100)
Topic of the month
3 (0.7)
3 (100)
Awards
2 (0.5)
2 (100)
Meeting proceedings
1 (0.2)
1 (100)
Submission category: European office
Number of articles (%), N = 580
Number of articles accepted by submission category (%)
Original articles
403 (69.5)
95 (23.6)
Case reports
88 (15.2)
0 (0)
Short communications
28 (4.8)
6 (21.4)
Image/video of the month
6 (1.0)
6 (100)
Letters to the editor
19 (3.3)
14 (73.7)
Reviews
12 (2.1)
4 (33.3)
Society papers
8 (1.4)
7 (87.5)
Supplements
9 (1.6)
9 (100)
Rapid communications
3 (0.5)
1 (33.3)
Editorials
3 (0.5)
3 (100)
Topic of the month
1 (0.2)
1 (100)
Awards
0 (0)
0 (N/A)
Meeting proceedings
0 (0)
0 (N/A
TABLE 2 -
Editor's initial decision and final decision
Editor's initial decision
Final decision
Accept, N = 38 (%)
Accept with revision, N = 110 (%)
Revise no guarantee, N = 269 (%)
Reject, N = 606 (%)
Total, N = 1023 (%)
Accept
38 (100)
109 (99.1)
223 (82.9)
2 (0.3)∗
372 (36.4)
Revise
0 (0)
0 (0)
16 (5.9)
0 (0)
16 (1.6)
Reject
0 (0)
1 (0.9)†
30 (11.2)
604 (99.7)
635 (62.1)
P value < 0.001.Exact test, chi-square test.
∗ Further examination of the two submissions with initial editor decision to R but final decision to A: invited commentary, initial editor decision to outright reject; letter to the editor, initial editor decision to outright reject.
† Further examination of the submission with initial editor decision to accept with revisions (AWR) but final decision to reject (R): original article, three reviewers, initial editor decision to AWR, final decision to R.
Reviewer Data
Of the 1023 total submissions, 543 underwent peer review by at least one reviewer. Reviewer manuscript scores correlated with reviewers’ recommendations for manuscripts. Manuscripts with reviewer recommendations to A, AWR, and RNG had better manuscript scores compared with manuscripts with reviewer recommendations to R (A—Reviewer X: 2.2 ± 1.2, P < 0.001; Reviewer Y: 2.6 ± 0.9, P < 0.001; AWR—Reviewer X and Y: 2.6 ± 0.7, P < 0.001; RNG—Reviewer X and Y: 3.1 ± 0.6, P < 0.001; R—Reviewer X: 3.6 ± 0.8, P < 0.001; Reviewer Y: 3.7 ± 0.8, P < 0.001) (mean ± SD, P < 0.001) (Fig. 1 ). This supports that the reviewer tool is valid and consistently utilized. The self-reported hours spent by reviewers did not correlate with the final outcome of the manuscript.
FIGURE 1: (A) Average reviewer manuscript score and reviewer recommendation. (B) Inter-reviewer agreement and editor's initial decision.
Reviewer and Editor Recommendations
Regardless of whether the editor agreed or disagreed with the reviewer's recommendation, the editor's rating of the reviewer's review (representing the editor's perceived quality of the review) was generally the same, with a score of 86.6 ± 10.8 (for Reviewer X, mean ± SD) and 86.5 ± 12.2 (for Reviewer Y, mean ± SD). This suggests that the editor evaluated the review itself rather than concordance of the review with the editor's decision. The self-reported hours spent by reviewers (mean 2.25, range 1.0–4.0) did not correlate with the editor's ratings of the reviewers’ reviews (Spearman correlation coefficient < 0.2).
Inter-Reviewer Agreement
Subset analysis of the 486 manuscripts that had at least two peer reviewers was completed to further understand the role of inter-reviewer agreement in the MRP. The recommendations of the two reviewers were in agreement 43% of the time. The ICC between the two reviewers suggests moderate agreement (ICC = 0.40, 95% confidence interval 0.29–0.50) (Table 3 ). If both reviewers recommended to not Reject the submission (ie, either A, AWR, or RNG), the editor agreed in 93% of cases (269/289); when both reviewers agreed to Reject, the editor agreed in 100% of cases (55/55) (Fig. 1 ). If reviewer recommendations were discordant (ie, one reviewer recommended to not Reject [A, AWR, or RNG] and the other recommended to Reject) (142/486), the editor rejected the submission in 66% of cases (95/142). Overall, inter-reviewer agreement strongly correlated with the editor's initial decision (P < 0.001).
TABLE 3 -
Agreement between two reviewers’ recommendations
Reviewer Y's recommendation
Reviewer X's recommendation
Accept
Accept with revision
Revise no guarantee
Reject
Total
Accept
2
17
6
3
28
Accept with revision
9
131
46
45
231
Revise no guarantee
5
52
21
29
107
Reject
5
33
27
55
120
Total
21
233
100
132
486
Intra-class correlation = 0.40, 95% confidence interval is [0.29–0.50].
Original Articles
Of the 632 original articles (61.8%), slightly over one-third were rejected without peer review (229/632, 36.2%). The remaining 403 original article submissions underwent peer review; 53.1% (214/403) were ultimately accepted and 44.9% (181/403) were ultimately rejected after peer review. Eight original article submissions (8/403) could not proceed to a final decision due to the lack of submission of revised resubmissions and thus remained categorized with an editor initial decision and final decision to revise.
Editorial Offices
Data Overview
Of 1023 total submissions, 443 manuscripts were submitted to the NA office and 580 to the EU office. Analysis revealed a higher overall acceptance rate in the NA office (51%, 226/443) compared with the EU office (25%, 146/580). The NA office had a lower rate of Outright Reject decisions (submissions rejected without peer review) (22%, 98/443) compared with the EU office (53%, 309/580).
Inter-Reviewer Agreement
Of the 486 submissions with at least two peer reviewers, 257 were reviewed within the North American (NA) office and 229 within the European (EU) office. The ICC between the two reviewers in the NA office was 0.27 (suggesting fair agreement) compared with an ICC of 0.51 (suggesting moderate agreement) in the EU office.
Original Articles
Of 443 submissions to the NA office, 229 manuscripts were original articles. Of these, 15.7% (36/229) were outright rejected. The remaining 193 original article submissions underwent peer review, and of these, 61.7% (119/193) were accepted and 34.7% (67/193) were ultimately rejected after peer review. Seven original article submissions (7/193) to the NA office could not proceed to a final decision due to the lack of submission of revised manuscripts and thus remained categorized with an editor initial decision and final decision to revise. Of 580 submissions to the EU office, 403 manuscripts were original articles. Of these, 47.9% (193/403) were outright rejected. The remaining 210 original article submissions underwent peer review, and of these, 45.2% (95/210) were accepted and 54.3% (114/210) were ultimately rejected after peer review; one submission to the EU office (1/210) could not proceed to a final decision due to the lack of a submission of a revised manuscript and thus remained categorized with an editor initial decision and final decision to revise.
DISCUSSION
This analysis reports data on the MRP at a well-established journal and provides insight into the MRP. Some cite the first major journal credited to pioneer the peer review process as the British Medical Journal in 1893, when non-editorial submissions were sent to an external expert (6) Philosophical Transactions , in 1662 (2)
Prompted by the increasing complexity of medical research, most biomedical journals have since implemented their own MRP to screen the quality of submissions. The Peer Review Congress held its first meeting in 1989 in Chicago, Illinois to improve the conduct, reporting, quality, and integrity of scientific research (7) (7)
Data from JPGN demonstrated that the overall acceptance rate was 36.4%, which appears to be within typically reported norms for scientific journals. Of note, though case reports were handled somewhat differently between the NA and EU offices (24% acceptance rate (13/54) versus 0% (0/88)), overall acceptance rate of the journal remained similar even when case reports were removed from the analysis (40.7%). Bjork reported that the estimated global average acceptance rate for manuscripts submitted to a wide variety of journals was around 35–40% and varied considerably based on factors such as study discipline and the country from which the research was completed (1) Academic Emergency Medicine showed that the journal had an acceptance rate of 32%, but that nearly two-thirds of all rejected manuscripts were ultimately successfully published in other journals (8) JPGN , the percentage of submissions ultimately accepted after the editor's initial decision of RNG was high (82.9%) and favorable for prospective authors. Just under 40% of all submissions were rejected without peer review, which appears to be consistent with norms for other scientific journals.
As an expert, a peer reviewer is expected to assess studies in their specialty for procedural logic, originality, and impact on the field. Regarding the impact of peer review on editor decisions at JPGN , the editor most often agreed with reviewers’ assessments when there was concordance between the two reviewers’ recommendations. About a third of peer reviews resulted in discordant recommendations between the two reviewers. Of these, two-thirds of manuscripts were ultimately rejected. This was supported in a review of all manuscripts submitted to Stroke over an 8-year period, where the authors found that each reviewer's opinion had a significant influence on the approval of a manuscript; <5% of manuscripts were accepted if even one reviewer chose to reject the study (9)
The discordance between reviewers in one-third of peer reviews at JPGN provides an opportunity to assess and improve the peer review process . However, Bruce et al (10) (10)
Limitations of this study include generalizability given that the submissions reviewed were from only one year and specific to one journal. In addition, there is a risk of missing data, as in any retrospective study, though chances of this are minimal as we utilized a well-established administrative database, the Editorial Manager.
In conclusion, peer review is considered a central component of the publication process to ensure the dissemination of high-quality research. At JPGN , we found that peer review typically resulted in moderate agreement between the reviewers. Inter-reviewer agreement correlated strongly with the editor's initial decision and the final disposition of the submission. Future research is necessary to identify ways to enhance the MRP and improve inter-reviewer agreement.
REFERENCES
1. Björk B-C. Acceptance rates of scholarly peer-reviewed journals: a literature survey.
El profesional de la información 2019; 28:e280407
https://doi.org/10.3145/epi.2019.jul.07 2. Lowe NK. Peer review in scientific scholarship.
J Obstet Gynecol Neonatal Nurs 2017; 46:799–800.
3. Manchikanti L, Kaye AD, Boswell MV, et al. Medical journal peer review: process and bias.
Pain Physician 2015; 18:E1–E14.
4. Glonti K, Cauchi D, Cobo E, et al. A scoping review on the roles and tasks of peer reviewers in the
manuscript review process in biomedical journals.
BMC Med 2019; 17:118.
5. Spier R. The history of the peer-review process.
Trends Biotechnol 2002; 20:357–358.
6. Triaridis S, Kyrgidis A. Peer review and journal impact factor: the two pillars of contemporary medical publishing.
Hippokratia 2010; 14: (Suppl 1): 5–12.
7. Ioannidis JPA, Berkwits M, Flanagin A, et al. Ninth international congress on peer review and scientific publication: call for research.
BMJ 2019; 366:l5475.
8. Grant WD, Cone DC. If at first you don’t succeed: the fate of manuscripts rejected by academic emergency medicine.
Acad Emerg Med 2015; 22:1213–1217.
9. Sposato LA, Ovbiagele B, Johnston SC, et al. A peek behind the curtain: peer review and editorial decision making at stroke.
Ann Neurol 2014; 76:151–158.
10. Bruce R, Chauvin A, Trinquart L, et al. Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis.
BMC Med 2016; 14:85.
