Share this article on:

Peer Review: Through the Looking Glass

Glass, Peter S. MB ChB, FFA (SA)

doi: 10.1213/ANE.0000000000000701
Editorials: Editorial

From the Department of Anesthesia, State University of New York, Stony Brook, New York.

Accepted for publication January 29, 2015.

Funding: None.

The author declares no conflicts of interest.

Reprints will not be available from the author.

Address correspondence to Peter S. Glass, MB ChB, FFA (SA), 2708 P St. NW, Washington DC 20007. Address e-mail to glass001@gmail.com.

The peer-review process has recently been under significant attack. Much of the concern is well placed, as evidenced by publication of the hoax manuscript “Fuzzy, Homogeneous Configurations” by Margaret Simpson, Kim Jong Fun, and Edna Krabappel,a the acceptance of multiple manuscripts written by random word generators,b and the acceptance by 157 open-source “peer-reviewed” journals of a bogus paper written by John Bohannon with obvious and egregious flaws, submitted to test the peer-review process.c

The peer-review process started >300 years ago with the introduction of the first 2 scientific journals. In January of 1665, the Journal Des Scavans (Journal of the Learned) was first published in Paris. This was followed in March of the same year by Philosophical Transactions of the Royal Society in London.1 It was quickly recognized that the publication of an article by the Royal Society provided legitimacy to the findings of the paper. It was also recognized that if the data were found to be fraudulent, the reputation of the Royal Society would be tarnished. To ensure that only high-quality papers were published, the Royal Society had papers reviewed by the editor and critiqued by members of the society who were knowledgeable on the paper’s topic. Thus was born the peer-review process. The French Academie had an additional caveat. If a manuscript was reviewed and determined to be logical and reasonable, but the content could not be directly verified, the paper would be published with the note sit penes authorum fides (let the author take responsibility for it).1,2 This later practice has since been dropped, likely reflecting general understanding that only the authors, and the institution, can take final responsibility for their research.

Like every subjective human endeavor, peer review is fallible. Nonetheless, the legitimacy conferred on a paper by acceptance in a peer-reviewed journal challenges every editor. Most of us who have been part of the process (author or reviewer) have experienced these challenges. As a personal example, probably the most innovative and far-reaching manuscript I have published was my study on the increase in the potency of desflurane (i.e., decrease in minimum alveolar concentration) by fentanyl.3 The manuscript was initially rejected by Anesthesiology, based on an assertion by the reviewer and editor that the data were not believable. It was then submitted to and rejected by Anesthesia & Analgesia, and by the British Journal of Anaesthesia. One of the original reviewers from Anesthesiology queried what had happened to the manuscript. Told of the triple rejection, the reviewer appealed on our behalf to Larry Saidman, Editor-in-Chief of Anesthesiology. He permitted us to again submit the paper, and it was ultimately accepted.

The article by Nagappa et al.4 in this issue of the journal has gone through a similarly circuitous route. The manuscript is a systematic review and meta-analysis of the effects of continuous positive airway pressure (CPAP) on postoperative outcomes in patients with obstructive sleep apnea (OSA). In the original review, one of the reviewers pointed out that the number of studies included in the meta-analysis was inadequate to determine the true value of CPAP. Because the value of a meta-analysis is to reconcile the findings of disparate studies having varying outcomes to provide a more accurate answer to the research question, I rejected the manuscript.

The senior author of manuscript appealed to me, and Dr. Shafer, Editor-in-Chief, for reconsideration. Recognizing the fallibility of peer review, we invited a response to the set of reviews. This is the policy of Anesthesia & Analgesia, part of our efforts to address the shortcomings of peer review. Eventually, the authors prevailed. As an aside, Anesthesia & Analgesia has a higher acceptance rate for appealed rejections than for original submissions. This is probably because only authors certain that the rejection was a mistake bother asking for reconsideration.

Making the right decision during the peer-review process is challenging. On many occasions, I have struggled with my own decisions in accepting or rejecting a manuscript. A recent editorial by Larson and Chung2 listed the ethical responsibilities of the reviewer: (1) Remain unbiased toward gender, nationality, and previously published work; (2) Inappropriate promotion of like-minded work; (3) Inappropriate delay or prevention of publication of work with which the reviewer personally disagrees but has no grounds for such action; and (4) Editors should not have the final publication decision for any paper that they coauthor. These may seem fairly straightforward, but they are wrought with issues that are not obvious when one considers the vagaries of bias and the challenges of self-assessment. In recent years, we have placed a lot of emphasis on declaring one’s bias in the academic environment, and several guidelines have been created. But do we really know the extent or direction of a person’s bias simply because he or she has declared obvious elements of this bias?

In an interesting article looking at decision making in the legal system, Jonathan Levav of Columbia Business School in New York and his colleagues analyzed 1112 parole hearings for inmates made by 8 judges during a 10-month period.5 The authors divided the judges’ days into 3 sessions separated by 2 meal breaks: a morning snack and lunch. Judges had control of when to break but had no control over the ordering of cases. At the beginning of each session, a prisoner had a 65% chance of being paroled. This declined to almost zero by the end of a session and leaped back to 65% after a break. The severity of the crime, the time served in prison, any previous incarcerations, and the availability of rehabilitation programs were not enough to explain the effect on the probability of parole. Neither the nationality nor the sex of the prisoner made a difference. The data suggest that lack of food, rest, or both resulted in a bias in the judicial decisions. Applied to the editorial process, these results suggest that very simple things (e.g., mood, lack of rest, glycemic state, etc.) that vary continuously during the day in any given individual might profoundly influence one’s bias toward accepting or rejecting a manuscript. After reading this, it confirmed my concern that I possibly rejected papers because of simple elements such as mood or hunger.

Probably more sinister is the Dunning-Kruger effect.d Not only do we often not know we don’t know, but lacking knowledge we may overestimate what we know. We see this when celebrities rail against vaccines or politicians dismiss climate change as a hoax. “Meta-cognition” refers to “knowing about knowing.”e Part of meta-cognition is recognizing those fields in which one does not have expertise. Not being aware of one’s own relative ignorance or “meta-ignorance” has become known as the Dunning-Kruger effect.6 As stated by Huang “First, attaching the label ‘Dunning-Kruger effect’ to peer reviewers makes a triple claim: reviewers (1) can be ignorant of the subject matter concerned; (2) are not aware of it; and (3) act as if they are experts when in fact they often are not, thereby misleading editorial boards.”

Editors make every effort to have manuscripts reviewed by true experts in the subject matter. Huang goes on to appropriately argue, “the increasingly interdisciplinary nature of research creates an asymmetry of knowledge: the reviewer as a single person faces the daunting combined knowledge of an entire team of coauthors. Thus, statistically, we can safely accept our first claim and assume that on average, reviewers nowadays are with high probability less knowledgeable about the subject matter of a manuscript than its authors.”6

The Dunning-Kruger effect is not the only recognized bias in peer review. It is also recognized that, “the process of peer review can be prone to bias toward ideas that affirm the previous convictions of reviewers and against innovation and radical new ideas.”7 Innovative hypotheses are thus highly vulnerable to being filtered out or made to accord with conventional wisdom by the peer-review process.7 Kumar makes a similar circuitous argument; “How can someone with collateral interest (subject experts) in the research areas of the scientist’s work he/she is reviewing give unbiased opinion on the scientific work, which potentially impacts his/her research as well? If such a person does indeed exist, then it is not peer review. Over time the peer-review system can give rise to a vicious cycle of peer reviewers and authors, who mutually favor each other eventually leading to raise in pollution of scientific literature (Infollution) and skewing the scientific facts (several examples of these are available in the scientific domain).”8 The opposing view that papers should only be reviewed by non-experts also makes no sense. There is increasing support for open review as done by Wikipedia. Heron using a mathematical model demonstrated that post publication review exceeded the accuracy of traditional peer review.9

Much of the scientific literature is compromised by inadequate statistical rigor. This is based, in part, by failure to adjust the bar for statistical significance to the previous likelihood that findings are true. John and Ionnadis at Stanford have studied the accuracy of published studies. They conclude, “Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.”10 In another study on modeling the effects of subjective and objective decision making in scientific peer review, it was demonstrated that this phenomenon of herding was further substantiated.11 Although reviewers and editors are motivated to publish the truth, this study indicated that their behavior and decision are dominated more by information published by their peers, rather than by their personal understanding of the subject.11

Most of us try to follow the rules listed by Larson and Chung, but following these rules is not so easy.2 There is a problem with accepting that which should not be published and rejecting that which should be published. A recent evaluation of the decision-making process at 3 elite medical journals (Annals of Internal Medicine, British Medical Journal, and The Lancet) reported that of the 808 eventually published articles submitted to the journals, the 14 most highly cited papers were rejected by at least 1of the 3 journals and published elsewhere.12 In a review of articles accepted or rejected in the journal Stroke, the authors found that a single negative review reduced the likelihood of acceptance to <5%.13 Thus negative reviews have a very powerful impact on acceptance of an article.

For manuscripts accepted that should not be, tincture of time will ultimately determine their value. Most poor quality articles are recognized by the readers as simply not worthwhile and are ignored. However, some papers should not have been accepted because they are fraudulent or egregiously flawed. Anesthesia & Analgesia has plenty of experience with these. In the past decade, the journal has retracted more papers than any other medical journal. We have also made peer review far more rigorous to prevent such papers from appearing in the journal in the future.

However, with this increase in the rigor of peer review, how can we avoid rejecting good manuscripts? The article by Nagappa et al. is an example of how Anesthesia & Analgesia tries to deal with the potential that the reviewers and editor may have made a mistake. First, the journal encourages reviewers to sign their reviews. Although this is encouraged for all reviewers, as of January 2015 all editors of Anesthesia & Analgesia are required to sign their reviews. In my own experience, I have always felt that this has created an important self-check on the opinions I express in the review. The journal also follows a policy that any rejected manuscript can be appealed. Anesthesia & Analgesia’s Editorial Board takes very seriously every appeal of a rejected manuscript. In fact for the manuscript by Nagappa et al. numerous emails were exchanged between the authors and the editors. In addition, a phone call between the senior author, the section editor, and editor-in-chief helped us reach the best decision.

In trying to determine the value and rigor of the manuscript, the critical question was whether the information in the article would ultimately enhance patient care. Patients with OSA make up a significant portion of our patients. In addition, OSA has a significant impact on the anesthetic management of patients and potential impact on their perioperative outcomes. Thus, work relating to the management of such patients is important for the anesthesia community. Trying to answer the question whether perioperative CPAP provides enhanced outcomes presents a conundrum. As explained in the manuscript, the presently published data are not clear as to the value of CPAP on perioperative outcomes. However, randomizing patients to receive or not receive CPAP may be frowned upon both by IRBs and patients themselves. Thus, the challenge was to create a manuscript that had sufficient rigor, yet was able to present the limited data to the anesthesia community so they could use this information to optimize the care of their OSA patients. Ultimately, time will determine the value of this manuscript. In the interim, the peer-review process worked. We listened to the authors. We listened to the reviewers. We made a decision that placed at the top of the decision process “what is in the best interest of our patients?”

Returning to meta-cognition, it is important to think about thinking, cogitate about cognition. We (editors and reviewers) need to be aware of the vagaries of our biases, our tendency to accept the familiar and reject the unfamiliar, and our fear of being defrauded by malicious authors. Accepting the limitations of peer review, being willing to revisit decisions, being willing to admit mistakes, and placing the interests of improving patient care above everything else helps us overcome some of the inevitable shortcomings of peer review.

Back to Top | Article Outline

DISCLOSURES

Name: Peter S. Glass, MB ChB, FFA (SA).

Contribution: This author wrote the manuscript.

Attestation: Peter S. Glass is the sole author and attests to the contents of this editorial.

This manuscript was handled by: Steven L. Shafer, MD.

Back to Top | Article Outline

FOOTNOTES

a Available at: http://www.vox.com/2014/12/7/7339587/simpsons-science-paper. Accessed January 4, 2015.
Cited Here...

b Available at: http://pdos.csail.mit.edu/scigen/. Accessed January 4, 2015.
Cited Here...

c Available at: https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%3F. Accessed January 4, 2015.
Cited Here...

d Available at: https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect. Accessed January 4, 2015.
Cited Here...

e Available at: https://en.wikipedia.org/wiki/Metacognition. Accessed January 4, 2015.
Cited Here...

Back to Top | Article Outline

REFERENCES

1. Siegelman SS. The genesis of modern science: contributions of scientific societies and scientific journals. Radiology. 1998;208:9–16
2. Larson BP, Chung KC. A systematic review of peer review for scientific manuscripts. Hand (N Y). 2012;7:37–44
3. Sebel PS, Glass PS, Fletcher JE, Murphy MR, Gallagher C, Quill T. Reduction of the MAC of desflurane with fentanyl. Anesthesiology. 1992;76:52–9
4. Nagappa M, Mokhlesi B, Wong J, Wong DT, Kaw R, Chung F. The effects of continuous positive airway pressure on postoperative outcomes in obstructive sleep apnea patients undergoing surgery: a systematic review and meta-analysis. Anesth Analg. 2015;120:1013–23
5. Danziger S, Levav J, Avnaim-Pesso L. Extraneous factors in judicial decisions. Proc Natl Acad Sci U S A. 2011;108:6889–92
6. Huang S. When peers are not peers and don’t know it: the Dunning-Kruger effect and self-fulfilling prophecy in peer-review. Bioessays. 2013;35:414–6
7. Steinhauser G, Adlassnig W, Risch JA, Anderlini S, Arguriou P, Armendariz AZ, Bains W, Baker C, Barnes M, Barnett J, Baumgartner M, Baumgartner T, Bendall CA, Bender YS, Bichler M, Biermann T, Bini R, Blanco E, Bleau J, Brink A, Brown D, Burghuber C, Calne R, Carter B, Castaño C, Celec P, Celis ME, Clarke N, Cockrell D, Collins D, Coogan B, Craig J, Crilly C, Crowe D, Csoka AB, Darwich C, Del Kebos T, Derinaldi M, Dlamini B, Drewa T, Dwyer M, Eder F, de Palma RE, Esmay D, Rött CE, Exley C, Falkov R, Farber CI, Fearn W, Felsmann S, Flensmark J, Fletcher AK, Foster M, Fountoulakis KN, Fouratt J, Blanca JG, Sotelo MG, Gittler F, Gittler G, Gomez J, Gomez JF, Polar MG, Gonzalez J, Gösselsberger C, Habermacher L, Hajek M, Hakala F, Haliburton MS, Hankins JR, Hart J, Hasslberger S, Hennessey D, Herrmann A, Hersee M, Howard C, Humphries S, Isharc L, Ivanovski P, Jenuth S, Jerndal J, Johnson C, Keleta Y, Kenny A, Kidd B, Kohle F, Kolahi J, Koller-Peroutka M, Kostova L, Kumar A, Kurosawa A, Lance T, Lechermann M, Lendl B, Leuchters M, Lewis E, Lieb E, Lloyd G, Losek A, Lu Y, Maestracci S, Mangan D, Mares AW, Barnett JM, McClain V, McNair JS, Michael T, Miller L, Monzani P, Moran B, Morris M, Mößmer G, Mountain J, Phuthe OM, Muñoz M, Nakken S, Wambui AN, Neunteufl B, Nikolić D, Oberoi DV, Obmode G, Ogar L, Ohara J, Rybine NO, Owen B, Owen KW, Parikh R, Pearce NJ, Pemmer B, Piper C, Prince I, Reid T, Rindermann H, Risch S, Robbins J, Roberts S, Romero A, Rothe MT, Ruiz S, Sacher J, Sackl W, Salletmaier M, Sanand J, Sauerzopf C, Schwarzgruber T, Scott D, Seegers L, Seppi D, Shields K, Siller-Matula J, Singh B, Sithole S, Six F, Skoyles JR, Slofstra J, Sole DA, Sommer WF, Sonko M, Starr-Casanova CJ, Steakley ME, Steinhauser W, Steinhoff K, Sterba JH, Steppan M, Stindl R, Stokely J, Stokely K, St-Pierre G, Stratford J, Streli C, Stryg C, Sullivan M, Summhammer J, Tadesse A, Tavares D, Thompson L, Tomlinson A, Tozer J, Trevisanato SI, Trimmel M, Turner N, Vahur P, van der Byl J, van der Maas T, Varela L, Vega CA, Vermaak S, Villasenor A, Vogel M, von Wintzigerode G, Wagner C, Weinberger M, Weinberger P, Wilson N, Wolfe JF, Woodley MA, Young I, Zuraw G, Zwiren N. Peer review versus editorial review and their role in innovative science. Theor Med Bioeth. 2012;33:359–76
8. Kumar AH. Anatomy and physiology of peer review process, can it be unbiased? J Nat Sci Biol Med. 2014;5:1–2
9. Herron DM. Is expert peer review obsolete? A model suggests that post-publication reader review may exceed the accuracy of traditional peer review. Surg Endosc. 2012;26:2275–80
10. Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2:e124
11. Park IU, Peacey MW, Munafò MR. Modelling the effects of subjective and objective decision making in scientific peer review. Nature. 2014;506:93–6
12. Siler K, Lee K, Bero L. Measuring the effectiveness of scientific gatekeeping. Proc Natl Acad Sci U S A. 2015;112:360–5
13. Sposato LA, Ovbiagele B, Johnston SC, Fisher M, Saposnik GStroke Outcome Research Working Group (www.sorcan.ca). . A peek behind the curtain: peer review and editorial decision making at Stroke. Ann Neurol. 2014;76:151–8
© 2015 International Anesthesia Research Society