Journal Logo

SURGICAL PERSPECTIVES

Reviewer Blinding in Peer Review

Perspectives From Reviewers at Three Stages of Their Careers

Gupta, Vaibhav MD, PhD∗,†; Coburn, Natalie G. MD, MPH∗,†,‡; Detsky, Allan S. MD, PhD†,§,¶

Author Information
doi: 10.1097/SLA.0000000000003959

Peer review is widely accepted as the criterion standard in medicine and other academic disciplines for assessing and evaluating the quality of scientific work submitted for publication and dissemination to the broader community. The best way of conducting peer review for articles submitted to academic journals, however, continues to be debated. Three common forms of peer review are single-blind, double-blind, and open peer review. Single-blind review, the traditional method of peer review, keeps the reviewer anonymous while allowing the reviewer to know who wrote the article. Double-blind review means both authors and reviewers are kept anonymized. Proponents of double-blind review believe it results in more objective assessments of research from less well known authors and institutions. Open peer review is where authors and reviewers know each other's identities during the review process, and sometimes the peer-review reports are published alongside accepted articles.

Despite research evaluating different methods of blinding,1–4 practices still vary across journals. We surveyed the peer-review policies at 54 leading clinical medical journals in the United States across 26 specialties to understand current practices (Table 1; full data in Supplemental Digital Content. This file describes how journals were sampled and surveyed for their peer review practices. It contains a table showing the peer review practice of each of the 54 journals surveyed; http://links.lww.com/SLA/C148).

TABLE 1
TABLE 1:
Peer Review Practices at 54 Leading Clinical Medical Journals in the United States

Majority of journals use single-blind peer review, and allow reviewers to reveal their identity by signing the review, if they so choose. Among the journals we surveyed, 7 use double-blind peer review (2 in emergency medicine, 2 in radiology, 2 in orthopedics, and 1 in pathology). The American Journal of Gastroenterology (like all Nature Research journals) offers an optional double-blind peer-review process. Authors can choose to deidentify their manuscript and remain anonymous to reviewers. Only the British Medical Journal and BMC Medicine practice open peer review.

Preferences for the form of peer review may vary for reviewers who are trainees, mid-career, and late-career academics in medicine. We represent each of these groups (V.G. is a surgery resident and just completed his PhD; N.G.C. was recently promoted to professor of surgery; A.S.D. has been professor of medicine for 30 years) and present our views as reviewers.

TRAINEE'S PERSPECTIVE (V.G.)

As a surgery resident pursuing a PhD in clinical epidemiology, I was invited to be a statistical reviewer for a leading cardiothoracic surgery journal because of my increasing expertise in research methodology, critical appraisal, and statistics. Cardiothoracic surgery is a small academic community, and over the course of my first 25 reviews for the journal, I have submitted recommendations for articles written by well-known and highly influential surgeons in the field. These individuals may be involved in selecting me for cardiothoracic surgery training or a faculty position in the future. This power differential would create a conflict of interest if my identity were revealed in peer review, and might result in a more favorable review than is warranted for 2 reasons. First, I would be concerned that a critical review could risk my relationship with superiors. Second, I could be inclined to provide a favorable review to develop or enhance my relationship with them.

Blinding reviewers takes the relationship out of a process designed to enhance the quality of published science. I have found methodological flaws or incorrect inferences in papers from prominent faculty and institutions. In these situations, blinding has eased my worry about repercussions from a critical review, and let me focus on an honest and objective assessment of the work. I remain accountable to the editor, associate editor, and associate statistical editor for my review, and the editors remain accountable to authors for a fair review.

Proponents of open peer review create an ethical argument for authors to know who is providing “judgment” on their work. Reviewers, however, provide advice, and judgment is ultimately from the editors. Editors hold responsibility for accurate, timely, and unbiased review, and can address concerns about the process if any arise.

MID-CAREER PERSPECTIVE (N.G.C.)

Double-blinded clinical trials are the criterion standard, due to the ability to reduce bias. Double blinding—of authors and reviewers—has the same power to reduce bias in journal submissions. We all hold biases. Cognitive heuristics, including unconscious associations, are necessary in pattern recognition and clinical decision making. While useful in provision of care, effort reduction, and problem-solving, these associations (or biases) can also lead to unintended consequences. For some of us, bias is well-managed, and when we make, or hear, a biased statement, we recognize it and correct it. There may, however, be various levels of bias that creep into reviews: the country in which the research was conducted, institution, type of funding, and race, sex, and seniority of authors. Studies have shown sex and racial bias in judging students, candidates, manuscripts, grants, and speaker selection.5,6 Furthermore, when reviewers are blinded to the authors, work from women and minorities is ranked more highly.7,8 Double blinding may be an important intervention to level the playing field for women and minorities for publication9—a step in fixing the leaky pipeline in academics and leadership.

In addition to sex and ethnicity bias, research shows that manuscripts from better known authors and universities are more likely to be accepted in a single-blind review, compared to a double-blind review.2,10 Furthermore, as a junior or mid-level faculty, how do we manage the conflict of writing a signed poor review for someone who is able to determine committee placements and speaking assignments? We should be judging the work on its scientific merits, not based upon the track record of its authors. Finally, and importantly, in a randomized, double-blinded study of blinding, the quality of the review was significantly better for the blinded reviews.1

In medicine, double-blinded studies are rare, as they are difficult to conduct. Although it may be difficult to operationalize, to minimize bias, blinding of the reviewer and the author provides the least biased review.

LATE-CAREER PERSPECTIVE (A.S.D.)

When I began my career as a faculty member, David Sackett told me that he always signed his reviews. It seemed like a novel idea, one that would make me stand out from the crowd; so I adopted that practice myself. Over the years, I have recognized 2 advantages from revealing my identity. It allows the authors of the article to evaluate my expertise in the subject of their article. I review many articles; some are right in my area of research, others are not. As an author myself, I sometimes struggle with how to revise articles I have written after receiving reviews. Suggestions from a person reviewing the paper who is truly an expert in the field carry more weight with me, so I assume knowledge of my areas of expertise are also helpful to the authors whose papers I review. Second, I believe that revealing my identity makes me more accountable and polite. Every author has the experience of getting a review from someone who has clearly not read the article carefully; displaying obvious errors in their review (eg, suggesting a method of analysis I clearly used, suggesting a reference that is already included.) I do recognize, however, that when I write a negative review and reveal my identity, that the author may react negatively, using a now famous phrase, “what goes around comes around.” I guess I have lived with the consequences of my practice, but see that people at the beginning of their careers might be more risk averse. So I am not sure I would recommend this practice to everyone.

COMMENT

Career stage may influence reviewers’ preference for blinding in peer review. We believe that single-blind (reviewer-blind) peer review, with an option for reviewers to sign their reports if they choose, is the most favorable policy. We recognize that reviewers may choose to leave their identity hidden when their review is negative and only reveal it when they are supportive. Most editors, however, prefer that reviewers offer comments and suggestions while leaving out recommendations about acceptance or revision, so this may be less of a problem than it seems.

Open review will not work for everyone and will especially be hard for junior people; it brings the potential for everyone to become less critical, negating the value of objective, unbiased approaches to evaluate the validity of research. Double blind likely has the least chance of bias related to who the authors and reviewers are. The downside is that some reviewers feel, rightly or wrongly, that knowledge of the authors tells them something about the potential for ethical concerns. Identifying fraud, plagiarism, duplicate submission, and publications, inappropriate authorship, clinical trial registration, or other such concerns, requires content and sociologic knowledge of the field that reviewers have. We cannot rely on editors to know all of this; thus, blinding of authors would leave out valuable information. Double-blind review may produce the least-biased assessments, but is associated with these trade-offs. Single-blind reviews can be honest, provide valuable field-specific knowledge, and protect reviewers from influential colleagues. As we, however, self-regulate our clinical practice, we must collectively identify and self-manage our unconscious bias to offer fair reviews.

REFERENCES

1. McNutt RA, Evans AT, Fletcher RH, et al. The effects of blinding on the quality of peer review. A randomized trial. JAMA 1990; 263:1371–1376.
2. Okike K, Hug KT, Kocher MS, et al. Single-blind vs double-blind peer review in the setting of author prestige. JAMA 2016; 316:1315–1316.
3. Van Rooyen S, Godlee F, Evans S, et al. Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA 1998; 280:234–237.
4. Jefferson T, Alderson P, Wager E, et al. Effects of editorial peer review: a systematic review. JAMA 2002; 287:2784–2786.
5. Women in science: women's work. Nature 2013; 495:21.
6. Knobloch-Westerwick S, Glynn CJ, Huge M. The matilda effect in science communication: an experiment on gender bias in publication quality perceptions and collaboration interest. Sci Commun 2013; 35:603–625.
7. Roberts SG, Verhoef T. Double-blind reviewing at EvoLang 11 reveals gender bias. J Lang Evol 2016; 1:163–167.
8. Snodgrass R. Single- versus double-blind reviewing: an analysis of the literature. SIGMOD Record 2006; 35:8–21.
9. Budden AE, Tregenza T, Aarssen LW, et al. Double-blind review favours increased representation of female authors. Trends Ecol Evol 2008; 23:4–6.
10. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. Proc Natl Acad Sci U S A 2017; 114:12708–12713.
Keywords:

academic publishing; peer review; reviewer blinding

Supplemental Digital Content

Copyright © 2020 Wolters Kluwer Health, Inc. All rights reserved.