Secondary Logo

Journal Logo

Editorials: Editorial

Publishing Methods Without Results: A First That Hopefully Will Not Last

Crosby, Gregory MD

Author Information
doi: 10.1213/ANE.0000000000002905
  • Free

This issue marks a first for Anesthesia & Analgesia (A&A), publication of a stand-alone methods paper by Mincer et al1 for an ongoing clinical research study. Do not look for data though. There are none here or elsewhere by these authors using these methods. This is just a paper to describe study methods, and for that reason, I hope this first for A&A is also the last.

The issue here is not the quality of the study. It is a terrific study! The Trajectory of Recovery in the Elderly (TORIE) study may yield significant insights into the clinical trajectory and underlying neurobiology of cognitive recovery from general anesthesia. The approach, which involves cognitive testing before and after general anesthesia in the absence of surgery along with imaging for functional connectivity, white matter integrity, and cerebral blood flow, is intriguing and novel. The study has a target enrollment of 76 subjects (19 per decade between 40 and 80 years of age) and was appropriately registered in ClinicalTrials.gov in 2014. It was reviewed and funded for 5 years by the National Institute on Aging in September 2014, and it was in its fourth year and recruiting.

The issue is also not the merit of making clinical trial protocols publically available. Acknowledging the reproducibility crisis in research,2,3 where among other things authors sometimes misreport prespecified outcomes or selectively report outcomes different from those prespecified, many major journals now require authors of manuscripts reporting clinical trial results to simultaneously submit an original preenrollment protocol with any amendments that were made, and others strongly encourage authors of accepted studies to post the protocol or provide a web link to it.

The problem is that the paper is essentially an infomercial for the study. Unencumbered by data, it tells a lovely story. It is a story with strong protagonists (authors), compelling hypotheses, elaborate cognitive testing, fancy imaging, mind-numbing statistics, and clinically worthy implications. But, we do not know whether this particular story is fact or fiction because there are no results. We do not know whether the methodological details and assumptions about recruitment and retention, cognitive testing and neuroimaging protocols, and power analyses and missing data will be met, or whether results will be robust enough to be interpretable. Because we cannot judge whether the study protocol is adequate, the reader is left hanging. How does it end and what have we learned? Neither the reader nor, presumably, the TORIE investigators have any idea at this point. That being the case, the TORIE protocol and methods should not have been published before actual results. In doing so, the Journal is expecting the reader to judge this book by its cover.

There are also practical reasons not to publish methods papers before the reporting of study results/outcomes. One is that it probably will not improve the quality of the study. That is the case here. The TORIE study is in its 2nd to last year, so the study protocol cannot be changed based on feedback from peer reviewers or journal readership at large. Another reason is low interest; a stand-alone methods paper will likely appeal to only a handful of investigators with a career interest in the topic and the training and ability to conduct similar research. Finally, it sets a precedent. Should A&A publish independent methods papers for small, in- progress studies too? And what about preclinical studies? There is a reproducibility crisis in that area of research too,4 and the methods are typically more complex, arcane, and unfamiliar to clinicians. Should we publish methods papers for those studies also?

Methods and results are interdependent in scientific work. They should be treated as such in the published literature. One cannot evaluate methods without knowing what they produce in the hands of specific investigators any more than one can interpret results without knowing how they were generated. Can anyone imagine seeing a study dataset published in a reputable journal without information about the methods? How is it then that publishing methods independent of results is acceptable? If the goal is to ensure transparency and reproducibility in the published literature, there are other options. Most reputable journals, including A&A, already require that clinical trials be registered in ClinicalTrials.gov, and this one was. A journal could also require that a newly submitted original manuscript include the preenrollment protocol and amendment documents to assure concordance between what was planned and what is reported. This approach would eliminate the risk, and potential embarrassment, of publishing methods for a study that flops and never yields publishable results. What would a journal do then? Withdraw the methods paper or leave it, as a testament to bad editorial judgment?

One cannot fault Mincer et al1 for trying to publish the protocol for the TORIE study. It is an important, potentially groundbreaking study and some journals publish methods papers, or even concentrate on them. But, A&A should not be one of them. Protocols and methods are an undeniably important and indispensible aspect of research, but they are just a part of the work. They reflect ideas—good ideas in the case of Mincer et al1—but nonetheless, just ideas. Ideas are cheap; execution is hard. Publication should be a reward for successful execution of good ideas but methods that stand alone have no place in a journal dedicated to the highest scientific standards.

ACKNOWLEDGMENTS

The author is grateful to Drs Michael Todd and Steven Shafer for sharing their wisdom and reviewing drafts of this editorial.

DISCLOSURES

Name: Gregory Crosby, MD.

Contribution: This author wrote, edited, and prepared this manuscript for publication.

Conflicts of Interest: G. Crosby receives funding from the National Institutes of Health. He is an executive section editor for Anesthesia & Analgesia.

This manuscript was handled by: Thomas R. Vetter, MD, MPH.

Acting EIC on final acceptance: Thomas R. Vetter, MD, MPH.

REFERENCES

1. Mincer JS, Baxter MG, McCormick PJ, et al. Delineating the trajectory of cognitive recovery from general anesthesia in older adults: design and rationale of the TORIE (Trajectory of Recovery in the Elderly) project. Anesth Analg. 2018;126:1675–1683.
2. Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2:e124.
3. Vetter TR, McGwin G Jr, Pittet JF. Replicability, reproducibility, and fragility of research findings–ultimately, caveat emptor. Anesth Analg. 2016;123:244–248.
4. Ioannidis JP. Acknowledging and overcoming nonreproducibility in basic and preclinical research. JAMA. 2017;317:1019–1020.
Copyright © 2018 International Anesthesia Research Society