Secondary Logo

Journal Logo

The Importance of Being Negative

Leslie, Kate MBBS, MD, M Epi, FANZCA*,†; Short, Timothy G. MBChB, MD, FANZCA

doi: 10.1213/ANE.0b013e318229d693
Editorials: Editorials
Free
SDC

From the *Department of Anaesthesia and Pain Management, Royal Melbourne Hospital, Melbourne, Australia; Department of Pharmacology, University of Melbourne, Melbourne, Australia; and Department of Anaesthesiology and Perioperative Medicine, Auckland City Hospital, Auckland, New Zealand.

Funding: Departmental funds.

The authors declare no conflict of interest.

Reprints will not be available from the authors.

Address correspondence to Kate Leslie, MBBS, MD, M Epi, FANZCA, Department of Anaesthesia and Pain Management, Royal Melbourne Hospital, Melbourne, Australia. Address e-mail to kate.leslie@mh.org.au.

Accepted June 16, 2011

The truth is rarely pure and never simple,” wrote Oscar Wilde in his play, The Importance of Being Earnest (1895). The truth is also rarely revealed in a blinding flash, but builds up over time in an incremental fashion, with each new study contributing to the emerging understanding of “the truth.” Informed readers do not act on the results of a single study; whether it is an observational study or a randomized controlled trial, they interpret that study in the light of the accumulated evidence and evaluate the residual gaps in knowledge before making a decision about changing their practice.1 To assist in this process, it is vital that the scientific community avoid bias in the dissemination of research results.

Dissemination bias occurs when the dissemination profile of a study's results depends on the direction or strength of the findings.2 It occurs in many forms, including publication bias (publication depends on the direction or strength of the findings), outcome reporting bias (reporting of only those outcomes with “significant” findings), time lag bias (speed of publication depends on the direction or strength of direction of the findings), gray literature bias (findings reported in journals systematically differ from those reported in abstracts, theses, or other reports [“gray literature”]), full publication bias (in which the publication of the full results depends on the reception of the abstract), language bias (language of publication depends on the direction or strength of the findings), duplicate publication bias (“significant” findings being more likely to generate multiple publications), place of publication bias (place of publication depends on the direction or strength of the findings), citation bias (the chance of citation depends on the direction or strength of the findings), indexing bias (biased indexing of published studies), and media attention bias (in which “significant” results are more likely to draw media attention).24 The dissemination profile of a study may also depend on whether the results are statistically significant (“positive”) or not (“negative”), clinically important or clinically unimportant, and favored or disliked by the researchers or potential readers.2 Biases are also built into the peer review process that leads to publication.5 Studies published in leading journals are currently graded by reviewers for their novelty, clinical or scientific impact, definitiveness, and perceived interest to the specialty. The net effect is that results supporting a substantial positive treatment effect, especially for a common or difficult-to-treat condition, are more likely to be disseminated, just as a spectacular win in the football World Cup is more likely to be noticed than yet another draw.6,7

This tendency to attach importance to positive trial results has led to a phenomenon known as the “decline effect,” in which readers attach a great deal of importance to a strongly positive result, and change their practice, but then find confirmatory studies to be less impressive.810 This is a particular problem with small trials and trials with low event rates, or randomized trials that follow sensational observational study results. The statistical reality is that when we design studies to achieve a significant difference at a probability of P < 0.05, at one extreme, 1 in 20 of these studies will be negative (P > 0.05) and at the other extreme, 1 in 20 of these studies will be impressively positive (P ≪ 0.05). If a study in the latter category is the first to be published on a subject, subsequent studies are destined to be disappointing.

Journal editors have made a concerted effort in recent years to reduce discrimination against negative studies, although these efforts relate more to randomized trials than observational or post hoc analyses.5 In addition, it is now enshrined in the Declaration of Helsinki that clinical trials must be registered before recruitment is commenced.11 Registries require a statement of the primary outcome variable, a priori calculation of sample size, and consideration that appropriate, meaningful endpoints are chosen, rather than those that will yield positive results (www.clinicaltrials.gov). This decreases the risk that the primary outcome can be manipulated for statistical significance. It is also hoped that registries will reduce inadvertent duplication of studies, although whether this is an advantage could be disputed, given the need to confirm important results. It is of note that registries do not contain an obligation to publish, and indeed publication could never be enforceable. Consequently, registries have become another source of gray literature that must be searched before embarking on a major trial or during a systematic review. There are interesting examples of nonpublication of negative results in the medical literature, for which trial results have been available to those who search for them (e.g., U.S. Food and Drug Administration filings by drug companies).12 These initiatives ensure that a complete view of research is accessible to all those involved in health care decision making. It is expected that they will eventually strengthen the validity and value of the scientific evidence base.

There is also greater appreciation of designing trials to look for equivalence, for which a “positive” result is no difference between groups.13 These equivalence trials are becoming more important, with the effectiveness of many medical treatments and procedures making placebo-controlled trials unethical. They require much larger patient numbers than do placebo-controlled trials because it is unusual for one treatment to be dramatically better or worse than another, particularly when there is no clinical equipoise as to which treatment is best, or event rates are also likely to be low.14

In this issue of Anesthesia & Analgesia, Lindholm et al. report on their research into a possible association between general anesthesia and the subsequent development of cancer, and their result was “negative.”15 The hypothesis was generated as a result of research suggesting that volatile anesthetics impair immune function, which is vital for restraining populations of cancer cells in a quiescent state and preventing the development of clinical and potentially fatal disease.16,17 For this purpose, the authors used an existing cohort of patients collected to determine the incidence of awareness with bispectral index (BIS) monitoring.18 Subsequently, the authors also report on the association between long-term all-cause mortality and duration of BIS values <45 during anesthetic maintenance.19 Malignancy was the cause of death in 71% of these patients. Longer duration of low BIS was associated with higher all-cause mortality when preexisting malignancy status was not included among the covariates (hazard ratio 1.18; 95% confidence interval: 1.08 to 1.29). However, when preexisting malignancy status was included, the result no longer reached statistical significance (1.08; 0.99 to 1.18).

In the present study,15 Lindholm et al. report that neither duration of sevoflurane anesthesia nor increased duration of low BIS during maintenance was associated with an increased risk of new malignant diseases within 5 years of surgery in previously cancer-free patients. Interestingly, 4.3% of patients in the study developed a new cancer within the 5-year observation period. Patients will often associate the occurrence of cancer to significant events in their life, such as a major operation, and so it is easy to see how a simple audit may erroneously conclude that there is an association. This case-controlled study does not support the hypothesis that anesthesia may cause cancer, although the power of the study to draw this conclusion is not addressed by Lindholm et al.

Nevertheless, the publication of this negative result is important for several reasons. First, it is good news that general anesthesia might not cause or promote cancer! It would be deeply ironic and very sad if anesthesia, regarded as one of the most important medical discoveries of all time, proved to be harmful to the long-term health of our patients. Second, this result may encourage further observational studies or secondary analyses attempting to replicate the result. Third, this result may give pause to researchers considering a large randomized trial on the effect of long duration or deep general anesthesia on death from cancer, because it suggests that, if there is a small effect that has been missed by this study, a very large and expensive trial would be required to detect it. And finally, it represents this journal's commitment to publishing well-conducted studies on important topics, including those with a negative result.

So in conclusion, with apologies to Oscar Wilde, we trust you realize (hopefully not for the first time in your life), the vital importance of being negative.

Back to Top | Article Outline

DISCLOSURES

Name: Kate Leslie, MBBS, MD, M Epi, FANZCA.

Contribution: This author helped design the study and write the manuscript.

Attestation: Kate Leslie approved the final manuscript.

Name: Timothy G. Short, MBChB, MD, FANZCA.

Contribution: This author helped design the study and write the manuscript.

Attestation: Timothy G. Short approved the final manuscript.

This manuscript was handled by:Sorin J. Brull, MD.

Back to Top | Article Outline

REFERENCES

1. Angell M, Kassier J. Clinical research—what should the public believe? New Engl J Med 1994;331:189–90
2. Song F, Parekh S, Hooper L, Soke Y, Ryder J, Sutton A, Hing C, Kwok C, Pang C, Harvey I. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 2010;14:1–193
3. Koren G, Klein N. Bias against negative studies in newspaper reports of medical research. JAMA 1991;266:1824–6
4. Lee A, Copas J, Henmi M, Gin T, Chung R. Publication bias affected the estimate of postoperative nausea in an acupoint stimulation systematic review. J Clin Epi 2006;59:980–3
5. Sridharan L, Greenland P. Editorial policies and publication bias: the importance of negative studies. Arch Intern Med 2009;169:1022–3
6. Sterne J, Egger M, Smith G. Systematic reviews in health care: investigating and dealing with publication and other biases in meta-analyses. BMJ 2001;323:101–5
7. Apfel C, Saxena A, Cakmakkaya O, Gaiser R, George E, Radke O. Prevention of postdural puncture headache after accidental dural puncture: a quantitative systematic review. Br J Anaesth 2010;105:255–63
8. Lehrer J. The truth wears off. Is there something wrong with the scientific method? The New Yorker, . , 2010
9. Poldermans D, Boersma E, Bax J. The effect of bisoprolol on perioperative mortality and myocardial infarction in high-risk patients undergoing vascular surgery. N Engl J Med 1999;341:1788–94
10. POISE Study Group. Effects of metoprolol succinate extended-release in patients undergoing noncardiac surgery: a randomized controlled trial. Lancet 2008;371:1839–47
11. World Medical Association. WMA Declaration of Helsinki—ethical principles for medical research involving human subjects. Available at http://www.wma.net/en/30publications/10policies/b3/index.html. Accessed on June 1, 2011
12. Tfelt-Hansen P. Unpublished trials with sumatriptan [correspondence]. Lancet 2009;374:1501–2
13. Snappinn S. Noninferiority trials. Curr Control Trials Cardiovasc Med 2000;1:19–21
14. Rigg J, Jamrozik K, Myles P, Silbert B, Peyton P, Parsons R, Collins K. Epidural anaesthesia and analgesia and outcome of major surgery: a randomised trial. Lancet 2002;359:1276–82
15. Lindholm M, Granath F, Eriksson L, Sandin R. Malignant disease within 5 years after surgery in relation to duration of sevoflurane anesthesia and time with bispectral index under 45. Anesth Analg 2011;113:778–83
16. Gottschalk A, Sharma S, Ford J, Durieux M, Tiouririne M. The role of the perioperative period in recurrence after cancer surgery. Anesth Analg 2010;110:1636–43
17. Snyder G, Greenberg S. Effect of anaesthetic technique and other perioperative factors on cancer recurrence. Br J Anaesth 2010;105:106–15
18. Ekman A, Lindholm M, Lennmarken C, Sandin R. Reduction in the incidence of awareness using BIS monitoring. Acta Anaesthesiol Scand 2004;48:20–6
19. Lindholm M, Traff S, Granath F, Greenwald S, Ekbom A, Lennmarken C, Sandin R. Mortality within 2 years after surgery in relation to low intraoperative bispectral index values and preexisting malignant disease. Anesth Analg 2009;108:508–12
© 2011 International Anesthesia Research Society