Secondary Logo

Journal Logo

Medically Clear: Discovering the Truth in Science

Ballard, Dustin MD

doi: 10.1097/01.EEM.0000499534.09315.d6
Medically Clear

Dr. Ballardis an emergency physician at San Rafael Kaiser, co-chair of the KP CREST (Clinical Research on Emergency Services and Treatments) Network, and the medical director for Marin County Emergency Medical Services. He is also the creator of the Medically Clear podcast on iTunes or athttps://medium.com/medically-clear. Follow him on Twitter at @dballard30. Read his past columns athttp://bit.ly/BallardEMN.

Figure

Figure

Figure

Figure

Perhaps you read in the New York Times about the slew of promising anti-aging drugs in development. Metformin is one example.

The antidiabetic has been used since the 1920s to help control blood sugar levels, but it may actually do much more, like prolong life in diabetics and non-diabetics alike. Purportedly, metformin modulates metabolism and cellular damage through its effects on gluconeogenesis in the liver. Physiologically plausible? Yes. Should we write ourselves a prescription and start supplementing life expectancy tomorrow? Not so fast.

As we know all too well, it's an unfortunate reality that many biomedical research findings are ultimately proven to be illusory. This goes for drugs, dietary associations, and diagnostics. Remember when Vioxx was considered a safe treatment for arthritis, and oat bran was thought to prevent heart disease?

How frequently do facts become fiction? Sadly, it's more often than we'd like. As a matter of fact, findings in medical research may be only slightly more reliable than Donald Trump's facts and only about as good as sport pundit expert picks against the spread. Feeling doubtful, Dr. Thomas? Here's a look at the evidence behind medical evidence.

First, let's be clear on what we mean by a research finding: It is finding is any relationship discovered in a study that meets the criteria for statistical significance. The typical standard for this is a p value of less than 0.05, which essentially means only a one in 20 chance that the research finding was due to chance rather than a real association. The concept of 95 percent confidence in the truth of a finding may seem intuitively appealing, but evidence suggests it may not be conservative enough.

Consider a 2005 paper by John P. A. Ioannidis, MD, “Why Most Published Research Findings are False.” (PLoS Med 2005;2[8]:e124.) Dr. Ioannidis builds a theoretic model of the chances of obtaining a true research finding starting with sample size and p-value. Then, he mixes in additional terms to account for possible confounders that could cause a finding to be falsely true. These include investigator bias (recognized or not), the pre-test probability, and the number of different possible associations being tested. From his model, Dr. Ioannidis concludes that the chances of a finding being true range from about 85 percent to lower than 20 percent. The highest chance was if the study was a well-controlled, randomized trial of an association with reasonably high prestudy odds of being true, and the lowest chance was if the study was retrospective and examined rather broad associations such as diet and disease (think processed meat and cancer risk) or thousands of associations at once (think genetic-discovery research where 30,000 genes may be tested to find 30 or so true culprits).

Subsequent theoretic analyses have lent support to Ioannidis' notion and generalize that no better than 50 percent of research findings are actually correct. Indeed, in our recent history, a litany of research found could not be replicated or were outright refuted (e.g., routine lidocaine for acute MI, The Lancet's vaccine-autism study, etc.). The topic of how to distinguish the truth from the false-positive has picked up considerable steam.

If this has you feeling skeptical about the value of biomedical research, you are not alone. Fortunately, this is not a presidential debate where all conclusions should be considered false until impartially verified. Certain principles can help researchers, physicians, and the public arrive at a better state of biomedical knowledge. Some of these principles may be rather obvious, but it is worth enumerating them nonetheless, because even those well-versed in the literature may need a reminder when addressing the excitement and publicity that often accompanies an exciting new finding.

  • Bigger is much better. The work by Ioannidis and others has demonstrated that small numbers of observations or study subjects are more likely to produce spurious and nonreplicable results. Place more credence in studies that have thousands or tens of thousands of subjects rather than those with just a handful.
  • Be wary of bias. Any studies published by entities with possible conflicts of interest or on a topic of newsy interest have a higher chance of false findings. This is not to say that every drug company study is rigged, but caution is always advised in interpretation.
  • Face validity. The probability of a particular truth in medicine before a study is done greatly affects the likelihood of a finding being actually true. Just because some people who eat a lot of ice cream are thin does not mean that ice cream is an effective weight loss tool.
  • Ask if the findings were randomized and replicated. As we know, a series of studies in 2015 were all randomized trials that demonstrated the benefit of a new clot retrieval technique for certain types strokes. These studies all employed the best type of study design (randomized controlled trial), and had strongly positive results that were repeated in other separate investigations. We can believe these results.
  • Five sigma. Different scientific disciplines use different thresholds for defining a research finding. The threshold is two sigma in medicine, which is two standard deviations or 95 percent confidence. It is five sigma in physics, however, which equates to 99.99 percent confidence. If you see a finding in medicine that meets this five sigma criteria, you better believe that Sir Isaac Newton would approve.

Before you start popping metformin or whatever trendy supplement or super food is all the rage, it is worth considering the spotty record of science's ability to discover truth. Fortunately, we can take heart in the fact that science remains the most vigorous supervisor of its own truths: False findings are discovered and discarded, and the total body of evidence moves forward. It may be messy, but ultimately, science gets it right a majority of the time.

Copyright © 2016 Wolters Kluwer Health, Inc. All rights reserved.