ARTICLE IN BRIEF
Investigators reviewed 35 widely cited research reports about disease biomarkers published between 1991 and 2006 in 10 biomedical journals, and found that 86 percent produced results that didn't hold up to subsequent validation efforts. Leading neurology researchers discuss how these findings apply to neurology.
Without reliable biomarkers for disease, researchers cannot ‐easily identify new mechanisms of disease, tests to help diagnose patients, or determine the effectiveness of treatments being tested. But the eagerness of researchers to find reliable biomarkers, combined with the tendency of well‐regarded journals to publish only positive findings, often result in the promulgation of unrealistically strong associations between biomarkers and disease, according to John Ioannidis, MD, DSc, the C.F. Rehnborg Professor in Disease Prevention at Stanford, and chief of the Stanford Prevention Research Center.
In a June 1 paper in the Journal of the American Medical Association, Ioannidis and his co‐author, Orestis A. Panagiotou, MD, of the University of Ioannina School of Medicine in Ioannina, Greece, reviewed 35 widely cited research reports about disease biomarkers published between 1991 and 2006 in 10 biomedical journals. They found that 86 percent produced results that didn't hold up to subsequent validation efforts. Fewer than half retained any statistically significant associations at all, and only two proved to be stronger than the original study claimed.
What makes this even more troubling, according to Dr. Ioannidis, is that misleading results often linger in the literature long after subsequent research finds a weaker link — or no link at all. “There's a high risk that biomarkers without solid documentation may move to borderland of clinical practice and public health,” Dr. Ioannidis told Neurology Today.
This problem poses serious challenges for neurology, where biomarkers are urgently needed to test the effectiveness of new treatments for Alzheimer disease (AD), Parkinson disease (PD), multiple sclerosis (MS), and other neurological conditions that appear to begin years, if not decades, before symptoms appear.
CHALLENGES FOR NEUROLOGY
Consider, for example, recent findings on a biomarker for MS. An April 5 paper in Neurology reported that a neurofilament heavy chain protein found in the CSF of people developing MS correlates with the severity of symptoms. However, the authors also acknowledged that the same protein appears in the CSF of people as they age, potentially compromising the reliability of the protein as a biomarker.
Even one of the most reliable AD biomarkers — the presence of amyloid beta (Abeta) oligomers believed to be toxic to neurons — was challenged in a May 25 paper in the Journal of Neuroscience. By subjecting a new AD mouse model to a panel 12 antibodies, Virginia M‐Y Lee, PhD, and colleagues at the University of Pennsylvania School of Medicine concluded that the antibodies used to identify the presence of Abeta within neurons actually identify not the troublesome fragments of Abeta cleaved from the amyloid precursor protein (APP), but rather a portion of the normal APP that includes the fragments before they are cleaved and become toxic.
This calls into question not only the reliability of Abeta as a biomarker of disease progression, but also the prevailing model of the disease process itself. Not surprisingly, this finding has elicited vigorous challenges from other AD researchers who support their position by citing previous studies — the type of studies that tend to linger in the literature even after attempts at replication fail, according to Dr. Ioannidis.
David M. Holtzman, MD, Andrew B. and Gretchen P. Jones Professor and chair of the department of neurology at the Washington University School of Medicine in Seattle, and neurologist‐in‐chief at Barnes‐Jewish Hospital, has done extensive research into AD biomarkers. He agrees that although large well‐designed studies and clinical trials offer good protection against misleading results, there are many reasons for the inconsistent results, particularly for neurological conditions.
“You have to make sure all the sites are using proper inclusion and exclusion criteria, for example, and you have to make sure the trials are long enough,” he said. “A lot of neurological diseases are chronic diseases, and when a trial is short you don't know if the effects will be persistent. The longer the definitive study is, the more likely it is that the result will be correct.”
Better technology, including robotic equipment, helps reduce procedural variability in large trials, especially those conducted at multiple sites, according to Samia J. Khoury, MD, Jack, Sadie and David Breakstone Professor of Neurology, and co‐director of the Partners MS Center at Harvard Medical School.
“For example, until now people have done their own surface staining of molecules, and that is very operator‐dependent, so a lot depends on who's doing the assay,” she said.
A robotic method contributed significantly to the development of a new antigen microarray that appears able to distinguish relapsing‐remitting, secondary progressive, and primary progressive forms of MS from each other and from AD, lupus, and other neurologic and autoimmune diseases, Dr. Khoury said.
In a 2008 paper in the Proceedings of the National Academy of Sciences, Dr. Khoury and her colleagues found unique autoantibody signatures for the various clinical forms of MS that proved to be more revealing than single antigen‐antibody relationships. One form of MS, for example, displays increased immunoglobulin G antibodies to heat shock protein 60, myelin oligodendrocyte glycoprotein (MOG), and myelin proteolipid protein, while another displays antibodies to lactosylceramide and L‐‐lysophosphatidylserine.
“We're now doing this in large numbers of patients,” she said of the new procedure. “We're trying to replicate what we had previously published in longitudinal studies, focusing on different subgroups of patients. We're looking at responders vs. non‐responders to therapy, so we will try to ask questions about the different subpopulations of MS.”
The search for new PD drugs is hampered by the lack of reliable biomarkers that reveal the disease process in its earliest stages, Todd Sherer, PhD, CEO of the Michael J. Fox Foundation for Parkinson's Research told Neurology Today.
“Given current clinical trial methodology, it is unclear whether an effective disease‐modifying treatment for PD could even be detected,” he said in a recent article in Science Translational Medicine. “The problem does not lie in a shortage of hypotheses or compounds; multiple compounds targeting the many proposed underlying disease mechanisms are already being tested clinically.…Rather, the major problem lies in the lack of objective measurements to assess whether a PD intervention is altering disease progression.”
This provides a major barrier to larger investment by pharmaceutical companies and venture capitalists, according to Dr. Sherer.
In its search for biomarkers, the ‐Parkinson's Progression Markers ‐Initiative (PPMI), a $45 million ‐clinical study, is seeking to insulate itself from some of the dangers Dr. Ioannidis identifies by consolidating research data when possible, said Mark Frasier, PhD, director of research programs for the Michael J. Fox Foundation.
“The Foundation has funded biomarker research since 2002, and what we realized about four years ago was that the work of every investigator we supported had to be verified and validated, which meant recruiting different patients,” he said. “This was really inefficient, so in the last year we started to develop a more comprehensive data set that could be used for biomarkers we hadn't even thought of yet.”
By compiling large amounts of data on each participant, including biospecimens, imaging results, and clinical data about symptoms, new biomarkers can be investigated more easily, according to Dr. Frasier.
“Most of the analyses and meta‐analyses that Dr. Ioannidis looked at were studies of one particular marker in a population,” he said. “We are compiling all sorts of different data on the same population, and it's all publicly available. Anyone can mine it and publish results from it. With this kind of transparency, if there's manipulation of data it should be rapidly exposed through attempts at ‐replication by ‐others. That should ‐insulate us from some of the problems Dr. ‐Ioannidis identifies.”
Ioannidis JPA, Panagiotou OA. Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta‐analyses. JAMA 2011;305(21):2200‐2210.
Kuhle J, Leppert D, Lindberg RLP, et al. Neurofilament heavy chain in CSF correlates with relapses and disability in multiple sclerosis. Neurology 2011;76:1206‐1213.
Winton MJ, Lee EB, Lee VM‐Y, et al. Intraneuronal APP, not free Aβ Peptides in 3xTg‐AD mice: Implications for tau versus A‐mediated Alzheimer neurodegeneration. J Neurosci 2011;31(21):7691‐7699.
Quintana FJ, Farez MF, Weiner HL, et al. Antigen microarrays identify unique serum autoantibody signatures in clinical and pathologic subtypes of multiple sclerosis. Proc Natl Acad Sci USA 2008;105(48):18889‐18894.
Sherer TB. Biomarkers for Parkinson's disease. Science Transl Med 2011;3(79):ps14.
Ioannidis JPA. Why most published research findings are false. PLoS Med 2005;2(8):e124.