Meta-analyses and systematic reviews are considered the highest quality evidence available in medicine. Yet the methods are certainly not impervious to various biases. Therefore several new efforts are underway to improve their quality, or at least to call attention to potential problems.
Last month the Centre for Reviews and Dissemination at the University of York announced the launch of PROSPERO, a database for systematic reviews and meta-analyses.
The database, which provides a permanent record for review protocols, is designed to reduce publication bias. If review protocols are entered in the database prospectively, then other researchers can compare the original goals of the project with what eventually gets published.
Of course, registration of review protocols in the database is entirely voluntary at this point. If, however, the International Committee of Medical Journal Editors were to decide that reviews had to be registered prior to publication, as they’ve done for clinical trials, study authors would have more incentive to participate.
So far, that doesn’t seem likely to happen, at least not in the short run. The editors of PLoS Medicine wrote an editorial supporting the effort. PLoS journal editors, they wrote, will start asking authors of meta-analyses and systematic reviews if they registered their study and will include the registry number in publications.
However, the PLoS move seems to be unique amongst journal editors currently. (I’ve asked several leading journals if they have similar efforts in the works. So far the only one I’ve heard from, New England Journal of Medicine, does not have plans to require it.)
Meanwhile, a study published last week in the Journal of the American Medical Association raises issues of transparency and conflict of interest in meta-analyses.
The authors, led by Brett Thombs, PhD, of McGill University in Montreal, found that only 2 out of 29 meta-analyses reviewed included funding sources for the primary randomized controlled trials reviewed in the meta-analysis. And none of the 29 meta-analyses included information about the primary study authors’ industry ties or employment.
Yet in seven meta-analyses, all of the primary randomized clinical trials reported at least one industry-related conflict of interest.
Given that there has been clear demonstration that industry-funded trials are much more likely to be published if they have positive results than negative ones, Dr. Thombs and colleagues argue that ignoring funding sources of trials included in a meta-analysis could lead to bias in the literature.
The authors, though, are clear to say that they did not actually assess bias in the current study, just the presence or absence of conflict of interest information.
I was caught off guard when I read Dr. Thombs’s paper. And not because the conflict of interest information was missing. Rather I was caught off guard because I had simply not considered this variable at all – even though I know there is lots of evidence for publication bias, particularly in industry-funded trials.
And apparently I’m not the only one. When I asked Don Berry, PhD, Head of the Division of Quantitative Sciences at The University of Texas MD Anderson Cancer Center, whether he thought the absence of conflict of interest from the primary studies was likely to have a large impact on bias, he paused and said, “I’ve never thought about it.”
After weighing the issue for a few minutes, Dr. Berry said he thought other sources of bias in meta-analyses probably outweighed the lack of conflicts of interest data from the primary studies.
After all, he noted, the conflict of interest information was likely reported in the original trial papers. “So the information is out there,” he said.
More substantial sources of bias are in the design and reporting of the meta-analyses and systematic reviews, according to Dr. Berry.
He notes that while he doesn’t know the magnitude of benefit (or hazard ratio) for a given therapy before he performs a meta-analysis, he has a good idea of the approximate measure before he even starts, because he’s familiar with the relevant literature.
Therefore, how a researcher designs a meta-analysis and the exact question they choose to ask probably has a larger impact than whether or not they report the conflicts of interest from the trials themselves.
Dr. Thombs and colleagues meanwhile have asked for the primary conflict of interest data to be included in the preferred reporting guidelines for systematic reviews and meta-analyses.
By the way, those guidelines, called the PRISMA statement, do ask for inclusion of funding sources for the meta-analysis or systematic review itself.
As for me, I think I will start looking at meta-analyses with a bit more scrutiny – including taking note of who funded the original studies.