There could be a potentially harmful number of abstracts of randomized clinical cancer trials that have been tailored to emphasize more positive findings than are supported by data in the studies in full—especially incomplete reporting of primary outcome results, a team of researchers has found.
They conducted their own study—available online ahead of print in the Journal of Clinical Oncology (doi: 10.1200/JCO.2014.56.7503)—which randomly assigned abstracts with or without previously acknowledged “spin” to two teams of cancer clinicians with more than 15 years of clinical experience as well as experience in randomized clinical trial (RCT) reporting.
Spin was defined as misrepresentation of study findings that can be used to positively influence the interpretation of statistically nonsignificant RCTs—for example, by emphasizing the apparent benefit of a secondary outcome or findings from a subgroup of patients.
The study, called “SPIIN,” which follows earlier research by the same group, was supported by a grant from the French Ministry of Health, the National Institute for Cancer Research, and the Fondation Pour la Recherche Médicale. First author is Isabelle Boutron, MD, PhD, at Centre d'Epidémiologie Clinique at Hôpital Hôtel Dieu de Paris, and other researchers in France, England, Canada, and the United States also participated in the study.
The participants were asked to identify abstracts that did not meet acceptable standards based on guidelines developed in 2010 by the Consolidated Standards of Reporting Trials (CONSORT) group.
The clinicians were assigned a sample of 30 abstracts of articles describing cancer RCTs with differences that were not statistically significant in their primary outcome, but with spin in their abstract conclusion. A total of 150 were assigned to abstracts without spin, and 150 reviewed those written with spin.
Using five existing collections of abstracts with spin, the French team then rewrote each abstract in two formats: one with spin and the other without, and the clinicians were randomly assigned to evaluate one abstract without knowing if it had spin or not. One abstract was read by several clinicians, with a clustering between readers of the same abstract.
As an inducement to participate, each clinician was told that by evaluating an abstract they would be eligible for a drawing to win an iPad.
Even though the two teams did not have access to the full text of the studies, the clinicians reviewing abstracts with spin flagged several areas where they fell short of what abstracts should include. In addition to failing to clearly state primary outcomes or providing incomplete or no data on them, many abstracts instead emphasized more positive findings in secondary endpoints and/or focused on statistically significant subgroup or secondary analysis.
“To our knowledge, ours is the first randomized clinical trial evaluating the impact of spin, but the results need to be replicated,” said another of the authors, Ian F. Tannock MD, PhD, Professor of Medical Oncology at Princess Margaret Cancer Center in Toronto. “Our results showed that spin in an abstract of a RCT can have an impact on cancer clinicians interpreting trial results, such that they are more likely to rate a treatment as being beneficial despite the primary outcome not being statistically significant.”
This is important, Tannock told OT, because it shows that only a slight modification in the focus of an abstract can have an impact on clinicians' interpretation of a trial's result.
Spin is “fairy widespread,” he said, noting a 2013 study that he and his colleagues published in Annals of Oncology (2013; 24:1238-1244): In that paper—“Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer”—the group analyzed 164 published breast cancer RCTs and found that 33 percent had bias in their primary endpoint reporting and 67 percent did not provide adequate reporting of toxicity findings.
“Outcome reporting bias is especially common in studies with a negative primary endpoint, and toxicity reporting is often poor, especially in studies reporting a positive primary endpoint,” he said.
But what about the journals—Isn't there an obligation to fully review study results, including tables and figures, and compare them with the abstract? Tannock said yes, that he believes they should have a checklist for reviewers, which should also be followed by editorial staff reviewing submitted studies.
“This is both a systemic problem with the peer-review process and with researchers submitting papers and trial sponsors,” Tannock said. “Investigators and sponsors have the responsibility to present unbiased reports. I believe that some journals are now addressing it—For example, The Lancet and Lancet Oncology, as well as the Journal of the American Medical Association are all very rigorous in their editorial reviews.”
Asked for his perspective, Navneet Majhail, MD, Director of the Cleveland Clinic's Blood & Marrow Transplant Program, said it is important to keep in mind that the survey was completed by a very select group of physicians— 44 percent had PhDs as well as MDs, 72 percent were at a tertiary care center, 47 percent had training in epidemiology, and 78 percent had training in clinical trial design/methods. Also, most conducted their own clinical trials and most had published research papers. “So this group is not representative of most clinicians,” he said.
“Overall, given that the abstracts reviewed had shown non-significant results and were negative trials, it is reassuring that most of the clinicians who participated in this study rated the beneficial effects of treatment as low in both groups, and even though the effects were significantly different, the effect size was small in both the spin and no-spin categories.”
The clinical research expertise of the study participants was also evident in the other two findings, where the spin abstracts were felt to represent less rigorous studies and the respondents wanted to see the full text of the articles: “I am assuming they wanted more details before believing the results were accurate and true,” he said. “Also, there isn't much data on this issue.”
Majhail said that it is difficult to say how widespread the issue of “spinning” study findings might be because of this fact. “I have not seen many trial reports with ‘spin’ at prominent venues for research dissemination, such as respectable peer-reviewed journals or oral/plenary presentations at high-quality scientific meetings where true practice-changing research is presented.”
He said that at premier hematology/oncology meetings such as the American Society of Clinical Oncology or the American Society of Hematology annual meetings, “the reviewers do an excellent job of weeding out abstracts where the conclusions are not supported by the results or there are methodological flaws in the research. Only very high-quality research abstracts are selected for oral presentations.”
So the question is whether peer-reviewed journals have an obligation to fully review study results and compare the findings to what is said in the abstract. “At the end of the day, the gold standard for dissemination of medical research is publication in a high-impact peer-reviewed journal. I believe that respectable peer-reviewed journals, such as the Journal of the American Medical Association, the New England Journal of Medicine, the Journal of Clinical Oncology, Blood, and several others, do an excellent job of identifying spin in submitted research papers,” Majhail said.
“The editors and reviewers for these journals are mostly experienced researchers who are experts in their fields. And most astute reviewers can identify papers where the results don't support the conclusions or there are methodological issues. Many such journals may also get a statistical review to ensure that the study being reported is of high quality and integrity.
“With the present revolution in how information is disseminated, I would argue it has become easier to identify and call out studies that are not rigorous. With all journals having an online presence they are able to reach a larger audience and it is easier to write and submit a letter to the editor. Several journals allow online comments to papers. And finally, periodicals and journals, such as Oncology Times and others, get commentary from other experts to put clinical trials into perspective.”
Even the current study may need closer scrutiny, he noted. “With the way the abstract for this paper is structured, one can argue that the abstract for the SPIIN trial has a spin in it as well.”
Also asked for his opinion, Vladimir Hachinski, MD, Professor of Neurology and Epidemiology at London Health Sciences Center, Western University Hospital, in Ontario, who was Editor-in-Chief of Stroke from 2000 to 2010, said that at Stroke, as many as 95 percent of submitted studies were not published. “We did not survey the reviewers about why this was—i.e., whether or not it was due to spin or other factors. Our reviewers often did the abstracts or asked the authors to re-write them if they were not supported by the contents of a study—often helping them with this.”
He said he does not want to discourage researchers from submitting articles, but only wants them to be more mindful about the necessity of following established abstract guidelines.
Consolidated Standards of Reporting Trials (CONSORT) Guidelines for Abstracts
- In the Context section:
- Delete all information that could distort the understanding of the aim of the trial.
In the Methods section:
- i. The aim is to evaluate the treatment effect on a secondary outcome;
- ii. The aim is to evaluate the treatment effect for a subgroup; and
- iii. The aim is to evaluate overall improvement.
In the Results section:
- Clearly report the primary outcome;
- According to space constraints, report all secondary outcomes evaluated in the Methods section or report no secondary outcome evaluated in the Methods section to avoid specifically highlighting statistically significant secondary outcomes; and
- Delete information that could distort the understanding of the aim (e.g. within-group comparison, modified population analysis, subgroup analysis).
In the Conclusions section:
- Delete subgroup analyses that were not prespecified, based on the primary outcome, and interpreted in light of the totality of prespecified subgroup analyses undertaken;
- Delete within-group comparisons;
- Delete linguistic spin;
- Report the results for the primary outcome with numbers in both arms (if possible with some measure of variability) with no wording of judgment;
- Report results for all secondary outcomes, for no secondary outcome, or for the most clinically important secondary outcome; and
- Report safety data including reason for withdrawals; report treatment discontinuation when applicable.
- Delete the author conclusion, and only add the following standardized conclusion: “the treatment A was not more effective than comparator B in patients with....”;
- Specify the primary outcome in the conclusion when some secondary outcomes were statistically significant: “the treatment A was not more effective on overall survival than the comparator B in patients with....”