As Editor-in-Chief of the Journal of the National Cancer Institute, Barnett Kramer, MD, has received studies on every facet of cancer research, but never did he expect to see one with his own journal as the subject. Yet there it was, a study from the University of Florida, reporting that not only JNCI, but also Blood, Cancer, the Journal of Clinical Oncology, and the New England Journal of Medicine lack important details in their published trials.
That came as a surprise to an editor who claims that “most of the comments we get are about the high quality of our papers.”
Yet he saw enough truth in the study to publish it (JNCI 2010;102:702-705).
“The topic is timely in an era of making sure health care is optimized,” he said. “I think self-reflection on the part of journals and editors is important.”
Self-reflection led Dr. Kramer and other journal editors to realize they may indeed be selling readers short. Now they're inspired to fill the gap.
“The authors have done an important service,” Dr. Kramer said. “There's room for improvement.”
Assessing the Journals
The study's senior author, Thomas J. George, Jr., MD, Director of the Hematology Oncology Fellowship Program and Assistant Professor, explained that he launched the study in clinical frustration over what he considered a lack of data in published trials.
“We never could find the details we needed to deliver therapy to the patients,” he said.
He and first author Jennifer M. Duff, MD, and their coauthors began the study by asking a separate, multidisciplinary group of oncology providers what they consider the most important points in a published trial. From the answers, they developed a list of 10 essential elements and then searched for those in 262 randomized controlled trials published between January 1, 2005, and November 30, 2008.
Nearly every article contained such obvious elements as the drug name, dose, cycle length, and maximum cycles. Not as many, though, included the route of administration, and fewer still of the studies reported such incidental factors as premedication, growth-factor support, and dose adjustments for hematologic toxicity and organ dysfunction.
Only 11% of the trials contained all 10 elements. With an average of 80% of the elements, the JNCI's trials were the most complete. Other journals had 71% to 73%.
“Finding just how bad the articles were really surprised us,” Dr. George said.
When he unveiled the study as a poster at the 2009 ASCO Annual Meeting, he realized he'd struck a chord, he said.
“Dozens of program directors and oncologists came by to show their gratitude for conducting the study,” he said. “It's clearly been on the minds of many fellowship-program directors and practicing clinicians.”
One of them was Christian Cable, MD, Director of the Blood and Marrow Transplant Program at Texas A&M Health Science Center College of Medicine: “I thought it was brilliant—I agree with the authors that details are lacking.”
Debate Over the Essentials
But whether those elements are critical is open to debate. The journal editors acknowledge the value of thorough reporting of oncology trials. Some are making changes the authors recommend to give readers more trial details. Yet they dismiss the authors' 10 elements, saying that the elements missing from trials in their journals were either unnecessary, or obvious to anyone who's treated cancer patients.
“We assume people know how to use antiemetics, we assume they know that tamoxifen is not given IV,” said Daniel Haller, MD, Editor-in-Chief of JCO
Similarly Dr. Kramer, who said he feels the researchers' elements leave little room for judgment. “There are guidelines [elsewhere] on how to give growth factors and when to give them,” he said. “You wouldn't have to repeat the guidelines in every article.”
Eric Seifter, MD, Associate Professor of Medicine and Oncology at Johns Hopkins University School of Medicine, said he's never felt a need for more information when replicating a published trial.
“Most of us know how to give drugs and what the side effects are,” he said.
Same for Patrick Loehrer, MD, Associate Dean for Cancer Research at Indiana University School of Medicine, who also feels comfortable replicating trials with only the usual published data: “Dosage adjustments are relatively common,” he said. “We tend to do this without even thinking about it.”
Still, more details would help less experienced physicians—“For young clinicians,” he said, “it's critically important.”
Dr. George goes so far as to suggest that sketchy trial data compromise patient care: “We've seen that when study results are first reported, the outcomes are always better than after the drug is widely used in practice,” he said. “It may be that a lot of doctors are making adjustments in a haphazard fashion based on the lack of details [in the published trial].”
The spare format of published trials stems in part from journal space restrictions. Most print journals limit studies to 3,000 to 5,000 words.
“Reproducing every dose modification and every premedication...would take up so many words you'd have nothing to report,” Dr. Haller said.
Dr. George and his coauthors recommend an online appendix for such supplemental data. Four of the five journals in the study have sections on their sites for supplemental materials, but only two used theirs for the trials the researchers studied. And only one of the five online supplements provided all the data that the study recommended.
On this point, the editors and authors agree. Online supplements would enable editors to give clinicians access to more trial data without bogging down the report.
Blood Editor-in-Chief Cynthia Dunbar, MD, noted that after several readers asked the journal to provide more practical information about administering drugs, she recently decided to add online appendices to the published trials. Authors now must provide supplemental data geared toward clinicians replicating the study.
“There was a disconnect in translating things from academic centers into practice,” she said. “I think [the supplements] will go a long way toward making the studies more useful.”
One of the simplest ways to satisfy readers' demand for trial information is to provide online access to the protocols, something Dr. Thomas and his colleagues recommend. The New England Journal started posting protocols this year, which was after the UF team ended their study; and Dr. Haller said he expects to follow suit on his journal's website within a couple months. He started requiring authors to submit their protocols a year ago, for the benefit of reviewers.
“We've had no objections from anyone,” he said. “The vast majority simply downloaded the protocols.”
Dr. Haller regards the UF study like a veteran oncologist regards the latest drug trial. He'll learn from it. he'll make some changes the authors recommend, but when it comes to the details—what elements to include in every published trial—he'll stick with what he knows works, he said. “I don't need someone saying you need to have the name of a drug in your study. That's just good journalism.”