More than six months after their publication, there is still controversy swirling around the two historic systematic reviews on the safety and effectiveness of the bone-grafting drug bone morphogenetic protein-2 (BMP-2, Medtronic) commissioned by the Yale Open Data Access (YODA) Project—and the way those reviews are being interpreted.
The two reviews were historic in several ways. They not only provided the first rigorous examination of the totality of early patient-level data on any major commercial spine intervention but also have provided a test of the way the spine care community responds to important new evidence.
How Will the Spine Field Interpret the Evidence?
The two reviews collectively suggested that: (1) there did not appear to be any clear indication for the use of BMP-2 in spinal fusion surgery; (2) industry-sponsored clinical trials misrepresented study findings, exaggerated the effectiveness of BMP-2, and overstated risks associated with traditional iliac crest bone grafting; and (3) these studies failed to disclose probable harms related to BMP-2, including an increased risk of cancer.
How will the spine field interpret and implement these findings? By curtailing the use of BMP-2 in spinal fusion and other forms of spine surgery? By restricting it to high-risk patients? By initiating a major new research effort to fill in the many gaps in the evidence on this drug? Or will it ignore the new data and go back to business as usual?
Anecdotal evidence in this area suggests that spine surgeons have pulled back substantially in their use of BMP-2 in spinal fusion surgery. In a show of hands at a recent session on fusion surgery for back pain at the Federation of Spine Associations meeting in New Orleans, only a handful of attendees raised their hands when asked if they are still using BMP-2 routinely in single-level posterolateral spinal fusion surgery for axial back pain. (See Federation of Spine Associations, 2014.)
A Revisionist View of the YODA Reviews?
But some proponents of BMP-2 have recently attempted to recharacterize the conclusions of the two YODA reviews—and suggest that one review supports the continuing use of BMP-2 and one does not.
Several speakers at a recent symposium on this issue at the annual meeting of the North American Spine Society (NASS) articulated that view. (See NASS, 2013.)
They suggested that the review by researchers at the University of York in the UK showed that BMP-2 is effective and did not find any conclusive evidence of serious risks, and that it was only the Oregon Health and Science University review that concluded that BMP-2 has not been proven to be safe and effective. (See Fu et al., 2013; Rogers et al., 2013; Simmonds et al., 2013.) Those taking that interpretation suggested that the 12% advantage in radiographic fusion rates (based on an unvalidated and idiosyncratic method of fusion assessment) in patients receiving BMP-2 in the York review might justify the continuing use of this bone-growth factor in routine clinical practice.
But this seems to be an erroneous interpretation of the results of the reviews, as explained below.
Harlan Krumholz, MD, the principal investigator of YODA, recently exacerbated this controversy with a brief characterization of the two reviews in a New York Times commentary. “Interestingly, the reviews produced somewhat conflicting results,” wrote Krumholz. “One found that the device was no better than a bone graft and might be associated with a slight increase in cancer, while the other found that the device [i.e. BMP-2] was effective and the cancer risk inconclusive,” he said. (See Krumholz, 2014; and further comments below.)
Suggesting that one of the YODA reviews showed BMP-2 to be effective in a newspaper with 1.8 million daily readers has certainly brought this issue to the fore. There are now concerns that this kind of casual characterization might translate into thousands of surgeries with a bone-grafting agent of uncertain safety and effectiveness.
Several prominent researchers and patient representatives recently commented to a BackLetter editor that they thought this statement was misleading and is not an accurate reflection of the York review—or the overall conclusions of the YODA project.
Similar but Not Identical Conclusions
The statement by Krumholz contradicts the conclusions of the editors of Annals of Internal Medicine, who supervised the peer review and revision of the statistical analysis of both reviews.
“After looking at the data from the YODA review with two completely independent teams of statisticians and analysts, the editors of Annals were in the best position to characterize these outcomes,” said Eugene J. Carragee, MD, of Stanford University in a recent email. Carragee, of course, was the lead author of the incendiary review of BMP-2 that prompted the YODA project. (See Carragee et al., 2011.)
“The editors of Annals were closest to the data from the two reviews and the complex statistical analyses employed in the reviews and the revision process. And they came to a clear conclusion: there was no clear benefit for BMP-2, and there were increased harms. The reviews documented biased reporting, inappropriate study conduct and misrepresentation of the evidence. And they found no clear indications for continuing use of BMP-2,” Carragee added.
Here are the actual conclusions by Christine Laine, MD, and other editors of Annals: “After systematic evaluation and synthesis of all available evidence, both systematic reviews published here independently conclude that rhBMP-2, compared with iliac crest bone grafting, does not improve pain or function and increases adverse events, possibly including cancer. For the outcome of fusion, the focus of initial enthusiasm for rhBMP-2, 1 review concluded that it improved by a marginally statistically significant amount with rhBMP-2 (overall fusion, 1.14 [95% CI, 1.03 to 1.25]), and the other concluded that it did not (anterior fusion, 1.05 [CI, 0.88 to 1.24]).” (See Laine et al., 2013.)
In a symposium on the YODA project in late 2013 at the annual meeting of the North American Spine Society in New Orleans, Laine reiterated the view that the two reviews came to similar conclusions. (See Laine, 2013.)
She explained that the editors of Annals went to great lengths to keep the two reviews completely separate. She divided the editors and statisticians at the journal into two groups and assigned each group one of the reviews. The two groups were not allowed to communicate about the two reviews during the editing, peer review, and revision processes.
Laine asserted that it is the points of agreement between the two completely independent reviews that lend the overall conclusions of the YODA project credibility.
“If one team [of systematic reviewers] had done this review, there would be more debate about BMP-2,” said Laine. “But the fact that you had two independent teams come to what is basically the same answer gives us more confidence in the validity of the results.” (See Laine, 2013.)
She noted that there are ascending levels of evidence on medical interventions. “One study doesn't give you an answer, many studies get you a little closer, and analyses [of the whole body of evidence] give greater confidence in the conclusions,” she said. In addition, the fact that systematic reviews by two world-class review teams were in general agreement about BMP-2 gives greater confidence still.
She suggested that the two reviews were similar in their conclusions about effectiveness, or lack thereof. “When compared to iliac crest bone grafting, rhBMP-2 results in no or modest improvements in radiologic fusion and does not yield a clinically significant improvement in clinical outcomes at 2 years,” according to Laine.
What about adverse events and safety? “Safety outcomes were described differently in the two reviews,” said Laine. “But the bottom line was basically the same: adverse events were more common with BMP-2. Cancer was infrequent, but there was a signal that it was about twice as common with BMP-2 ...”
On the basis of this evidence, Laine suggested moving away from the routine use of BMP-2 in spine surgery. “It seems that from the perspective of a medical doctor that this should not be used routinely in spinal fusion surgery,” she said.
Perspective of One of the Systematic Reviewers
Roger Chou, MD, is the Director of the Pacific Northwest Evidence-Based Practice Center at Oregon Health and Science University—where he coauthored one of the two YODA reviews. (See Fu et al., 2013.) In a recent e-mail, he said he also believes that the two reviews came to similar conclusions.
“The main reason for the difference in the two reviews is that the York team pooled trials of different surgical techniques (e.g., posterior lumbar interbody fusion, anterior lumbar interbody fusion). For our review, we made an a priori decision that the surgeries are different enough that pooling wasn't appropriate clinically,” Chou explained.
“There may be some differences related to the statistical techniques used. We used a one-step model for the Independent Patient Data analysis, but the York folks decided to stick with the two step-model. However, the differences in statistical methods probably had a pretty small effect,” according to Chou.
“In terms of the findings of the review, even though the York review found some statistically significant differences they noted that the differences [in patient-related outcomes] probably aren't clinically significant. So I think the findings of the two reviews are more consistent than inconsistent,” Chou concluded.
Response From the YODA Principal Investigator
A BackLetter editor recently queried Krumholz about his characterization of the York review, asking whether it represented a shift in the stance of the YODA leadership—which had remained studiously neutral in its interpretation of the two reviews until the New York Times article.
Here is the question posed to Krumholz: [A BackLetter] article is going to argue that the York review does not show BMP-2 to be effective. Rather it produced mixed findings on the three primary outcome measures—and only showed marginal benefits for BMP-2. Do you want to expand on why you think the York review shows BMP-2 to be “effective,” and why YODA would diverge from the balanced interpretation of the Annals' editors?
In a subsequent e-mail, Krumholz denied any intention to redefine the results of the YODA reviews.
My point was to highlight that the reviews were not completely aligned. I took my comments in the op-ed straight from the results of the abstracts that were published in the articles. One group highlighted no benefit and [some] risk, and the other highlighted some benefit and little evidence of risk.
I don't think that is much in dispute if you look at what they wrote in their abstracts. I do agree that digging deeper into their analyses brings them a little bit closer, but still I was impressed that they did not come up with identical findings and identical interpretations.
I have no dog in this fight. I was just reporting what was in the abstracts. If I'm mistaken, please show me where that mistake was. I was not seeking to interpret their findings. I was seeking to report their findings.
I purposely sought not to add an additional interpretation on top of what they explicitly stated. Your questions are perfectly legitimate but I do not agree that I was interpreting their findings.
If I were doing the reviews, I certainly would have emphasized the effect on patient outcomes rather than an outcome such as fusion, but I purposely sought groups that were a step away from the company to do their own analysis in the way that they thought was most appropriate.
These are world-class organizations. I am merely reporting what they determined was their main conclusion from the review that they performed. Had I done an interpretation of what I thought their analysis showed, then I would've said something quite different. I resisted saying in public what my interpretation might be because of my interest in staying independent and not expressing an opinion about this particular drug/device. So no disrespect intended, but I think you're off base.
Krumholz sent a capsule summary of the two abstracts, upon which he based his characterization:
Here are the 2 conclusions:
[Oregon review]: In spinal fusion, rhBMP-2 has no proven clinical advantage over bone graft and may be associated with important harms, making it difficult to identify clear indications for rhBMP-2.
[York review]: At 24 months, rhBMP-2 increases fusion rates, reduces pain by a clinically insignificant amount, and increases early postsurgical pain compared with [iliac crest bone graft] ICBG. Evidence of increased cancer incidence is inconclusive.
Krumholz then sent a second e-mail reiterating that he supports active, open debate regarding the two reviews.
I hope you know that I really appreciate the work that you and your colleagues are doing.
And I want to make a distinction again between reporting what the two centers said in their conclusion and what the body of their work truly says.
I think you're perfectly within your rights to interpret the body of their work and come to your own conclusions about what their most important findings are.
If science is to be progressive and self-correcting then people have to stop just accepting what people say and be willing to dig into the data and the findings and the studies and think for themselves and then be willing to put that out for public dialogue.
This may have an innocent characterization, but it still does not seem to be an accurate one. The York review defined three primary outcome measures and documented an advantage for BMP-2 in only one of them. BMP-2 resulted in a slightly higher fusion rate (12%), but it found no clinically significant advantage in patient-centered outcomes. The York abstract never used the word “effective” to describe the overall results of the review.
“I think the article by Dr. Krumholz clearly illustrates the dangers of characterizing the results of scientific studies based on their abstracts,” commented Terry Corbin, a Consumer Representative at the Cochrane Collaboration Back Review Group in a recent e-mail.
Corbin said he would have preferred to see Krumholz cite the actual conclusion that the York review group published at the YODA website, though the editors of the New York Times might have balked at the longer, more accurate characterization.
“For the period up to 24 months after surgery, treatment with rhBMP-2 increases the probability of successful fusion (according to Medtronic definitions) but this does not appear to translate to clinically meaningful benefits in pain reduction, function, or quality of life. The small benefits in these outcomes observed from six months onwards come at the expense of more pain in the immediate post-operative period and a possible increased risk of cancer, although the absolute risk of cancer is small. Chances of experiencing other complications may also be increased, although these findings are derived from less robust study designs. We suggest it is very important that these findings are expressed clearly and discussed with patients so that they can make informed choices about the type of surgery they would prefer,” according to the York University review group. (See Brown et al., 2014.)
Corbin also said he doesn't think it is worthwhile debating whether the 12% advantage in fusion rates documented in the York review is significant or not. He pointed out that there were both study design and study conduct issues in the early clinical trials sponsored by Medtronic. And these issues would compromise any take-home messages based on marginally significant statistical advantages.
“The thing that seems to be lost in the post-YODA discussion is that the trials had biased designs and conduct so BMP-2 should have won those comparisons handily,” according to Corbin. As a result, he does not think that the 12% advantage in fusion rates would actually translate into an actual advantage in real-world surgical settings.
And at the end of the day, the leadership of YODA, including Krumholz, appears to have also embraced this point of view.
Just before this issue of the BackLetter went to the printer, Krumholz and colleagues published a new editorial in Spine regarding the outcomes of the two reviews—apparently based on a new statistical analysis.
“Although at times the groups used different methodological strategies and techniques, results were broadly similar. One group found a small advantage to rhBMP-2 compared with iliac crest bone graft in inducing spinal fusion combining data across all approaches, but this difference was perhaps too small to be detected when data were stratified by surgical approach,” according to Jeffrey B. Low, AB, and colleagues. (See Low et al., 2014.)
This interpretation should certainly discourage the view that the systematic review from York University shows BMP-2 to be “effective.”
And one hopes this new editorial will put this entire controversy to rest.
Disclosures: None declared.
Carragee EJ, et al., A critical review of recombinant human bone morphogenetic protein-2 trials in spinal surgery: Emerging safety concerns and lessons learned. , Spine Journal. , 2011;, 1:471–491. doi: 10.1016/j.spinee.2011.04.023.
Federation of Spine Associations, Debate I: Axial Pain, Section II: North American Spine Society. , 27th Annual Specialty Day Meeting, Annual Conference. , American Academy of Orthopaedic Surgeons, New Orleans, 2014; as yet unpublished.
Fu R, et al., Effectiveness and harms of recombinant human bone morphogenetic protein-2 in spine fusion. , Annals of Internal Medicine. , 2013; 158:890–902.
Laine C., Editor-in-Chief's perspective, Symposium: Lessons From the YODA Project. , Annual Meeting of the North American Spine Society. , New Orleans, 2013; https://www.spine.org/pages/myaccount.aspx
. Accessed April 18, 2014.
Laine C, et al., Closing in on the truth about recombinant human bone morphogenetic protein-2: Evidence synthesis, data sharing, peer review, and reproducible research. , Annals of Internal Medicine. , 2013; 158:916–8.
Low JB, et al., Moving forward from rhBMP-2. , Spine. , 2013; 39:531–2.
North American Spine Society, Update on BMP-2: Current Usage, YODA Findings, and Risk Management. , presented at the Annual Meeting of the North American Spine Society. , New Orleans, 2013; as yet unpublished.
Rogers MA, et al., Reporting of industry funded study outcome data: Comparison of confidential and published data on the safety and effectiveness of rhBMP-2 for spinal fusion. , BMJ. , 2013; 346:; f3981
Simmonds MC, et al., Safety and effectiveness of recombinant human bone morphogenetic protein-2 for spinal fusion. , Annals of Internal Medicine. , 2013; 158:877–89.
© 2014 Lippincott Williams & Wilkins, Inc.