Turner, Judith A. PhD*; Hollingworth, William PhD†; Comstock, Bryan MS‡; Deyo, Richard A. MD, MPH¶
Policymakers are often faced with difficult decisions concerning coverage of medical treatments that are promising, but for which scientific evidence is limited. An example is spinal cord stimulation (SCS), a treatment for chronic back and leg pain after spine surgery (failed back surgery syndrome [FBSS]). In Washington State, the Department of Labor and Industries (DLI) has never covered SCS for workers' compensation recipients due to concerns about its efficacy and safety.1 SCS is widely covered by Medicare (with an estimated 596% increase in utilization from 1997 to 2006),2 private insurers, and other US states workers' compensation programs.3 In response to industry pressure for coverage and the need for more evidence concerning the long-term effectiveness and cost-effectiveness of SCS in the workers' compensation environment, the DLI funded a coverage with evidence development (CED) study. CED requires patient participation in a registry or study to receive insurance coverage of a technology.4 This allows coverage of a promising technology while gathering data to better understand its risks, benefits, and costs.4
SCS involves surgical placement of electrodes, which are connected to an electrical pulse generator, into the epidural space. Typically, patients first have a stimulation trial; a permanent stimulator implantation follows only if the trial is successful in relieving pain (usually by at least 50%).5,6 Although SCS has been used for over 40 years, its efficacy for FBSS has been evaluated in only 2 randomized clinical trials (RCTs).7–9 Both of these industry-sponsored studies found evidence for efficacy at 6 months,7–9 but high crossover rates hampered conclusions about long-term efficacy. A 2008 National Institute for Health and Clinical Excellence appraisal recommended SCS for chronic neuropathic pain, but also recommended further research to evaluate benefit duration.10
Evidence from these RCTs is of uncertain relevance for workers' compensation because patients receiving workers' compensation were less likely to be included in the first RCT7,8 and were not included in the second.9 Workers' compensation recipients have worse outcomes from a variety of pain treatments.11–13 Furthermore, questions concerning benefits of SCS for pain and function beyond 6 months remain unanswered. In Washington State workers' compensation, treatment must be either curative or rehabilitative. Rehabilitative treatment, including SCS, must be associated with functional recovery and changes must be long-term.14
We previously reported the study methods and patient outcomes.15 We now report results from a second SCS patient cohort; study costs; the decision-making process concerning SCS coverage by Washington State payers; and the efforts of industry and providers to influence these decisions. This information may be useful to policymakers and investigators involved in or contemplating comparative effectiveness research. Current controversies concerning application of scientific evidence to clinical practice (eg, efficacy of vertebroplasty,16 mammography screening17) illustrate the uncertainties involved in how to translate comparative effectiveness research findings into policy and practice. To provide context, we first summarize the study design, methods, and major findings; the previous study publication contains more detail.15
Overview of Design, Participants, Setting, and Procedures
The study was designed to evaluate outcomes and costs for patients with FBSS in 3 treatment groups: SCS, Pain Clinic, and Usual Care. Study participants were receiving work time loss compensation and met clinical criteria intended to select patients likely to benefit from SCS (and similar to the inclusion criteria in the multinational RCT9). All patient-reported data collection, data analyses, and report writing were performed independently of DLI. Participants provided informed consent and were informed that the study was independent of the workers' compensation program, that all information would be confidential, and that their decision regarding participation would not affect their workers' compensation benefits (other than providing access to SCS coverage).
Health care providers were notified in November 2004 that DLI would cover SCS for patients with a Washington State Fund workers' compensation claim who met study eligibility criteria and enrolled. The State Fund covers most of Washington State employers and about two-third of the state's workforce.18 Physicians informed potential SCS candidates that DLI would cover SCS only if they participated in the study. Patients who desired SCS, were willing to participate in the study, and were approved upon administrative review were referred to the research team for final screening and enrollment. Patients potentially eligible for the Pain Clinic or Usual Care groups were identified from DLI administrative databases and approached by research staff for enrollment.
Patients completed standardized, validated, clinically responsive measures of pain, physical functioning, and mental health, and reported work status and other information at baseline and 6, 12, and 24 months later. The primary outcome was a composite measure, defined as ≥50% reduction in 0 to 10 ratings of leg pain intensity, at least modest (≥2-point) improvement on the Roland Disability Questionnaire (RDQ)19 (0–24 scale; higher scores indicate worse function), and less than daily opioid medication use. These outcomes are consistent with the clinical goals of SCS, which include leg pain relief, improvement in function, return to work, and medication reduction.5,20 At the study onset, 50% improvement in pain was a common primary outcome in SCS studies and was regarded as a standard for clinically meaningful improvement in chronic pain.5,21 We also examined separately each of the components of the composite outcome, and the proportion of patients with 30% improvement in pain (currently regarded as the minimal clinically important difference21). The primary end point was 12 months. Work time loss compensation and claim status data through 24 months were obtained from workers' compensation administrative databases. Medical records were reviewed for SCS-related adverse events.
Fifty-one patients underwent trial SCS (27 received a permanent implant), 39 underwent Pain Clinic evaluation (22 received pain clinic treatment), and 68 patients were in the Usual Care group.15 Follow-up response rates (98%, 94%, and 87%, at 6, 12, and 24 months, respectively) did not differ across groups. At baseline, the groups were similar on most demographic, work, and pain characteristics, but SCS patients were significantly more likely to have an attorney. The SCS group also had significantly longer duration of compensation and leg pain as compared with the Pain Clinic, but not the Usual Care, group. Finally, SCS patients reported significantly greater leg pain than did Usual Care patients and significantly worse physical function than did Pain Clinic or Usual Care patients, although these differences were small. Table 1 shows characteristics of the study participants as compared with those in the RCTs.
The primary analyses, like those of the multinational RCT, included all patients who received trial SCS, regardless of trial success. Our rationale was that patients who have a good response to trial SCS would be expected to have better pain outcomes over time, not only with SCS, but potentially with other therapies. Excluding patients who failed the therapy initially in one group, but not the others, would introduce bias. Analyzing only patients who had a successful trial would suggest the need to similarly analyze only patients in the other treatment groups who had a favorable initial response to therapy. No such screening process exists in usual care, but not all patients evaluated at a pain clinic receive pain clinic treatment. Therefore, we also compared in secondary analyses patients who received a permanent SCS to patients who received some pain clinic treatment, regardless of content or duration.
Comparisons of Patients Who Received Trial SCS to Patients Who Received Pain Clinic Evaluation (Regardless of Treatment) or Usual Care Only
At each timepoint, on the primary composite outcome, few (<10%) patients in any group achieved success and the SCS group did not differ significantly from the other groups.15 The SCS group also did not differ significantly from the other groups on any of the individual components (≥50% improvement in leg pain, ≥2-point improvement in RDQ score, less-than-daily opioid medication use) of the composite outcome at 12 or 24 months.15 At 6 months, however, more SCS patients showed ≥50% improvement in leg pain (18% versus 3%–5% in the other groups) and more SCS patients than Pain Clinic patients reported daily opioid use (88% versus 66%).15 When success was defined without the opioid medication component (ie, ≥2-point improvement on the RDQ and ≥50% leg pain improvement), the success rate was higher (but still low) in the SCS group (18%) at 6 months (P = 0.01 in comparison with the Pain Clinic [0%] and Usual Care [3%] groups). The 3 groups showed modest and similar improvements in leg pain intensity and physical function over the 24 months of the study, adjusting for baseline differences between groups.15 In each group, <10% of patients were working at 12 months and <25% at 24 months. The groups did not differ significantly on any work status, time loss compensation, or claim status measure at any time point.15 Although the sample size was limited, post hoc power calculations indicated good power to detect modest differences between groups: >80% power for a 2.5-point difference on the 25-point RDQ and >95% power for a 1.5-point difference on the 11-point leg pain intensity measure.
Comparisons of SCS Patients Who Received a Permanent Stimulator to Patients Who Received Some Pain Clinic Treatment
Slightly over half of the SCS patients (53%) received a permanent stimulator and a similar proportion of Pain Clinic patients (56%) received at least some pain clinic treatment.15 Fewer than 10% of patients who received permanent stimulators or at least some pain clinic treatment achieved success on the primary outcome at any time point, with no significant differences between these 2 subgroups on the primary outcome at any time point.15 Examining the components of the composite primary outcome, there were trends toward higher rates of both ≥50% improvement in leg pain (33% versus 10%, P = 0.06) and daily opioid use (85% versus 57%, P = 0.05) at 6 months among those who received permanent stimulators versus those who received pain clinic treatment, but no other differences.15
Adverse Events in the SCS Group
There was one life-threatening SCS trial-related event (deep infection requiring extensive surgery, followed by respiratory arrest).15 One patient had a dural puncture during a permanent implantation attempt, resulting in hospitalization for neurologic monitoring. Among patients with permanent stimulators, adverse events over 18 months included superficial (11%) and deep (4%) infections, persistent pain in the region of the equipment (19%), and other biologic complications (11%). Five patients (19%) had a total of 7 revision operations and 5 had the stimulator removed.
Patients Treated After the End of the Study
When study enrollment ended in June 2006, the DLI needed to make a coverage decision for claimants referred for SCS while study results were pending. DLI decided to cover SCS for patients who met the study inclusion criteria and who agreed to provide baseline and 12-month data. Patients enrolled in this “SCS-2” group from July 2006 to April 2007. This enabled us to examine whether there were any major changes over time in patient baseline characteristics or response to SCS.
The SCS-2 group (n = 30) was similar to the original SCS group on most baseline characteristics, although all SCS-2 participants were male (Table 2). Similar proportions of patients in the SCS-2 group (57%) and the original cohort (53%) received permanent stimulators, and their baseline characteristics were generally comparable (Table 3), although mental health scores were worse in the SCS-2 group.
At 12 months, the SCS-2 follow-up rate was high (90%). Comparing those who received permanent stimulators in the 2 cohorts, there were no significant differences on 7 of 8 success definitions examined (Table 4). However, more SCS-2 patients reported less-than-daily opioid use (P = 0.05). The 2 groups did not differ significantly on RDQ and leg pain intensity scores, adjusted for age, gender, and baseline scores (Table 5).
Costs of the Study
The original 4-year U.W.-DLI contract and SCS-2 supplement totaled $602,050. Reimbursements by DLI for medical procedures, services, and hardware on the days of SCS trials and permanent implants totaled ∼$1,085,000 for the 51 original SCS patients. These totals do not include costs associated with utilization review, DLI staff time required to provide administrative and medical records information, treatment of SCS-related adverse events, revision operations, and device removals or SCS-related costs for the SCS-2 group, and subsequent patients treated with SCS before the DLI resumed its noncoverage policy. DLI has covered and will continue to cover medically necessary SCS-related care for patients who received SCS while covered by DLI.
Response to the Study From Industry and SCS Providers and Study Impact on Policy
The final study report was submitted on September 30, 2008. The Principal Investigator presented the final study results at the October 2008 meeting of the Washington State Industrial Insurance Medical Advisory Committee (IIMAC).22 This statutorily authorized committee, which includes practicing clinicians representative of those who treat injured workers, advises DLI on all major health care coverage decisions.23 In November 2008, DLI resumed its SCS noncoverage policy based on the original intent to cover SCS only until the study was completed, the final study results, and the increasing volume of requests for SCS (114 in 2008).1
SCS device manufacturers and a neuromodulation therapy advocacy organization, whose members include pain interventional societies and device manufacturers, submitted criticisms of the study at all stages, beginning in the design phase, to DLI and to the IIMAC. In May 2008, national and local physician advocates of SCS, on behalf of the neuromodulation therapy advocacy organization, met with the DLI Director to express concerns about the study.24 After the October 2008 IIMAC meeting, at which no public comment was allowed, the advocacy group's newsletter stated that at the meeting, the IIMAC “continued to (a) minimize or ignore the limitations in the study design, including its nonrandomized approach, problems in patient selection, etc; and (b) improperly interpret the results, which actually show good results for SCS in the study, despite the design limitations.”24 The advocacy group continued to express concerns to DLI that the study conclusions were at odds with other evidence on SCS and that the study had “serious methodological shortcomings, which significantly limit its ability to meaningfully inform the discussion on the efficacy of SCS for properly selected patients. These concerns include problems with the overall study design, patient selection, screening trials, data collection, and the definition of success.”25 The group also contracted an evidence-based medicine assessment organization to critique the study.25
At the January 2009 IIMAC meeting, public comment on the study was allowed.26 The neuromodulation group requested time on the agenda for an “expert presentation that will place the results of the study in the broader context of literature on SCS. Ultimately, we intend to clearly demonstrate the underlying methodological concerns with the study; that it should be seen as an outlier relative to published literature on SCS; and that payers throughout the United States and the world have come to strikingly different conclusions.”27 The neuromodulation advocacy organization arranged for a presentation by an academic researcher who consults for an SCS device manufacturer, critiquing the study and summarizing the results of the more favorable multinational RCT.24,28 Another consultant for the device manufacturer presented an economic model using data from a nonworkers' compensation population, concluding that DLI could expect a marginal reduction to its annual budgeted costs if it were to cover SCS.29 After the presentations, the IIMAC made the advisory recommendation to not change the DLI SCS noncoverage policy.29 The next day, advocacy organization representatives and the academic researcher they sponsored met with the DLI Director to again discuss the limitations of the study, the more favorable evidence in the literature, and the National Institute for Health and Clinical Excellence recommendations; subsequently, they wrote a follow-up letter to the Director.24
The DLI then asked the State Health Technology Assessment (HTA) Program to review the study and the larger literature on SCS. In the HTA Program, developed to make state-purchased health care safer, more cost-effective, and more consistent across state agencies,30 practicing clinicians evaluate scientific evidence for the safety, effectiveness, and cost-effectiveness of health technologies. The HTA Program will review SCS in 2010.
The success rates for SCS trials and permanent implants in our study were considerably lower than in the 2 RCTs.7–9 Patient outcomes with SCS may be worse in a workers' compensation setting11–13 and as delivered in community practice. Although RCTs are the best study design for determining a therapy's efficacy, often they have limited external validity and do not answer questions of whether the therapy works in clinical practice or in a specific patient population or setting, or whether the benefits are worth the harms or costs.31,32 An RCT of a standardized procedure performed by physicians selected for their technical expertise on patients selected using rigorous inclusion/exclusion criteria is likely to find more benefits and fewer harms for the therapy than a study of the operation performed in community settings by providers with varied experience and skill, on more clinically heterogeneous patients.31–33
The study was criticized by an industry-backed neuromodulation advocacy group at all stages, in the form of letters to and meetings with the director of DLI and presentations to the DLI medical advisory group. Specific criticisms were that there were no standardized patient selection criteria, screening trial parameters, or treatment; the study was not randomized; and the composite outcome was not standard in the literature. We acknowledge that our study, like all studies, had methodological limitations. However, all patients were carefully screened for study inclusion/exclusion criteria intended to select patients most likely to benefit and similar to those in the multinational RCT; the study was designed to evaluate patient outcomes in community practice rather than under ideal conditions; and we analyzed outcomes using standard measures in addition to the composite primary outcome, with similar results. Interestingly, the study was criticized for enrolling workers' compensation recipients who had been off work a long time—exactly the population likely to be offered SCS and thus of interest to the agency.
The DLI interim coverage decision allowed us to assess whether patients or outcomes changed over time. The SCS-2 cohort was similar to the original SCS group in most baseline characteristics, rates of permanent SCS implantation, and permanent SCS outcomes, supporting the reliability of the study results.
The study was expensive for the payer. However, the costs of a CED study of a previously noncovered therapy may be recouped over time if the therapy is found to substantially improve patient outcomes, becomes covered, and results in medical and/or wage replacement costs savings. Alternatively, if the therapy is found not to improve patient outcomes and a noncoverage policy is maintained, the costs of the study could be substantially less than the costs of covering the therapy. The costs to conduct our study were substantially less than the costs of SCS provided to the participants, much less future patients.
Because of the importance of patient-reported outcomes in studies of pain therapies, we could not rely solely on administrative data for assessing outcomes. Study follow-up response rates were good in each study group, supporting the feasibility of obtaining patient-reported outcomes data in CED studies. Furthermore, the SCS group enrollment rate was good due to accurate patient contact information and the study participation requirement for SCS coverage. However, we had more difficulty contacting and enrolling patients in the comparison groups, resulting in some uncertainty about their representativeness of the populations from which they were drawn.
Researchers contemplating CED studies of a therapy for which there are substantial industry and professional interests should be prepared for criticism. Nonetheless, CED studies have the potential to contribute important knowledge concerning the benefits and risks of a therapy in actual clinical practice. In the case of SCS for FBSS, despite 2 RCTs, as a recent editorial emphasized, “..the risk of adverse events..is far from trivial..and the evidence base is in no way robust..It is quite conceivable that future studies might lead to different conclusions regarding clinical effectiveness, and we do not know with any degree of certainty whether SCS is worth it.”34
In conclusion, CED studies can be useful to payers in making coverage decisions for medical procedures that are promising but for which there are questions regarding patient selection criteria, safety, or benefits over time in clinical practice.35 However, the costs and other challenges are barriers to payers conducting such studies. An unresolved issue in making treatment coverage decisions is how to weigh evidence from RCTs of a therapy delivered in an ideal manner to a select sample of patients versus evidence from observational studies of the therapy as delivered in clinical practice to a specific population of interest. CED studies and their investigators will receive critical attention from industry, providers, and legislators, as was the case with the National Emphysema Treatment Trial.35 Nonetheless, CED studies may yield important information that would not be apparent in RCTs concerning long-term benefits and risks of a therapy as applied either in broad clinical practice or for specific patient groups.
2. Manchikanti L, Singh V, Pampati V, et al. Analysis of growth of interventional techniques in managing chronic pain in the Medicare population: a 10-year evaluation from 1997 to 2006. Pain Physician
4. Tunis SR, Pearson SD. Coverage options for promising technologies: Medicare's ‘coverage with evidence development.' Health Affairs
5. North R, Shipley J, Prager J, et al. Practice parameters for the use of spinal cord stimulation in the treatment of chronic neuropathic pain. Pain Medicine
6. Ontario Medical Advisory Secretariat. Spinal Cord Stimulation for Neuropathic Pain:AnEvidence-Based Analysis
. Ontario Health Technology Assessment Series. Toronto, Ontario: Medical Advisory Secretariat;2005:5.
7. North RB, Kidd DH, Lee MS, et al. A prospective, randomized study of spinal cord stimulation versus reoperation for failed back surgery syndrome: initial results. Stereotact Funct Neurosurg
8. North RB, Kidd DH, Farrokhi F, et al. Spinal cord stimulation versus repeated lumbosacral spine surgery for chronic pain: a randomized, controlled trial. Neurosurgery
9. Kumar K, Taylor RS, Jacques L, et al. Spinal cord stimulation versus conventional medical management for neuropathic pain: a multicentre randomised controlled trial in patients with failed back surgery syndrome. Pain
10. National Institute for Health and Clinical Excellence. SpinalCord StimulationforChronic PainofNeuropathicorIschaemic Origin, NICETechnology Appraisal Guidance159.
London: National Institute for Health and Clinical Excellence; 2008. Available at: http://guidance.nice.org.uk/TA159/Guidance/pdf/English
11. Harris I, Mulford J, Solomon M, et al. Association between compensation status and outcome after surgery: a meta-analysis. JAMA
12. Atlas SJ, Chang Y, Keller RB, et al. The impact of disability compensation on long-term treatment outcomes of patients with sciatica due to a lumbar disc herniation. Spine
13. Anderson PA, Schwaegler PE, Cizek D, et al. Work status as a predictor of surgical outcome of discogenic low back pain. Spine
15. Turner JA, Hollingworth W, Comstock BA, et al. Spinal cord stimulation for failed back surgery syndrome: outcomes in a workers' compensation setting. Pain
16. Kallmes DF, Comstock BA, Heagerty PJ, et al. A randomized trial of vertebroplasty for osteoporotic spinal fractures. N Engl J Med
17. US Preventive Services Task Force. Screening for breast cancer: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med.
18. Schurke J, Malooly RJ. 2008 Annual Report for the Washington State Fund: Washington's State-run Workers' Compensation Program.
Washington: Washington State Department of Labor & Industries; 2008. Available at: http://www.lni.wa.gov/IPUB/101–086–000.pdf
19. Roland M, Morris R. A study of the natural history of back pain. Part 1: development of a reliable and sensitive measure of disability in low-back pain. Spine
20. Carter ML. Spinal cord stimulation in chronic pain: a review of the evidence. Anaesth Intensive Care
21. Farrar JT, Young JP, LaMoreaux L, et al. Clinical importance of changes in chronic pain intensity measured on an 11-point numerical pain rating scale. Pain
30. Washington State Health Care Authority. Health Technology Assessment Program (HTA). Available at: http://www.hta.hca.wa.gov/
. Accessed August 26, 2009.
31. Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?” Lancet
32. Atkins D. Creating and synthesizing evidence with decision makers in mind-integrating evidence from clinical trials and other study designs. Med Care
33. Westfall JM, Mold J, Fagnan L. Practice-based research–“blue highways” on the NIH roadmap. JAMA
34. Chou R. Generating evidence on spinal cord stimulation for failed back surgery syndrome: not yet fully charged. Clin J Pain
35. Carino T, Sheingold S, Tunis S. Using clinical trials as a condition of coverage: lessons from the National Emphysema Treatment Trial. Clin Trials
36. Ware JE, Kosinski M, Dewey JE. How to Score Version Two of the SF-36 Health Survey. Lincoln, RI: QualityMetric Incorporated; 2000.
© 2010 Lippincott Williams & Wilkins, Inc.