The Highest Level of Evidence in a High Impact Journal: Is This the Final Verdict?

Fisher, Charles G. MHSc, MD*; Vaccaro, Alexander R. MD, PhD†

doi: 10.1097/BRS.0b013e3181e41f87
Systematic Reviews
Author Information

From the *Combined Neurosurgical & Orthopaedic Spine Program, Department of Orthopaedics, University of British Columbia, Vancouver, BC, Canada; and †Departments of Orthopaedic Surgery and Neurological Surgery, Thomas Jefferson University and The Rothman Institute, Philadelphia, PA.

The device(s)/drug(s) is/are FDA-approved or approved by corresponding national agency for this indication.

No funds were received in support of this work. No benefits in any form have been or will be received from a commercial party related directly or indirectly to the subject of this manuscript.

Address correspondence and reprint requests to Alexander R. Vaccaro, MD, PhD, Department of Orthopedic Surgery, The Rothman Institute, Philadelphia, PA; E-mail:

Article Outline

Evidence-based medicine involves the deliberate integration of clinical research into therapeutic decision making.1 A prospective, randomized control trial (PRCT) is assumed to equally distribute unknown confounding variables and only manipulate the “treatment variable.” In medical PRCTs, this usually occurs in a well-defined population to determine efficacy; that is, does the intervention work on the participants in the study (internal validity)? In most medical (drug) studies, the greater challenge is determining whether the results can be extended to patients not in the study, so called external validity. Surgical or interventional studies face the same generalizabilty or external validity issues; however, one of the greatest challenges in surgical trials is patient recruitment, and the establishment of a valid study population to ensure internal validity.

The recent history of medicine has been punctuated by PRCTs, which have established, reinforced, and challenged traditional clinical beliefs.2 One example of the astounding effect of level I trials is the recent literature on percutaneous vertebroplasty (PVP) compared with sham procedures3,4 and to conservative treatment5 for treatment of osteoporotic vertebral compression fractures. Many clinicians experience significant cognitive dissonance6 between the astounding early clinical improvement of many patients and the popular media perception of the results of these trials.

An excellent review in this issue of the journal effectively dissects 2 of these studies and highlights strengths and weaknesses.5 However, the impact of these studies and others really distills down to the difficulties with establishing both internal and external validity.

The major challenge of PRCTs is the requirement of equipoise by both patients and physicians regarding 2 apparently equally effective treatments. Clinicians may be biased to recommend direct interventions to some patients and only enroll patients in whom there is less severe intensity of symptoms, although this is controlled for once the study begins, the bias can be still appear through the inclusion-exclusion criteria developed by the clinicians. It is often the inclusion-exclusion measures that are adjusted based on expected recruitment. Patient consent to participate in interventional studies is by far the greatest challenge as systematic differences evolve between patients willing to participate in randomization. There may be significant differences in risk taking, expectations, and perseverance in people who are willing to relegate their treatment to chance versus patients who refused to participate. This volunteer bias would be akin to selection bias in observational studies. Most interventional studies have roughly 33% enrollment of those patients eligible. Patient preference is a big part of evidence-based medicine, and perhaps, the reason for the observational study advocates touting it as the best methodology for ensuring external validity. This dilemma was addressed in the Spine Patient Outcomes Research Trial by including an observational cohort to capture patients who were not enrolled in the prospective, randomized study.7,8

PRCTs are expected to have appropriate and accurate long term follow-up. However, in some situations long term follow-up may be less relevant than early functional results. For example, in the orthopedic literature, the natural history of most femur fractures is healing by 6 to 12 months regardless of treatment.9,10 The goal of internal fixation is early mobilization and pain control to avoid the sequelae of prolonged immobilization, possibly at the expense of soft tissue stripping and healing. Similarly, because the vertebral bodies have an excellent blood supply and soft tissue envelope, it is not surprising that the natural history of vertebral compression fractures is healing by 6 to 12 months. Is it fair to judge the long-term outcome of vertebroplasty compared with conservative treatment?

Would anyone for go internal fixation of a femur fracture because of the equivocal long-term fracture healing? The real question is whether the “internal fixation” facilitates substantial early pain improvement and mobilization. Indeed, early pain relief after PVP has been observed in all of the PVP trials.11 An important overlooked result in the Rousing et al study was the significant (1.3 point, P < 0.02) improvement in Barthel functional score in the PVP group over the conservative group at 12 months. This functional improvement may be a reflection of earlier pain control and mobilization compared with conservatively treated patients.

In practicing evidence-based medicine, we as clinicians and researchers must be wary of assigning truths to PRCTs, even if published in high impact journals. The so-called pinnacle of the research design hierarchy is not immune to methodological limitations often camouflaged by the unique and coveted PRCT design for evaluation of a surgical intervention. Common sense, continued investigation, and an appreciation of the principles around internal and external validity of high-level evidence studies will hopefully guide our practices in the future of spinal care.

Back to Top | Article Outline


1. Sackett DL, Rosenberg WM, Gray JA, et al. Evidence based medicine: what it is and what it isn't. BMJ 1996;312:71–2.
2. Mandell BF. Evidence, limes, and cement. Cleve Clin J Med 2010;77:7.
3. Kallmes DF, Comstock BA, Heagerty PJ, et al. A randomized trial of vertebroplasty for osteoporotic spinal fractures. N Engl J Med 2009;361:569–79.
4. Buchbinder R, Osborne RH, Ebeling PR, et al. A randomized trial of vertebroplasty for painful osteoporotic vertebral fractures. N Engl J Med 2009;361:557–68.
5. Rousing R, Hansen KL, Andersen MO, et al. Twelve-months follow-up in forty-nine patients with acute/semiacute osteoporotic vertebral fractures treated conservatively or with percutaneous vertebroplasty: a clinical randomized study. Spine (Phila Pa 1976) 2010;35:478–82.
6. Orr RD. Vertebroplasty, cognitive dissonance, and evidence-based medicine: what do we do when the ‘evidence’ says we are wrong? Cleve Clin J Med 2010;77:8–11.
7. Weinstein JN, Lurie JD, Tosteson TD, et al. Surgical vs nonoperative treatment for lumbar disk herniation: the Spine Patient Outcomes Research Trial (SPORT) observational cohort. JAMA 2006;296:2451–9.
8. Weinstein JN, Tosteson TD, Lurie JD, et al. Surgical vs nonoperative treatment for lumbar disk herniation: the Spine Patient Outcomes Research Trial (SPORT): a randomized trial. JAMA 2006;296:2441–50.
9. Thomas TL, Meggitt BF. A comparative study of methods for treating fractures of the distal half of the femur. J Bone Joint Surg Br 1981;63B:3–6.
10. St Pierre RK, Fleming SS, Fleming LL. Fractures of the femoral shaft: a prospective study of closed intramedullary nailing, modified open intramedullary nailing, and cast-bracing. South Med J 1982;75:827–35.
11. Rousing R, Andersen MO, Jespersen SM, et al. Percutaneous vertebroplasty compared to conservative treatment in patients with painful acute or subacute osteoporotic vertebral fractures: three-months follow-up in a clinical randomized study. Spine (Phila Pa 1976) 2009;34:1349–54.
© 2010 Lippincott Williams & Wilkins, Inc.