Levels of Evidence in Research Published in The Journal of Bone and Joint Surgery (American Volume) Over the Last Thirty Years

Hanzlik, Shane MS; Mahabir, Raman C. MSc, MD, FRCSC; Baynosa, Richard C. MD; Khiabani, Kayvan T. MD, MSc, FRCSC

Journal of Bone & Joint Surgery - American Volume:
doi: 10.2106/JBJS.H.00108
Scientific Articles
Abstract

Background: The recent emphasis on evidence-based medicine has led to increasing levels of evidence being published in surgical journals. The purpose of the present study was to review the levels of evidence in reports published in The Journal of Bone and Joint Surgery (American Volume) over the last thirty years.

Methods: We reviewed all of the articles published in The Journal in the years 1975, 1985, 1995, and 2005. Cadaver studies, animal studies, basic-science studies, review articles, Instructional Course Lectures, and correspondence were excluded. Articles were scored according to The Journal's levels of evidence for a primary research question.

Results: A total of 1058 articles were reviewed. Of these, 134, 123, 120, and 174 articles met the inclusion criteria for the years 1975, 1985, 1995, and 2005, respectively, and were ranked according to level of evidence. The number of articles for each level of evidence rating was then expressed as a percentage of the total number of articles meeting the inclusion criteria for that year. There was a significant trend toward higher levels of evidence, with the combined percentage of Level-I, II, and III studies increasing from 17% to 52% (p < 0.01). The percentage of Level-I studies increased from 4% in 1975 to 21% in 2005. The average level of evidence rating improved from 3.72 to 2.90 during the study period.

Conclusions: The level of evidence in The Journal has improved significantly over the last thirty years.

Author Information

1University of Nevada School of Medicine, 2040 West Charleston Boulevard, Suite 301, Las Vegas, NV 89102

2Division of Plastic Surgery, Scott and White Memorial Hospital, 2401 South 31st Street, Temple, TX 76508. E-mail address: rmahabir@swmail.sw.org

Article Outline

The goal of medical research—to improve patient care and outcomes—is best realized when the methods that are utilized are subject to the least possible error or bias. With the popularization of evidence-based medicine, a standardized system of interpreting medical research was proposed by Sackett1. Research designs were assigned a level of evidence, with Level I being the highest (prospective randomized blinded controlled trials) and Level V being the lowest (case reports)1. This system introduced both authors and readers to the concept of level of evidence, with the underlying emphasis that certain levels of evidence were more relevant than others. Evidence-based medicine quickly became an integral part of the practice of clinical research, and the randomized controlled trial was recognized as the gold standard. Since January 2003, The Journal of Bone and Joint Surgery (American Volume) has included a level of evidence rating for each of its clinical scientific papers2.

The purpose of the present study was to evaluate the changes in the level of evidence of research published in The Journal from 1975 to 2005 and to document changes in the quality of research from practices before the development of evidence-based medicine to the present.

Back to Top | Article Outline

Materials and Methods

Articles that were published in The Journal of Bone and Joint Surgery (American Volume) in the years 1975, 1985, 1995, and 2005 were reviewed. Excluded from this analysis were any publications found under the heading of Basic Science, Cadaver Studies, Editorials, Continuing Medical Education, Special Topics, Correspondence and Communication, Current Concept Reviews, and Instructional Course Lectures. Each article was assigned a level of evidence by the lead author (S.H.) according to The Journal's levels of evidence for a primary research question2. Senior authors (R.C.M. and K.T.K.) randomly reviewed one of every ten articles to provide interobserver reliability. No inconsistencies were found in the reviewed articles. Reviewers were not blinded. The format of The Journal has changed substantially over the years and limited the ability to blind the reviewers.

Back to Top | Article Outline
Statistical Methods

A chi-square test was used to analyze percentages across the decades, and the Cochran-Armitage test was used to test for trends. Descriptive statistics were also used. Results are expressed as percentages.

Back to Top | Article Outline
Source of Funding

No external funding was received for this study.

Back to Top | Article Outline

Results

A total of 1058 articles were reviewed. Of these, 134, 123, 120, and 174 articles met the inclusion criteria for the years 1975, 1985, 1995, and 2005, respectively, and were ranked according to level of evidence (Fig. 1). The number of articles for each level-of-evidence rating was then expressed as a percentage of the total number of articles meeting the inclusion criteria for that year (Fig. 2). Data are presented in Table I. There was a significant trend toward higher levels of evidence, with the combined percentage of Level-I, II, and III studies increasing from 17% (twenty-three of 134) in 1975 to 52% (ninety-one of 174) in 2005 (p < 0.01). The percentage of Level-I studies increased from 4% (five of 134) in 1975 to 21% (thirty-seven of 174) in 2005. The average level of evidence improved from 3.72 to 2.90 during the study period.

Back to Top | Article Outline

Discussion

The results of the present study demonstrate a clear trend toward improved levels of evidence in The Journal from 1975 to 2005. A recent study showed that other orthopaedic surgery journals have begun to increase their publication of Level-I and II studies as well3. Despite the many obstacles to randomized trial research in surgical practice, investigators are increasingly finding ways to design and conduct randomized controlled trial studies in orthopaedics. This increase in level of evidence has been shown to directly affect the reliability and confidence with which others view a journal, as measured on the basis of the journal impact factor3. The impact factor of The Journal increased from 1.22 to 2.33 over the same time period4.

This trend compares favorably with the research published in other surgical disciplines. In the general surgery literature, for example, Diseases of the Colon and Rectum contained five, thirteen, and seventeen randomized controlled trials for the years 1990, 1995, and 2000, respectively5. A review of Plastic and Reconstructive Surgery revealed that only seven (1.5%) of 459 articles published in the year 2003 were randomized controlled trials and demonstrated an improvement in the mean level of evidence from 4.42 to 4.16 from 1983 to 20036. In The Journal of Bone and Joint Surgery (American Volume), twenty-seven (16%) of 174 studies published in 2005 were randomized controlled trial studies and the mean level of evidence improved from 3.72 to 2.90 between 1975 and 2005.

The question remains as to why the average level of evidence in surgical journals lags behind those in medical journals. While the scope of this question is outside the objective of the present study, two key issues are pertinent. Surgical researchers often claim that a potential ethical barrier limits the use of sham procedures in surgical research7. However, with the proper design, randomized trials with sham controls can be performed, and the effectiveness of such trials was demonstrated in a study of arthroscopic knee débridement for the treatment of osteoarthritis8. A second, frequently cited potential barrier to the use of randomized controlled trials in surgical practice is the variable nature of a particular procedure in a particular patient. The application of a rigid surgical methodology may actually undermine the effectiveness of a technique and negatively impact the measurable outcome9.

Researchers with predominantly clinical training do not perform as well in the grant application process as do PhD-researchers10-13. In an effort to increase the volume of research, surgical training programs have placed an emphasis on research. This emphasis was shown in the Association of American Medical Colleges study entitled “Charting Outcomes in the Match,” which demonstrated that, in 2006, the majority of applicants accepted into orthopaedic residency had at least one publication14. Prospective candidates must demonstrate their commitment through publication to increase the probability of matching to a residency position14. To circumvent the lack of time and funding to conduct higher-level-of-evidence studies, they demonstrate their commitment to research by conducting lower-level-of-evidence research, thereby diluting the level of evidence in the literature15.

Research can be improved with the use of strict inclusion and exclusion criteria; prospective patient enrollment; a standard treatment protocol; predetermined, well-defined outcome measures; follow-up of patients at specified time-intervals; and patient-derived validated instruments for functional assessment5,16. It is worth noting that many of the Level-IV studies that we reviewed would have become much stronger research (Level II or III) with the simple inclusion of a control group. Investigators should be encouraged to include a control group and thus increase the level of evidence. Of course, the best-case scenario for the future is to focus on more Level-I and II studies.

Two recent studies have examined the percentage of interventions in surgical practice that have been evaluated in randomized controlled trials. Those studies showed that, in general surgery, 11% to 24% of interventions were supported by randomized controlled trial data17,18. Solomon and McLeod provided an interesting addendum to this issue by calculating the percentage of clinical questions in a surgical practice that were amenable to study in a randomized controlled trial and found that as many as 40% of clinical questions were amendable to study in a randomized controlled trial9.

The present study is limited by the clinical bias of the ranking scale. The original level-of-evidence scale as described by Sackett1 was specifically designed to assess the clinical level of evidence. Basic-science research either was excluded or was ranked as the lowest level, regardless of the quality of the work. A second potential limitation of the present study is that each article was reviewed by the lead author. However, senior authors also randomly reviewed one of every ten articles to provide interobserver reliability. No inconsistencies were found in the articles reviewed. It is worth pointing out that, in two previous studies, the level of experience of the reviewer did not change the scoring of the article with regard to level of evidence as long as a level-of-evidence table was available at the time of the review3,19.

The Journal has shown an improvement in the level of evidence in its publications over the past thirty years. While the increase in high-level-of-evidence studies compares favorably with other surgical specialties, there remains substantial room for improvement, and it is the obligation of the surgeon-scientist community to continue to accomplish this improved quality of research.

Disclosure: The authors did not receive any outside funding or grants in support of their research for or preparation of this work. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the authors, or a member of their immediate families, are affiliated or associated.

Investigation performed at the Division of Plastic Surgery, University of Nevada School of Medicine, Las Vegas, Nevada

1. Sackett DL. Rules of evidence and clinical recommendations on the use of antithrombotic agents. Chest. 1986;89(2 Suppl):2S-3S.
2. Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85:1-3.
3. Obremskey WT, Pappas N, Attallah-Wasif E, Tornetta P 3rd, Bhandari M. Level of evidence in orthopaedic journals. J Bone Joint Surg Am. 2005;87:2632-8.
4. Institute for Scientific Information. 2007 journal citation reports. Philadelphia, PA: Thomson Reuters; 2008.
5. Ko CY, Sack J, Chang JT, Fink A. Reporting randomized, controlled trials: where quality of reporting may be improved. Dis Colon Rectum. 2002;45:443-7.
6. Loiselle F, Mahabir RC, Harrop AR. Levels of evidence in plastic surgery research over 20 years. Plast Reconstr Surg. 2008;121:207e-11e.
7. Stirrat GM. Ethics and evidence based surgery. J Med Ethics. 2004;30:160-5.
8. Moseley JB, O'Malley K, Petersen NJ, Menke TJ, Brody BA, Kuykendall DH, Hollingsworth JC, Ashton CM, Wray NP. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med. 2002;347:81-8.
9. Solomon MJ, McLeod RS. Should we be performing more randomized controlled trials evaluating surgical operations? Surgery. 1995;118:459-67.
10. McArdle CS, Hole D. Impact of variability among surgeons on postoperative morbidity and mortality and ultimate survival. BMJ. 1991;302:1501-5.
11. Bell PR. Surgical research. Lancet. 1996;347:1479; author reply 1482.
12. Beahrs OH. Clinical trials from the surgeon's view. Cancer. 1990;65(10 Suppl):2383-4.
13. Solomon MJ, Laxamana A, Devore L, McLeod RS. Randomized controlled trials in surgery. Surgery. 1994;115:707-12.
14. National Resident Matching Program; Association of American Medical Colleges. Charting outcomes in the match: characteristics of applicants who matched to their preferred specialty in the 2007 NRMP main residency match. 2nd ed. 2007 Aug. http://http://www.nrmp.org/data/chartingoutcomes2007.pdf. Accessed 2008 Oct 24.
15. Offer GJ, Perks AG. In search of evidence-based plastic surgery: the problems faced by the specialty. Br J Plast Surg. 2000;53:427-33.
16. Keller RB, Rudicel SA, Liang MH. Outcomes research in orthopaedics. Instr Course Lect. 1994;43:599-611.
17. Howes N, Chagla L, Thorpe M, McCulloch P. Surgical practice is evidence based. Br J Surg. 1997;84:1220-3.
18. Kenny SE, Shankar KR, Rintala R, Lamont GL, Lloyd DA. Evidence-based surgery: interventions in a regional paediatric surgical unit. Arch Dis Child. 1997;76:50-3.
19. Bhandari M, Swiontkowski MF, Einhorn TA, Tornetta P 3rd, Schemitsch EH, Leece P, Sprague S, Wright JG. Interobserver agreement in the application of levels of evidence to scientific papers in the American volume of the Journal of Bone and Joint Surgery. J Bone Joint Surg Am. 2004;86:1717-20.
Copyright 2009 by The Journal of Bone and Joint Surgery, Incorporated