Journal Logo

An Evaluation of Publication Bias in High-Impact Orthopaedic Literature

Scott, Jared BS1; Checketts, Jake X. BS1; Cooper, Craig M. BS1; Boose, Marshall DO2; Wayant, Cole BS1; Vassar, Matt PhD1

doi: 10.2106/JBJS.OA.18.00055
Scientific Articles
Open
SDC
Disclosures

Background: Statistical analysis of systematic reviews allows the results of previous studies to be combined and synthesized to assess the overall health effect of the intervention in question. Systematic reviews can also be used to guide the creation of clinical practice guidelines and are considered to have a high level of evidence. Thus, it is important that their methodological quality is of the highest standard. Publication bias presents 2 problems: (1) studies with significant results may be overrepresented in systematic reviews and meta-analyses (“false positives”) and (2) studies without significant results may not be included in systematic reviews and meta-analyses (“false negatives”) because each study, on its own, was underpowered, meaning that some treatment options that may have clinical benefit will not be adopted.

Methods: We performed a study to evaluate the techniques used by authors to report and evaluate publication bias in the top 10 orthopaedic journals as well as 3 orthopaedic-related Cochrane groups. Two authors independently screened the titles and abstracts to identify systematic reviews and meta-analyses. We assessed publication bias in the systematic reviews that did not assess publication bias themselves.

Results: Our final sample included 694 systematic reviews or meta-analyses that met our inclusion criteria. Our review included 502 studies (72%) that focused on clinical outcomes, with the majority of the remaining studies focused on predictive and prognostic accuracy (20%) or diagnostic accuracy (5%). Publication bias was discussed in 295 (42.5%) of the included studies and was assessed in 135 (19.5%). Of the studies that assessed publication bias, 31.9% demonstrated evidence of publication bias. Only 43% and 22% of studies that involved use of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines discussed and assessed publication bias, respectively.

Conclusions: Publication bias is infrequently discussed and assessed in the high-impact orthopaedic literature. Furthermore, nearly one-third of the studies that assessed for publication bias demonstrated evidence of publication bias. In addition to these shortcomings, fewer than half of these studies involved use of the PRISMA guidelines and yet only one-fourth of the studies assessed for publication bias.

Clinical Relevance: By understanding the degree to which publication bias is discussed and presented in high-impact orthopaedic literature, changes can be made by journals and researchers alike to improve the overall quality of research produced and reported.

1Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma

2Department of Orthopaedic Surgery, Oklahoma State University Medical Center, Tulsa, Oklahoma

E-mail address for J.X. Checketts: jake.checketts@okstate.edu

Investigation performed at the Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma

Disclosure: The authors indicated that no external funding was received for any aspect of this work. The Disclosure of Potential Conflicts of Interest forms are provided with the online version of the article (http://links.lww.com/JBJSOA/A92).

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

The quality of a systematic review and meta-analysis is directly proportional to its methodological quality and the quality of the individual studies it includes1,2. A potential shortcoming in systematic reviews and meta-analyses is the overrepresentation of studies with significant results. This overrepresentation is a remnant of publication bias in clinical trials. Publication bias results from the non-publication of studies that do not demonstrate significant results or studies that demonstrate negative results. However, even in the most methodologically sound systematic review, low levels of evidence and underpowered studies in the literature can still lead to distorted conclusions3 because of the overrepresentation of studies with significant results (“false positives”). Furthermore, treatment options that may have clinical benefit will not be adopted because studies without statistical significance may not be included in systematic reviews and meta-analyses because each study on its own was underpowered (“false negatives”). It has been demonstrated that, despite a large increase in the quality and quantity of orthopaedic studies over time, the levels of evidence still remain low4.

Evaluation of publication bias within systematic reviews is mentioned in many pertinent reporting guidelines5-7. For instance, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines clearly state that authors should “specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies)”7. Adherence to the PRISMA guidelines is recommended by many of the journals included in the present study8. Despite this recommendation, it has been found that publication bias is underreported in systematic reviews published in high-impact journals and that there are high levels of publication bias in those journals9. Studies have shown that, in systematic reviews published in anesthesiology and dermatology journals, assessment for publication bias is rare10,11. The consequences of not evaluating for publication bias can lead to treatments based on only part of the available evidence and can potentially lead to serious clinical consequences.

A powerful example of the ramifications of unchecked publication bias is the use of reboxetine for the treatment of depression. As a result of the positive findings of initial research, mainly sponsored by the pharmaceutical industry, reboxetine was approved for the acute treatment of major depression in the United Kingdom and parts of Europe. However, a systematic review that included more unpublished data regarding the safety and efficacy of the drug found that it was ineffective and in fact was harmful as an antidepressant12. This is a striking example of the potential harm that publication bias can cause and illustrates the urgent need for more transparency and accountability in published research.

Given the prevalence of publication bias in other medical specialties in addition to findings of publication bias in orthopaedic and surgical research13, the primary objective of the present study was to assess the method by which publication bias is evaluated within high-ranking orthopaedic journals. In instances in which publication bias had been evaluated, our secondary objective was to perform post-hoc publication bias assessments of the systematic reviews that met certain inclusion criteria.

Back to Top | Article Outline

Materials and Methods

We performed a study to evaluate the techniques used by authors to report and evaluate publication bias in the top 10 orthopaedic journals as well as 3 orthopaedic-related Cochrane groups: (1) the Bone, Joint and Muscle Trauma group, (2) the Back and Neck group, and (3) the Musculoskeletal group. The top 10 orthopaedic journals, which were identified by consulting Google Scholar’s H5-Index scores for “Orthopedic Medicine & Surgery”, included The American Journal of Sports Medicine; The Journal of Bone & Joint Surgery; Spine; Clinical Orthopaedics and Related Research; Arthroscopy: The Journal of Arthroscopic and Related Surgery; Knee Surgery, Sports Traumatology, Arthroscopy; The Journal of Arthroplasty; European Spine Journal; Journal of Shoulder and Elbow Surgery; and International Orthopaedics. The present study includes only previously published research and therefore is not subject to institutional review board oversight. We applied Statistical Analyses and Methods in the Published Literature (SAMPL) guidelines for reporting descriptive statistics when necessary.

Back to Top | Article Outline

Search Strategy

On August 11, 2017, we conducted a search of PubMed for systematic reviews and meta-analyses that had been published between 2013 and 2016. The search string is available in the Appendix.

We located relevant Cochrane reviews from the selected groups by accessing the associated Cochrane Library web page for the group and manually extracting references for those reviews that met our inclusion criteria.

Using Rayyan (Qatar Computing Research Institute), a tool for optimizing work flow of systematic reviews, 2 authors independently screened the titles and abstracts to identify systematic reviews and meta-analyses. We determined an article to be a systematic review if the methods involved summarizing evidence across multiple studies that had been identified through a comprehensive, well-described literature search. Meta-analyses were defined as any quantitative synthesis of data from multiple studies. We did not include any primary studies (i.e., randomized controlled trials, cohort studies, case-control studies), reviews of other reviews, letters to the editor, or commentaries in our review. We used Paperpile software to retrieve and access full texts of the studies meeting our inclusion criteria.

Back to Top | Article Outline

Data Extraction and Analysis

Using a pilot-tested Google Form, we extracted elements from each systematic review and/or meta-analysis. We stored the results in Microsoft Excel. Data were analyzed with use of Microsoft Excel and Comprehensive Meta-Analysis software (Biostat).

Back to Top | Article Outline

Assessing for Publication Bias

We assessed publication bias in the systematic reviews from our sample that did not assess publication bias themselves and that met the following eligibility criteria: (1) the systematic review had to include a meta-analysis, (2) the meta-analysis had to include ≥10 primary studies, and (3) sufficient data from the primary studies (e.g., odds ratios, risk ratios, mean differences, or other effect size measurements with confidence intervals) had to be included in the systematic review. Our minimum required size of 10 primary studies was based on previous research in The BMJ14 indicating that statistical power is too low to distinguish chance from actual asymmetry in <10 studies. In the event that a systematic review had >1 meta-analysis that met our inclusion criteria, we assessed publication bias of the meta-analysis with the most primary studies. When necessary in order to replicate the analyses and perform publication bias assessments, we made minor adjustments to the upper confidence interval until symmetry was reached. We used funnel plots, the Egger regression test, and the Duval and Tweedie trim-and-fill method to assess publication bias in those meta-analysis that met inclusion criteria. We set p < 0.05 as the significance level for the Egger regression test. All publication bias assessments were performed with use of Comprehensive Meta-Analysis software.

Back to Top | Article Outline

Search Results

Our PubMed and Cochrane Library searches returned 848 and 74 studies, respectively (Fig. 1). We uploaded our search results to Rayyan and screened the returned studies by title and abstract, yielding 717 studies. The retained studies were uploaded to Paperpile to assist in full-text retrieval and data extraction. Our final sample included 694 systematic reviews or meta-analyses that met our inclusion criteria.

Fig. 1

Fig. 1

Back to Top | Article Outline

Results

Characteristics of Included Studies

Our review includes 502 studies (72%) that focused on clinical outcomes, with the majority of the remaining studies focused on predictive and prognostic accuracy (20%) or diagnostic accuracy (5%). The majority of studies included >1 study type, with 56% of the systematic reviews including randomized controlled trials. Studies from 32 countries, most frequently the United States (206 studies; 30%) and China (138 studies; 20%) were included. Fifty percent (346) of the included studies mentioned the use of the PRISMA reporting guidelines. Grey literature searches (171 studies; 24.6%) and bibliography or hand searches (559 studies; 80.5%) were common methods used to reduce publication bias.

Back to Top | Article Outline

Publication Bias Reporting

Publication bias was discussed in 295 (42.5%) of the included studies and was assessed in 135 (19.5%) (Table I). The studies by the 3 Cochrane groups discussed publication bias 90% of the time, whereas the studies in Arthroscopy discussed publication bias only 18.8% of the time. Publication bias was assessed most often in The Journal of Arthroplasty (36.4%) and least often in Arthroscopy (6.0%) (Table II). Construction and visual analysis of funnel plots was the most common method for assessing publication bias (107 studies; 79.3%). Other methods for assessing publication bias included the Begg rank correlation test, Egger regression, Duval and Tweedie trim-and-fill test, and Rosenthal failsafe-N test. Of the studies that assessed publication bias, 31.9% reported evidence of publication bias and another 3.7% did not specify or were inconclusive as to whether publication bias was present. Only 43% and 22% of studies that were performed with use of the PRISMA guidelines discussed and assessed publication bias, respectively.

TABLE I - Publication Bias Variables and Counts
No. of Studies
Variable Yes No Unspecified
Publication bias discussed (n = 694) 295 (42.5%) 399 (57.5%)
Publication bias evaluated (n = 694) 135 (19.5%) 559 (80.5%)
Funnel plots presented (n = 135) 107 (79.3%) 28 (20.7%)
Publication bias present (n =135) 43 (31.9%) 87 (64.4%) 5 (3.7%)
Grey literature search (n = 694) 171 (24.6%) 523 (75.4%)
Bibliography or hand search (n = 694) 559 (80.5%) 133 (19.2%) 2 (0.3%)

TABLE II - Publication Bias Results by Journal
No. of Studies
Journal Yes No
Publication bias discussed
The American Journal of Sports Medicine (n = 72) 16 (22.2%) 56 (77.8%)
The Journal of Bone & Joint Surgery (n = 17) 8 (47.1%) 9 (52.9%)
Spine (n = 49) 25 (51.0%) 24 (49.0%)
Clinical Orthopaedics and Related Research (n = 56) 26 (46.4%) 30 (53.6%)
Arthroscopy: The Journal of Arthroscopic and Related Surgery (n = 117) 22 (18.8%) 95 (81.2%)
Knee Surgery, Sports Traumatology, Arthroscopy (n = 114) 35 (30.7%) 79 (69.3%)
The Journal of Arthroplasty (n = 55) 27 (49.1%) 28 (50.9%)
European Spine Journal (n = 82) 47 (57.3%) 35 (42.7%)
Journal of Shoulder and Elbow Surgery (n= 18) 8 (44.4%) 10 (55.6%)
International Orthopaedics (n = 44) 18 (40.9%) 26 (59.1%)
 Cochrane groups (n = 70) 63 (90.0%) 7 (10.0%)
Publication bias assessed
The American Journal of Sports Medicine (n = 72) 12 (16.7%) 60 (83.3%)
The Journal of Bone & Joint Surgery (n = 17) 5 (29.4%) 12 (70.6%)
Spine (n = 49) 17 (34.7%) 32 (65.3%)
Clinical Orthopaedics and Related Research (n = 56) 15 (26.8%) 41 (73.2%)
Arthroscopy: The Journal of Arthroscopic and Related Surgery (n = 117) 7 (6.0%) 110 (94.0%)
Knee Surgery, Sports Traumatology, Arthroscopy (n = 114) 16 (14.0%) 98 (86.0%)
The Journal of Arthroplasty (n = 55) 20 (36.4%) 35 (63.6%)
European Spine Journal (n = 82) 21 (25.6%) 61 (74.4%)
Journal of Shoulder and Elbow Surgery (n = 18) 5 (27.8%) 13 (72.2%)
International Orthopaedics (n = 44) 11 (25.0%) 33 (75.0%)
 Cochrane groups (n = 70) 6 (8.6%) 64 (91.4%)

Back to Top | Article Outline

Publication Bias Assessments

We found 52 studies that met our inclusion criteria for publication bias assessment. Sufficient information was available in 41 systematic reviews to perform publication bias assessment (see Appendix). Funnel plot asymmetry was present in 36 analyses (88%). Studies were most often missing from the left side of the pooled analysis (54%). The Duval and Tweedie trim-and-fill method indicated that a mean of 3.5 added studies (range, 0 to 12 added studies) would be needed to produce symmetry of the funnel plot. Fourteen meta-analyses (34%) had evidence of publication bias when assessed with Egger regression, whereas 9 (22%) had evidence when assessed with Begg rank correlation. Evidence of publication bias from all 3 methods was noted in 7 studies (17%). No evidence of publication bias from any of the 3 methods was found in 3 studies (7%). The mean absolute percent difference between adjusted and observed point estimates was 17.2% (median, 9.2%; range, 0% to 123.1%) (Table III).

TABLE III - Publication Bias Assessment Results*
Article Begg P Value No. of Studies Trimmed Side of Mean Missing Model Observed Point Estimate (95% CI) Adjusted Point Estimate (95% CI) Egger Intercept T Value
Andriolo et al. 0.06 2 Left Random 78.93 (71.13 to 86.74) 77.15 (69.63 to 84.67) 3.98 2.75
Avenell et al. 0.36 1 Right Random 0.74 (0.49 to 1.11) 0.77 (0.51 to 1.17) −0.17 0.28
Beckmann et al. 0.0001 5 Left Fixed 0.0019 (−0.006 to 0.0099) 0.0014 (−0.0064 to 0.0093) 0.47 5.19
Beckmann et al. 0.02 8 Left Random 0.0076 (0.0047 to 0.01) 0.0069 (0.004 to 0.0098) 1.02 5.54
Beckwee et al. 0.27 2 Left Random 1.09 (1.04 to 1.14) 1.07 (1.01 to 1.13) 1.30 1.75
Buckland et al. 0.31 12 Left Fixed 53.71 (53.47 to 53.95) 50.91 (50.68 to 51.15) 21.99 2.99
Chee et al. 0.26 3 Left Random 0.79 (0.46 to 1.39) 0.70 (0.42 to 1.16) −0.12 0.28
Desai et al. 0.47 0 NA Random 0.36 (0.22 to 0.51) 0.36 (0.22 to 0.51) −0.37 0.36
Hewison et al. 0.15 1 Right Random 0.49 (0.32 to 0.78) 0.51 (0.33 to 0.80) −0.93 1.02
Higgins et al. 0.33 0 NA Random 0.43 (−0.04 to 0.90) 0.43 (−0.04 to 0.90) 0.66 0.19
Houwert et al. 0.20 0 NA Random 1.32 (0.65 to 2.67) 1.32 (0.65 to 2.67) −0.62 0.58
Kamper et al. 0.24 3 Left Random 1.02 (0.74 to 1.40) 0.96 (0.70 to 1.32) 0.54 1.14
Kang et al. 0.31 4 Left Random 0.03 (0.02 to 0.04) 0.02 (0.01 to 0.03) 0.75 1.94
Keurentjes et al. 0.00006 3 Left Random 91.53 (89.53 to 93.53) 90.49 (87.63 to 93.36) −6.11 3.29
Kim et al. 0.45 1 Right Random 1.06 (0.98 to 1.15) 1.07 (0.98 to 1.18) 0.33 0.44
Kim et al. 0.31 6 Right Random 0.32 (0.14 to 0.49) 0.49 (0.31 to 0.67) 1.97 0.95
Li et al. 0.35 4 Left Random −0.13 (−0.51 to 0.25) −0.29 (−0.62 to 0.03) 5.41 1.43
Lieberman et al. 0.00001 0 NA Random −0.91 (−1.21 to −0.62) −0.91 (−1.21 to −0.62) −4.53 6.35
Liu et al. 0.09 2 Right Fixed 1.37 (0.99 to 1.88) 1.43 (1.05 to 1.95) −0.53 1.29
Ma et al. 0.21 3 Right Random 0.36 (0.08 to 0.65) 0.54 (0.24 to 0.85) 1.74 1.11
Medina et al. 0.0004 7 Left Random 17.94 (13.16 to 22.72) 12.98 (7.72 to 18.24) 2.68 3.98
Meijer et al. 0.24 2 Right Fixed 1.07 (0.78 to 1.48) 1.20 (0.88 to 1.64) −1.38 1.06
Negahban et al. 0.003 7 Left Fixed 0.36 (0.26 to 0.45) 0.24 (0.14 to 0.33) 3.69 2.72
Pavon et al. 0.33 4 Right Random 0.93 (0.58 to 1.47) 1.33 (0.81 to 2.19) −1.01 1.39
Peersman et al. 0.02 4 Left Random 0.0104 (0.0085 to 0.0123) 0.0103 (0.0083 to 0.0122) 0.51 1.71
Rebal et al. 0.15 2 Left Random 0.14 (0.09 to 0.18) 0.12 (0.07 to 0.17) 1.66 0.83
Riboh et al. 0.44 2 Left Random 1.28 (0.46 to 3.58) 1.15 (0.43 to 3.12) 0.29 1.01
Santesso et al. 0.07 6 Left Fixed 0.91 (0.88 to 0.94) 0.85 (0.83 to 0.88) 2.58 1.62
Saragiotto et al. 0.07 4 Right Random −0.49 (−0.70 to −0.29) −0.34 (−0.56 to −0.12) −1.69 1.79
Schneider et al. 0.00005 7 Left Random 0.012 (0.0028 to 0.021) 0.0085 (−0.000070 to 0.02) 1.12 4.28
Si et al. 0.29 1 Left Fixed 1.42 (1.07 to 1.88) 1.39 (1.06 to 1.84) 0.29 0.55
Song et al. 0.20 3 Right Fixed 2.45 (2.40 to 2.50) 2.48 (2.43 to 2.53) −0.51 0.48
Towle et al. 0.21 5 Left Random 1.29 (1.15 to 1.47) 1.13 (0.98 to 1.29) 0.48 0.35
Van Bodegom Vos et al. 0.17 2 Right Random 0.49 (0.39 to 0.59) 0.51 (0.41 to 0.62) −1.06 1.46
Wiggins et al. 0.04 6 Left Random 0.06 (0.05 to 0.07) 0.05 (0.04 to 0.06) 1.42 2.39
Woodmass et al. 0.11 4 Right Random 0.41 (0.31 to 0.51) 0.47 (0.37 to 0.57) −1.39 1.98
Wyles et al. 0.16 6 Left Fixed 0.12 (0.08 to 0.19) 0.07 (0.05 to 0.10) 2.51 6.10
Xiao et al. 0.26 1 Right Fixed 0.56 (0.41 to 0.76) 0.58 (0.43 to 0.78) −1.56 1.36
Yang et al. 0.16 7 Right Fixed −6.78 (−7.78 to −5.78) −4.90 (−5.79 to −4.01) −1.07 1.12
Yun et al. 0.21 0 NA Random 4.40 (3.19 to 5.61) 4.40 (3.19 to 5.61) 0.77 0.87
Zhang et al. 0.31 2 Left Random 0.24 (−0.11 to 0.59) 0.15 (−0.23 to 0.53) 1.87 1.22
*
CI = confidence interval, and NA = not applicable.
The references for these studies can be found in the Appendix.

Back to Top | Article Outline

Discussion

The primary objective of the present study was to assess the methods used to evaluate for publication bias in high-ranking orthopaedic journals as well as in journals from the orthopaedic-related Cochrane groups. We found that even though publication bias was discussed in 43% of the reviews published in these journals, a formal assessment for publication bias only occurred in roughly 20% of these reviews.

Our study demonstrated a higher percentage of reviews lacking formal publication bias assessment than the findings of other studies in anesthesiology11 and dermatology10, which demonstrated that publication bias was assessed in 43% and 21% of reviews, respectively. The same can be said in comparison with a similar study published in 20149 that investigated rates of publication bias assessments across multiple medical specialties. Of the 694 reviews that were included in our study, 559 gave no indication if publication bias affected their results. Of the studies that did formally evaluate for publication bias, 31.9% reported evidence of it. It appears that, compared with other fields of medical research, the orthopaedic literature is assessing for publication bias in systematic reviews at a lower rate. This finding is concerning given the disproportionately high levels of studies with positive findings in the surgical literature13. The effects of the overrepresentation of studies with positive findings as the only studies that should be investigated for clinical benefit leads us to the other shortcoming of publication bias: a drug or treatment that actually has substantial clinical benefit may not be adopted because, although several independent studies indicated a positive clinical effect, each study on its own lacked the power (did not include enough patients) to show this effect as being statistically significant (e.g., p < 0.05). For example, if 3 independent studies were to show a similar substantial improvement in patient outcome, but the p value in each study was p ≥ 0.05, the individual studies would be reported as demonstrating “no significant effect.” However, if the 3 studies were to be combined using meta-analysis, their combined p value may be significant.

There are many ways to prevent the effect of publication bias in medical literature. The driving force to implement these measures must come from the journals and publishers15-17. One method to increase the number of relevant studies included in systematic reviews is a search of the grey literature18. Grey literature has been defined as “that which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers,” as defined by the Luxembourg Convention on Grey Literature (i.e., conference proceedings, technical documents or government reports).19 Although searching for grey literature can be difficult, it provides a more complete view of the evidence. There are several guides that provide both reliable sources of grey literature and sound search methods18,20-23. Furthermore, publication bias also could be avoided if authors were required by journals to report the total sample size, the magnitude of the observed effect (mean and standard deviation), and the magnitude of the associated p value (and/or confidence interval), regardless of whether the p value was above or below an arbitrary value (e.g., p < 0.05) specified as corresponding to “statistical significance.” This would allow reviewers to more accurately assess the total magnitude of the observed effect and the significance of a study’s findings and would permit a meta-analysis that includes all of the studies, rather than just the ones with individually positive outcomes. In addition, we believe that if the medical community were to abandon the arbitrary use of the p value as a measure of “success,” the degree of publication bias would drastically decrease, as proposed in recent high-profile literature24-26. Such efforts have the power to mitigate the proven negative effects of publication bias and prevent an ineffective or harmful treatment from becoming the standard, or conversely, a potentially effective treatment from being dismissed as having “no significant effect” simply because the study or studies examining it individually lacked the power (i.e., sufficient numbers of patients) to show the effect at p < 0.05, or any other arbitrary value. As systematic reviews are considered high-level evidence in the creation of clinical practice guidelines, there is a need to address publication bias.

Reporting guidelines for systematic reviews also have the potential to address the issue of publication bias. However, these guidelines can be effective only if they are used properly and if adherence to the guideline is required by the journal. In our study, 50% of the reviews mentioned the use of the PRISMA guideline. Of those studies, 43% discussed publication bias and only 22% evaluated for it. The Cochrane group, well known for their support of reporting guidelines such as PRISMA27, discussed publication bias at the highest rate (90%).

One limitation of the present study is that other databases may have identified additional systematic reviews outside of PubMed and the Cochrane reviews. However, given that the journals included in our study are indexed in PubMed, this is a very low possibility. There are also other journal metrics, such as the impact factor, available to assess the rank of a journal. We chose Google Scholar’s H5-Index on the basis of its open availability. As the present study was limited to high-ranking journals, the results should not be generalized to lower-ranking journals.

In conclusion, the high level of publication bias found in the present study, coupled with the low levels of adherence to systematic review reporting guidelines, is of concern. Discussion and evaluation of publication bias in the development of systematic reviews should be standard practice. However, the quality of reporting for systematic reviews in high-ranking orthopaedic journals has been found to be poor28. Therefore, readers of systematic reviews should not assume that lack of assessment for publication bias indicates that publication bias was not present. The publication bias present in orthopaedic journals is worrisome, considering that randomized controlled trials published in orthopaedic journals likely have exaggerated treatment effects due to bias29. It has been shown that in high-ranking orthopaedic journals, such as The Journal of Bone & Joint Surgery, positive and nonpositive studies were accepted at similar rates30. These findings should encourage orthopaedic researchers to submit studies regardless of the direction of the results.

Back to Top | Article Outline

Appendix

Supporting material provided by the authors is posted with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A93).

Back to Top | Article Outline

References

1. Phan K, Mobbs RJ. Systematic reviews and meta-analyses in spine surgery, neurosurgery and orthopedics: guidelines for the surgeon scientist. J Spine Surg. 2015 Dec;1(1):19-27.
2. DiSilvestro KJ, Tjoumakaris FP, Maltenfort MG, Spindler KP, Freedman KB. Systematic reviews in sports medicine. Am J Sports Med. 2016 Feb;44(2):533-8. Epub 2015 Apr 21.
3. Provencher MT, Brand JC, Rossi MJ, Lubowitz JH. Are orthopaedic systematic reviews overly prevalent? Arthroscopy. 2016 Jun;32(6):955-6.
4. Little Z, Newman S, Dodds A, Spicer D. Increase in quality and quantity of orthopaedic studies from 2002 to 2012. J Orthop Surg (Hong Kong). 2015 Dec;23(3):375-8.
5. Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0. 2011 Mar. http://handbook-5-1.cochrane.org. Accessed 2019 Apr 1.
6. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. QUality Of Reporting Of Meta-analyses. Lancet. 1999 Nov 27;354(9193):1896-900.
7. Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group. Preferred Reporting Items for Systematic reviews and Meta-Analyses: the PRISMA statement. PLoS Med. 2009 Jul 21;6(7):e1000097. Epub 2009 Jul 21.
8. Checketts JX, Sims MT, Detweiler B, Middlemist K, Jones J, Vassar M. An evaluation of reporting guidelines and clinical trial registry requirements among orthopaedic surgery journals. J Bone Joint Surg Am. 2018 Feb 7;100(3):e15.
9. Onishi A, Furukawa TA. Publication bias is underreported in systematic reviews published in high-impact-factor journals: metaepidemiologic study. J Clin Epidemiol. 2014 Dec;67(12):1320-6. Epub 2014 Sep 4.
10. Atakpo P, Vassar M. Publication bias in dermatology systematic reviews and meta-analyses. J Dermatol Sci. 2016 May;82(2):69-74. Epub 2016 Feb 24.
11. Hedin RJ, Umberham BA, Detweiler BN, Kollmorgen L, Vassar M. Publication bias and nonreporting found in majority of systematic reviews and meta-analyses in anesthesiology journals. Anesth Analg. 2016 Oct;123(4):1018-25.
12. Eyding D, Lelgemann M, Grouven U, Härter M, Kromp M, Kaiser T, Kerekes MF, Gerken M, Wieseler B. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ. 2010 Oct 12;341:c4737.
13. Hasenboehler EA, Choudhry IK, Newman JT, Smith WR, Ziran BH, Stahel PF. Bias towards publishing positive results in orthopedic and general surgery: a patient safety issue? Patient Saf Surg. 2007 Nov 27;1(1):4.
14. Sedgwick P, Marston L. How to read a funnel plot in a meta-analysis. BMJ. 2015 Sep 16;351:h4718.
15. Dirnagl U, Lauritzen M. Fighting publication bias: introducing the negative results section. J Cereb Blood Flow Metab. 2010 Jul;30(7):1263-4.
16. Demaria AN. Publication bias and journals as policemen. J Am Coll Cardiol. 2004 Oct 19;44(8):1707-8.
17. Diguet E, Gross CE, Tison F, Bezard E. Rise and fall of minocycline in neuroprotection: need to promote publication of negative results. Exp Neurol. 2004 Sep;189(1):1-4.
18. Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods. 2014 Sep;5(3):221-34. Epub 2013 Dec 6.
19. Boekhorst AK, Farace DJ, Frantzen J. Grey Literature Survey 2004: a research project tracking developments in the field of grey literature. Grey Journal-Amsterdam. 2005;1(1):41.
20. Auger CP. Information sources in grey literature. Berlin: Walter de Gruyter; 2017. https://trove.nla.gov.au/work/8702344?q&versionId=45642315. Accessed 2018 Jun 25.
21. Canadian Agency for Drugs and Technologies in Health (CADTH). Grey Matters: a practical tool for searching health-related grey literature. https://www.cadth.ca/resources/finding-evidence/grey-matters. Accessed 2018 Jun 25.
22. Saleh AA, Ratajeski MA, Bertolet M. Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized. Evid Based Libr Inf Pract. 2014;9(3):28-50.
23. Giustini D, Thompson D. Finding the hard to finds: searching for grey literature. 2012. https://guides.mclibrary.duke.edu/sysreview/greylit
24. Baker M. Statisticians issue warning over misuse of p values. Nature. 2016 Mar 10;531(7593):151.
25. Woolston C. Psychology journal bans p values. Nature. 2015;519(7541):9.
26. Wayant C, Scott J, Vassar M. Evaluation of lowering the p value threshold for statistical significance from. 05 to. 005 in previously published randomized clinical trials in major medical journals. JAMA. 2018 Nov 6;320(17):1813-5.
27. Mann M, Sampson M, Cooper C, Stanfield C, Rader T. PRISMA in practice: keeping track of the literature search documentation. What are the problems? Presented at the 19th Cochrane Colloquium; 2011 Oct 19-22; Madrid. http://2011.colloquium.cochrane.org/abstracts/b2o4-prisma-practice-keeping-track-literature-search-documentation-what-are-problems.html. Accessed 2018 Jun 28.
28. Gagnier JJ, Kellam PJ. Reporting and methodological quality of systematic reviews in the orthopaedic literature. J Bone Joint Surg Am. 2013 Jun 5;95(11):e771-7.
29. Chess LE, Gagnier J. Risk of bias of randomized controlled trials published in orthopaedic journals. BMC Med Res Methodol. 2013 Jun 9;13:76.
30. Okike K, Kocher MS, Mehlman CT, Heckman JD, Bhandari M. Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg Am. 2008 Mar;90(3):595-601.

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2019 The Authors. Published by The Journal of Bone and Joint Surgery, Incorporated. All rights reserved.