De Oliveira, Gildasio S. Jr. MD; Chang, Ray BS; Kendall, Mark C. MD; Fitzgerald, Paul C. RN, MS; McCarthy, Robert J. PharmD
Clinical decisions and practice guidelines are preferably based on solid scientific evidence. Among the different types of clinical studies, the best evidence is obtained from randomized controlled trials. Unfortunately, many clinical trials lack a sufficient number of subjects to produce conclusive evidence, and meta-analysis of controlled trials is often used to aggregate the effects across multiple studies. Unfortunately, conclusions from meta-analysis may be susceptible to bias. Publication bias occurs because authors are more likely to submit and editorial boards are more likely to accept a trial with a positive effect of treatment and “file drawer” or reject studies that do not demonstrate a positive effect.1 In addition, when positive studies are more likely to be published in journals with larger circulation or higher impact than those with negative statistical or clinical findings, this practice can impact the conclusions of qualitative reviews, another source of solid scientific evidence.2
Although impact factor has been a topic for extensive debate,3 it is frequently used as a measure of the quality of articles published within the journal.4 The journal impact factor is obtained by examining all citations for a 1-year period that reference a journal's publications in the prior 2 years, divided by scholarly articles published in that journal in those 2 years.4 Leading journals in a field of medicine strive for a higher impact factor, which can affect the journals position on the number of studies published and likelihood of accepting a negative or inconclusive manuscript. This practice may erroneously drive future research and clinical practice statements within the specialty,5 because articles that refute findings published in high impact journals are frequently published in lower impact factor journals, limiting the perceived merit of the study.6
The presence and the size of the effect of publication bias have been demonstrated in different specialties,7–10 but no study has examined this topic in the anesthesiology literature. The purpose of this study was to evaluate the extent of publication bias in clinical trials in the anesthesiology literature and to evaluate whether studies with a positive result are more likely to be published in journals with higher impact and greater circulation when compared with articles with negative or inconclusive findings controlling for factors predictive of publication.
A PubMed search was performed for published controlled clinical trials in journals in the anesthesiology field, published in the English language, excluding journals exclusive of pain medicine for the years of 2008 and 2009. Included journals are shown in Table 1. The following search terms were used: double-blind$ or random$, controlled, clinical, or human. Searches were restricted for the years of 2008 and 2009 for each journal. The initial search yielded 1377 articles from 14 journals. Eleven hundred sixty-three studies met our inclusion criteria and were evaluated for the study variable (Fig. 1).
The full manuscripts of the trials identified by the search were independently evaluated by 2 investigators (GDO and RC) for inclusion in the analysis using criteria determined a priori. Articles that were selected for evaluation were full research reports of a controlled trial performed in humans. Reports of surveys, review articles, case repots, practice guidelines, and letters to the editor were excluded. Also excluded were articles in which no control group was identified. Retracted articles were excluded. Disagreements on inclusion of articles were resolved by consensus. If a consensus could not be reached, the final decision was left to a third investigator (RJM).
The primary independent variable for this study was a positive or negative study result as defined by the primary outcome reported by the investigators of the included articles.11 Studies were classified as positive if they showed a statistically significant effect (P < 0.05, 95% confidence interval [CI] for difference excluding 0, or 95% CI for a ratio excluding 1) of the primary outcome. If no primary outcome was defined, the article was classified based on the majority (positive or negative) of the reported outcomes. The primary independent variable was evaluated independently by 2 of the investigators (GDO and RC); disputes were resolved by consensus between the 2 investigators. If a consensus could not be reached, a decision was made with the help of a third investigator (RJM). The initial agreement of the primary independent variable between the raters was 0.75 (Cronbach α). All discrepancies were resolved before statistical analysis.
Data extracted from articles also included factors identified to be associated with the likelihood of publication12–16: size of the sample (more than or less than 100 subjects), country of origin of the publication (United States [US] and non-US), presence or absence of a sample size calculation, presence or absence of blinding of treatment, number of centers involved in the study (single center versus multicenter), presence or absence of reporting of withdrawals, and whether a method of randomization was described. Also extracted were the adherence to the Consort Guideline for reporting of a randomized clinical trial,17 registration of the trial, source of trial funding, and the statement of a study hypothesis.
The number of citations for each included article since publication was determined by searching Google Scholar and the Web of Science for the article title or PubMed identification number. The journal clinical trial impact factor, defined as the average citations for controlled clinical trials, was calculated by summating the total citations for the articles and dividing by the number of articles included in the analysis for each journal. The average journal impact factor for the years 2008 and 2009 was obtained from published sources.18 The circulation for each journal was determined using Ulrich's Periodicals Directory™ accessed June 20, 2011 or via inquiry with the circulation department of the journal.
Journals were divided into 2 groups. High impact journals were those with a clinical trial impact factor in the upper quartile (Anesthesiology, Regional Anesthesia and Pain Medicine, British Journal of Anaesthesia, and Anesthesia & Analgesia). Low impact journals were those with a clinical trial impact factor in the lower 3 quartiles. The primary study outcome was the publication in a journal with a clinical trial impact factor in the upper quartile of those evaluated. A flowchart depicting the number of articles at each step of the study is presented in Figure 1. The quartiles for the clinical trial impact factor for the journals included in the analysis were 4, 6.1, and 9.1. Of the studies analyzed, 588 were from the journals with a clinical trial impact factor in the upper quartile and 575 from journals with an impact factor below the upper quartile. The average circulation of the journals in the upper quartile is 19,608 compared with 3994 for the remaining journals.
We estimated that 270 articles per group would be necessary to achieve an 80% power to detect a 10% difference in the proportion of positive finding articles in the higher clinical trial impact factor group compared with the lower clinical trial impact factor group using a 2-sided Fisher exact test at P = 0.05, assuming that 75% of the articles in the lower group would demonstrate positive findings. Sample size calculations were performed using PASS 2008 (version 8.0.15, release date September 8, 2010; NCSS LLC, Kaysville, UT).
Data are presented as counts and percentages. Interrater reliability of the primary independent variable was assessed using Cronbach's alpha. Study variables associated with publication in a journal in the upper quartile were compared with those published in a journal in the lower quartiles by constructing cross-tabulation tables and the Fisher exact test (Table 2). Based on the univariate results, a classification and regression tree (CART) analysis was created to develop a decision tree for publication in a journal in an upper or lower quartile journal. The CART algorithm uses binary recursive splitting of the data to classify articles belonging to either the upper or lower quartile groups. The algorithm performance was evaluated by calculation of the sensitivity, specificity, and positive and negative predictive values of the predicted groups.
A binary logistic regression model was constructed to adjust the effect of positive study results for other characteristics associated with publication in the higher clinical trial impact factor group. Logistic regression analysis was performed using stepwise likelihood ratio elimination. Variables were entered into the model if the univariate association with publication in a high impact factor journal was P < 0.2. Based on the CART analysis, an interaction term between US and non-US origin and positive findings was also entered in the model. Criteria for entry and removal from the model at each step were set at 0.2 and 0.05, respectively. Concordance of the model with publication in an anesthesiology journal in the upper or lower group was determined by constructing a classification table.
Also based on the binary logistic model, a multinomial logistic regression model was fitted for the journals with a clinical trial impact factor in the upper quartile with adjustment for study trial registration, origin of publication, positive study findings, reporting of treatment blinding, reporting of subject withdrawals, study sponsorship, and description of the randomization method. The interaction between US and non-US origin and positive findings was also entered in the model. Publications in all journals with a clinical trial impact factor in the lower 3 quartiles were used as the reference for determination of the odds for publication in the upper quartile impact factor journals. Bias correction of regression coefficients and CIs for variables in the regression models were made using a 10,000-sample bootstrap. All reported P values are 2-sided, and statistical significance was accepted at P ≤ 0.001. Data were analyzed using IBM® SPSS® Statistics 19 (version 184.108.40.206; IBM Corporation, Somers, NY) and R version 2.13.1, release date July 8, 2011 (The R Foundation for Statistical Computing).
Positive study findings, US origin of data, blinding of treatments, and report of subject withdrawal were identified by CART analysis for classification of articles into upper or lower quartile groups (Fig. 2). CART correctly classified 72.4% of the 588 articles in the upper quartile group and 58.2% of the 575 articles published in journals below the upper quartile.
Variables identified by multivariate logistic regression are presented in Table 3. Positive findings was an independent predictor of publication in an anesthesiology journal with a clinical trial impact factor >9.1, with an odds ratio (95% CI) of 2.32 (1.78–3.00) (P < 0.0005) after adjusting for other factors associated with publication. The logistic regression model correctly classified 65.6% of the 588 studies in the upper quartile group and 63.7% of the 575 articles published in journals below the upper quartile. Performance characteristics of the logistic and CART models are compared in Table 4.
There were no statistically significant differences in factors associated with publication in articles that reported positive or negative findings (Table 5). The median shift (95% CI) for citations of positive findings articles compared with articles that did not report positive findings was 1 (0–2) (P = 0.0005). When stratified between the top and the lower quartile groups, the median shift (95% CI) in the number of citations between positive and negative findings articles was statistically significant in the upper quartile group 2 (0–3) (P = 0.008), but not in the lower quartile group 0 (−1 to 1) (P = 0.38).
Multinomial logistic regression found a positive study finding as a significant factor associated with publication in 3 of the 4 anesthesia journals in the upper quartile compared with the aggregate of the journals in the lower quartiles (Table 6). The odds ratio (95% CI) for publication of a negative trial in Regional Anesthesia and Pain Medicine was 1.90 (0.98–3.64), but did not achieve statistical significance (P = 0.06). Trial registration, US origin of data, reporting of treatment blinding and subject withdrawals, and study sponsorship were found to increase the odds of publication in one or more of the journals in the upper quartile. Interestingly, the odds ratio for publication with a lack of a description of the randomization method was associated with publications in Anesthesiology, Anesthesia & Analgesia, and the British Journal of Anaesthesia more frequently than in the aggregate of the journals with a clinical trial impact factor <9.1.
The important finding of this investigation is that a positive result study is an independent predictor for a manuscript to be published in an upper quartile clinical trial impact factor, peer-reviewed, indexed anesthesiology journal. A much higher proportion of positive studies were found in journals with an impact factor in the upper quartile compared with lower impact factor journals. Nonetheless, there were no significant differences between positive and negative result studies with respect to other factors associated with publication. There were more overall citations for positive finding studies, albeit this difference was of miniscule importance. Our findings suggest the presence of publication bias among high clinical trial impact journals in the anesthesiology literature.
An analysis of the high clinical trial impact journals compared with an aggregate of the publications in lower clinical trial impact factor journals also confirmed that positive results in a study are associated with publication in the higher impact journals in anesthesiology. After adjusting for factors that we found to be significant indicators of the likelihood of publication, a positive result study was still a predictor of publication in the high impact journal. Among the journals in the high clinical trial impact factor group, the odds of publication of a study with a positive result were significantly higher than the aggregate of the low impact group in 3 of the 4 journals. However, a limitation of this analysis is the low sample size in the individual journals as compared with the group in total.
The impact of publication bias can affect both clinical and research decision-making. Because higher impact journals have broader subscription rates and subsequently a higher number of readers, studies in these journals will be more likely evaluated for qualitative decision-making. Because only 28% of the interventions published in high impact factor journals will report a negative result, limitations and possible extensions of the effect beyond the scope of the study may be difficult to ascertain. The finding that higher impact journals published 72% of articles with positive results when compared with 54% in lower impact journals can affect clinical care because more anesthesiologists will be more likely to be exposed to more positive studies than negative ones. Therefore, clinical decisions based solely on data from high impact journals may be incomplete or inaccurate based on the literature bias.19 In addition, the effect of the bias is not easily correctable because information regarding negative results can be delayed in publication compared with that from studies with positive findings. Stern and Simes20 demonstrated that a negative study takes >3 years longer than a positive study to be published.
Our findings also have implications on research practices. The presence of publication bias can also impact a meta-analysis or qualitative systematic review done for determining future research and clinical implications of research publications. Investigators should be adjusting for publication bias in quantitative meta-analyses and qualitative systematic reviews to account for the effect of publication bias on the pooled effects by using techniques such as sensitivity analysis or meta-regression. Investigators may be less likely to address topics such as equivalence of techniques or safety of methods, because these study designs are less likely to produce a positive result and therefore will be less likely to be published in a high impact factor journal. This practice also promotes design of studies that examine only large effect size outcome differences that are inadequate to evaluate a secondary finding and adverse effects. Study trial registration has been suggested as a mechanism to promote transparency around clinical trials and their results, yet an analysis of studies registered on ClinicalTrials.gov found that of the 66.3% of trials that had published outcomes, 78.3% reported positive results.21 These authors also found an association of positive outcome with study funding, with industry-funded studies reporting positive results in 85% of published studies compared with 50% for government funding and in 72% of publications supported by nonprofit or nonfederal funding (P < 0.001).21
There are many possible explanations for the findings of this study. The most likely one is that there is publication bias in the anesthesiology literature similar to that found in other medical specialties.1 High impact journals have low rates of acceptance for manuscripts, generating high competition among authors. It is possible that authors acknowledge this competition by not submitting their negative studies to a high impact journal, but it is also possible that reviewers and editors are biased toward accepting articles with positive results. Available evidence points more toward investigators not submitting negative result studies (submission bias) as the main cause of publication bias. Dickersin,19 after following applications for studies at an IRB, concluded that publication bias originates primarily with investigators, not journal editors. Olson et al.22 examined the editorial process of a very high impact medical journal prospectively and they did not find an editorial factor contributing to publication bias; however, their findings cannot be generalized to editorial boards of different journals.
Our study has several limitations. We were unable to examine the cause of the discrepancy between the different percentages of positive studies among the groups. Editorial bias might be responsible for a specific journal and submission bias might be the cause for publication bias in other journals. The identification of specific causes to exclude editorial bias is an internal process for each journal and it requires full access to the editorial process. We also only examined controlled trials, which limits our capacity to generalize our findings to other types of articles such as meta-analysis and surveys.
In conclusion, we demonstrated that studies with positive findings are more likely to be published in higher impact factor anesthesiology journals than negative studies, even after adjusting for quality factors classically associated with publication. The presence of a publication bias in the anesthesia literature can have implications for future research and the clinical care of patients. Authors should be encouraged to submit negative studies to high impact journals and the journals should be encouraged to evaluate the editorial process as a potential cause of publication bias.
Name: Gildasio S. De Oliveira, Jr., MD.
Contribution: This author participated in the design, conduct, and manuscript preparation.
Name: Ray Chang, BS.
Contribution: This author participated in the conduct of the study.
Name: Mark C. Kendall, MD.
Contribution: This author participated in the conduct of the study.
Name: Paul C. Fitzgerald, RN, MS.
Contribution: This author participated in the conduct of the study.
Name: Robert J. McCarthy, PharmD.
Contribution: This author participated in the design, data analysis, and manuscript preparation.
This manuscript was handled by: Franklin Dexter, PhD, MD.
1. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337:867–72
2. Frank E. Authors' criteria for selecting journals. JAMA 1994;272:163–4
3. Garfield E. The history and meaning of the journal impact factor. JAMA 2006;295:90–3
4. McVeigh ME, Mann SJ. The journal impact factor denominator: defining citable (counted) items. JAMA 2009;302:1107–9
5. Hoeffel C. Journal impact factors. Allergy 1998;53:1225
6. Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA 2002;287: 2847–50
7. McGauran N, Wieseler B, Kreis J, Schüler YB, Kölsch H, Kaiser T. Reporting bias in medical research: a narrative review. Trials 2010;11:37
8. Willenheimer R. Statistical significance versus clinical relevance in cardiovascular medicine. Prog Cardiovasc Dis 2001;44:155–67
9. Littner Y, Mimouni FB, Dollberg S, Mandel D. Negative results and impact factor: a lesson from neonatology. Arch Pediatr Adolesc Med 2005;159:1036–7
10. Squitieri L, Petruska E, Chung KC. Publication bias in Kienböck's disease: systematic review. J Hand Surg Am 2010;35:359–67
11. Moher D, Dulberg CS, Wells GA. Statistical power, sample size, and their reporting in randomized controlled trials. JAMA 1994;272:122–4
12. Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA 1992;267:374–8
13. Chalmers I, Adams M, Dickersin K, Hetherington J, Tarnow-Mordi W, Meinert C, Tonascia S, Chalmers TC. A cohort study of summary reports of controlled trials. JAMA 1990;263:1401–5
14. Link AM. US and non-US submissions: an analysis of reviewer bias. JAMA 1998;280:246–7
15. Misakian AL, Bero LA. Publication bias and research on passive smoking: comparison of published and unpublished studies. JAMA 1998;280:250–3
16. Scherer RW, Dickersin K, Langenberg P. Full publication of results initially presented in abstracts: a meta-analysis. JAMA 1994;272:158–62
19. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 1990;263:1385–9
20. Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315:640–5
21. Bourgeois FT, Murthy S, Mandl KD. Outcome reporting among drug trials registered in clinicaltrials.gov. Ann Intern Med 2010;153:158–66
22. Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, Reiling J, Pace B. Publication bias in editorial decision making. JAMA 2002;287:2825–8