Secondary Logo

Journal Logo

Miscellaneous

The degree of adherence to CONSORT reporting guidelines for the abstracts of randomised clinical trials published in anaesthesia journals

A cross-sectional study of reporting adherence in 2010 and 2016

Chow, Jeffrey T.Y.; Turkstra, Timothy P.; Yim, Edmund; Jones, Philip M.

Author Information
European Journal of Anaesthesiology: December 2018 - Volume 35 - Issue 12 - p 942-948
doi: 10.1097/EJA.0000000000000880

Abstract

Introduction

Background

Abstracts are concise summaries of research studies such as randomised clinical trials (RCTs), and are often used by readers to determine whether reading the full-text article is warranted.1 Abstracts are also important tools for clinical decision-making because there may be financial, technological, temporal or linguistic barriers that prevent or reduce access to the full-text of articles.1–3 Indeed, just over 50% of studies published in 2009 and indexed on PubMed were found to be open access, leaving almost half of the studies in the biomedical field with the full-text only accessible by subscription.4 These factors suggest that abstracts are an integral part of the initial appraisal of the value of a trial, making the quality of abstract reporting of paramount importance.

To improve the quality of reporting RCT abstracts, an extension of the Consolidated Standards of Reporting Trials (CONSORT) statement was developed and published in 2008.5,6 This CONSORT for Abstracts statement specified a minimum set of items that authors should include in the abstract of a RCT, ensuring that readers have the necessary detail and clarity to assess a RCT's validity and applicability.5

No studies in the anaesthesiology literature have examined whether the items omitted from the abstract are present in the body of the article. If items required by the CONSORT for Abstracts statement are not reported in both the abstract and the full-text, the underlying problem may be inadequate trial reporting in general and not only inadequate abstract reporting in particular. However, if items are not reported in the abstract but are present in the full-text, this lack of adherence to the CONSORT for Abstracts guidelines may cause abstract readers to have a distorted impression of the RCT's validity or applicability. ‘Spin’ and misrepresentation of the salient trial details in the abstract can have an impact on clinicians’ interpretations of an RCT, wherein a small modification in an abstract's focus can make clinicians more likely to consider a treatment beneficial despite the primary outcome being statistically nonsignificant.7

Objectives

The objective of this study was to determine the degree of adherence to CONSORT's abstract reporting guidelines for RCTs published in major anaesthesiology journals in 2010 and 2016. For checklist items not reported in the abstract, this study also identified whether they were reported in the full-text article to determine the potential for inadequate abstract reporting distorting or ‘spinning’ a RCT's findings. This study updated the existing literature8 on anaesthesiology RCT abstract reporting by using RCTs published in 2016 and comparing them with RCTs published in 2010 to determine whether there was an improvement over time. The years 2010 and 2016 were chosen so that authors would have had 2 years to incorporate the CONSORT for Abstracts recommendations and the most recent full year of publications when the study was conducted could be used as a longitudinal comparison.

Materials and methods

Study design and setting

This study was designed as a cross-sectional analysis of RCTs published in 2010 and 2016. It was performed at the University of Western Ontario and University Hospital, London Health Sciences Centre. The data were extracted from October 2016 to March 2017.

Participants

An electronic database was created to identify and retrieve all RCTs published in the top six general anaesthesiology journals as determined by impact factor: Anaesthesia, Anesthesia & Analgesia (A&A), Anesthesiology, British Journal of Anaesthesia (BJA), Canadian Journal of Anesthesia (CJA) and European Journal of Anaesthesiology (EJA).9 Since the CONSORT for Abstracts statement was published in 2008,5,6 2010 was selected as the first year for analysis to allow authors to incorporate the recommendations into their publications. To be able to determine a longitudinal trend and provide the most current results, RCTs were only included if published in 2010 or 2016.

To identify eligible RCTs, the Tables of Contents as posted on each journal's websites, excluding supplemental issues, were hand-searched by a first reviewer and the results independently reconciled by a second reviewer. Because only specific journals were assessed, a more sensitive manual search was able to be used as opposed to an electronic database search. Rather than defining RCT as randomised controlled trial, the more specific term of randomised clinical trial was used to indicate a more clinically relevant set of trials involving healthcare interventions in human participants. All included trials had a control group, and potential studies were only included if they were a prospective study assessing randomly allocated healthcare interventions in human participants, with random allocation of participants to study arms clearly reported (i.e. the use of the words ‘random’, ‘randomized’ or ‘randomised’ in the full-text article).

As a result, observational studies (e.g. case–control or cohort studies), learning curve studies, cadaver studies, cost-effectiveness studies, dose-finding or dose–response studies, which were not designed to test a clinical intervention, diagnostic test accuracy studies, meta-analyses, editorials, narrative reviews, animal studies, manikin studies and re-analyses of previously published studies were all excluded. RCTs published in the correspondence or letters to the editor sections were also excluded. If two phases were reported in one study, the phase with the highest number of patients was used for data extraction. RCTs identified as meeting the inclusion/exclusion criteria were retrieved and stored in a prospectively created electronic database. More details regarding methodology can be found in the publicly available study protocol.10

Variables

Data extraction was performed for all RCTs using the created electronic database. General study characteristics were extracted, including journal, year, multicentricity, study design, sample size and trial registration status. The number of participants randomised (rather than the number of participants actually analysed) was used for the sample size. Adequate trial registration was defined as the trial being registered in a publicly available trial registry before the first participant was enrolled with a clearly defined primary outcome.

As no validated scoring system for abstract quality exists, the 16 checklist items from the CONSORT for Abstracts statement5,6 were used to create a convenience score as a proxy for RCT abstract reporting quality. A brief description of each item was adapted from the CONSORT for Abstracts statement explanations and presented alongside each checklist item during data extraction to improve consistency of data extraction (Appendix 1, http://links.lww.com/EJA/A171). Each checklist item was measured as being reported in abstract, not reported in abstract but reported in full-text article or not reported in abstract or full-text article; this was then used to calculate summary measures such as the number of trials with the checklist item reported and the number of trials with the checklist item reported in the full-text only. An overall score was calculated by counting the number of checklist items reported in each RCT's abstract.

Bias

Data extraction for included RCTs was performed by one of four reviewers. All reviewers were trained together to ensure consistency of data extraction. Ambiguous data points were assessed by a second reviewer with a decision made through consensus.

Study size

This study aimed to be descriptive. There was no clear hypothesis to be tested, and no information on the proportion of items that would be fulfilled in 2010 or 2016. Therefore, no sample size calculation was performed.

Statistical methods

Descriptive statistics were used to present the raw number and percentage of checklist items reported for included RCTs. To assess differences between years, the Pearson χ2 test was used for categorical variables and the Wilcoxon rank-sum (Mann–Whitney) test was used for continuous variables. The Kruskal–Wallis test was used to assess differences between journals stratified for each year of publication. To improve clarity in the results, the overall CONSORT for Abstracts score was reported as the abstract score and the number of full-text only items (reported in the article but not in the abstract) was reported as the full-text-only score. Stata 13 statistical software (StataCorp LLC, College Park, Texas, USA) was used for all analyses. P value less than 0.05 was considered statistically significant.

Results

Participants and descriptive data

Table 1 describes the general characteristics for the 395 included RCTs, with 219 published in 2010 and 176 published in 2016. Although the total number of RCTs decreased from 2010 to 2016, Anesthesiology and CJA had an absolute increase in the number of included RCTs. Most RCTs were conducted at a single-centre and had a parallel group superiority design. From 2010 to 2016, the number of single-centre RCTs decreased from 94 to 86%, the number of adequately registered RCTs increased from 4 to 39% and the median sample size increased from 68 to 86.

Table 1
Table 1:
Characteristics of randomised clinical trials included for analysis

Main results

Although the overall CONSORT for Abstracts abstract score increased from 2010 to 2016 for each journal, the median score was below 10 (out of a possible 16) for every journal (Fig. 1) in both years analysed. The distributions of the abstract scores were statistically significantly different between years; the median [interquartile range (IQR)] abstract score increased from 4 points [3 to 5] in 2010 to 6 points [5 to 8] in 2016 (Fig. 2; Table 2). When analyses were stratified by year, there was a statistically significant difference for the abstract score between the different journals of publication (P < 0.0001 for 2010 and P < 0.0001 for 2016) (Fig. 1).

Fig. 1
Fig. 1:
Distribution of overall CONSORT for Abstracts abstract scores by journal over time. White line represents median; box represents 25th and 75th percentiles; each whisker represents 1.5 times interquartile range; dots represent outside values.
Fig. 2
Fig. 2:
Distributions of overall CONSORT for Abstracts abstract scores in 2010 and 2016. Wilcoxon Rank-Sum test for difference in distribution: P < 0.0001.
Table 2
Table 2:
Adherence to CONSORT for Abstracts reporting items in randomised clinical trials

The median [IQR] full-text-only score increased from 7 points [6 to 9] in 2010 to 8 points [7 to 9] in 2016 (Table 2). There was also a statistically significant difference for the full-text-only score between the different journals of publication (P < 0.0001 for 2010 and P < 0.0001 for 2016). A breakdown of the number of trials with the checklist item reported in the abstract, not reported in abstract but reported in article and not reported in abstract or article is presented in Table 2. For each checklist item in 2016, the majority of items that were not adequately reported in the Abstract were adequately reported in the full text of the article (Table 2).

Other analyses

Appendix 2, http://links.lww.com/EJA/A171 presents the same information for RCTs published in 2016 stratified by each journal. Without adjusting for multiple comparisons or potential confounding, Pearson's χ2 tests for differences between journals suggest that the journal had a statistically significant effect on adherence to the CONSORT for Abstracts guidelines for every criterion except for randomisation, numbers randomised and numbers analysed (Appendix 2, http://links.lww.com/EJA/A171).

Discussion

Key results

The results of this study suggest that many anaesthesiology RCTs are not following the CONSORT for Abstracts guidelines when publishing their final articles. Furthermore, there is evidence of selective reporting of items in the abstract, as the majority of checklist items in 2016 that were inadequately reported in the Abstract were in fact reported in the full text of the article. This implies authors may be consciously (and selectively) deciding which items to report in the abstract. Although their reasons for doing so are unknown (and were not the subject of this study), it is possible one of the reasons was to ‘spin’ the results to favour the message the investigators wished to transmit rather than simply a factual account of the study's results.

Interpretation

A previous study examining adherence to the CONSORT for Abstracts guidelines for four general anaesthesiology journals found the mean proportion of items reported to be 27% in 2006 and 29% in 2009.8 The median number of items reported in this study was four in 2010 and six in 2016, which corresponds to 25 and 38%, respectively. Since the CONSORT for Abstracts guidelines were published in 2008, this increase may be due to more authors and editors being aware of them. Although previous studies have reported on the lack of essential information included in RCT abstracts,11,12 the development of the CONSORT for Abstracts guidelines created a consistent list of necessary criteria. Studies in general and specific medical journals have described the suboptimal adherence of RCTs to the CONSORT for Abstracts guidelines,13–19 and anaesthesiology as a specialty has many of the same deficiencies.

The proportions of RCTs that met each criterion in 2010 were similar to a previous study, with many RCTs not meeting the CONSORT for Abstracts guidelines.8 As that study found that more checklist items were met after the CONSORT for Abstracts guidelines were published,8 it would be expected that the current study would see continued increases in adherence over time. Other than the interventions and conclusions items, the percentage of RCTs in 2016 that reported each criterion in the abstract did in fact increase compared with 2010.

For each of the six included anaesthesiology journals, the instructions to authors do not specifically address adherence to the CONSORT for Abstracts statement. However, all six journals require or recommend compliance with the general CONSORT guidelines, which includes adhering to the CONSORT for Abstracts guidelines.20–25 In terms of abstract-specific instructions, Anaesthesia requires an unstructured abstract no more than 250 words,20A&A requires a structured abstract no more than 400 words,24Anesthesiology requires a structured abstract no more than 250 words,21BJA requires a structured abstract no more than 250 words,22CJA requires a structured abstract no more than 250 words23 and EJA requires a structured abstract no more than 300 words.25 Although there are strict word limits that authors cannot exceed in their abstract, 250 to 300 words is sufficient to address all items in the CONSORT for Abstracts guidelines.5,6 In addition, MEDLINE no longer truncates abstracts at a specific word count and instead allows up to 10 000 characters in an abstract,26 questioning whether journals should be imposing a specific word limit at all.

Many checklist items (participants, interventions, randomisation, blinding, numbers randomised, recruitment and numbers analysed) show similar trends in terms of reporting in the abstract compared with reporting in the full-text only, with a high percentage of RCTs reporting the information in the full-text only. The randomisation criterion shows the most extreme trend; the way randomisation was conducted was rarely described in the abstract, with less than 2% in both years, consistent with a previous study.8 However, 88% of the trials in 2016 described this information in the full-text, suggesting that authors were choosing not to report this information in the abstract, not that the information was unavailable. Most RCTs in general medical journals also failed to meet this criterion, with fewer than 8% of RCTs in the New England Journal of Medicine, Journal of the American Medical Association and BMJ reporting the method of random sequence generation in the abstract.14

Consistent with a previous study in anaesthesiology,8 the recruitment criterion also followed this trend, as RCTs that did not terminate early rarely reported the trial status. The proportion reporting in the full-text was almost 100%, as we considered RCTs to meet this criterion if the number recruited met or exceeded the target sample size. Either RCTs should always report the trial status or this item should only be relevant when a trial is terminated early.

Although the percentage of RCTs reporting the numbers randomised and numbers analysed items improved from 2010 to 2016, many RCTs did not have sufficient information in the abstract, as they often reported a total sample size and not the number per group. Likely due to the increasingly widespread use of the CONSORT flow diagram,27,28 all RCTs that did not meet these two items in the abstract reported the information in the full-text.

Although the percentage of RCTs that met the Results outcome item in the abstract increased over time, the majority of the RCTs in 2016 were still not adequately reporting this item. Most RCTs typically only reported a result for each group and a P value, with few studies reporting an effect size and precision for that effect size. By comparison, in 2006, 62% of RCTs in major general medical journals reported the effect size and confidence interval in the abstract.13 Of the anaesthesiology RCTs in 2016 that did not meet the criterion in the abstract, only 24% met the criterion in the full-text, so the CONSORT for Abstracts guidelines was not able to be followed because the general CONSORT guidelines were not being followed. The harms criterion has a similar trend, though only a minority of RCTs were not reporting the harms.

Corresponding with an increase in trials being registered over time,29 the proportion of RCTs reporting trial registration in the abstract also increased. However, many RCTs still only reported the trial registration in the full-text and not the abstract. A distinct journal difference was observed in 2016, with almost no RCTs from Anaesthesia, A&A and Anesthesiology reporting the trial registration in the abstract even though many of these RCTs had the trial registration reported in the full-text. In contrast, BJA, CJA and EJA reported the trial registration in the abstract for many (but certainly not all) of the RCTs.

Just as in a previous study,8 the funding source was never reported in the abstract. However, the percentage of RCTs that reported the funding source in the full-text did increase in 2016. Despite CONSORT for Abstracts guidelines, the funding source is always reported in the full-text only and not the abstract. Perhaps the authors of the CONSORT for Abstracts guidelines should consider removing this criterion or making it only applicable for conference abstracts, similar to the contact details for the corresponding author.

Limitations and generalisability

A limitation of this study arises from the lack of a validated scoring system for measuring a study's abstract quality. Despite the absence of studies assessing the psychometric properties of the CONSORT for Abstracts scores, these convenience scores were used as approximations of a trial's adherence to the CONSORT for Abstracts. As a result, overall adherence and full-text-only scores considered each CONSORT for Abstracts criterion as being equally important, preventing an assessment of how much distortion the failure to report would cause the typical reader. These scores allow for a rough summary of the overall results, but the focus for quality improvement should be on adherence to each individual criterion.

As this study was restricted to the top six general anaesthesiology journals by impact factor, results may not be generalisable to other specialties or anaesthesiology journals with a lower impact factor. Although this may lower the external validity, conclusions drawn from this study are intended for the six journals included.

In conclusion, RCTs in six general anaesthesiology journals have improved their adherence to the CONSORT for Abstracts guidelines from 2010 to 2016, but the level of adherence is still unsatisfactory. Even in 2016, around 75% of RCTs met fewer than half of the 16 checklist items with no RCTs reporting all 16 items in the abstract. Other than the Results outcome criterion, a majority of the RCTs have the information present in the full-text, but not reported in the abstract. This suggests that, per the CONSORT for Abstracts guidelines, the abstracts for many anaesthesiology RCTs are not a complete summary of the entire article. There is evidence of selective inclusion of checklist items into the Abstract and this raises a concern about the potential for ‘spinning’ of study results by study investigators. An alternative explanation is that some items are simply unrealistic for real-world application, with more investigation needed to determine which items would provide the most impactful information. More research needs to be conducted to determine what tools can be used to improve the abstract reporting for anaesthesiology RCTs and whether the CONSORT for Abstracts statement is appropriate.

Acknowledgements relating to this article

Assistance with the article: We appreciate the support and advice offered by Neil Klar (PhD, Department of Epidemiology and Biostatistics, The University of Western Ontario) and Janet Martin (PharmD, Department of Anesthesia & Perioperative Medicine and Department of Epidemiology and Biostatistics, The University of Western Ontario).

Financial support and sponsorship: This work was supported by an Ontario Graduate Scholarship: Queen Elizabeth II Graduate Scholarship in Science and Technology (JC) and internal departmental funding (PJ). No other authors received funding for this study.

Conflicts of interest: PJ is an Associate Editor at the Canadian Journal of Anesthesia (CJA).

Presentation: A version of this manuscript was published as part of an MSc thesis currently under embargo (https://ir.lib.uwo.ca/etd/4681/).

References

1. Sivendran S, Newport K, Horst M, et al. Reporting quality of abstracts in phase III clinical trials of systemic therapy in metastatic solid malignancies. Trials 2015; 16:341.
2. Altwairgi AK, Booth CM, Hopman WM, Baetz TD. Discordance between conclusions stated in the abstract and conclusions in the article: analysis of published randomized controlled trials of systemic therapy in lung cancer. J Clin Oncol 2012; 30:3552–3557.
3. Wang L, Li Y, Li J, et al. Quality of reporting of trial abstracts needs to be improved: using the CONSORT for abstracts to assess the four leading Chinese medical journals of traditional Chinese medicine. Trials 2010; 11:75.
4. Kurata K, Morioka T, Yokoi K, Matsubayashi M. Remarkable growth of open access in the biomedical field: analysis of pubmed articles from 2006 to 2010. Scherer RW, editor. PLoS One 2013; 8:e60925.
5. Hopewell S, Clarke M, Moher D, et al. CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration. von Elm E, editor. PLoS Med 2008; 5:e20.
6. Hopewell S, Clarke M, Moher D, et al. CONSORT for reporting randomised trials in journal and conference abstracts. Lancet 2008; 371:281–283.
7. Boutron I, Altman DG, Hopewell S, et al. Impact of spin in the abstracts of articles reporting results of randomized controlled trials in the field of cancer: the SPIIN randomized controlled trial. J Clin Oncol 2014; 32:4120–4126.
8. Can OS, Yilmaz AA, Hasdogan M, et al. Has the quality of abstracts for randomised controlled trials improved since the release of Consolidated Standards of Reporting Trial guideline for abstract reporting? A survey of four high-profile anaesthesia journals. Eur J Anaesthesiol 2011; 28:485–492.
9. Thomson Reuters. Journal citation reports: anesthesiology [Internet]. ISI Web of Knowledge. 2016 [cited 15 August 2016]. http://admin-apps.webofknowledge.com/JCR/JCR.
10. Chow J, Jones P. Identifying areas to improve the quality of reporting in randomized clinical trials published in anesthesiology journals: a study protocol for a series of literature surveys assessing quality of trial registration, adherence to abstract reporting guidelines, adequacy of sample size calculations, and impact of funding source. 21 December 2016; https://doi.org/10.6084/m9.figshare.4490582.v1.
11. Wong H, Truong D, Mahamed A, et al. Quality of structured abstracts of original research articles in the British Medical Journal, the Canadian Medical Association Journal and the Journal of the American Medical Association: a 10-year follow-up study. Curr Med Res Opin 2005; 21:467–473.
12. Burns KEA, Adhikari NKJ, Kho M, et al. Abstract reporting in randomized clinical trials of acute lung injury: an audit and assessment of a quality of reporting score. Crit Care Med 2005; 33:1937–1945.
13. Berwanger O, Ribeiro RA, Finkelsztejn A, et al. The quality of reporting of trial abstracts is suboptimal: Survey of major general medical journals. J Clin Epidemiol 2009; 62:387–392.
14. Ghimire S, Kyung E, Kang W, Kim E. Assessment of adherence to the CONSORT statement for quality of reports on randomized controlled trial abstracts from four high-impact general medical journals. Trials 2012; 13:77.
15. Fleming PS, Buckley N, Seehra J, et al. Reporting quality of abstracts of randomized controlled trials published in leading orthodontic journals from 2006 to 2011. Am J Orthod Dentofac Orthop 2012; 142:451–458.
16. Chen Y, Li J, Ai C, et al. Assessment of the quality of reporting in abstracts of randomized controlled trials published in five leading Chinese Medical Journals. Earnest CP, editor. PLoS One 2010; 5:e11926.
17. Shin WJ, Kim YO, Oh JH, et al. Is there any quality improvement in the randomized controlled trial abstracts in the Korean Journal of Anesthesiology after the publication of the CONSORT abstract guidelines in 2008? Korean J Anesthesiol 2015; 68:420–422.
18. Ghimire S, Kyung E, Lee H, Kim E. Oncology trial abstracts showed suboptimal improvement in reporting: a comparative before-and-after evaluation using CONSORT for Abstract guidelines. J Clin Epidemiol 2014; 67:658–666.
19. Jin L, Hua F, Cao Q. Reporting quality of randomized controlled trial abstracts published in leading laser medicine journals: an assessment using the CONSORT for abstracts guidelines. Lasers Med Sci 2016; 31:1583–1590.
20. Klein AA, Anaesthesia Editorial Team. Author guidelines [Internet]. Anaesthesia. [cited 15 August 2016]. http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1365–2044/homepage/ForAuthors.html.
21. Anesthesiology Editorial Office. Instructions for authors [Internet]. Anesthesiology 2015 [cited 1 August 2016]. http://anesthesiology.pubs.asahq.org/public/instructionsforauthors.aspx.
22. Mahajan RP. British Journal of Anaesthesia editorial team. Instructions to authors [Internet]. Br J Anaesth 2016.
23. Backman S, Baker A, Beattie S, et al. 2011 Canadian journal of anesthesia guide for authors. Can J Anesth 2011; 58:668–696.
24. Anesthesia & Analgesia Editorial Office. Instructions for authors [Internet]. Anesth Analg 2016 [cited 15 August 2016]. http://edmgr.ovid.com/aa/accounts/ifauth.htm.
25. European Journal of Anaesthesiology Editorial Office. Guidance for authors on the preparation and submission of manuscripts to the European Journal of Anaesthesiology [Internet]. Eur J Anaesthesiol 2016.
26. U.S. National Library of Medicine. MEDLINE/PubMed data element (field) descriptions [Internet]. U.S. National Library of Medicine. 2016 [cited 15 August 2016]. https://www.nlm.nih.gov/bsd/mms/medlineelements.html.
27. Toerien M, Brookes ST, Metcalfe C, et al. A review of reporting of participant recruitment and retention in RCTs in six major journals. Trials 2009; 10:52.
28. Hopewell S, Hirst A, Collins GS, et al. Reporting of participant flow diagrams in published reports of randomized trials. Trials 2011; 12:253.
29. Jones PM, Chow JTY, Arango MF, et al. Comparison of registered and reported outcomes in randomized clinical trials published in anesthesiology journals. Anesth Analg 2017; 125:1292–1300.

Supplemental Digital Content

© 2018 European Society of Anaesthesiology