Clinical practice guidelines are commonly defined as systematically developed statements to assist practitioner and patient decisions about appropriate healthcare for specific clinical circumstances.1-3 Development of clinical practice guidelines is a complex process that takes time and requires substantial financial and intellectual investment.4-7 Guidelines need to be based on the best available evidence, of high quality, suitable for their purpose, and user friendly.4,6,8 Evaluating the quality of the evidence on which guidelines are based is an essential step in the development of appropriate guideline recommendations for practice, the use of which should be beneficial for patients.7-15
Previous studies of clinical practice guidelines have indicated that improvements to the quality, transparency and usability of guidelines are required and ongoing.4,7,10,11,13,14,16-22 These include improvements to the methods used for identifying, appraising, and synthesizing the evidence underpinning guideline recommendations.4,11,14,16,21 These studies have also emphasized the need for guideline developers to be transparent about, and guideline users to have an understanding of, the evidence basis for guideline recommendations that will guide clinical decision making and impact on health outcomes.12,16
Several tools exist for appraising clinical practice guidelines.12,15 The AGREE (Appraisal of Guidelines for Research and Evaluation) or AGREE II tool (2009 updated version of AGREE), published by the AGREE Collaboration, is among the most commonly used of these tools.23,24 The AGREE tool assesses the quality of guidelines, based on the methodological rigor of the development and reporting of guidelines.17,18,20,23 Many studies of clinical practice guidelines have evaluated the degree to which guidelines have met the AGREE or AGREE II tool criteria.7,10,12,17-21 While these studies have found some improvements over time, they have identified persistent problems with the development and reporting of guidelines.7,10,17,19
A 2010 study of Australian clinical practice guidelines examined guidelines produced or endorsed by national and state health-related organizations between 2003 and 2007.16 The study evaluated the extent to which guidelines documented the following: key health area, sources of funding, guideline producers, evidence search and appraisal processes, competing interests, plans for review and citation of Australian National Health and Medical Research Council (NHMRC) guidance for guideline development.6,16 This study found that only a third of guidelines documented the evidence on which recommendations were based (91/313), of which only 60 guidelines reported sufficient detail to replicate their methods.16 A subsequent NHMRC 2014 annual report of Australian clinical practice guidelines assessed similar characteristics to this 2010 study, for guidelines published between 2005 and 2013 that were included in the NHMRC Clinical Practice Guidelines Portal.4 However, at the time of publication of this NHMRC report, guidelines issued or approved by the NHMRC represented only 5% of Australian guidelines published in the previous five years.25
In Australia, several national initiatives exist to increase the accessibility and rigor of clinical practice guidelines. The NHMRC Clinical Practice Guidelines Portal is one of these.26 Launched in 2010,16 the portal is a repository of Australian clinical practice guidelines containing both NHMRC approved and non-approved guidelines.26 All guidelines included in this portal need to meet the following explicit evidence-based selection criteria:26
- Is the guideline evidenced-based? (Guidelines published prior to 2015 that did not meet, or partially met, this criterion were included in the portal.)
- Is the guideline Australian?
- Is the guideline current?
- Is the guideline freely available? (The guideline must be available to access online and free at the date of inclusion on the portal.)
- Is a funding statement included in the guideline?
- Was the guideline developed in a transparent manner with potential conflicts of interest stated?
- Was the guideline developed under the auspices of a professional college or association?
In addition to the above-mentioned NHMRC guidelines portal selection criteria, NHMRC approved guidelines are subject to a detailed methodological review to ensure that they meet the NHMRC guideline standard.5 A guideline for which NHMRC approval is sought must be based on the systematic identification and synthesis of the best available scientific evidence, and make clear recommendations for health professionals practising in Australia.5 Each clinical question of a NHMRC approved guideline is based on a systematic review and critical appraisal of the current scientific literature.5 The NHMRC specifies procedures and requirements for the conduct of these evidence reviews, which include a requirement to rate the quality of evidence underpinning each recommendation using one of two NHMRC approved methods.5 The NHMRC has adopted GRADE (Grading of Recommendations, Assessment, Development and Evaluation)1,2,8,27 for this purpose in its internally developed clinical practice and public health guidelines.5 In addition, the NHMRC strongly advocates the use of GRADE for all new third party guidelines that it approves.
GRADE is a structured and transparent approach to grade the quality (certainty/confidence) of evidence and the strength of recommendations in healthcare, proposed by the GRADE Working Group in 2000.1,2,8,27 GRADE provides a stepwise process to framing questions, selecting outcomes and rating their importance, evaluating the evidence, and considering evidence together with the values and preferences of patients and society to arrive at recommendations.1,2,8,27 GRADE is used for systematic reviews and guidelines that evaluate alternative management interventions or strategies.1,27 Unlike GRADE, AGREE does not assess the clinical content of a guideline.17,18,20,23 A rigorous and well-reported guideline development process, assessed with AGREE, should increase the trustworthiness of a guideline, but it will not ensure that a guideline is clinically important or has appropriate recommendations.17,20
It is unclear to what extent Australian clinical practice guidelines currently follow GRADE methods. Studies on the use of GRADE methods in guideline development in Australia, or internationally, do not appear to have yet been published.
Types of resources and search strategy
All guidelines available through the NHMRC Clinical Practice Guidelines Portal (whether NHMRC approved or not) since 2011 (which is when GRADE was mentioned in the requirements outlined in the Procedures and requirements for meeting the 2011 NHMRC standard for clinical practice guidelines5) until the present will be included in this review. These guidelines may be from any healthcare specialty or field for any disease or disease group.
All guidelines will be assessed by at least two independent reviewers and data will be extracted independently. Guidelines will be assessed to determine whether they have used the GRADE approach. If needed, guideline developers will be contacted to clarify any uncertainty. The data to be extracted will include:
- i) Guideline specialty/field
- ii) Guideline disease/disease area
- iii) Year of publication
- iv) Developing organization/society/body
- v) Did the guideline state that it would use the GRADE approach?
- vi) Did the guideline state that its approach would be informed by GRADE but did the guideline instead use a combination of methods including GRADE?
For the subset of guidelines that state that they have either followed or been informed by the GRADE approach, the following criteria (informed by the author team, the GRADE handbook,27 including Chapter 8: Criteria for determining whether the GRADE approach was used, and the criteria for applying or using GRADE document28) will be used to determine the extent to which GRADE methodology has been used:
- i) Definitions:
- a) Was the quality of evidence (also known as the certainty in the evidence or confidence in the estimated effects) defined consistently with the definitions used by the GRADE Working Group?
- b) Were the definitions for the ranking of certainty (such as high, moderate, low and/or very low) consistent with the definitions used by the GRADE Working Group?
- c) Were the definitions for each of the strength of recommendation wording categories (strong and weak/conditional) consistent with those used by the GRADE Working Group?
- ii) Framing the healthcare question:
- a) Was each healthcare question formatted according to the PICO (patient/population, intervention, comparison, outcomes) framework (or an equivalent framework)?
- iii) Selecting and rating the importance of outcomes:
- a) Were outcomes ranked as not important, important or critical to patients for decision making?
- b) Were grading assessments done on a per outcome basis (as compared to a per study basis)?
- iv) Domains for assessing the quality of evidence:
- a) Were all aspects contributing to the GRADE approach to rating the quality of evidence (methodological limitations, directness of evidence, consistency and precision of results, risk of publication bias, magnitude of the effect, dose-response gradient and influence of residual plausible confounding) considered for each outcome?
- v) Quality of evidence grades:
- a) Was the quality of evidence assessed for each outcome expressed using the GRADE categories (such as high, moderate, low and/or very low), based on consideration of the above factors (in question iv-a)?
- b) Was the overall quality of evidence across outcomes graded?
- vi) Summarizing the evidence:
- a) Was a GRADE Evidence Profile, Evidence Summary or Summary of Findings (SoF) presented?
- b) Were all GRADE-required details provided in the Evidence Profile, Evidence Summary or SoF?
- Was the evidence profile based on systematic reviews?
- Was the type of evidence that was assessed clearly described?
- Were the methods that were used to identify the evidence clearly described?
- Were the methods that were used to appraise the evidence clearly described?
- Were reasons for downgrading and upgrading the quality of evidence transparently described?
- vii) Going from evidence to recommendations:
- a) Were the GRADE criteria for determining the strength of a recommendation (balance of desirable and undesirable consequences, quality of evidence, values and preferences of those affected and resource use) considered?
- b) Was the strength of recommendations, for or against a specific management option, expressed using two categories (strong and weak/conditional)?
- c) Was a GRADE evidence to decision (EtD) framework used in the development of recommendations?
- d) Were decisions about the strength of recommendations transparently reported?
Guidelines that state that they have used GRADE will also be assessed using the AGREE II tool23 rigor of development domain (domain 3) to determine if guidelines that have used GRADE have had a rigorous guideline development process according to this tool. The rigor of development domain may be a stronger indicator of guideline quality compared to the other AGREE II domains.10,12 These domain 3 criteria are below and will be assessed with the AGREE II guidance:
- i) Were systematic methods used to search for evidence?
- ii) Were the criteria for selecting the evidence clearly described?
- iii) Were the strengths and limitations of the body of evidence clearly described?
- iv) Were the methods for formulating the recommendations clearly described?
- v) Were the health benefits, side effects and risks considered in formulating the recommendations?
- vi) Was there an explicit link between the recommendations and the supporting evidence?
- vii) Was the guideline externally reviewed by experts prior to its publication?
- viii) Was a procedure for updating the guideline provided?
Descriptive statistics, graphs and tables will be used to present and summarize the data. Use of GRADE methods will be assessed over the sampling frame of 2011–2018. Statistical associations will be explored (where feasible), between variables related to guideline development organizations (e.g. medical/professional societies, government bodies, non-governmental agencies), guideline specialties/fields/disease areas, the use of GRADE criteria and the use of the AGREE II rigor of development domain criteria. This will include analyzing whether there is an association between the use of GRADE and the AGREE II rigor of development score. In terms of reporting, guideline development organizations and the guidelines themselves will not be identified in the final report.
Potential impact of this research
This research will better inform Australian guideline developers about the historic trends of GRADE in guideline development, provide GRADE methodologists an improved understanding of how GRADE has been applied in Australian settings, and provide useful baseline data for future monitoring of the use of GRADE in guideline development. This research may be an important step towards determining priorities for improving and supporting the use of GRADE in guideline development.
We would like to acknowledge the assistance of senior NHMRC staff in providing expert advice regarding the use of GRADE by the NHMRC, particularly Geraint Duggan, Director of Clinical Guidelines, NHMRC.
1. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol
2011; 64 4:383–394.
2. Brozek JL, Akl EA, Alonso-Coello P, Lang D, Jaeschke R, Williams JW, et al. Grading quality of evidence and strength of recommendations in clinical practice guidelines. Part 1 of 3. An overview of the GRADE approach and grading quality of evidence about interventions. Allergy
2009; 64 5:669–677.
3. Field MJ, Lohr KN. Clinical practice guidelines: directions for a new program [Internet]. Washington, DC: Institutes of Medicine; 1990 [cited 2018 Apr 20]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK235751/pdf/Bookshelf_NBK235751.pdf
4. National Health and Medical Research Council. Australian Clinical Practice Guidelines 2014 Annual Report [Internet]. 2014 [cited 2018 Mar 5]; NH165. Available from: https://www.nhmrc.gov.au/_files_nhmrc/publications/attachments/nh165_2014_nhmrc_clinical_guidelines_annual_report_140805.pdf
5. National Health and Medical Research Council. Procedures and requirements for meeting the 2011 NHMRC standard for clinical practice guidelines [Internet]. 2011 [cited 2018 Mar 5]; cp133 and cp133a. Available from: https://www.nhmrc.gov.au/_files_nhmrc/publications/attachments/cp133_nhmrc_procedures_requirements_guidelines_v1.1_120125.pdf
6. National Health and Medical Research Council. Reporting on Australian Clinical Practice Guidelines [Internet]. 2016 [cited 2018 Mar 5]. Available from: https://www.nhmrc.gov.au/guidelines-publications/information-guideline-developers/reporting-australian-clinical-practice
7. Al-Ansary LA, Tricco AC, Adi Y, Bawazeer G, Perrier L, Al-Ghonaim M, et al. A systematic review of recent clinical practice guidelines on the diagnosis, assessment and management of hypertension. PLoS One
2013; 8 1:e53744.
8. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ
2008; 336 7650:924–926.
9. Taylor-Phillips S, Stinton C, Ferrante di Ruffano L, Seedat F, Clarke A, Deeks JJ. Association between use of systematic reviews and national policy recommendations on screening newborn babies for rare diseases: systematic review and meta-analysis. BMJ
10. Alonso-Coello P, Irfan A, Sola I, Gich I, Delgado-Noguera M, Rigau D, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care
2010; 19 6:e58.
11. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. J Am Med Assoc
1999; 281 20:1900–1905.
12. Hoffmann-Eßer W, Siering U, Neugebauer EA, Brockhaus AC, Lampert U, Eikermann M. Guideline appraisal with AGREE II: systematic review of the current evidence on how users handle the 2 overall assessments. PLoS One
2017; 12 3:e0174831.
13. Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet institute of medicine standards: two more decades of little, if any, progress. Arch Intern Med
2012; 172 21:1628–1633.
14. Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet
2000; 355 9198:103–106.
15. Siering U, Eikermann M, Hausner E, Hoffmann-Eßer W, Neugebauer EA. Appraisal tools for clinical practice guidelines: a systematic review. PLoS One
2013; 8 12:e82915.
16. Buchan HA, Currie KC, Lourey EJ, Duggan GR. Australian clinical practice guidelines: a national study. Med J Aust
2010; 192 9:490–494.
17. Padjas A, Kehar R, Aleem S, Mejza F, Bousquet J, Schunemann HJ, et al. Methodological rigor and reporting of clinical practice guidelines in patients with allergic rhinitis: QuGAR study. J Allergy Clin Immunol
2014; 133 3:777–783.e4.
18. Shen J, Sun M, Zhou B, Yan J. Nonconformity in the clinical practice guidelines for subclinical Cushing's syndrome: which guidelines are trustworthy? Eur J Endocrinol
2014; 171 4:421–431.
19. Castellani A, Girlanda F, Barbui C. Rigour of development of clinical practice guidelines for the pharmacological treatment of bipolar disorder: systematic review. J Affect Disord
20. Bush SH, Marchington KL, Agar M, Davis DH, Sikora L, Tsang TW. Quality of clinical practice guidelines in delirium: a systematic appraisal. BMJ Open
2017; 7 3:e013809.
21. Van de Velde S, Heselmans A, Donceel P, Vandekerckhove P, Ramaekers D, Aertgeerts B. Rigour of development does not AGREE with recommendations in practice guidelines on the use of ice for acute ankle sprains. BMJ Qual Saf
2011; 20 9:747–755.
22. Kryworuchko J, Stacey D, Bai N, Graham ID. Twelve years of clinical practice guideline development, dissemination and evaluation in Canada (1994 to 2005). Implement Sci
23. The AGREE Next Steps Consortium. Appraisal of Guidelines for Research and Evaluation II: AGREE II Instrument [Internet]. 2013 [cited 2018 Mar 5]. Available from: https://www.agreetrust.org/wp-content/uploads/2013/06/AGREE_II_Users_Manual_and_23-item_Instrument_ENGLISH.pdf
24. Brouwers MC, Kerkvliet K, Spithoff K. The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines. BMJ
25. National Health and Medical Research Council. Report on Australian Clinical Practice Guidelines 2014 [Internet]. 2014 [cited 2018 Mar 5]. Available from: https://www.nhmrc.gov.au/guidelines-publications/nh165
26. National Health and Medical Research Council. Australian clinical practice guidelines: Clinical practice guidelines portal [Internet]. 2017 [cited 2018 Apr 20]. Available from: https://www.clinicalguidelines.gov.au/portal
27. Schunemann H, Brozek J, Guyatt G, Oxman A. GRADE handbook for grading the quality of evidence and the strength of recommendations [Internet]. The GRADE Working Group; 2013 [cited 2018 Mar 5]. Available from: http://gdt.guidelinedevelopment.org/app/handbook/handbook.html
28. Schunemann H. Guyatt G, Oxman A. Criteria for applying or using GRADE. GRADE Working Group and GRADE Guidance Group. 2016: 1–2.