Clinical practice guidelines (CPGs) are evolving as a means of translating evidence into clinical practice in the intensive care unit (ICU) (1–5). Research teams and professional societies have developed CPGs to optimize care in the ICU (6–14).
Evaluations (15–17) of internal medicine guidelines and those for the management of chronic obstructive pulmonary disease (18, 19) found shortcomings in their methods of gathering and grading evidence and formulating recommendations. Almost a decade after guidelines became prevalent in the ICU (2), their quality remains to be assessed. Poor quality may contribute to inappropriate recommendations and low CPG adherence (20). Improving the quality of critical care CPGs may increase adherence and improve the processes of care and patient outcomes.
Numerous consensus conference statements (CCSs) have appeared since the first National Institutes of Health conference in 1983 (21, 22); many jointly developed by international critical care societies. Although critical care clinicians use CCSs to guide patient care (23), their quality has not been evaluated. Given the profusion of CPGs and the international movement to standardize the presentation of practice recommendations, the ideal development, format, funding, and future of CCSs are unclear (24).
Our objective was to determine the quantity and quality of professional society CPGs and CCSs on selected topics within two categories relevant to ICU practice: 1) mechanical ventilation and 2) prevention of complications of critical illness associated with mechanical ventilation. Although CCSs provide recommendations similar to that of CPGs, there are no validated tools to specifically appraise their quality; therefore, we used CPG quality appraisal tools.
Literature Search and Study Selection.
We searched OVID MEDLINE (1966 through the second week of May 2007), PubMed (including “related articles” through May 2007), EMBASE (1980 through the second week of May 2007), and the Cochrane Library (all databases) (Issue 2, 2007). We also searched personal files and conference proceeding abstracts (1990 through May 2007) of critical care professional societies and their Web sites. References of CCSs and CPGs were hand searched. We contacted primary authors of the CCSs and CPGs to inquire about additional relevant CCSs or CPGs but identified no additional citations. We applied no language restrictions (25).
One author (T. Sinuff) performed the literature search. Independently, two reviewers (T. Sinuff, R. V. Patel) reviewed all citations and applied the following inclusion criteria: original reports of CCSs or CPGs developed by professional societies, specific to predefined areas of critical care (airway management, mechanical ventilation, acute respiratory distress syndrome, weaning from mechanical ventilation, noninvasive ventilation, prevention of ventilator-associated pneumonia, ventilator-induced lung injury, venous thromboembolism, upper gastrointestinal bleeding), and specific recommendations made regarding management or prevention. A single version of multiple simultaneous publications of the same topic was included. Disagreements regarding the eligibility of articles were resolved by consensus or by a third investigator (D. J. Cook).
We used authors’ definitions of CPG and CCS and classified articles as CPGs or CCSs if specified as such. If the authors did not specify the article type, we classified articles based on the description of the details of the CPG and CCS development process.
Quality Instrument Selection.
There are ≥24 instruments to assess CPG quality (26). Items in these instruments may be grouped into ten domains: validity, reliability, clinical applicability, clinical flexibility, clarity, scheduled review, development team, implementation, dissemination, and evaluation (26). We developed criteria (Table 1) based on these domains and selected instruments if they met at least eight of the ten criteria, including the criterion of instrument validation. From the 24 available instruments (26), we selected Grilli et al. (3 items) (15), Shaneyfelt et al. (25 items) (16), and the Appraisal of Guideline Research and Evaluation (AGREE) Collaboration (23 items) (27).
We selected the short Grilli (15) instrument to evaluate its correlation with the longer instruments. The Shaneyfelt (16) instrument is intermediate in length. The AGREE instrument (27) is the longest, and it is a revised version of the instrument by Cluzeau et al. (28) but does not include the implementation domain. We chose it because it includes a domain on editorial independence (acknowledging the possibility of conflict of interest) and is the only validated instrument with a numerical scoring scale (Appendix 1 provides instrument details).
We abstracted data independently and in duplicate on: publication status and year, professional society, sponsoring agency, declaration of conflict of interest of the panel members, expertise of panel members (content, methodology), document format and presentation, provision of clinical aids, document length, summary of recommendations, supporting evidence, and quantification of benefit. We classified recommendations as evidence-based if the document reported a literature search, evidence appraisal, and systematically assessed the quality of the evidence. Alternatively, we classified recommendations as expert opinion–based (interpretation of the evidence in the context of other knowledge or expertise) or consensus-based (used RAND or Delphi methods). All disagreements were resolved by consensus.
Each reviewer provided an overall assessment of the guideline, based on the AGREE instrument (27), and whether it should be strongly recommended, recommended with provisos, or not recommended. Both reviewers made these assessments independently, without previous calibration.
We evaluated agreement between reviewers for document selection using Cohen's κ (29) and report raw agreement for assessment of quality and data abstraction.
Before calculating the total scores for the Grilli (15) and Shaneyfelt (16) instruments, we converted their yes/no, single-item scoring system into a numerical scale (0 = no, 1 = yes). For the AGREE instrument (27), first, we used the AGREE method to calculate standardized domain scores (27) for each CPG and CCS. Second, we calculated a total score by summing the absolute domain scores, acknowledging the AGREE authors’ stated limitations of using total scores (27).
We compared the quality of CPGs with CCSs using the Student's t-test for the difference in mean values of the total scores for each quality instrument (29) after verification of normality. We calculated inter-instrument Pearson's correlation using mean total scores after verification of normality (29). We used linear regression to assess trends over time of the total AGREE score. We report binary data as proportions or percentages, continuous data as mean ± sd or median (interquartile range), and differences as mean and 95% confidence interval. We interpreted statistically significant (two-sided) p values as <.05. All analyses were performed using SPSS 11.0 (SPSS, Chicago, IL).
Document Selection and Characteristics.
We included 25 CPG and CCS publications (13 CPGs, 12 CCSs) (Fig. 1  and Appendix 2), with perfect agreement (κ = 1.0) on study selection. Topics included airway management (31), mechanical ventilation (32–34), mechanical ventilation for acute respiratory distress syndrome patients (35, 36), weaning (37, 38), noninvasive ventilation (39–41), and prevention of complications (ventilator-associated pneumonia [42–48], ventilator-induced lung injury , venous thromboembolism [50–53], and stress ulceration [54, 55].
Few documents reported the development panel's composition, including content (24%) (33, 35, 43, 45, 48, 54) and methods experts (8%) (43, 45). Nine (36%) reported involvement of the professional society (32, 37, 39, 40, 43, 45, 48, 49, 54). Six (24%) disclosed pharmaceutical (32, 43, 47, 52) or ventilator company (39, 41) funding. Fifteen (60%) reported on financial conflicts of interest (32, 37–41, 46, 48, 49, 51, 54, 55); three (12%) disclosed conflicts (42, 43, 45).
Document format was consistent: 23 were text only, and two (36, 54) combined text and an algorithm. Only one (45) provided a clinical decision aid in the form of a quick reference guide. Documents were a median of 19 (7, 35) pages in length; the median number of tables was 2 (0, 4). We classified recommendations as evidence-based in 16 (64%) documents (33, 36–43, 45, 46, 48–51, 55). We were unable to determine the type of recommendations (evidence-based, expert-based, or combination) in eight (32%) (31, 34–36, 44, 47, 52, 53). Only nine (36%) provided an executive summary of recommendations (31, 38, 40, 42, 45–46, 50, 51, 54), 20 (80%) provided a synopsis of supporting evidence (33, 35–40, 42–54), and three (12%) quantified the benefit of the CPG (50, 51, 53). Interrater raw agreement was 98% for data abstraction.
Quality and Determinants of Quality.
Individual document quality scores are presented in Appendix 2. Raw agreement between reviewers for document quality was 85%. Overall quality was low (Table 2). CPGs had higher quality than CCSs; the differences in mean (95% confidence interval) values of the total scores (CPG − CCSs) were 1.3 (0.5, 2.2; p = .003) for the Grilli instrument, 4.4 (1.4, 7.3; p = .006) for the Shaneyfelt instrument, and 16.2 (7.2, 25.2; p = .001) for the AGREE instrument. There was significant pair-wise correlation between the instruments of the quality mean scores of these documents (Pearson's correlation coefficients: 0.91 for AGREE–Shaneyfelt, 0.71 for Shaneyfelt–Grilli, 0.63 for AGREE–Grilli; p = .01 for all correlations).
CPGs scored significantly higher than CCSs in the evidence identification and summary and formulation of recommendations subsections of the Shaneyfelt instrument (16) (Table 3) (mean differences of 2.5 [1.0, 3.9; p = .002] and 1.0 [0.3, 1.8; p = .007], respectively). CPGs scored higher in other subsections, but differences were not statistically significant. CPGs performed well (mean ≥50% of total possible score) in two subsections: details of development process and details of recommendations. CCSs performed poorly in all subsections.
CPGs scored significantly higher than CCSs in four of six domains of the AGREE instrument (27) (Table 3): rigor of development (p = .002), clarity and presentation (p < .001), applicability (p = .05), and editorial independence (p = .001). Using AGREE definitions (27), 12 documents (48%) were not recommended for use (31–33, 34, 42, 46–48, 51–53, 55), six (24%) were recommended for use with provisos or alterations (33, 35, 39, 41, 44, 50), and only six (24%) were strongly recommended for use (37, 38, 40, 43, 45, 54). Limited data suggest that the quality of CPGs has improved significantly in the past 20 yrs (Fig. 2) (40-point increase in the total AGREE score, p < .001) (27), whereas the quality of CCSs remained essentially unchanged (10-point increase, p = .52) out of a possible total score of 92.
In this systematic review of the quality of 25 critical care CCSs and CPGs, we found modest adherence to current methodologic standards. CPGs were of higher quality than CCSs, although both types of documents had methodologic weaknesses in the domains of description of stakeholder involvement, rigor of development, applicability in practice, and editorial independence. Using standard CPG quality appraisal instruments, we found that the quality of CPGs seems to have improved over time, whereas that of CCSs has not.
Although the original mandate of CCSs included assessing and summarizing research (21, 22, 56), their role has evolved to incorporate clinical recommendations (35, 41, 46, 50, 51). Without instruments to evaluate CCSs, we used CPG appraisal instruments to evaluate their quality. Given the low scores of CCSs, and because they include recommendations for clinical practice (23), their content, format, or role could be reexamined. For example, CCSs could be reformulated as distilled sources of referential knowledge with the aim of focusing on the description of new or sparse evidence. Because clinicians have an unmet need for rapid-advice documents, streamlined CCS development may be helpful for emerging healthcare problems in the face of changing, incomplete knowledge. A recent example is the World Health Organization statement on pharmacologic management of avian flu (57). Rapid-advice documents could help to produce more timely clinical recommendations. In addition, the process of developing critical care CCSs has other benefits. Often, consensus conferences are focused on challenging, topical, and controversial issues. The meetings themselves are sometimes open to members of the general ICU community. CCSs organized by groups, such as the Society of Critical Care Medicine, American Thoracic Society, and the European Society of Intensive Care Medicine, provide a unique opportunity for international experts to convene and exchange research findings of global importance, which often produces valuable summary statements or broadly applicable recommendations. Finally, an important component of these documents in the future may be to serve as a primary role for generating research agendas.
Poor guideline quality is an important barrier to their use (58). Clinicians report placing little value on non–evidence-based CPGs (58). We found that more explicit identification and interpretation of the evidence contribute to higher-quality CPGs. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) (24) framework provides a systematic and explicit approach to making judgments about the validity of research evidence. More consistent use of the GRADE approach may increase the quality of CPGs and make more transparent the methods by which scientific evidence, experience, and values are integrated into clinical recommendations.
We found that both CPGs and CCSs performed poorly in the AGREE category of editorial independence. This is consistent with previous work showing that conflicts of interest influence research conduct and reporting of study results (59). We also found that 40% of documents did not report on conflicts of interest, as others have found (60, 61). Although panel members disclosed no financial conflicts in 80% (12/15) of reports addressing this issue, such conflicts are likely underreported (60, 61). Recently, the ATS Conference on the ethical conduct of clinical research involving critically ill patients strongly recommended that the critical care community should “disclose [conflicts of interest] whenever possible during the process of research dissemination” (59).
Strengths of our systematic review include the quality assessment of two types of knowledge-translation documents (CPGs and CCSs) prevalent in the critical care literature. For the topics we addressed, our literature search was comprehensive. Finally, we undertook duplicate appraisal of quality using three different instruments.
Limitations of this systematic review include our reliance on self-reports of document development, which may underestimate true document quality. On the other hand, reporting of process (e.g., application of an evidence grading system) does not guarantee optimal implementation of the methods. To address these limitations, CPG reporting standards have recently been developed (62), and major international professional guideline development societies (e.g., the National Institute for Health and Clinical Excellence [NICE], Scottish Intercollegiate Guidelines Network [SIGN]) have detailed procedures to ensure adequate training of guideline developers (63, 64). The National Institute for Health and Clinical Excellence is likely the organization devoting the most resources and considerable expertise to improving guideline development. The methods of the National Institute for Health and Clinical Excellence are under continuous development and have undergone a number of improvements. Interestingly, experts (and, thus, classic expert consensus) are primarily involved as consultants but not as authors of recommendations. Second, only one of the three instruments we used was extensively validated (AGREE) (27), which may limit inferences drawn from these results. Third, the appraisal scores corresponding to a clinically important difference in document quality has not been defined. Finally, we did not incorporate every CPG topic relevant to critical care. Other publication types such as task force documents (addressing the structure of the ICU and critical care training programs) could also be formally evaluated in the future.
Overall, the current quality of critical care CPGs and CCSs leave room for improvement. Because these documents are common and resource-intensive to produce, our findings have important implications. This analysis supports the rationale for guideline development standards (GRADE ) and enhanced reporting transparency (Conference on Guideline Standardization [COGS]) (62). These advances could help to produce higher-quality documents that both educate clinicians and inform the process of care in the ICU. Further research evaluating the effect of CPGs and CCSs on patient outcomes is needed.
We thank France Clarke for help with the data abstraction of the French documents and Dr. Roman Jaeschke for thoughtful suggestions.
1. Cook DJ, Ellrodt AG: Systematic reviews, economic evaluations, and practice guidelines
: Can research synthesis help us care for the critically ill? New Horiz
2. Cook DJ: Practice guidelines
: An emerging synthetic science. Intensive Care Med
3. Wall RJ, Dittus RS, Ely EW: Protocol-driven care in the intensive care unit: A tool for quality
. Crit Care
4. Ibrahim EH, Kollef MH: Using protocols to improve outcomes of mechanically ventilated patients: Focus on weaning and sedation. Crit Care Clin
5. Kollef MH: Improving the appropriateness and effectiveness of ICU practices. Intensive Care Med
6. Ely EW, Meade MO, Haponik EF, et al: Mechanical ventilation weaning protocols driven by nonphysician health-care professionals: Evidence-based clinical practice guidelines
7. Dodek P, Keenan S, Cook D, et al: Development of clinical practice guideline for prevention of ventilator-associated pneumonia (VAP): The Canadian Critical Car Trials Group and the Canadian Critical Care
Society. Am J Respir Crit Care Med
8. Ibrahim EH, Ward S, Sherman G, et al: Experience with a clinical guideline for the treatment of ventilator-associated pneumonia. Crit Care Med
9. McKinley BA, Moore FA, Sailors RM, et al: Computerized support for mechanical ventilation of trauma induced ARDS: Results of a randomized clinical trial. J Trauma
10. Brook AD, Ahrens TS, Schaiff R, et al: Effect of nursing-implemented protocol on the duration of mechanical ventilation. Crit Care Med
11. Doig GS, Martin CM, Sibbald WJ: Implementing evidence-based guidelines
for ICU nutrition improves hospital outcome: A cluster-randomization trial. Am J Respir Crit Care Med
12. Pitimana-aree S, Forrest D, Brown G, et al: Implementation of a clinical practice guideline for stress ulcer prophylaxis increases appropriateness and decreases cost of care. Intensive Care Med
13. Sinuff T, Cook DJ, Randall J, et al: Evaluation of a practice guideline for noninvasive positive pressure ventilation for acute respiratory failure. Chest
14. McMullin J, Cook D, Griffith L, et al: Minimizing errors of omission: Behavioural Reenforcement of Heparin to Avert Venous Emboli. The BEHAVE study. Crit Care Med
15. Grilli R, Magrini N, Penna A, et al: Practice guidelines
developed by subspecialty societies: The need for a critical appraisal. Lancet
16. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J: Are guidelines
? The methodological quality
of clinical practice guidelines
in the peer-reviewed medical literature. JAMA
17. Burgers JS, Cluzeau FA, Hanna SE, et al: Characteristics of high-quality guidelines
: Evaluation of 86 clinical guidelines
developed in ten European countries and Canada. Int J Technol Assess Health Care
18. Lacasse Y, Ferreira I, Brooks D, et al: Critical appraisal of clinical practice guidelines
targeting chronic obstructive pulmonary disease. Arch Intern Med
19. Heffner JE, Ellis R: The guideline approach to chronic obstructive pulmonary disease: How effective? Respir Care
20. Sinuff T, Cook DJ, Heyland D, et al: Facilitating guideline adherence in the intensive care unit: A qualitative study. Crit Care Med
21. National Institutes of Health (NIH) Consensus Development Conference on Critical Care
Medicine. Crit Care Med
22. National Institutes of Health Guidelines
for the Selection and Management of Consensus Development Conferences. Bethesda, MD, Office of Medical Applications of Research, September 1988
23. Sinuff T, Eva K, Meade M, et al: Information resources for ICU decision-making: Attitudes towards clinical practice guidelines
. Proc Am J Respir Crit Care Med
24. Atkins D, Best D, Briss PA, et al: Grading quality
of evidence and strength of recommendations. BMJ
25. Moher D, Fortin P, Jadad AR, et al: Completeness of reporting of trials published in languages other than English: Implications for conduct and reporting of systematic reviews. Lancet
26. Vlayen J, Aertgeerts B, Hannes K, et al: A systematic review
of appraisal tools for clinical practice guidelines
: Multiple similarities and one common deficit. Int J Qual Health Care
27. AGREE Collaboration: Development and validation of an international appraisal instrument for assessing the quality
of clinical practice guidelines
. The AGREE project. Qual Saf Health Care
28. Cluzeau F, Littlejohns P, Grimshaw J, et al: Development and application of a generic methodology to assess the quality
of clinical guidelines
. Int J Qual Health Care
29. Norman GR, Streiner DL: Measures of association for categorical data. In:
Biostatistics: The Bare Essentials. Second Edition. Hamilton, Canada, BC Decker, 2000, pp 217–223
30. Moher D, Cook DJ, Eastwood S: Improving the quality
of reports of meta-analyses of randomized controlled trials: QUOROM statement. Lancet
31. Plummer AL, Gracey DR: Consensus conference on artificial airways in patients receiving mechanical ventilation. Chest
32. Richard C, Arich C, Gajdos P, et al: XIIIth Consensus Conference in Intensive Care and Emergency Medicine: Assisted respiration during acute decompensation in chronic respiratory failure in adults (excluding neuromuscular pathology and weaning). Therapie
33. Slutsky AS: Mechanical ventilation: American College of Chest Physicians’ Consensus Conference. Chest
34. Society of Critical Care
for standards of care for patients with acute respiratory failure on mechanical ventilatory support. Crit Care Med
35. Artigas A, Bernard GR, Carlet J, et al: The American-European Consensus Conference on ARDS: Part 2. Ventilatory, pharmacologic, supportive therapy, study design strategies, and issues related to recovery and remodeling: Acute respiratory distress syndrome. Am J Respir Crit Care Med
1998; 157(4 Pt 1):1332–1347
36. Sevransky JE, Levy MM, Marini JJ: Mechanical ventilation in sepsis-induced acute lung injury/acute respiratory distress syndrome: An evidence-based review
. Crit Care Med
37. MacIntyre NR, Cook DJ, Ely EW Jr, et al: Evidence-based guidelines
for weaning and discontinuing ventilatory support: A collective task force facilitated by the American College of Chest Physicians, the American Association for Respiratory Care, and the American College of Critical Care
38. Ely EW, Meade MO, Haponik EF, et al: Mechanical ventilator weaning protocols driven by nonphysician healthcare professionals: Evidence-based clinical practice guidelines
39. Evans TW: International Consensus Conferences in Intensive Care Medicine: Non-invasive positive pressure ventilation in acute respiratory failure. Organised jointly by the American Thoracic Society, the European Respiratory Society, the European Society of Intensive Care Medicine, and the Societe de Reanimation de Langue Francaise, and approved by the ATS Board of Directors, December 2000. Am J Respir Crit Care Med
40. British Thoracic Society Standards of Care Committee: Non-invasive ventilation in acute respiratory failure: BTS Guideline. Thorax
41. MacIntyre NR, Hess DR, Brougher P, et al: Consensus conference: Noninvasive positive pressure ventilation. Respir Care
42. American Thoracic Society: Guidelines
for the management of adults with hospital-acquired, ventilator-associated, and healthcare-associated pneumonia. Am J Respir Crit Care Med
43. Dodek P, Keenan S, Cook D, et al: Evidence-based clinical practice guideline for the prevention of ventilator-associated pneumonia: The Canadian Critical Care
Trials Group and the Canadian Critical Care
Society. Ann Intern Med
44. Tablan OC, Anderson LJ, Besser R, et al: Guidelines
for preventing health-care-associated pneumonia, 2003: Recommendations of CDC and the Healthcare Infection Control Practices Advisory Committee. MMWR Recomm Rep
46. Hubmayr RD, Burchardi H, Elliot M, et al: Statement of the 4th International Consensus Conference in Critical Care
on ICU-Acquired Pneumonia: Chicago, Illinois, May 2002. Intensive Care Med
47. Hospital-acquired pneumonia in adults: Diagnosis, assessment of severity, initial antimicrobial therapy, and preventive strategies. A consensus statement, American Thoracic Society, November 1995. Am J Respir Crit Care Med
48. Tablan OC, Anderson LJ, Arden NH, et al: Guideline for prevention of nosocomial pneumonia: The Hospital Infection Control Practices Advisory Committee. Am J Infect Control
49. International Consensus Conferences in Intensive Care Medicine: Ventilator-associated Lung Injury in ARDS. This official conference report was cosponsored by the American Thoracic Society, The European Society of Intensive Care Medicine, and The Societé de Réanimation de Langue Française, and was approved by the ATS Board of Directors, July 1999. Am J Respir Crit Care Med
50. Geerts WH, Pineo GF, Heit JA, et al: Prevention of venous thromboembolism: The Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest
51. Geerts WH, Heit JA, Clagett GP, et al: Prevention of venous thromboembolism. Chest
2001; 119(1 Suppl):132S–175S
52. Nicolaides AN, Breddin HK, Fareed J, et al: Prevention of venous thromboembolism: International Consensus Statement. Guidelines
compiled in accordance with the scientific evidence. Int Angiol
53. Nicolaides AN, Bergvist D, Hull R, et al: Prevention of venous thromboembolism. International Consensus Statement (guidelines
according to scientific evidence). Int Angiol
54. American Society of Health-System Pharmacists (ASHP) therapeutic guidelines
on stress ulcer prophylaxis: ASHP Commission on Therapeutics and approved by the ASHP board of directors on November 14, 1998. Am J Health Syst Pharm
55. Tempe JD, Colin R, Florent C, et al: Consensus Conference on resuscitation and emergency medicine: Prevention of stress ulcer related gastroduodenal hemorrhage. Rean Soins Intens Med Urg
58. Cabana MD, Rand CS, Powe NR, et al: Why don't physicians follow clinical practice guidelines
? A framework for improvement. JAMA
59. Luce JM, Cook DJ, Martin TR, et al: The ethical conduct of clinical research involving critically ill patients in the United States and Canada: Official Workshop Report of the American Thoracic Society. Am J Respir Crit Care Med
60. Choudhry NK, Stelfox HT, Detsky AS: Relationship between authors of clinical practice guidelines
and the pharmaceutical industry. JAMA
61. Reichhardt T: Cash interests taint drug advice. Nature
62. Shiffman RN, Shekelle P, Overhage JM, et al: Standardized reporting of clinical practice guidelines
: A proposal from the conference on guideline standardization. Ann Intern Med
*See also p. 1360.