Figure 2 illustrates a second kind of indirect evidence that we included. In some studies, an intervention was introduced to the entire study sample (without a comparison group). Change in an important health outcome was measured, and then authors looked for associations between surrogate measures and the health outcome. For example, in the chapter on Cerebral Perfusion Pressure, Downard et al (3) conducted a retrospective analysis of 118 patients who were all treated for severe traumatic brain injury and assessed for Glasgow Outcome Scale score at ≥3 months, dichotomized as “good” or “poor.” Then, using a logistic regression analysis, they looked for significant associations between cerebral perfusion pressure, a surrogate measure, and outcome. Lower cerebral perfusion pressure was associated with poorer outcomes. This association was used as weak class III evidence for the chapter's recommendations. In Figure 2, arc A′ represents an uncontrolled association between an intervention and an important health outcome, and arc B′ represents a correlation between measures on the surrogate outcome and the important health outcome. Studies were included if they contained both components of the indirect evidence illustrated in Figure 2.
IV. DATA ABSTRACTION AND SYNTHESIS
Remaining blinded, coauthors read each publication and abstracted data using an evidence table template (Appendix E). They compared results of their data abstraction and through consensus finalized the data tables that constitute the evidence on which the recommendations are based. As a result of heterogeneity of studies within topics, and the lack of literature of adequate quality, data were not combined quantitatively.
Coauthors drafted manuscripts for each topic. The entire team gathered for a 2-day work session to discuss the literature base and craft the recommendations. Manuscripts were revised. Virtual meetings were held with a subset of the coauthors to complete the editing process. The final draft manuscript was circulated to the peer review panel and was revised incorporating selected peer review input.
V. QUALITY ASSESSMENT OF INDIVIDUAL STUDIES AND CLASSIFICATION OF EVIDENCE
In April of 2004, the BTF established a formal collaboration with the Evidence-Based Practice Center from Oregon Health & Science University. Center staff worked with two Evidence-Based Practice Center epidemiologists to develop criteria and procedures for the quality assessment of individual studies and classification of level of evidence provided by each included study. These criteria are designed to assess risk of bias for individual studies based on study design and conduct. Criteria for classification of evidence are in Table 1 and are derived from criteria developed by the U.S. Preventive Services Task Force (4), the National Health Service Centre for Reviews and Dissemination (U.K.) (5), and the Cochrane Collaboration (6). These criteria were used to assess the literature.
Three members of the Center staff, two of whom are Evidence-Based Practice Center epidemiologists, conducted all of the quality assessments. Two assessors, blinded to each other's work and to publication identification, read the selected studies and classified them as class I, II, or III based on the criteria in Table 2. Discrepancies were resolved through consensus or through a third person's review.
Class I evidence is derived from randomized controlled trials. However, some randomized controlled trials may be poorly designed, lack sufficient patient numbers, or suffer from other methodologic inadequacies.
Class II evidence is derived from clinical studies in which data were collected prospectively and retrospective analyses that were based on clearly reliable data. Comparison of two or more groups must be clearly distinguished. Types of studies include observational, cohort, prevalence, and case–control. Class II evidence may also be derived from flawed randomized controlled trials.
Class III evidence is derived from prospectively collected data that are purely observational and retrospectively collected data. Types of studies include case series, databases, or registries. Class III evidence may also be derived from flawed randomized controlled trials or flawed observational, cohort, prevalence, or case–control studies.
VI. QUALITY OF BODY OF EVIDENCE AND STRENGTH OF RECOMMENDATIONS
At the beginning of each recommendation section in this document, the recommendations are categorized in terms of strength and quality of evidence. The strength of the recommendation is derived from the overall quality of the body of evidence used to assess the topic.
Quality of Body of Evidence
The underlying methods for assessing risk of bias for individual studies are represented in Table 1. However, ultimately the individual studies must be considered in aggregate, whether through meta-analyses or through qualitative assessment. Thus, the strength of recommendations must be derived from the quality of the overall body of evidence used to address the topic.
Consistent with recommendations for grading a body of evidence adopted by the Agency for Healthcare Research and Quality (7), we assessed the overall quality of the body of evidence considering the domains of 1) risk of bias from individual studies; 2) consistency of findings across studies; 3) directness of evidence; and 4) precision of estimates of effect. The quality of the overall body of evidence for each recommendation in this document is classified as high, moderate, or low. Factors that may decrease the quality include potential bias, differing findings across studies, the use of indirect evidence, or lack or precision. For example, if two or more class I studies demonstrate contradictory findings for a particular topic, the overall quality most probably will be low because there is uncertainty about the effect. Similarly, class I or II studies that provide indirect evidence may only constitute low-quality evidence, overall.
Strength of Recommendations
Consistent with methods generated by the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) Working Group, recommendations in this document are categorized as either strong or weak. As stated in the American Thoracic Society's official statement (8), in which they endorsed the GRADE methods for their guidelines endeavors, “The strength of a recommendation reflects the degree of confidence that the desirable effects of adherence to a recommendation outweigh the undesirable effects.”
Strong recommendations are derived from high-quality evidence that provides precise estimates of the benefits or downsides of the topic being assessed. With weak recommendations, 1) there is lack of confidence that the benefits outweigh the downsides; 2) the benefits and downsides may be equal; and/or 3) there is uncertainty about the degree of benefits and downsides.
a One randomized controlled trial had a sample of 24 patients (Kloti, 1987) and one a sample of 18 (Fisher, 1992).
b One retrospective review had a sample of 24 patients (Pfenninger, 1983).
c One study included 16% of patients with moderate traumatic brain injury (Downard, 2000).
1. Badjatia N, Carney N, Crocco T, et al.: Guidelines for the prehospital management of severe traumatic brain injury, 2nd edition. Prehosp Emerg Care 2007; 12:S1–S52
2. Bratton S, Bullock R, Carney N, et al.: Guidelines for the management of severe traumatic brain injury, 3rd edition. J Neurotrauma 2007; 24(Suppl 1):S1–S105
3. Downard C, Hulka F, Mullins RJ, et al.: Relationship of cerebral perfusion pressure and survival in pediatric brain-injured patients. J Trauma 2000; 49:654–658; discussion 658–659
4. Harris RP, Helfand M, Woolf SH, et al.: Current methods of the third U.S. Preventive Services Task Force. Am J Prev Med 2001; 20:21–35
5. Anonymous: Undertaking Systematic Reviews of Research on Effectiveness: CRD's Guidance for Those Carrying Out or Commissioning Reviews. CRD Report Number 4. Second Edition). York, UK, NHS Centre for Reviews and Dissemination, 2001, p 4
6. Mulrow CD, Oxman AD: How to Conduct a Cochrane Systematic Review. Version 3.0.2. Presented at Cochrane Collaboration, San Antonio, TX, 1997
7. Owens D, Lohr K, Atkins D, et al.: Grading the strength of a body of evidence when comparing medical interventions—Agency for Healthcare Research and Quality and the Effective Health Care Program. J Clin Epidemiol 2010; 63:513–523
©2012The Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies
8. Schunemann H, Jaeschke R, Cook D, et al.: An official ATS statement: grading the quality of evidence and strength of recommendations in ATS guidelines and recommendations. Am J Respir Crit Care Med 2006; 174:605–614