A thoughtful critique in a recent News and Views described the dangers of consensus statements and the weaknesses of practice guidelines developed by specialty societies. The author concluded that journals should establish standards for consensus statements or it might be better to depend on traditional narrative reviews (1). There is an alternative approach, submitted here, that societies themselves should adopt and adhere to rigorous standards for the development of high-quality, evidence-based guidelines.
Studies on the methods used to develop clinical guidelines have described in detail the attributes of a good guideline and the means of creating one. One study which looked at compliance with 25 methodological standards found that 43% of standards were adhered to in 279 guidelines published from 1985 to 1997 (2). Another study which assessed compliance with three standards found that only 5% of 431 guidelines produced by specialty societies from 1988 to 1998 met all three standards (3). This failure of specialty societies to produce methodologically sound guidelines does not mean that it is an impossible task. In fact, in 1997 the North American Society for Pediatric Gastroenterology, Hepatology and Nutrition (NASPGHAN) adopted rigorous standards for development of its clinical practice guidelines. It has subsequently produced three guidelines that adhere to these standards and is currently preparing three more (4–6). The NASPGHAN method, based on a similar approach of the American Academy of Pediatrics, is described in the following paragraphs.
1. A disorder is identified in which there is variability in practice, a body of scientific evidence, and the likelihood that a change in management could result in a significant improvement in outcome.
2. A guideline committee for that disorder is established. Participants are selected on the basis of their research and publications, reputation in the field, the ability to collaborate, and a willingness to commit both time and effort. The participants are chosen to represent a wide range of opinions. The guideline committee must include representatives from primary care and clinical epidemiology as well as pediatric gastroenterology.
3. The committee specifically defines the problem to be addressed, the type of patients to whom the guideline will apply, the diagnostic and therapeutic interventions to be considered, the practice setting (office, hospital), the desired outcome of the guideline, and the target population of physicians (primary care, specialists, nurse practitioners).
4. The committee develops a seed algorithm of the step-by-step process by which a patient would be managed. The algorithm is based on the current knowledge and experience of the members.
5. The seed algorithm is used to prepare a list of critical steps for which a literature search will be performed.
6. A rigorous literature search is performed to collect evidence for each critical step. The content and experimental methodology of each article is rated: level I (randomized controlled trials), level II-1 (controlled trials without randomization), level II-2 (cohort or case-control analytic studies), level II-3 (multiple time series), and level III (descriptions of clinical experience) (7).
7. For each critical step an evidence table is prepared, with an analysis and a quality rating of each article.
8. The guideline committee examines, discusses, and interprets the evidence on each critical step.
9. The committee develops recommendations on patient management. These recommendations are based on a combination of the evidence reviewed and the clinical experience of the members. When useful, algorithms are developed to illustrate the recommended management.
10. All recommendations and other decisions are based on consensus achieved by the Nominal Group Technique (8,9). This technique is a structured quantitative method of secret ballot that gives equal importance to the opinion of each committee member, and requires agreement by all members for consensus to be achieved. On each critical step discussion continues until consensus is achieved or until it is decided that consensus cannot be achieved (a rare occurrence). Minutes are kept regarding all discussions and votes.
11. A manuscript is drafted and revised by the entire committee. Once a satisfactory draft is completed it is sent to as many as 50 primary care, pediatric gastroenterology, and other pediatric specialists for review. Constructive suggestions are incorporated into subsequent drafts. A penultimate draft is distributed to the entire NASPGHAN membership for review and comment.
12. A final draft is submitted to the NASPGHAN Executive Council for review and approval. Guidelines have also received the endorsement of the governing board of the American Academy of Pediatrics.
13. The clinical practice guideline is published in the Journal of Pediatric Gastroenterology and Nutrition without further review, and is posted on the NASPGHAN web site, www.naspghan.org.
NASPGHAN believes that this process, which takes two to three years, produces an objective, practical, and useful clinical practice guideline. Each guideline explicitly describes the process of guideline development and contains the caveat that the recommendations are general and not intended to substitute for clinical judgment or as a protocol for the treatment of all patients. The quality of the evidence for each recommendation is indicated. Furthermore, the guidelines require updating as the findings of new research alter recommendations for management.
Do guidelines work? An analysis of 59 published evaluations of clinical guidelines that met defined criteria for scientific rigor demonstrated that 93% of the guidelines produced significant improvement in the process of care and 71% produced significant improvement in the outcome of care (10). Ideally, development of a guideline is the first step in a process of quality improvement. Education, auditing, and incentives are necessary to bring about changes in clinical practice that produce improvements in the quality of care.
It is important to distinguish between informal consensus development, formal consensus development, and evidence-based guideline development (11). In pediatric gastroenterology, as in other pediatric subspecialties, there is a dearth of published level I evidence. As a result, only one-third of NASPGHAN clinical guideline recommendations for management are based on level I evidence. Nonetheless, it is important for NASPGHAN to continue to develop clinical guidelines based on a combination of evidence and clinical experience. If it does not, physicians or organizations seeking to improve, standardize, or evaluate care will only have available to them less objective guidelines or guidelines inappropriate for the pediatric patient (12).
1. Drumm B. The dangers of consensus. J Pediatr Gastroenterol Nutr 2001; 33:429–30.
2. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 1999; 281:1900–05.
3. Grilli R, Magrini N, Penna A, Liberati A. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 2000; 355:103–06.
4. Baker SS, Liptak GS, Colletti RB, Croffie JM, DiLorenzo C, Ector W, Nurko S. Constipation in infants and children: evaluation and treatment. J Pediatr Gastroenterol Nutr 1999; 29:612–26.
5. Gold BD, Colletti RB, Abbott M, Czinn SJ, Elitsur Y, Hassall E, MacArthur C, Snyder J, Sherman PM. Helicobacter pylori infection in children: recommendations for diagnosis and treatment. J Pediatr Gastroenterol Nutr 2000; 31:490–97.
6. Rudolph C, Mazur LJ, Liptak GS, Baker R, Boyle JT, Colletti RB, Gerson W, Werlin S. Guidelines for evaluation and treatment of gastroesophageal reflux in infants and children: recommendations of the North American Society for Pediatric Gastroenterology and Nutrition. J Pediatr Gastorenterol Nutr 2001; 32:S1–S31.
7. Canadian Task Force on the Periodic Health Examination. The periodic health examination. Can Med Assoc J
8. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995; 311:376–80.
9. McMurray AR. Three decision-making aids. Brainstorming, nominal group, and Delphi technique. J Nurs Staff Dev
10. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993; 342:1317–22.
11. Woolf SH. Practice guidelines, a new reality in medicine. II. Methods of developing guidelines. Arch Intern Med 1992; 152:946–52.
12. American Academy of Pediatrics Task Force on Medical Management Guidelines. Guiding principles, attributes, and process to review medical management guidelines. Pediatr