Evaluation of the Quality of Guidelines for Acute Gastroenteritis in Children With the AGREE Instrument

Abstract

Aim: 

The aim of the study was to assess the quality of clinical practice guidelines (CPGs) using the Appraisal of Guidelines for Research and Evaluation (AGREE) instrument, a validated international tool.

Materials and Methods: 

CPGs were identified by searching MEDLINE (1966–January 2009) and Embase (1988–January 2009), CPG databases, and relevant Web sites of agencies and organizations that produce and/or endorse guidelines. Included in the study were CPGs in English that addressed the management of acute gastroenteritis in children. Retrieved CPGs were evaluated with the AGREE instrument for quality assessment by 6 independent reviewers. AGREE consists of 6 domains for a total of 23 items.

Results: 

Nine CPGs were identified. Four were evidence based (EB) and 2 of these included tables of evidence. Eight CPGs (88%) scored <50% for “applicability,” 7 (77%) for “stakeholder involvement,” and 6 (66%) for “editorial independence.” Compared with non-EB CPGs, EB CPGs had higher quality scores for all AGREE domains, with a better score for “rigor of development” (P < 0.001), “stakeholder involvement” and “clarity of presentation” (P < 0.01), and applicability (P < 0.05). Over time, the quality of guidelines tended to improve. The main recommendations of CPGs were similar. However, there were differences in the treatment of diarrhea, namely based on the settings and circumstances in which CPGs were produced.

Conclusions: 

The overall quality of CPGs on acute gastroenteritis management in children is fair. Aims, target population, synthesis of evidence, formulation of recommendations, and clarity of presentation are points of strength. Weak issues are applicability, including identification of organizational barriers and adherence parameters, and cost/efficacy analysis.

Clinical practice guidelines (CPGs) are systematically developed statements to assist practitioners in making decisions about appropriate health care in specific clinical circumstances ⁽¹⁾. Their purpose is to make explicit recommendations with a definite intent to influence what clinicians do. The primary goal of CPGs in pediatrics is to improve the health of infants and children by ensuring that they receive up-to-date, evidence-based (EB) care. CPGs are a major tool to improve the quality of care. Several studies have shown that adherence to EB guidelines leads to improvement in the quality of care provided ^(2,3). For many health conditions, there is a gap between what medical science has shown to be effective practice and what is actually done ⁽⁴⁾.

The number of CPGs is rapidly mounting also in pediatrics. However, the plethora of CPGs has been accompanied by growing concern about differences among guideline recommendations and about the quality of guidelines ^(5–8). How does one define the quality of guidelines? A “good” guideline should be scientifically valid, usable, and reliable, and should improve the outcome of patients; however, it is rarely known how a guideline performs in clinical practice. Evaluation of CPGs should include both methods used to develop recommendations and applicability of recommendations (benefits, adverse effects, and costs).

An international group of researchers, the Appraisal of Guidelines for Research and Evaluation (AGREE) Collaboration, developed and validated a specific instrument to assess the quality of CPGs based on theoretical assumptions ⁽⁹⁾. A recent assessment of the quality of pediatric guidelines with the AGREE instrument demonstrated better results for pediatric than for adult CPGs ⁽¹⁰⁾. The best performers were CPGs published and endorsed by the American Academy of Pediatrics (AAP) or registered in the National Guidelines Clearinghouse (NGC).

Acute gastroenteritis (AGE) remains a common cause of morbidity and mortality among infants and children worldwide. In industrialized countries, the disease is relatively mild and generally self-limiting, but nevertheless can have a major effect on the quality of life of infected children and their families. AGE is a major cause of outpatient visits and hospital admissions in developed countries, and consequently it has a substantial effect on health costs. Several guidelines for the management of AGE in children are available. However, only a minority of physicians fully comply with guidelines, and recommendations are only slowly put into practice ^(11–14).

The aims of the present study were to assess the quality of guidelines on AGE in children using the AGREE instrument, and to identify strengths and weaknesses, with the ultimate aim of improving the quality and applicability of guidelines. Despite that the specific recommendations contained in the CPGs were not a specific aim of the present article, we briefly reviewed the main recommendations of the CPGs to see whether there were major differences. We therefore also investigated the main indications of CPGs in parallel with the evaluation of their quality.

MATERIALS AND METHODS

Literature Search

We searched MEDLINE (1966–January 2009) and Embase (1988–January 2009) with the following terms: gastroenteritis (MeSH or text word), or diarrhea (MeSH or text word), and practice guidelines publication type, and infants or child preschool or child (MeSH), with the limit “English language.” We also looked at relevant Web sites of agencies that produce and/or endorse CPGs, namely, the AAP (www.aap.org), Morbidity and Mortality Weekly Report (www.cdc.gov/mmwr), European Society for Pediatric Gastroenterology, Hepatology, and Nutrition (www.espghan.org), and North American Society of Pediatric Gastroenterology, Hepatology, and Nutrition (www.naspghan.org). We also searched the NGC (www.guideline.gov).

Inclusion/Exclusion Criteria

We included CPGs on the management of AGE that included specific recommendations for infants and children. In cases of studies that referred to or endorsed previous publications, we evaluated the original document. We excluded guidelines that referred also to adults, as well as guidelines on diarrhea prevention, vaccination, surgery, or other rare diseases.

Appraisal of Guidelines With the AGREE Instrument

The AGREE instrument is an international validated instrument for the evaluation of guidelines development methodology. This instrument consists of 23 items organized in 6 domains. Each item is scored with a 4-point Likert scale. The scale ranges from 4 (strongly agree) to 1 (strongly disagree), with 2 midpoints: 3 (agree) and 2 (disagree). Each of the 6 domains assesses a dimension of guideline quality: scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, applicability, and editorial independence. The domain score is expressed as a percentage of the maximum possible score for that domain and is obtained by summing the scores of individual items. A 3-point scale (1 = not recommended, 2 = recommended with provisos or modifications, and 3 = strongly recommended) provides an overall judgment on whether the guideline ought to be recommended for use. Although there is no threshold score, a domain scoring <50% is usually considered to be of limited use.

According to AGREE, each guideline should be assessed by at least 2 appraisers to increase the reliability of the assessment. We invited 12 scientists who have experience in CPG evaluation to collaborate in the present study. Five accepted (S.D.M., A.Gi., M.T.O., L.P., and C.D.) and were enrolled together with A.L.V. to evaluate the 9 CPGs identified using the AGREE instrument.

Appraisal of Agreement Between Reviewers

We used the free-marginal multirater kappa (multirater k_free) as a measure of agreement between reviewers. This statistical instrument is appropriate for a typical agreement study ⁽¹⁵⁾. The Fleiss multirater kappa, generally used to assess agreement between more than 2 raters, is influenced by prevalence and biases, which can lead to the paradox of high agreement but low kappa. To overcome this problem, we decided to dichotomize the response categories agree/strongly agree versus disagree/strongly disagree. The result estimates the degree of agreement in classification over that expected by chance and is scored as a percentage. The k-statistic was then applied to each of the 23 items of the AGREE instrument. Quality scores of EB and non-EB CPGs and, successively, scores of CPGs published before and after AGREE instrument validation were compared by the t test. We investigated the indications to use oral rehydration solution (ORS; recommended osmolality), refeeding, and indications to active treatment.

Finally, we investigated the similarities and differences of the main recommendations of the guidelines in parallel with the evaluation of their quality.

RESULTS

A total of 237 citations identified through a computerized search and Web site consultation performed with selected search terms were screened. Nine CPGs that met inclusion and exclusion criteria were identified ^(16–24). All selected guidelines were available in the MEDLINE database. Two of them were listed also in the NGC, 1 in the European Society for Pediatric Gastroenterology, Hepatology, and Nutrition site, and 1 appeared in both the Morbidity and Mortality Weekly Report Web site and in the AAP site. Table 1 lists the AGREE domain scores of the 9 CPGs evaluated. Only 3 CPGs ^(21,23,24) were overall assessed as “strongly recommended” by reviewers and did not need any provisos or alteration. Other 4 CPGs ^(18–20,22) were not recommended because the majority of domains did not reach scores >50%. Two CPGs ^(16,17) could be recommended with modifications.

TABLE 1: AGREE domain scores for CPGs for the management of acute gastroenteritis in children

Evaluation of the AGREE Domains of Guidelines Analyzed

Scope and Purpose

This domain evaluates the overall aim of a guideline, the clinical questions addressed, and the target population covered by the guideline. The mean score for this domain was 78.2% (range 24%–100%), with only 2 CPGs scoring <50%. This result indicates that as much as 80% of the criteria of this domain were satisfied, although a detailed description of the clinical questions covered by the guideline was rarely provided.

Stakeholder Involvement

This domain evaluates the degree of involvement of all of the parties taking part in the preparation and dissemination of the document, as well as the target of recommendation. This domain consists of questions about the composition of the working group and evaluates the influence of the patients' experiences and expectations on the development of guideline, the correct definition of target users, and pretesting among the end users. The overall score in this domain was poor with a mean of 39.8% (range 14%–88.9%). Seven of 9 (77%) CPGs scored <50%. Most of the guidelines (5/9) included relevant professional groups in the development stage, but only 1 of them was pretested by end users.

Rigor of Development

The domain evaluates the methods used to search for and select evidence, determines whether systemic methods were used to formulate recommendations, and evaluates explicit links between evidence and recommendations. The domain evaluates whether health benefits, adverse effects, and risks were considered in the formulation of the statements, whether the CPGs were externally reviewed before publication, and, finally, whether a procedure for updating was provided. The mean score for this domain was 59.1% (range 12%–95.2%), with 4 guidelines scoring <50%. Specifically, 5 guidelines described systematic methods for searching and selecting evidence and described methods to formulate recommendations. One guideline did not provide explicit links between statements and supporting evidence. All of the guidelines, to different degrees, considered health benefits, adverse effects, and risks when formulating recommendations. Prepublication external review was reported by 6 of the 9 CPGs.

Clarity and Presentation

This domain is concerned with guidelines clarity and format. The items evaluate the clarity of recommendations, the presentation of different management options, and the availability of tools for application. The mean score was 71% (range 47.2%–97.2%). Five guidelines scored >80%. However, only 3 provided specific tools for application (algorithms), and 2 other guidelines included a single page with a summary of the main recommendations that may be used as a tool for consultation.

Applicability

This domain evaluates issues pertinent to guideline implementation. More specifically, it considers organizational barriers, cost implications, and monitoring criteria. The mean score of this domain was 27.9% (range 4%–63%), the lowest of all domains. Eight of 9 guidelines scored <50%. Cost implications and potential organizational barriers, which are key elements for implementation and local application of recommendations, were poorly addressed.

Editorial Independence

Editorial independence addresses possible conflicts of interest for members of guideline expert groups, external funding or financial contributions provided to authors to support the development, publication, or dissemination of the documents, by charity organizations, government grants, or pharmaceutical companies, and their explicit declaration. The mean score for this domain was 45.4% (range 2.8%–100%).

Appraisal According to Type of Guideline and Date of Publication

Documents whose recommendations were based on the integration of clinical expertise with the best-available scientific knowledge obtained from a systematic research and a critical appraisal of the quality of proofs were considered EB CPGs. Of the 9 guidelines evaluated, 4 were EB ^{(12,16,18,19)} (only 2 included tables of evidence [18,19]). The other 5 documents were considered non-EB CPGs because the link between recommendations and supporting evidence was not explicit, and the final recommendations were usually based on the opinion of experts. EB guidelines had higher quality scores for all of the AGREE domains, when compared with non-EB CPGs (Fig. 1). The difference was statistically significant for the domains stakeholder involvement (P < 0.01), rigor of development (P < 0.001), clarity and presentation (P < 0.01), and applicability (P < 0.01). Over time, the quality of guidelines tended to improve (Fig. 2).

FIGURE 1: Comparison of mean quality score between 4 evidence-based clinical practice guidelines (EB CPGs) and 5 non-evidence-based clinical practice guidelines for each AGREE domain. ^* P < 0.01, ^§ P < 0.001, ^# P < 0.05. AGREE = Appraisal of Guidelines for Research and Evaluation.

FIGURE 2: Quality domain scores according to guideline publication date. 1. American Academy of Pediatrics ⁽¹⁶⁾; 2. Armon et al ⁽¹⁷⁾; 3. Sandhu et al ⁽¹⁸⁾; 4. Guarino and Albano ⁽¹⁹⁾; 5. King et al ⁽²⁰⁾; 6. Cincinnati Children's Hospital Medical Center ⁽²¹⁾; 7. Bhatnagar et al ⁽²²⁾; 8. Guarino et al ⁽²³⁾; 9. Harris et al ⁽²⁴⁾. All included guidelines have been plotted according to publication date. A trend toward improvement of quality scores is evident for all AGREE domains. AGREE = Appraisal of Guidelines for Research and Evaluation.

All domain scores appear slightly higher for CPGs published after AGREE instrument development and validation (2003), the difference results statistically significant for rigor of development (P = 0.04) and editorial independence (P = 0.03) domains.

Agreement Among Reviewers

The 6 raters who applied the AGREE instrument came from different medical fields. All are medical doctors with an interest in the evaluation of evidence and guideline production—2 of them are hospital pediatricians and members of the Accreditation and Quality Improvement Working Group of Italian Society of Pediatrics (M.T.O., L.P.), 1 is an epidemiologist specializing in hygiene and preventive medicine and a member of the National Guidelines System of the Italian Institute of Health (S.D.M.), 1 is a pediatric gastroenterologist with experience in guidelines production (C.D.), 1 is a pediatrician PhD (A.Gi.), and 1 is a resident in pediatrics (A.L.V.), with experience in both scientific reviewing and guideline development—and collaborators on the National Guidelines System of the Italian Institute of Health. To evaluate possible differences in quality evaluation due to cultural background, we calculated general agreement among raters and agreement according to the AGREE domain. Table 2 summarizes the degree of agreement beyond chance (multirater k_free) for the 23 items of the AGREE instrument. The k_free values indicate that overall agreement between reviewers was excellent (>80%) for 9 of 23 of the items (40%) and substantial (>60%) for 14 of 23 (60%) of the items. We next looked at the mean degree of agreement between raters according to each domain, and found no differences (data not shown).

TABLE 2: Agreement among reviewers for AGREE instrument items

Similarities and Differences Between Recommendations in the CPGs

All of the CPGs recommended early refeeding or continuous feeding with no changes in child nutrition, with full-strength infant formula or milk, without restrictions. Antibiotics were not recommended unless in clearly defined (and unusual) circumstances, and often their use was explicitly discouraged. Differences between CPGs were observed for treatment of diarrhea (Table 3). Although all of the guidelines recommended ORS as the milestone of treatment, ORS osmolality was not consistent, ranging from 45 to 90 mmol Na⁺. In addition, active therapeutic options, in adjunct to ORS, were included in selected CPGs, generally those developed more recently. Treatment included probiotics, zinc administration, or drugs active against intestinal ion secretion such as racecadotril or smectite.

TABLE 3: Differences of main therapeutic options according to guidelines

Recommendation on active treatment was generally based on careful evaluation of evidence. However, considerations were also included in selected CPGs as to whether evidence obtained in selected settings would be valid in other settings. Examples included administration of zinc or probiotics. Zinc was recommended in guidelines developed in India ⁽²²⁾, but evidence was considered insufficient in a guideline developed for European children ⁽²³⁾, based on the fact that in the latter setting there is no evidence of zinc deficiency and evidence on zinc efficacy had been obtained in settings with high prevalence of malnutrition and hence with an expected high prevalence of zinc deficiency. Conversely, selected probiotic strains were recommended in European guidelines, based on published evidence, but not in India. In addition, probiotics were not included in a CPG developed in Australia because it was clearly reported that those strains that are effective according to published evidence are not available in Australia ⁽²⁴⁾. As a general comment on facts rather than on pure available evidence, considerations were introduced in some CPGs that affected the recommendations of selected guidelines. Overall, there was no clear relation between the quality of a CPG and its specific recommendations.

DISCUSSION

According to the AGREE criteria, the overall quality of published CPGs devoted to AGE is fair, and only 3 were strongly recommended without any provisos or alteration. The quality of EB CPGs was higher than that of non-EB CPGs for all of the AGREE domains. The AGREE standards can be also used for planning, execution, and monitoring of guidelines. According to our data, the quality of CPGs for the management of AGE improved after publication of the article describing the AGREE instrument, and domains of improvement were those related to methodology and editorial independence. This may suggest that compliance with validated criteria may contribute to the development of high-quality guidelines.

The scores obtained from the AGREE evaluation were similar to those reported by Boluyt et al ⁽¹⁰⁾ in a recent study on the quality of EB pediatric CPGs for 10 frequent conditions in pediatrics. Our mean scores were slightly lower for some domains (scope and purpose 78.2% vs 84%, stakeholder involvement 39.8% vs 42%, and clarity of presentation 71% vs 78%) when compared with those reported by Boluyt. This discrepancy could be related to the inclusion of non-EB CPGs in our survey. We obtained slightly higher scores for rigor of development and editorial independence, and a considerably higher score for applicability, 27.9% vs 19%. It is likely that recommendations for the management of AGE are easier to apply given the straightforward criteria for this condition as compared with more severe conditions.

The rigor of development domain correlated directly to the extent to which an EB procedure was applied. Although all of the guidelines evaluated peer-reviewed literature, many did not report the review methodology used or the mechanisms by which recommendations were formulated. This information is required to determine whether recommendations are truly EB and to understand how evidence was analyzed. The EB CPGs had relatively high scores for clarity and presentation and stakeholder involvement.

Although the involvement of professionals and users in the steering group is a point of strength in the development process, all but 2 of the CPGs failed to provide information about patients' preferences/expectations and experiences. Only 1 guideline committee included patients' representatives as required by AGREE criteria ⁽²⁴⁾. Patient's dimension should be factored into decisions regarding clinical care, especially in common diseases such as AGE.

Almost all of the guidelines performed poorly with respect to editorial independence. This may reflect conflicts of interest between funding sources and guideline development panels or, alternatively, it may result from poor reporting on these topics. It is generally recommended that guidelines be updated every 3 years. However, in our survey, only 4 guidelines described updating procedures ^{(17,21,23,24)}.

Guidelines should consider potential barriers to implementation and provide monitoring criteria to assess a guideline's impact. In our study, the domain of applicability scored poorly in most instances. Only 1 CPG ⁽²⁴⁾ reported procedures and outcome measures for CPGs implementation. Although AGREE requires guideline committees to undertake pilot testing before publication to ensure that the guideline can be put into practice, only 1 CPG reported the results of a pilot test ⁽²⁴⁾.

Our results show that the overall quality of some pediatric AGE guidelines may be questionable. However, this is true also of other fields. Recently, with regard to cardiovascular disease CPGs, Tricoci et al ⁽²⁵⁾ warned against the increase of low-grade recommendations and statements for which the evidence available was inconclusive. This may be due to the large number of interventions for which clinical evidence is often weak, and to the “illusory attempt to embrace the entire clinical reality” ⁽²⁶⁾. Because of this ambition, small trials reporting weak evidence are often included in CPGs.

In contrast with data on CPGs for cardiovascular diseases, we observed a progressive improvement of guideline quality with time in parallel with a more frequent application of the validated and standardized criteria of EB medicine. This could be related to the straightforward management of AGE and to the restricted number of possible scenarios.

Only a minority of physicians fully comply with AGE guidelines and recommend that guidelines be tested in local settings to increase compliance ^(11–14). Implementation and dissemination strategies affect the probability of guidelines being effective ⁽²⁷⁾. Implementation depends on acceptance of specific recommendations by physicians and on the applicability of indications and acceptance by customers.

Little is known about pediatricians' attitudes toward AGE guidelines. Flores et al ⁽²⁸⁾ found that pediatricians used CPGs for gastroenteritis less frequently than guidelines for asthma, hyperbilirubinemia, and otitis media, and only one third of participants knew that CPGs for AGE existed. A study ⁽¹¹⁾ conducted in 29 European countries on children with mild to moderate gastroenteritis showed that pediatricians adhered poorly to CPGs recommendations. Similarly, failure to provide recommended care has been reported in the United States (adherence to recommendations is 37%) by an authoritative study that analyzed the delivery of care to children affected by diarrhea ⁽²⁹⁾.

Implementation of recommendations depends much on local practice patterns. A recent controlled field trial on the implementation of guidelines for the management of AGE showed that a simple, brief educational intervention (2-hour course) directed toward Italian pediatricians had a significant effect on the disease, reducing the duration of diarrhea and improving child weight gain ⁽³⁰⁾. Poor knowledge and application of CPGs for AGE management in children may be related to underestimation of the disease and treatment that is mainly nonpharmacological. AGE should be managed by ORS and no change in feeding.

The evaluation of the main indications was not a specific aim of the study. However, the main recommendations included in guidelines matching the inclusion criteria were reviewed and, interestingly, it was observed that the content of some guidelines is not the pure result of available evidence or of criteria included in the AGREE instrument, based on scientific quality only; rather, it often includes pragmatic considerations related to local setting. Those are probably helpful if not essential in many circumstances in that they affect the applicability of a recommendation and hence the impact of CPG. Accordingly it would be senseless to recommend a drug that is not available on the market or suggest interventions that go against the cultural background in a specific setting. This, however, does not change the need for having guidelines based on the best evidence; rather, it introduces the concept of the extent to which evidence obtained in a specific setting is generally valid or applicable in other settings or circumstances.

CONCLUSIONS

The overall quality of CPGs for AGE management in children is fair. Aims, target population, synthesis of evidence, formulation of recommendations, and clarity of presentation are points of strength. However, there are several weaknesses among published CPGs, namely applicability, including identification of organizational barriers and adherence parameters, cost–efficacy analysis, and conflict of interests. Patient preferences and experiences were rarely sought. Most guidelines did not provide evidence of pilot testing, which is an essential issue required by AGREE. Compared with guidelines for other pediatric disorders applicability scores better.

The simple and straightforward management of AGE based more on “not doing” than on active interventions is a point of strength for dissemination and local application. On the contrary, recommendations need to be translated and applied into specific settings, taking into account geographical, cultural, and even economic considerations. The rigors of development and quality of CPGs are important features, but they do not ensure the efficacy and applicability of recommendations. Some AGREE domains take care of this aspect, recommending users and family involvement in the panel and suggesting a local pretesting of recommendations.

Interestingly, these domains that were not effectively fulfilled by most guidelines were those related to the implementation of CPG at local level. This suggests that AGREE, if thoroughly fulfilled, would effectively reflect not only the quality but also the efficacy and applicability of a CPG.

Acknowledgment

We are grateful to Jean Ann Gilder for editing the text.