Secondary Logo

Journal Logo

Identifying Persistent Asthma in Children: A Comparison of Healthcare Effectiveness Data and Information Set Criteria and National Heart, Lung, and Blood Institute Guidelines

Hall, Ashleigh; Smolij, Carol; Moughan, Beth; Kechli, Amer; Aronoff, Stephen

The Journal for Healthcare Quality (JHQ): March/April 2018 - Volume 40 - Issue 2 - p e20–e25
doi: 10.1097/JHQ.0000000000000098
Original Article
Open
SDC

ABSTRACT Persistent asthma requires a daily controller medication. National Heart, Lung, and Blood Institute (NHLBI) guidelines and Healthcare Effectiveness Data and Information Set (HEDIS) quality measures are used to identify persistent asthma, the former method used by clinicians, and the latter by managed care organizations (MCOs). This study aims to determine the degree of agreement between these criteria in classifying pediatric persistent asthma. Asthmatic patients aged 5–18 years old in a university pediatric practice who were continuously enrolled with one MCO from January 1, 2012 to December 31, 2013 (Cohort 1: 329 patients) and January 1, 2013 to December 31, 2014 (Cohort 2: 212 patients) were identified. Subjects with at least one asthma visit were included. Independence of NHLBI and HEDIS criteria was determined by the McNemar test. Agreement was determined by confusion matrix analysis and Cohen kappa statistic with continuity correction. Agreement in controller assignment was 82% in Cohort 1 and 88% in Cohort 2. Agreement was fair in Cohort 1 (Cohen kappa = 0.364; 95% confidence interval [CI] = 0.217–0.511) and moderate in Cohort 2 (Cohen kappa = 0.447; 95% CI = 0.247–0.646). The HEDIS misclassified persistent asthma by 16.4% (95% CI: 11.5–21.2%) and 11.8% (95% CI: 6.8–16.7%) in Cohorts 1 and 2, respectively. NHLBI and HEDIS criteria show fair to moderate agreement; however, the HEDIS consistently misclassified persistent asthma, suggesting that it is a poor measure of practice performance.

For more information on this article, contact Ashleigh Hall at Ashleigh.Hall@tuhs.temple.edu.

The authors declare no conflicts of interest.

This is an open access article distributed under the Creative Commons Attribution-NoDerivatives License 4.0 (CC BY-ND) which allows for redistribution, commercial and non-commercial, as long as it is passed along unchanged and in whole, with credit to the author.

Back to Top | Article Outline

Introduction

Asthma is one of the most common chronic diseases of childhood, affecting more than 6.1 million American children with asthma rates increasing by as much as 28% between 2001 and 2011.1,2 In addition, minority and low-income children have been shown to share a greater burden of asthma morbidity and mortality.3 Assessing asthma severity, defined as either intermittent or persistent, is key to initiating appropriate therapy. Persistent asthma in children requires the use of a controller medication, usually a daily inhaled steroid preparation, to control asthma symptoms.

There are two different asthma classification systems assessing asthma severity and predicting the need for daily controller therapy. The Healthcare Effectiveness Data and Information Set (HEDIS) quality measures are used by managed care organizations (MCOs) to assess the quality of care delivered by individual primary care practices. For pediatric asthma care, the metrics are generated from billing data over a 2-year period and are used to assess the need for daily controller therapy and to provide quality report cards as well as payments for performance to primary care practices4 (Table 1). Typically, the data, which measure more resource utilization, are provided to the clinician by the MCO retrospectively, at fixed time intervals.

Table 1

Table 1

By contrast, the National Heart, Lung, and Blood Institute (NHLBI) asthma guidelines are used clinically and in real time to classify asthma severity and to determine the need for a controller medication. The NHLBI guidelines, which are more based on the frequency of symptoms, are based on best practices and as such are the touchstone for evidence-based practitioners. The guidelines were developed after a comprehensive literature search, assessing the weight of current evidence, and discussion and interpretation of the evidence by experts5 (Table 2).

Table 2

Table 2

How these two measurement systems correlate is largely unknown. A single, previous study using post hoc determinations of NHLBI scores by interviews suggested misclassification of controller need by the HEDIS criteria.6 The objective of this study was to determine the degree of agreement between NHLBI and HEDIS criteria in classifying persistent asthma in an urban, publicly insured pediatric population.

Back to Top | Article Outline

Methods

Patients

Children between 5 and 18 years of age, who attended an urban, academic health center–affiliated primary care practice, and were continuously enrolled with one MCO from January 1, 2012 to December 31, 2013 (Cohort 1) and from January 1, 2103 to December 31, 2014 (Cohort 2) and carried a diagnosis of asthma were identified. Those subjects with at least one ambulatory visit during the respective study period when NHLBI scoring was performed were included in the study.

Back to Top | Article Outline

Data Collection

Based on HEDIS criteria for 2014, the MCO categorized each subject's disease as intermittent, with no controller required, or persistent asthma, requiring a controller medication (Table 1). Based on medical histories obtained during the appropriate study period, the treating physician categorized the subject using the criteria from the NHLBI guidelines (Table 2).

Back to Top | Article Outline

Statistical Analysis

Continuous variables between the two cohorts were compared using the Welch two sample t-test. Discrete variables were compared by contingency table analysis using Yates correction. In those cases in which the contingency table was 2 × 2, the Fisher exact test was used. In all instances, p values less than .05 were considered significant. The McNemar test was used to determine if categorization by the two sets of controller criteria was independent (null hypothesis); p values less than .05 were used to reject the null hypothesis. The McNemar test is a specialized 2 × 2 version of the χ2 test where two cells record agreement between the two measurements (Intermittent–Intermittent and Persistent–Persistent) and two cells show disagreement (Intermittent–Persistent and Persistent–Intermittent). Confusion matrix analysis was used to determine the percentage of agreement and areas of disagreement between the two classification systems. The Cohen kappa statistic quantifies the degree of agreement between two raters, in this case HEDIS and NHLBI scores. The Cohen kappa statistic with continuity correction was used to determine the strength of agreement between the two categorization methods. According to the Landis and Koch interpretation of Kappa scores, 0–0.2 = slight agreement, 0.21–0.40 = fair agreement, 0.41–0.60 = moderate agreement, 0.61–0.80 = substantial agreement, and 0.81–1 = almost perfect agreement.7

Back to Top | Article Outline

Results

This study was approved by our institution's Human Research Protection Program.

Back to Top | Article Outline

Patients

The characteristics of both patient cohorts are shown in Table 3. The mean age of Cohort 1 was 8.52 years, and the mean age of Cohort 2 was 11.79 years. The difference in mean ages was significantly different (3.27 years; 95% confidence interval [CI]: 2.65–3.89, p < .0001). The racial distribution between the two cohorts was not significantly different (χ2 = 0.12, p = .21). Using the NHLBI guidelines, 238/326 children in Cohort 1 and 174/212 children in Cohort 2 required daily inhaled controllers; significantly less children in Cohort 1 required controllers than in Cohort 2 (odds ratio [OR] = 0.591; 95% CI: 0.374–922, p = .017). By the HEDIS criteria, 299/326 children in Cohort 1 and 198/212 in Cohort 2 required inhaled controllers; these rates were not significantly different (OR = 0.78; 95% CI: 0.37–1.59, p = .51).

Table 3

Table 3

Back to Top | Article Outline

Comparison of National Heart, Lung, and Blood Institute Guidelines and Healthcare Effectiveness Data and Information Set Criteria

A comparison of controller assignment using the NHLBI guidelines and HEDIS criteria for both patient cohorts is shown in Table 4. Individual comparisons of controller assignments by the NHLBI guidelines and HEDIS criteria were not independent for Cohort 1 (McNemar χ2 = 35.9, p < .00001) or Cohort 2 (McNemar χ2 = 20.3, p < .00001). The agreement in assignment by the NHLBI guidelines and HEDIS criteria was fair in Cohort 1 (Cohen kappa = 0.364; 95% CI = 0.217–0.511) and moderate in Cohort 2 (Cohen kappa = 0.447; 95% CI = 0.247–0.646).

Table 4

Table 4

The proportions of patients in each matched and mismatched group were similar between the two cohorts (χ2 = 0.12, p = .21). Compared with the NHLBI guidelines, the HEDIS criteria misclassified controller requirements in both Cohort 1 (16.4%; 95% CI: 11.5–21.2%) and Cohort 2 (11.8%; 95% CI: 6.8–16.7%); this rate was not significantly different between the two cohorts (OR = 1.46; 95% CI: 0.86–2.55; p = .17).

Using the recommendation for controller therapy as the positive outcome and the NHLBI criteria as the comparative standard, the sensitivity/specificity of the HEDIS is 0.98/0.013 and 0.99/0.34 for Cohorts 1 and 2, respectively.

Back to Top | Article Outline

Limitations

Limitations of the study include possible generalizability to practices that serve different populations. The patient population in the study is an urban, primarily minority, patient population so may not be generalizable to patient populations outside these demographics. Variability in scoring among practitioners within a single practice may be another limitation. All the providers in the study are within one practice and therefore may have similar practice styles. Completeness of data collection needed for HEDIS scoring is another possible limitation. In addition to these limitations of the study, limited evidence is available about the benefits of pay for performance on pediatric asthma outcomes.8 Studies have shown some improvement in quality measures but often multiple interventions have been implemented at the same time, making interpretation difficult.9

Back to Top | Article Outline

Discussion

In two cohorts of publicly insured children with asthma, two methods for controller classification (NHLBI guidelines and HEDIS criteria) moderately agree. Compared with the NHLBI asthma guidelines, the HEDIS criteria misclassified controller requirements by 12–16%. Cabana et.al6 also compared the agreement of NHLBI and HEDIS criteria to assign asthmatic children to controller groups in a cross-sectional, post hoc study. A sample of children with existing HEDIS assignments previously enrolled in a randomized, asthma education study was contacted after the study. Using a structured interview based on the NHLBI guidelines, each subject's need for controller medication was determined and compared with the HEDIS assignment. From a sample of 896 children, 338 (38%) were designated as persistent asthmatics by the NHLBI criteria and 656 (73%) by the HEDIS criteria.

With the HEDIS criteria misclassifying more patients as persistent asthmatics, if practices adjust their management to better fit the HEDIS criteria, this could result in unintended consequences. Children would be placed on inhaled corticosteroids that they do not need. Although the medication is relatively safe, like every medication, there are potential side effects, including sore mouth or throat, thrush, and a slight reduction in overall linear growth for children.10 In addition, there is the cost of the medication, the time families must spend learning how to administer, and then actually giving the medication twice a day. It may also increase office visits for medication refills or reassessment of asthma more frequently leading to missed school time for children, missed work time for parents, and increased costs.

Unlike the NHLBI guidelines which are evidence based and use clinical measures, the HEDIS criteria are resource driven. Using medical claims data, patients need to meet criteria in both the measurement year and the previous year.4 As noted in Table 1, each criterion is based on resource utilization, including pharmacy, emergency department, and hospital. Moreover, scoring is mandated over a 2-year period. As such, the recommendation for controller therapy can only be made retrospectively from data that are not easily available to the child's healthcare provider. The NHLBI guidelines are applied at the time of examination and are based on clinical criteria readily available to the child's healthcare provider, permitting a contemporary point in time assessment as to the requirement for a daily controller medication (Table 2). The NHLBI criteria are used as part of best practices for the management of pediatric asthma and are endorsed by the American Academy of Pediatrics.11

Reproducibility of the HEDIS criteria is also an issue when used to measure individual pediatric practice performance. Two previous retrospective studies looking at patient data sets showed poor year to year variability using the HEDIS criteria.12,13 Fuhlbrigge et al14 attempted to differentiate individual practices using the HEDIS asthma criteria for controller usage in a pay for performance simulation. Using a large patient database developed from 39 individual practices, rates of emergency department visits, hospitalizations, oral steroid bursts, and HEDIS scores were determined for each practice in a bootstrap, reiterative analysis to determine reproducibility in ranking. With this model, acceptable rank reproducibility with the HEDIS criteria required large numbers of asthmatic patients leading the authors to conclude that this method should only be applied at the level of a healthcare organization, not individual practices.

Taking into account the misclassification using the HEDIS criteria, alternative methods of measurement should be considered. Although chart audits that actually review a physician's assessment of the gold standard NHLBI criteria would be best case scenario, it currently is not a feasible option, as health plans are limited to claims data. An alternative option may include the development of zero dollar codes. If a standardized asthma assessment tool that represented the NHLBI criteria was completed at each patient encounter and then coded, a physician could get credit for doing proper assessment at least every 6 months. The insurer could document that the code for asthma assessment was completed every 6 months as recommended. In addition to the presence of the assessment tool, if codes were created that more accurately reflected symptoms such as frequency of wheezing over a 6-month period or frequency of albuterol use, those assessments would be more in line with the symptom assessment of the NHLBI criteria rather than the more resource utilization assessment of the HEDIS criteria. The downside would be additional codes a provider would have to list. A third option that helps account for practices taking care of higher risk patients and address health disparities is to use peer comparison between practices.3 Within a region or for a single payer, insurances could look at hospital resource utilization (e.g., emergency room visits and admissions) per asthma patient-month. This would evaluate the practice as a whole in comparison with similar practices rather than assessing individual patients. A study in Cincinnati created an asthma improvement collaborative using multiple practices within one geographic area and developed a three-tiered pay for performance system that did show some increases in certain outcome measures.15

Back to Top | Article Outline

Conclusions

In summary, the NHLBI pediatric asthma guidelines and HEDIS asthma criteria agree moderately well as to the identity of those asthmatic children who require ongoing controller therapy. Compared with the NHLBI guidelines, the HEDIS criteria misclassified controller usage by 12–16%, on average.

Back to Top | Article Outline

Implications

This observation coupled with the existing evidence suggests that the HEDIS asthma criteria provide an inaccurate measure of the quality of care individual pediatric practices deliver to children with asthma. As such, these criteria should not be used as a part of a pay for performance scorecard for individual pediatric practices.

Back to Top | Article Outline

References

1. Centers for Disease Control and Prevention. Summary health statistics: National health interview survey, table C-1b, p 3. 2015.
2. Centers for Disease Control and Prevention. Asthma Facts- CDC's National Asthma Control Program Grantees. Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention; 2013.
3. James CV, Rosenbaum S. Paying for quality care: Implications for racial and ethnic health disparities in pediatric asthma. Pediatrics. 2009;123(suppl 3):S205–S210.
4. National Committee for Quality Assurance. HEDIS 2013. Use of Appropriate Medications for People with Asthma. Washington, DC: National Committee for Quality Assurance; 2012.
5. National Asthma Education and Prevention Program. Guidelines for the Diagnosis and Management of Asthma. National Asthma Education and Prevention Program Expert Panel Report 3. Bethesda, MD. National Institutes of Health, National Heart, Lung, and Blood Institute; 2007. NIH Publication 07-4051.
6. Cabana MD, Slish KK, Nan B, Clark NM. Limits of the HEDIS criteria in determining asthma severity for children. Pediatrics. 2004;114(4):1049–1055.
7. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174.
8. Okelo SO, Butz AM, SHarma R, et al. Interventions to modify health care provider adherence to asthma guidelines: As systematic review. Pediatrics. 2013;132(3):517–534.
9. Christianson JB, Leatherman S, Sutherland K. Lessons from evaluations of purchaser pay-for-performance programs: A review of the evidence. Med Care Res Rev. 2008;65(6 suppl):5S–35S.
10. Fuhlbrigge AL, Kelly HW. Inhaled corticosteroids in children: Effects on bone mineral density and growth. Lancet Respir Med. 2104;2(6):487–496.
11. American Academy of Pediatrics Asthma, Allergy, and Immunology Clinical Guidelines. http://www2.aap.org/sections/allergy/guidelines.cfm. Accessed January 20, 2016.
12. Mosen DM, Macy E, Schatz M, et al. How well do the HEDIS asthma inclusion criteria identify persistent asthma? Am J Manag Care. 2005;11:650–654.
13. Dombkowski KJ, Wasilevich EA, Lyon-Callo SK. Pediatric asthma surveillance using Medicaid claims. Public Health Rep. 2005;120(5):515–524.
14. Fuhlbrigge AL, Carey VJ, Finkelstein JA, et al. Validity of the HEDIS criteria to identify children with persistent asthma and sustained high utilization. Am J Manag Care. 2005;11(5):325–330.
15. Mandel KE, Kotagal UR. Pay for performance alone cannot drive quality. Arch Pediatr Adolesc Med. 2007;161(7):650–655.
Back to Top | Article Outline

Authors' Biographies

Ashleigh Hall, DO is an Assistant Professor of Clinical Pediatrics at Lewis Katz School of Medicine at Temple University in Philadelphia, PA.

Carol Smolij, BSN, RN, CPHQ is a Nurse at Health Partners Plans in Philadelphia, PA.

Beth Moughan, MD is the Section Chief of Ambulatory Pediatrics, Assistant Dean, Affiliate Faculty Development, and a Professor of Clinical Pediatrics at Lewis Katz School of Medicine at Temple University in Philadelphia, PA.

Amer Kechli, MD is an Associate Professor of Clinical Pediatrics at Lewis Katz School of Medicine at Temple University in Philadelphia, PA.

Stephen Aronoff, MD is the Waldo E Nelson Chair of the Department of Pediatrics and a Professor of Pediatrics at Lewis Katz School of Medicine at Temple University in Philadelphia, PA.

Keywords:

asthma; HEDIS; pediatrics

© 2018 National Association for Healthcare Quality