Asthma is one of the most common chronic diseases of childhood, affecting more than 6.1 million American children with asthma rates increasing by as much as 28% between 2001 and 2011.1,2 In addition, minority and low-income children have been shown to share a greater burden of asthma morbidity and mortality.3 Assessing asthma severity, defined as either intermittent or persistent, is key to initiating appropriate therapy. Persistent asthma in children requires the use of a controller medication, usually a daily inhaled steroid preparation, to control asthma symptoms.
There are two different asthma classification systems assessing asthma severity and predicting the need for daily controller therapy. The Healthcare Effectiveness Data and Information Set (HEDIS) quality measures are used by managed care organizations (MCOs) to assess the quality of care delivered by individual primary care practices. For pediatric asthma care, the metrics are generated from billing data over a 2-year period and are used to assess the need for daily controller therapy and to provide quality report cards as well as payments for performance to primary care practices4 (Table 1). Typically, the data, which measure more resource utilization, are provided to the clinician by the MCO retrospectively, at fixed time intervals.
By contrast, the National Heart, Lung, and Blood Institute (NHLBI) asthma guidelines are used clinically and in real time to classify asthma severity and to determine the need for a controller medication. The NHLBI guidelines, which are more based on the frequency of symptoms, are based on best practices and as such are the touchstone for evidence-based practitioners. The guidelines were developed after a comprehensive literature search, assessing the weight of current evidence, and discussion and interpretation of the evidence by experts5 (Table 2).
How these two measurement systems correlate is largely unknown. A single, previous study using post hoc determinations of NHLBI scores by interviews suggested misclassification of controller need by the HEDIS criteria.6 The objective of this study was to determine the degree of agreement between NHLBI and HEDIS criteria in classifying persistent asthma in an urban, publicly insured pediatric population.
Children between 5 and 18 years of age, who attended an urban, academic health center–affiliated primary care practice, and were continuously enrolled with one MCO from January 1, 2012 to December 31, 2013 (Cohort 1) and from January 1, 2103 to December 31, 2014 (Cohort 2) and carried a diagnosis of asthma were identified. Those subjects with at least one ambulatory visit during the respective study period when NHLBI scoring was performed were included in the study.
Based on HEDIS criteria for 2014, the MCO categorized each subject's disease as intermittent, with no controller required, or persistent asthma, requiring a controller medication (Table 1). Based on medical histories obtained during the appropriate study period, the treating physician categorized the subject using the criteria from the NHLBI guidelines (Table 2).
Continuous variables between the two cohorts were compared using the Welch two sample t-test. Discrete variables were compared by contingency table analysis using Yates correction. In those cases in which the contingency table was 2 × 2, the Fisher exact test was used. In all instances, p values less than .05 were considered significant. The McNemar test was used to determine if categorization by the two sets of controller criteria was independent (null hypothesis); p values less than .05 were used to reject the null hypothesis. The McNemar test is a specialized 2 × 2 version of the χ2 test where two cells record agreement between the two measurements (Intermittent–Intermittent and Persistent–Persistent) and two cells show disagreement (Intermittent–Persistent and Persistent–Intermittent). Confusion matrix analysis was used to determine the percentage of agreement and areas of disagreement between the two classification systems. The Cohen kappa statistic quantifies the degree of agreement between two raters, in this case HEDIS and NHLBI scores. The Cohen kappa statistic with continuity correction was used to determine the strength of agreement between the two categorization methods. According to the Landis and Koch interpretation of Kappa scores, 0–0.2 = slight agreement, 0.21–0.40 = fair agreement, 0.41–0.60 = moderate agreement, 0.61–0.80 = substantial agreement, and 0.81–1 = almost perfect agreement.7
This study was approved by our institution's Human Research Protection Program.
The characteristics of both patient cohorts are shown in Table 3. The mean age of Cohort 1 was 8.52 years, and the mean age of Cohort 2 was 11.79 years. The difference in mean ages was significantly different (3.27 years; 95% confidence interval [CI]: 2.65–3.89, p < .0001). The racial distribution between the two cohorts was not significantly different (χ2 = 0.12, p = .21). Using the NHLBI guidelines, 238/326 children in Cohort 1 and 174/212 children in Cohort 2 required daily inhaled controllers; significantly less children in Cohort 1 required controllers than in Cohort 2 (odds ratio [OR] = 0.591; 95% CI: 0.374–922, p = .017). By the HEDIS criteria, 299/326 children in Cohort 1 and 198/212 in Cohort 2 required inhaled controllers; these rates were not significantly different (OR = 0.78; 95% CI: 0.37–1.59, p = .51).
Comparison of National Heart, Lung, and Blood Institute Guidelines and Healthcare Effectiveness Data and Information Set Criteria
A comparison of controller assignment using the NHLBI guidelines and HEDIS criteria for both patient cohorts is shown in Table 4. Individual comparisons of controller assignments by the NHLBI guidelines and HEDIS criteria were not independent for Cohort 1 (McNemar χ2 = 35.9, p < .00001) or Cohort 2 (McNemar χ2 = 20.3, p < .00001). The agreement in assignment by the NHLBI guidelines and HEDIS criteria was fair in Cohort 1 (Cohen kappa = 0.364; 95% CI = 0.217–0.511) and moderate in Cohort 2 (Cohen kappa = 0.447; 95% CI = 0.247–0.646).
The proportions of patients in each matched and mismatched group were similar between the two cohorts (χ2 = 0.12, p = .21). Compared with the NHLBI guidelines, the HEDIS criteria misclassified controller requirements in both Cohort 1 (16.4%; 95% CI: 11.5–21.2%) and Cohort 2 (11.8%; 95% CI: 6.8–16.7%); this rate was not significantly different between the two cohorts (OR = 1.46; 95% CI: 0.86–2.55; p = .17).
Using the recommendation for controller therapy as the positive outcome and the NHLBI criteria as the comparative standard, the sensitivity/specificity of the HEDIS is 0.98/0.013 and 0.99/0.34 for Cohorts 1 and 2, respectively.
Limitations of the study include possible generalizability to practices that serve different populations. The patient population in the study is an urban, primarily minority, patient population so may not be generalizable to patient populations outside these demographics. Variability in scoring among practitioners within a single practice may be another limitation. All the providers in the study are within one practice and therefore may have similar practice styles. Completeness of data collection needed for HEDIS scoring is another possible limitation. In addition to these limitations of the study, limited evidence is available about the benefits of pay for performance on pediatric asthma outcomes.8 Studies have shown some improvement in quality measures but often multiple interventions have been implemented at the same time, making interpretation difficult.9
In two cohorts of publicly insured children with asthma, two methods for controller classification (NHLBI guidelines and HEDIS criteria) moderately agree. Compared with the NHLBI asthma guidelines, the HEDIS criteria misclassified controller requirements by 12–16%. Cabana et.al6 also compared the agreement of NHLBI and HEDIS criteria to assign asthmatic children to controller groups in a cross-sectional, post hoc study. A sample of children with existing HEDIS assignments previously enrolled in a randomized, asthma education study was contacted after the study. Using a structured interview based on the NHLBI guidelines, each subject's need for controller medication was determined and compared with the HEDIS assignment. From a sample of 896 children, 338 (38%) were designated as persistent asthmatics by the NHLBI criteria and 656 (73%) by the HEDIS criteria.
With the HEDIS criteria misclassifying more patients as persistent asthmatics, if practices adjust their management to better fit the HEDIS criteria, this could result in unintended consequences. Children would be placed on inhaled corticosteroids that they do not need. Although the medication is relatively safe, like every medication, there are potential side effects, including sore mouth or throat, thrush, and a slight reduction in overall linear growth for children.10 In addition, there is the cost of the medication, the time families must spend learning how to administer, and then actually giving the medication twice a day. It may also increase office visits for medication refills or reassessment of asthma more frequently leading to missed school time for children, missed work time for parents, and increased costs.
Unlike the NHLBI guidelines which are evidence based and use clinical measures, the HEDIS criteria are resource driven. Using medical claims data, patients need to meet criteria in both the measurement year and the previous year.4 As noted in Table 1, each criterion is based on resource utilization, including pharmacy, emergency department, and hospital. Moreover, scoring is mandated over a 2-year period. As such, the recommendation for controller therapy can only be made retrospectively from data that are not easily available to the child's healthcare provider. The NHLBI guidelines are applied at the time of examination and are based on clinical criteria readily available to the child's healthcare provider, permitting a contemporary point in time assessment as to the requirement for a daily controller medication (Table 2). The NHLBI criteria are used as part of best practices for the management of pediatric asthma and are endorsed by the American Academy of Pediatrics.11
Reproducibility of the HEDIS criteria is also an issue when used to measure individual pediatric practice performance. Two previous retrospective studies looking at patient data sets showed poor year to year variability using the HEDIS criteria.12,13 Fuhlbrigge et al14 attempted to differentiate individual practices using the HEDIS asthma criteria for controller usage in a pay for performance simulation. Using a large patient database developed from 39 individual practices, rates of emergency department visits, hospitalizations, oral steroid bursts, and HEDIS scores were determined for each practice in a bootstrap, reiterative analysis to determine reproducibility in ranking. With this model, acceptable rank reproducibility with the HEDIS criteria required large numbers of asthmatic patients leading the authors to conclude that this method should only be applied at the level of a healthcare organization, not individual practices.
Taking into account the misclassification using the HEDIS criteria, alternative methods of measurement should be considered. Although chart audits that actually review a physician's assessment of the gold standard NHLBI criteria would be best case scenario, it currently is not a feasible option, as health plans are limited to claims data. An alternative option may include the development of zero dollar codes. If a standardized asthma assessment tool that represented the NHLBI criteria was completed at each patient encounter and then coded, a physician could get credit for doing proper assessment at least every 6 months. The insurer could document that the code for asthma assessment was completed every 6 months as recommended. In addition to the presence of the assessment tool, if codes were created that more accurately reflected symptoms such as frequency of wheezing over a 6-month period or frequency of albuterol use, those assessments would be more in line with the symptom assessment of the NHLBI criteria rather than the more resource utilization assessment of the HEDIS criteria. The downside would be additional codes a provider would have to list. A third option that helps account for practices taking care of higher risk patients and address health disparities is to use peer comparison between practices.3 Within a region or for a single payer, insurances could look at hospital resource utilization (e.g., emergency room visits and admissions) per asthma patient-month. This would evaluate the practice as a whole in comparison with similar practices rather than assessing individual patients. A study in Cincinnati created an asthma improvement collaborative using multiple practices within one geographic area and developed a three-tiered pay for performance system that did show some increases in certain outcome measures.15
In summary, the NHLBI pediatric asthma guidelines and HEDIS asthma criteria agree moderately well as to the identity of those asthmatic children who require ongoing controller therapy. Compared with the NHLBI guidelines, the HEDIS criteria misclassified controller usage by 12–16%, on average.
This observation coupled with the existing evidence suggests that the HEDIS asthma criteria provide an inaccurate measure of the quality of care individual pediatric practices deliver to children with asthma. As such, these criteria should not be used as a part of a pay for performance scorecard for individual pediatric practices.
1. Centers for Disease Control and Prevention. Summary health statistics: National health interview survey, table C-1b, p 3. 2015.
2. Centers for Disease Control and Prevention. Asthma
Facts- CDC's National Asthma
Control Program Grantees. Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention; 2013.
3. James CV, Rosenbaum S. Paying for quality care: Implications for racial and ethnic health disparities in pediatric asthma
. 2009;123(suppl 3):S205–S210.
4. National Committee for Quality Assurance. HEDIS
2013. Use of Appropriate Medications for People with Asthma
. Washington, DC: National Committee for Quality Assurance; 2012.
5. National Asthma
Education and Prevention Program. Guidelines for the Diagnosis and Management of Asthma
. National Asthma
Education and Prevention Program Expert Panel Report 3. Bethesda, MD. National Institutes of Health, National Heart, Lung, and Blood Institute; 2007. NIH Publication 07-4051.
6. Cabana MD, Slish KK, Nan B, Clark NM. Limits of the HEDIS
criteria in determining asthma
severity for children. Pediatrics
7. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174.
8. Okelo SO, Butz AM, SHarma R, et al. Interventions to modify health care provider adherence to asthma
guidelines: As systematic review. Pediatrics
9. Christianson JB, Leatherman S, Sutherland K. Lessons from evaluations of purchaser pay-for-performance programs: A review of the evidence. Med Care Res Rev. 2008;65(6 suppl):5S–35S.
10. Fuhlbrigge AL, Kelly HW. Inhaled corticosteroids in children: Effects on bone mineral density and growth. Lancet Respir Med. 2104;2(6):487–496.
11. American Academy of Pediatrics Asthma
, Allergy, and Immunology Clinical Guidelines. http://www2.aap.org/sections/allergy/guidelines.cfm. Accessed January 20, 2016.
12. Mosen DM, Macy E, Schatz M, et al. How well do the HEDIS asthma
inclusion criteria identify persistent asthma
? Am J Manag Care. 2005;11:650–654.
13. Dombkowski KJ, Wasilevich EA, Lyon-Callo SK. Pediatric asthma
surveillance using Medicaid claims. Public Health Rep. 2005;120(5):515–524.
14. Fuhlbrigge AL, Carey VJ, Finkelstein JA, et al. Validity of the HEDIS
criteria to identify children with persistent asthma
and sustained high utilization. Am J Manag Care. 2005;11(5):325–330.
15. Mandel KE, Kotagal UR. Pay for performance alone cannot drive quality. Arch Pediatr Adolesc Med. 2007;161(7):650–655.
Ashleigh Hall, DO is an Assistant Professor of Clinical Pediatrics at Lewis Katz School of Medicine at Temple University in Philadelphia, PA.
Carol Smolij, BSN, RN, CPHQ is a Nurse at Health Partners Plans in Philadelphia, PA.
Beth Moughan, MD is the Section Chief of Ambulatory Pediatrics, Assistant Dean, Affiliate Faculty Development, and a Professor of Clinical Pediatrics at Lewis Katz School of Medicine at Temple University in Philadelphia, PA.
Amer Kechli, MD is an Associate Professor of Clinical Pediatrics at Lewis Katz School of Medicine at Temple University in Philadelphia, PA.
Stephen Aronoff, MD is the Waldo E Nelson Chair of the Department of Pediatrics and a Professor of Pediatrics at Lewis Katz School of Medicine at Temple University in Philadelphia, PA.