Secondary Logo

Journal Logo

Evidence for Misspecification of a Nationally Used Quality Measure for Substance Use Treatment

Mattke, Soeren; Predmore, Zachary; Sloss, Elizabeth; Wilks, Asa; Watkins, Katherine E.

The Journal for Healthcare Quality (JHQ): July/August 2018 - Volume 40 - Issue 4 - p 228–235
doi: 10.1097/JHQ.0000000000000106
Original Article

Objectives: The National Committee for Quality Assurance's (NCQA) measure “Initiation and Engagement of Alcohol and Other Drug Dependence Treatment” captures the proportion of substance use patients with (1) treatment initiation within 14 days and (2) treatment engagement within 30 days thereafter. The definition of treatment considers only counseling but not medication-assisted treatment (MAT), although MAT is supported by current guidelines. Our research question is whether this omission results in meaningful measurement error.

Study Design and Methods: We analyze claims data for members of commercial health plans to investigate whether including MAT would meaningfully change the measure rate and health plan rankings.

Results: Including MAT increased both the initiation and engagement rates. The initiation and engagement rates increased 2.4% (38.9–39.8%) and 9.9% (12.9–14%), respectively. These differences imply that 19% of health plans would change their ranking by at least one quintile for the initiation measure and 27% for the engagement measure.

Conclusions: The current specifications result in erroneous conclusions about the quality of care provided by different health plans. Our results suggest that aligning the measure specifications with guideline recommendations, as recently proposed by NCQA, would result in more accurate information.

For more information on this article, contact Soeren Mattke at

The study was funded by Alkermes, Inc.

This research was funded by Alkermes, Inc. The sponsor reviewed a draft of the manuscript but the decisions on research design, interpretation of the finding and submission were made by the authors. The authors report no other conflicts.

All authors for this paper are affiliated with RAND Corporation. There are no other conflicts of interest to declare.

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

Back to Top | Article Outline


The current opioid epidemic in the United States serves as a reminder that access to high-quality substance use treatment remains limited. In the United States, an estimated 400,000 individuals have used heroin in the past month, four million report misuse of prescription opioids and there are almost 17,000 deaths per year related to opioids.1-3 Despite this, only 26% of persons with opioid use disorders report receiving treatment in the past year.4,5 In 2015, only 1.3 million of the 15.1 million adults with alcohol use disorders received specialty alcohol treatment.6

Against this background, measuring and reporting on the quality of substance use care becomes increasingly important for patients, clinicians, and other stakeholders. A prominent quality measure in this space is the National Committee for Quality Assurance's (NCQA) measure “Initiation and Engagement of Alcohol and Other Drug Dependence Treatment,” which is a two-part measure that consists of the proportion of patients with an initial diagnosis of alcohol or other drug abuse and/or dependence who receive (1) treatment within 14 days of the initial diagnosis and (2) follow-up treatment within 30 days thereafter.7 The measure is currently endorsed by the National Quality Forum (NQF) for use in multiple care settings and is part of several federal accountability programs,8 such as Meaningful Use Stage 2 and the Physician Quality Reporting System, as well as NCQA's widely used Healthcare Effectiveness Data and Information Set (HEDIS) measure set for health plan quality.

It is noteworthy that the measure specifications define “treatment” as receipt of professional encounters with a diagnosis of a substance use disorder, including both abuse and dependence, such as group or individual psychosocial counseling and inpatient rehabilitation, but not as receipt of medication-assisted treatment (MAT), which combines behavioral therapy and medications to treat substance use disorders. Pharmacotherapy for alcohol or opioid dependence reduces withdrawal symptoms and psychological cravings and are recommended by several national guidelines9,10 as well as the federal Substance Abuse and Mental Health Services Administration (SAMHSA), but is underused.11 NCQA recently proposed12 incorporating MAT into the measure definition.

Our research question is whether the inclusion of MAT in the measure definition results in meaningful changes to the measure rates. On the one hand, guidelines recommend that MAT ought to be accompanied by counseling, and therefore, measure rates might not change by including MAT as a type of treatment. On the other hand, patients may seek counseling outside the formal health care setting, such as in self-help groups, pay for it out-of-pocket or forego counseling altogether. In these situations, no claim for a treatment encounter would be generated, and the omission of MAT from the measure definition might change the measure rate. We use a large dataset of insurance claims from members of commercial health plans to investigate if including MAT in the definition of treatment would meaningfully change the overall measure rate.

Back to Top | Article Outline


Measure Construction

We followed the 2015 definitions of the NCQA Initiation and Engagement of Alcohol and Other Drug Dependence Treatment measure as released in the documentation of the federal Quality Rating System.2 The measure identifies adolescent and adult health plan members with a new episode of alcohol or other drug abuse and/or dependence, which is called the “index visit,” and calculates the percentage of those members who initiate treatment within 14 days of diagnosis (Initiation) and have at least two additional encounters for substance use care within 30 days of the index visit (Engagement). To ensure the availability of sufficient follow-up time, the measure restricts the “intake period,” during which episodes are identified to the period from January 1 to November 15. A so-called “clean period” of 60 days before the qualifying event excludes members in ongoing treatment, that is, those who have claims for substance use care during that period.

Back to Top | Article Outline


Data for the analysis were derived from the Truven MarketScan (Ann Arbor, MI) Commercial Database (including data from Standard Quarterly Updates) for calendar years 2013 and 2014. This database has long been a commonly used data source to generate real-world insights into care patterns of members of commercial health plans.13 The database contains fully adjudicated, patient-level claims for inpatient care, outpatient care, and prescription drugs as well as enrollment data. All records in these files were used as input to identify individuals who met the measure's eligibility criteria. We used calendar year 2014 as the measurement year and the last 2 months of calendar year 2013 to ensure an appropriate clean period and restricted the sample to members with continuous enrollment of at least 60 days before and at least 44 days after the index episode. The final analytic file contained a total of 6,838,940 unique members. The study was reviewed and considered exempt by our institutional review board.

Back to Top | Article Outline

Definition of Medication-Assisted Treatment

We included all FDA-approved drugs in the definition of MAT. Those are acamprosate, disulfiram, oral naltrexone, and extended-release injectable naltrexone for treating patients with alcohol use disorder and buprenorphine, methadone, oral naltrexone, and extended-release injectable naltrexone for opioid use disorder. We also included topiramate for alcohol use disorder treatment, which is not FDA-approved but guideline-recommended based on strong evidence from a meta-analysis14,15 and included in a measure16 proposed by the American Society for Addiction Medicine. National Drug Codes (NDCs) for the oral drugs were identified by searching the “Active Ingredients” category in the FDA's NDC Directory17 for “acamprosate,” “disulfiram,” “topiramate,” “naltrexone,” and “buprenorphine.” Search results were then manually reviewed for relevance. Injectable naltrexone was identified based on a Healthcare Common Procedure Coding System (HCPCS) code (J2315) in the medical claims. As methadone may only be used for MAT through in-office dispensing in licensed treatment centers, it was also identified based on a HCPCS code (H0020). MAT was defined as the presence of at least one NDC or HCPCS code on a claim in the required timeframes.

Back to Top | Article Outline

Methods Used to Calculate Measure Rates

We followed the HEDIS measure specifications18 to calculate the original measure rates. These specifications include for each of the two rates (initiation and engagement): definitions of and periods for the numerator and denominator, detailed instructions for creating the numerator and denominator, and numerator and denominator exclusions. We calculated a revised version of the measure that counted MAT as treatment in the specifications of initiation and engagement.

Back to Top | Article Outline

Estimation of Misclassification

We calculated and compared the original and revised measure rates for initiation and engagement based on index episodes for any substance use, as defined in the original measure, and for alcohol dependence only, opioid dependence only, and combined alcohol and opioid use disorders at the population level.

In addition, we estimated the effect of the changes in definitions on the relative rankings of health plans. This analysis required an approximation because the data do not contain a unique identifier that would allow formally attributing patients to a health plan. As the MarketScan data came from larger employers, we formed pairs of U.S. Census codes for industry and metropolitan statistical areas (MSA) under the assumption that those pairs will represent the health plan members of one employer.

We calculated measure rates under the original and revised definitions for all MSA-industry pairs with at least 30 eligible episodes, based on the recommendations of the federal Quality Rating System Measure Technical Specifications that considers denominators of less than 30 patients as too small to report a valid measure rate.19 We calculated the average absolute difference, relative percentage difference, difference in rank and percentile rank and proportion of health plans, whose rank would change by at least one quintile, for the initiation and engagement rates separately. As a sensitivity analysis, we restricted the sample to plans with at least 50 and 100 episodes to investigate whether the greater stability of rates in larger samples changed our conclusions.

Back to Top | Article Outline


Population Level

Table 1 displays the initiation and engagement rates for the original and revised versions of the measure at the population level. We identified 296,750 index episodes. The results for the original version indicate that in 115,347 (38.9%) of those episodes, an inpatient or outpatient visit occurred within 14 days of the index episode (i.e., meeting the initiation criterion), and in 38,289 of those episodes (12.9%), two or more inpatient or outpatient visits occurred within 30 days of the index visit (i.e., meeting the engagement criterion).

Table 1

Table 1

Including MAT in the numerator increased both the initiation and engagement rates. The initiation rate increased from 38.9% to 39.8%, representing a 2.4% relative increase. The engagement rate increased from 12.9% to 14.2%, representing a 9.9% relative increase.

Changing the definition also meant that fewer index episodes were eligible for the denominator. The measure requires a period of 60 days before the index visit without any diagnosis-related visits. Patients with encounters for substance use treatment during this clean period are considered in active treatment and are therefore excluded from the measure. If claims for MAT are included in the measure definition, patients on MAT in the 60 days preceding the index episode are excluded. The number of eligible episodes in the denominator decreased from 296,750 in the original version to 284,412 in the revised version of the measure. In other words, using the original specifications, approximately 12,000 episodes in patients with ongoing treatment (approximately 4.5% of the episodes) were incorrectly labeled as new episodes and included in the measure.

Back to Top | Article Outline

Opioid Versus Alcohol Use

We analyzed the effect of adding MAT on the measure rate for diagnosis-specific subgroups related to alcohol and opioid use disorders, including both abuse and dependence, (Table 2), but excluding the approximately 80,000 index episodes for use of substances other than alcohol and opioids, for which there are no currently approved or recommended medications for treatment. In the alcohol subgroup, the initiation and engagement rates increased from 39.6% to 40.0% and from 12.1% to 12.9%, respectively, after MAT was included; these represent 1.1% and 6.2% relative increases for the initiation and engagement rates, respectively. The opioid subgroup exhibited a larger increase in the initiation and engagement rates. When MAT was included in the measure numerator, the initiation rate rose from 36.8% to 40.9% (a 11.1% relative increase) and the engagement rate rose from 16.7% to 21.6% (a 29.1% relative increase).

Table 2

Table 2

Back to Top | Article Outline

Approximation of Health Plan-Level Results

As described above, we estimated the effect of the change in definitions on the relative ranking of health plans, by calculating rates for industry-MSA pairs with 30 or more index episodes (N = 729), as well as for pairs with at least 50 (N = 550) and 100 (N = 319) index episodes.

For rates based on 30 index episodes, including MAT increased the measure rate by 1.4 and 1.8% points on average for initiation and engagement respectively, with a maximum difference of around 10% points for both initiations (Table 3). The average increases represent substantially larger relative increases for the engagement rate (16.2%) than for the initiation rate (34.8%).

Table 3-a

Table 3-a

Table 3-b

Table 3-b

The change in measure definition also influences relative ranking of pairs, with the rank changing by an average of 33 and 49 places for the initiation and engagement rates, respectively. “Health plans” changed their percentile rank by 4.7 and 6.4 on average for the initiation and engagement measure, respectively. We calculated the proportion of the “health plans,” for which the relative rankings changed by at least one quintile when MAT was included in the numerator definition and found that approximately one-fifth of pairs changed rank by at least one quintile for the initiation and one-fourth for the engagement measure.

The results were stable in the sensitivity analyses that restricted to pairs with at least 50 and 100 index episodes for changes in absolute and relative rates as well as percentile ranks and proportion of pairs that changed rank by at least a quintile. As expected, given the smaller subsamples, absolute changes in ranks were smaller.

Back to Top | Article Outline


We acknowledge that we can only approximate the implications for the relative ranking of health plans but suggest that the presented evidence is strong enough to justify the proposed change to the measure definition, as the current definition may lead to a biased assessment of the performance of health plans. Also, our findings are based on a sample of patients who receive health insurance from their employers and might not be generalizable to other populations, such as Medicaid beneficiaries.

Back to Top | Article Outline


We analyzed the effects on measure rates of not including MAT in the specifications for the HEDIS treatment initiation and engagement quality measure for substance use disorders using claims data of members of commercial health plans. Our results suggest that including MAT changes the measure results and health plan rankings in a meaningful way. Not surprisingly, the effect is stronger for the subsample with opioid use disorders than that for those with alcohol use disorders, as pharmacotherapy for opioid use disorders is more common than that for alcohol use disorders.20,21

The findings show that a sizeable share of patients on MAT do not receive counseling or other encounter-based care that generate claims and would thus be misclassified as not receiving adequate care based on the current specifications of the measure. Indeed, two studies suggest that many patients receive counseling through participation in 12-Step programs such as Alcoholics Anonymous. According to a study by Dawson et al,22 12% of patients, who sought help for alcohol problems, had participated in 12-Step programs only, 22% had received formal treatment only, and 67% had participated in both a 12-Step program and received formal treatment. Another study23 found that only 8% of those with a 12-month alcohol use disorder had sought treatment in the previous 12 months; 59% of those who sought treatment reported receiving treatment from a 12-Step program. The same study reported that 20% of those with a lifetime alcohol use disorder had sought treatment for alcohol dependence, of which 78% reported receiving treatment from a 12-Step program.

The misspecification of the measure may be a historical artifact. The measure was developed by the Washington Circle, a collaboration to develop performance measures for substance use disorders, in the late 1990s – early 2000s, when few MAT options were FDA-approved (e.g., methadone). According to the FDA website, buprenorphine was approved in 2002, acamprosate in 2004, and injectable naltrexone in 2006 for alcohol use and 2010 for opioid use. This finding highlights the importance of conducting regular maintenance of such high-profile measures to ensure that they continue to reflect the state of evidence,24 as the current definition can result in the unintended consequence of discouraging the use of MAT and reinforce the existing barriers against an effective and cost-effective treatment option.25-27 Although others have conducted studies on the validity of this measure,28,29 our analysis is the first to point out the importance of including MAT. As MAT is becoming more commonly used,30 the impact of this change will increase over time.

Back to Top | Article Outline


Our results suggest that the omission of MAT in the current version of the HEDIS measure “Initiation and Engagement of Alcohol and Other Drug Dependence Treatment” results in meaningful measurement error. Thus, providers, who use one of the evidence-based options for substance use treatment, and health plans, which cover MAT, will not receive proper credit. With the greater emphasis on using quality measures in recognition and accountability schemes, the omission could distort clinical decision making and lead to lower quality of care.

Back to Top | Article Outline


The findings support the recently proposed12 inclusion of MAT in the measure definition to align the measure specifications with current guideline recommendations. The change would allow clinicians to recommend the best-suited treatment modality to their patients without concerns for not receiving proper credit in quality measurement and reporting schemes for one treatment option.

Back to Top | Article Outline


1. Hedden SL. Behavioral Health Trends in the United States: Results from the 2014 National Survey on Drug Use and Health. Rockville, MD: Substance Abuse and Mental Health Services Administration, Department of Health & Human Services; 2015.
2. Brady KT, McCauley JL, Back SE. Prescription opioid misuse, abuse, and treatment in the United States: An update. Am J Psychiatry. 2015;173(1):18–26.
3. Dart RC, Surratt HL, Cicero TJ, et al. Trends in opioid analgesic abuse and mortality in the United States. N Engl J Med. 2015;2015(372):241–248.
4. Jones CM, Campopiano M, Baldwin G, McCance-Katz E. National and state treatment need and capacity for opioid agonist medication-assisted treatment. Am J Public Health. 2015;105(8):e55–e63.
5. Wu LT, Zhu H, Swartz MS. Treatment utilization among persons with opioid use disorder in the United States. Drug Alcohol Depend. 2016;169:117–127.
7. Initiation and Engagement of Alcohol and Other Drug Dependence Treatment. 2016. Accessed September 15, 2016.
9. Kleber HD, Anton RF Jr, George TP, et al. Treatment of patients with substance use disorders. Am J Psychiatry 2006;163(8 suppl):5–82.
10. Management of Substance Use Disorders Working Group. VA/DoD Clinical Practice Guideline for Management of Substance Use Disorders (SUD), 2009. Washington, DC; 2013.
11. Medication and Counseling Treatment. 2017. Accessed 14 August 2017.
12. National Committee for Quality Assurance (NCQA). Revised Measures, HEDIS 2018. Initiation and Engagement of Alcohol and Other Drug Abuse or Dependence Treatment (IET) and Identification of Alcohol and Other Drug Services (IAD). 2017. Accessed 14 August 2017.
13. Adamson D, Chang S, Hansen L. Health Research Data for the Real World: The Marketscan Databases. Ann Arbor, MI: Thomson Reuters; 2010.
14. Group MoSUDW. VA/DoD Clinical Practice Guideline for the Management of Substance Use Disorders. NW Washington, D.C.: Department of Veterans Affairs, Department of Defense; 2015 (Version 3.0).
15. Blodgett JC, Del Re A, Maisel NC, Finney JW. A meta-analysis of Topiramate's effects for individuals with alcohol Use disorders. Alcohol Clin Exp Res. 2014;38(6):1481–1488.
16. Harris AH, Weisner CM, Chalk M, et al Quality measures for the American Society of addiction Medicine's standards of care. J Addict Med. 2016;10(3):148–155.
17. United States Food and Drug Administration. National Drug Code Directory. Silver Spring, MD: USFDA; 2016. Accessed November 13, 2015.
20. Mattson M, Lynch S. Medication prescribing and behavioral treatment for substance use disorders in physician office settings. 2013.
21. Rieckmann T, Muench J, McBurnie MA, et al. Medication-assisted treatment for substance use disorders within a national community health center research network. Subst Abus. 2016;37(4):625–634.
22. Dawson DA, Grant BF, Stinson FS, Chou PS. Estimating the effect of help-seeking on achieving recovery from alcohol dependence. Addiction. 2006;101(6):824–834.
23. Grant BF, Goldstein RB, Saha TD, et al. Epidemiology of DSM-5 alcohol use disorder: Results from the national epidemiologic Survey on alcohol and related conditions III. JAMA Psychiatry. 2015;72(8):757–766.
24. Mattke S. When should measures be updated? Development of a conceptual framework for maintenance of quality-of-care measures. Qual Saf Health Care. 2008;17(3):182–186.
25. Volkow ND, Frieden TR, Hyde PS, Cha SS. Medication-assisted therapies—tackling the opioid-overdose epidemic. N Engl J Med. 2014;370(22):2063–2066.
26. Weiss RD, Potter JS, Griffin ML, et al. Long-term outcomes from the national drug abuse treatment clinical trials network prescription opioid addiction treatment study. Drug Alcohol Depend. 2015;150:112–119.
27. Baser O, Chalk M, Rawson R, Gastfriend DR. Alcohol dependence treatments: Comprehensive healthcare costs, utilization outcomes, and pharmacotherapy persistence. Am J Manag Care 2011;17:S222–S234.
28. Harris AH, Ellerbe L, Phelps TE, et al. Examining the specification validity of the HEDIS quality measures for substance use disorders. J Subst Abuse Treat. 2015;53:16–21.
29. Harris AH, Reeder RN, Ellerbe LS, Bowe TR. Validation of the treatment identification strategy of the HEDIS addiction quality measures: Concordance with medical record review. BMC Health Serv Res. 2011;11(1):73.
30. The N SSATS Report: Trends in the Use of Methadone and Buprenorphine at Substance Abuse Treatment Facilities: 2003 to 2011. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2013.
Back to Top | Article Outline

Authors' Biographies

Soeren Mattke, MD, DSc is a senior scientist at RAND, Boston, MA and focuses on measuring and improving the quality of care for patients with chronic conditions. He received his MD from the University of Munich and his MPH and DSc. from the Harvard School of Public Health.

Zachary Predmore, BA is an adjunct research assistant at RAND and a doctoral student in Health Policy and Management at the Johns Hopkins Bloomberg School of Public Health in Baltimore, MD.

Elizabeth Sloss, PhD is an epidemiologist/health services researcher at RAND, Arlington, VA. Her current research focuses on quality measures for psychological health conditions for the Department of Defense, and measurement of access to and quality of health care provided to veterans.

Asa Wilks, MPA is a statistical programmer at RAND, Santa Monica, CA. His work is primarily focused on data analysis projects in health, education, and public policy. Asa holds an MPA from Columbia University and a BA from Cornell University.

Katherine Watkins, MD, MSHS is a senior natural scientist at RAND. The overall goal of Dr. Watkins' research is to improve the quality of care for individuals with behavioral health disorders, by developing, implementing, and evaluating innovative treatments and treatment models of health care delivery. She received her MD from the University of Pennsylvania and her MSHS from UCLA.


substance use; medication-assisted treatment; quality measure

© 2018 National Association for Healthcare Quality