Accountable Care Organizations (ACOs) emerged as an alternative to the Medicare fee-for-service payment model with the signing of the Affordable Care Act (ACA). Under the ACA, the Medicare Shared Savings Program (MSSP) provides economic incentives aligned with accountability for reducing costs and improving the quality of care for a defined Medicare population.1,2 Participating ACOs consist of physicians, hospitals, and other health providers that work together to achieve measurable quality improvements and reductions in spending growth.3,4 In return, if the ACOs meet a set of quality standards, providers can share in the generated savings up to 50% of the difference between the actual costs and the spending benchmarks set by the Center for Medicare and Medicaid Services (CMS).5
Despite early evidence that MSSP ACO initiative has achieved some success in improving the quality of care and reducing Medicare costs,6–8 recent work has argued that the ACO rewarding model may disproportionately affect low-spending organizations that have fewer options to cut unnecessary health services while improving or maintaining quality of care compared with organizations that have high-spending level at the start of the program.6,9–11 Initially, CMS set an ACO's benchmark based on 3-year historical spending. There have been concerns raised about the fairness of this benchmarking method; that ACOs with higher costs and overutilization have an advantage in meeting targets and sharing in cost reduction savings compared with ACOs that were already more efficient at providing low-cost health services.9
Research has examined ACO performance regarding national Medicare spending and quality of care improvements.6,8,9,12 However, little is known about the effect of an ACO's historical spending level on its performance. Understanding how different spending levels relate to generating savings and quality of care performance could suggest possible strategies for health service organizations to continue their participation or join the ACO program or seek other payment arrangements. A greater understanding of past MSSP-ACO performance year (PY) results can provide additional insight to support the future development of payment policy in the Medicare ACO programs. Therefore, we examined the effects of spending level on ACO financial and quality of care performance, comparing ACOs with spending above local averages (high-spending) with ACOs with spending below local averages (low-spending).
The primary motivation of physician groups or other healthcare entities to participate in an ACO is to obtain patients (i.e., resources) and to reduce related costs through a coordinated continuum of healthcare services (i.e., improved quality of care), thereby maximizing their savings. Empirical evidence has demonstrated that integrated healthcare systems could provide higher quality of care with lower overall costs.13,14 However, given that the ACO benchmark for total cost of care for assigned beneficiaries is determined by the ACO's historical spending,5 organizations with high-cost structure (considered as less efficient) may generate more savings with more opportunities to reduce utilization and costs compared with organizations with low-cost structure (efficient).9 In the first year of Pioneer ACO program, two-third of the savings were generated from the organizations that run or practice at the most expensive hospital and in the region that had the highest per capita Medicare spending in the United States.15 In addition, organizations that left the program commonly reported that the unsustainability of the reward model for organizations that already provide care at lower costs was their primary reason for withdrawal.15,16 Therefore, our hypothesis was that ACOs with a high-cost structure would have larger gains in shared savings and better quality of care performance compared with those with a low-cost structure.
Data and Study Design
We conducted a cross-sectional retrospective study of ACOs in the MSSP. The principal sources of data were the MSSP-ACO PY results and the ACO Public Use File (PUF) in 2014 and 2015. The MSSP-ACO PY results data are composed of CMS annual reports on the quality of care and financial performance of all participants in the MSSP, and the ACO PUF contains ACO-specific metrics, characteristics of assigned beneficiaries, and provider information.17 We merged these data using the ACO identification number and legal business name for each organization as the matching variables. We also used the ZIP codes of the ACOs' addresses from the list of shared savings program ACO file and, if missing, from information on the ACOs' websites. Using the ACOs' ZIP codes, we identified their local hospital referral regions (HRRs). Then, HRR-level data of the Medicare fee-for-service population for each ACO were derived from the CMS database.18,19
We used the 2014 MSSP data to identify ACOs that had joined the program in 2012–2014 and analyzed the 2015 performance data of study eligible ACOs. We identified 333 ACOs in the PY 2014, and 303 ACOs remained in the PY 2015 (n = 30 dropouts). Of those, we excluded ACOs that failed to meet or report their quality of care performance measures (n = 6). A total of 297 ACOs in the MSSP were included for the study; of those, 108 ACOs were defined as low-spending ACOs (LS-ACOs), and 189 ACOs were defined as HS ACOs. Table 1 presents the characteristics of MSSP ACOs by spending level.
We defined ACO's spending level according to whether their risk-adjusted historical 3-year spending per beneficiary was above (HS-ACO) or below (LS-ACO) the standardized, risk-adjusted 3-year Medicare spending per beneficiary in its HRR.
The amount of generated savings per beneficiary (the difference between ACO benchmark and actual spending divided by the number of assigned beneficiaries) was used to measure financial performance and presented in 2015 U.S. dollars.
Quality of Care Performance
Center for Medicare and Medicaid Services provides the ACO 33 quality of care metrics across 4 domains, including patient experience, coordination of care and patient safety, preventive health, and at-risk population care. For most of the 33 measures, a higher score represents better performance on a scale of 0–100, but 8 measures for admissions, readmissions, and diabetes control have a reverse scoring that a lower score is indicative of better performance. We conducted an exploratory factor analysis (EFA) to see if all 33 ACO measures span the initial 4 quality of care domains and to capture and measure core dimensions of quality of care performance. This assessment validated quality of care performance domains and allowed the generation of composite scores to represent each captured domain. The raw quality of care performance scores was converted to percentile scores using ranking transformation to address non-normal distributions and to be compared on the same scale of 1–100 (reverse-scored where required). We calculated composite scores of each quality domain that were significantly loaded and an overall quality of care performance score.
Accountable Care Organizations Characteristics
Organizational characteristics included ACO size (sorted into quartiles based on the number of beneficiaries: small [<8,000], medium [8,000–12,499], medium-large [12,500–20,999], and large [>21,000]), health provider composition (number of primary care physicians, specialists, and non–physician providers per 1,000 assigned beneficiaries), and inclusion of hospital (including critical access hospital and teaching amendment hospital) or health clinic/center (including Federally Qualified Health Center [FQHC] and Rural Health Clinic [RHC]) participating in ACO. Beneficiary characteristics included proportions of beneficiaries aged 64 or younger, 65–74, 75–84, and 85 or older; female; non–white minorities; disabled; with end-stage renal disease (ESRD); and dually eligible for both Medicare and Medicaid. Measures in health services utilization included number of primary care services per 1,000 person-years by a physician, specialist, and non–physician provider; rate of short-term acute care readmissions within 30 days of discharge from a hospital per 1,000 discharges; and number of emergency department visits (outpatient and inpatient) per 1,000 person-years.
Several other geographical and organizational characteristics were included to control for their potentially confounding effect on ACO performance: number of years enrolled in the MSSP, Advanced Payment ACO Model status, HRR fixed effects, rurality, number of postdischarge provider visits, skilled nursing discharges, computed tomography/magnetic resonance imaging events, primary care services provided by FQHC or RHC, and CMS Hierarchical Condition Category risk scores for patients who are disabled, with ESRD, aged 65 or older with dual eligibility, and aged 65 or older without dual eligibility.
An EFA was conducted first to identify core quality of care dimensions among the ACO 33 quality of care measures for all study ACOs (N = 297). The principal component analysis was used to extract factors with a Promax rotation allowing derivation of correlated factors.20 Cronbach alpha was then used to test the internal consistency and reliability among items loaded in the same quality of care dimension. Descriptive statistics were used to evaluate characteristics of ACOs by spending level. Statistical significance was assessed using two-sample t-tests for continuous variables, chi-square tests for categorical variables, and Fisher exact test for dichotomous variables. To examine the associations between ACO spending level and financial and quality of care performance, we estimated multivariate generalized linear regression models controlling for other ACO characteristics and the covariates listed above. All analyses were performed using SPSS Complex Samples (version 24; IBM Corp, Armonk, NY) and SAS statistical software (version 9.4; SAS Institute Inc, Cary, NC). Two-tailed p < .05 was considered statistically significant. The institutional review board of our institution determined that this study was exempt from review (IRB 201700946).
Validity of the Quality Performance Measures
Exploratory factor analysis was executed to capture and measure core dimensions of quality of care performance among ACO 33 quality measures. On the basis of the factor loadings (eigenvalues ≥ 1.0 and factor loading ≥ 0.4),20 we removed measures that did not load significantly onto their intended domains and reduced the number of measures to 24 items across 5 constructs that can be interpreted as (1) patient experience/satisfaction, (2) routine checkup/follow-up, (3) preventive care services, (4) hospital management, and (5) risk population care management. The internal consistency reliabilities of all five constructs were excellent (range of α: 0.772–0.903; Table 2).21
Descriptive Analyses Results
Figure 1 shows scatter plots of unadjusted relationships between generated savings per beneficiary, average local Medicare spending, and overall quality performance score. Average generated savings for MSSP ACOs enrolled in 2012–2014 were $133.6 per beneficiary in 2015. As shown in Figure 1A, we observed a positive linear relationship between the difference of spending between ACO and its HRR average and the amount of generated savings (albeit low variability), indicating that ACOs that provided care at costs above average local spending were more likely to generate savings. ACOs with spending above the local average (1–60% spending more) scatter widely compared with those with spending below average (0–20% spending less). Further examination showed that the unadjusted average savings generated per beneficiary was significantly higher for HS-ACOs than for LS-ACOs ($256.4 vs. −$81.3, p < .001). However, the unadjusted average overall quality of care performance score was higher for LS-ACOs than for HS-ACOs (54.4 vs. 49.5, p < .001; Table 2).
Table 2 presents ACO quality of care performance measures in the five constructs identified from the EFA and their factor loadings. Quality of care performance in “Patient Experience/Satisfaction” (Factor 1) was higher for LS-ACOs than for HS-ACOs (composite score: 52.3 vs. 48.8); however, this difference did not reach statistical significance (p = .099). By and large, LS-ACOs better performed than HS-ACOs in “Preventive Care Services” (Factor 3; 54.3 vs. 47.7, p < .008) and “Hospitalization Management” (Factor 4; 61.4 vs. 43.1, p < .001). We did not observe statistically significant differences in “Routine Checkup/Follow-up” (Factor 2) and “Risk Population Care Management” (Factor 5).
Adjusted Analyses Results
Table 3 presents regression-adjusted ACO financial and quality of care performance. After adjusting for covariates, HS-ACOs had $609 (95% confidence interval [CI], $438.9–$780.7, p < .001) significantly greater generated savings per beneficiary compared with LS-ACOs. In quality performance, LS-ACOs had higher overall quality of care performance score than HS-ACOs (mean difference, 4.5; 95% CI, 1.7–7.3, p = .002) and the difference in Patient Experience/Satisfaction became significant (LS-ACOs vs. HS-ACOs, mean difference, 5.1; 95% CI, 0.2–9.9, p = .002). Consistent with the unadjusted results, LS-ACOs had better performance in Preventive Care Services (mean difference, 7.6; 95% CI, 2.4–12.8, p = .004) and Hospitalization Management (mean difference, 4.8; 95% CI, 2.0–7.6, p = .001) than HS-ACOs. However, we found that HS-ACOs had better performance than LS-ACOs in routine checkup/follow-up (mean difference, 14.0; 95% CI, 8.6–19.4, p < .001) and Risk Population Management (mean difference, 10.8; 95% CI, 4.7–16.9, p = .048).
Our study has several limitations that need to be noted. First, our results may not be generalizable to all ACOs across the country because the data were limited to the ACOs participating in the MSSP and did not include provider groups or organizations with commercial ACO arrangements. Although commercial insurers have implemented their own ACO programs and similar payment models,22 commercial ACOs are not required to follow the Medicare's ACO definitions and specifications; thus, effects of spending level for commercial ACOs on healthcare organization performance may differ. Second, CMS provides limited information on the characteristics of MSSP ACOs; thus, we were not able to examine or control for other key ACO characteristics (use of health information technology, care coordination, or case management) that could influence ACO performance. Last, the HRR-level Medicare spending to determine ACO spending level was derived based on the location of the ACO's headquarter. The estimations of baseline spending for ACOs that had their beneficiaries serving in distant areas may be overestimated or underestimated. For example, it is possible that a health provider serving in a remote location has higher or lower Medicare spending due to regional and patient population characteristics. Nevertheless, our results are still useful in understanding effects of spending level on ACO performance. Moreover, we used 3-year historical spending of each ACO with identical weights as applied by CMS for benchmarking to determine spending level and adjusted for HRR-fixed effects to minimize the potential for geographical differences in ACO performance.
Our analyses of the MSSP ACO data indicate that, although HS-ACOs had lower scores in overall quality of care measures, they were more likely than LS-ACOs to generate savings. After adjustment for ACO characteristics and beneficiary characteristics, HS-ACOs had generated savings more than twice as much as those of LS-ACOs. These findings suggest that, although health service providers that provide care at higher costs may be capable of constraining their spending for patient care in line with the intent of ACO initiative,1,2,5 the Medicare ACO rewarding model was advantageous for those providers regardless of their quality of care performance. Our study provides evidence that holding spending for attributed beneficiaries below a benchmark level (historical spending level) seems to be the only metric to determine whether an ACO generates savings, which links directly to receiving bonuses.
Our results are consistent with earlier studies9,23 that have suggested the use of historical spending average as benchmarks could disproportionately affect ACOs. For example, “already efficient” providers that had low-spending history had less of a chance to receive bonuses with relatively low-set benchmarks.9 Moreover, it is difficult for providers that had generated higher savings in previous contract period to achieve better performance in future years with updated baseline spending,23 which may cause an exodus of such providers from the ACO program due to problems in sustainability.
Our findings could support CMS efforts to incorporate regional Medicare spending data into adjusting or updating an ACO's benchmark.24 However, our results also call for additional adjustments in the ACO rewarding model to better reflect quality of care performance of an ACO. There is consistent evidence to suggest improved patient case mix including clinical and social factors in the ACO measure set.23,25,26 Similarly, we found that HS-ACOs had poorer performance in hospitalization management, which reflects higher admission and readmission rates than LS-ACOs. Although this may have resulted from the fact that HS-ACOs had greater proportions of beneficiaries who were older, nonwhite, or with ESRD, it appears that HS-ACOs' better performance in routine checkup and follow-up or risk population management has not resulted in better patient outcomes like readmissions. Adjusting variations in patient case mix or developing standardized risk adjustments that are designed for ACO program could be one way to address this.23,27 When compared with HS-ACOs, LS-ACOs stood out for patient experience/satisfaction and preventive care services, which are considered less costly quality of care performance. Taken together, our findings suggest the need for developing value-adjusted quality of care performance (e.g., different weighting methodology) and additional patient-centered quality measures28 (e.g., accounting for patient preference and perceived value of care) to reflect quality of care in ways relevant to actual performance and patient outcomes.
High-spending ACOs generated greater shared savings than LS-ACOs, whereas they had lower overall quality of care performance. Our evaluation suggests that the current ACO model may be successful in lowering Medicare costs but may not be sustainable for promoting better quality of patient care.
Accountable Care Organization programs have been proposed as a remedy to the fragmented nature of healthcare delivery in the United States. Because the MSSP ACO model gains traction in the United States, it is important to evaluate the progress and keep the momentum moving forward. This study emphasizes the importance of evaluating ACO performance to understand opportunities for better shaping the next CMS benchmarking methodology. Further rigorous study of the MSSP is especially warranted to support refinement of such benchmarking methodologies.
Our findings hold important implications for policy makers regarding additional adjustments and development of quality measures to better reflect ACO performance. It may be useful for CMS to consider including solely quality of care performance-based bonuses for efficient and quality providers or application of fixed amount of incentives with different weighting methodology for costly care performance in the MSSP ACO program.
1. Berwick DM. Launching accountable care organizations—The proposed rule for the medicare shared savings program
. N Engl J Med. 2011;364(16):e32.
3. McClellan M, McKethan AN, Lewis JL, Roski J, Fisher ES. A national strategy to put accountable care into practice. Health Aff. 2010;29(5):982–990.
4. Fisher ES, McClellan MB, Bertko J, et al. Fostering accountable health Care: Moving forward in Medicare. Health Aff. 2009;28(2):w219–w231.
6. McWilliams JM, Hatfield LA, Chernew ME, Landon BE, Schwartz AL. Early performance of accountable care organizations in medicare. N Engl J Med. 2016;374(24):2357–2366.
7. Toussaint J, Milstein A, Shortell S. How the pioneer ACO model needs to change. JAMA. 2013;310(13):1341.
9. McWilliams JM. ACO payment models and the path to accountability. J Gen Intern Med. 2014;29(10):1328–1330.
10. McWilliams JM, Chernew ME, Landon BE, Schwartz AL. Performance differences in year 1 of pioneer accountable care organizations. N Engl J Med. 2015;372(20):1927–1936.
11. Schwartz AL, Chernew ME, Landon BE, McWilliams JM. Changes in low-value services in year 1 of the medicare pioneer accountable care organization
program. JAMA Intern Med. 2015;175(11):1815.
12. Sommers BD, Kenney GM, Epstein AM. New evidence on the affordable care Act: Coverage impacts of early medicaid expansions. Health Aff. 2014;33(1):78–87.
13. Hwang W, Chang J, Laclair M, Paz H. Effects of integrated delivery system on cost and quality. Am J Manag Care. 2013;19(5):e175–e184.
14. Weeks WB, Gottlieb DJ, Nyweide DJ, et al. Higher health care quality and bigger savings found at large multispecialty medical groups. Health Aff. 2010;29(5):991–998.
21. Nunnally J, Bernstein I. Psychometric Theory. 3rd ed. New York, NY: McGraw-Hill; 1994.
23. Rose S, Zaslavsky AM, McWilliams JM. Variation in accountable care organization
spending and sensitivity to risk adjustment: Implications for benchmarking. Health Aff. 2016;35(3):440–448.
25. Albright BB, Lewis VA, Ross JS, Colla CH. Preventive care quality of Medicare Accountable Care Organizations. Med Care. 2016;54(3):326–335.
26. Lewis VA, Fraze T, Fisher ES, Shortell SM, Colla CH. ACOs serving high proportions of racial and ethnic minorities lag in quality performance. Health Aff. 2017;36(1):57–66.
27. Figueroa JF, Wang DE, Jha AK. Characteristics of hospitals receiving the largest penalties by US pay-for-performance programmes. BMJ Qual Saf. 2016;25(11):898–900.
28. Hong YR, Holcomb D, Ballard M, Schwartz L. Multigroup path analysis of the influence of healthcare quality, by different health insurance types. J Healthc Manag. 2017;62(2):93–105.
Young-Rock Hong, MPH, is a PhD student in the Department of Health Services Research, Management and Policy in the College of Public health and Health Profession, University of Florida, Gainesville, FL. His primary research interests focus on designing and analysis of service quality measures in health services, integrating evaluated patient-centered outcomes into Value-Based Payment (VBP) models, and improvements in clinician burnout and non-clinical employee safety.
Frederick Kates, MBA, PhD, is an assistant professor in the Department of Health Services Research, Management and Policy at the University of Florida. He has research interests related to health policy, most recently outcome-based evaluation of service quality in Accountable Care Organizations (ACOs) and the impact of the Medicaid expansion.
Soon Ju Song, MA, is a PhD student in the Department of Health Services Research, Management and Policy in the College of Public health and Health Profession, University of Florida, Gainesville, FL. Her research interests are health communication between health providers and patients, and disparity in access and utilization of oral health care.
Nayoung Lee, BA, is an administrative assistant volunteer in the Department of Volunteer Program, Alachua County Library District, Gainesville, FL. She has a solid background in analytical methods having worked as a marketing analyst. Her research interests include consumer behavior and patient experience in health services delivery.
R. Paul Duncan, PhD, is the Malcom and Christine Randall Professor of Health Services Research, and the Senior of Associate Dean of the Graduate School at the University of Florida. His research interests are focused on access and utilization, with particular emphasis on the role of health insurance as a facilitator of health care use.
Nicole M. Marlow, PhD, MSPH, is an assistant professor in the Department of Health Services Research, Management and Policy at the University of Florida in Gainesville, FL. Dr. Marlow's primary research focus is on improving healthcare quality by providing evidence to support patient-provider communication needs, particularly for persons with disabilities. Her work involves the analysis of large health databases, which has resulted in multiple collaborations with interdisciplinary and translational science investigators.
Keywords:© 2018 National Association for Healthcare Quality
Accountable Care Organization; Medicare shared savings program; value-based payment; quality of care delivery; fee-for-service