Secondary Logo

Journal Logo

Benchmarking Implications

Analysis of Medicare Accountable Care Organizations Spending Level and Quality of Care

Hong, Young-Rock; Kates, Frederick; Song, Soon Ju; Lee, Nayoung; Duncan, R. Paul; Marlow, Nicole M.

The Journal for Healthcare Quality (JHQ): November/December 2018 - Volume 40 - Issue 6 - p 344–353
doi: 10.1097/JHQ.0000000000000123
Original Article

ABSTRACT: Early evidence has shown that Accountable Care Organizations (ACOs) have achieved some success in improving the quality of care and reducing Medicare costs. However, it has been argued that the ACO rewarding model may disproportionately affect relatively low-spending (LS; considered as efficient) organizations that have fewer options to cut unnecessary services compared with high-spending (HS; inefficient) organizations. We conducted a cross-sectional retrospective study to compare ACO financial and quality of care performance between HS-ACO and LS-ACO. After adjusting for ACO organizational factors and beneficiary characteristics, we found that HS-ACOs generated greater savings per beneficiary than LS-ACOs ($501 vs. −$108, p < .001); however, HS-ACOs had a lower quality of care performance (48.79 vs. 53.29, p = .002). Specifically, LS-ACOs had better quality performance than HS-ACOs in patient experience/satisfaction (p = .02), preventive care services (p = .004), and hospitalization management (p = .001), whereas HS-ACOs better performed in routine checkup/follow-up (p < .001) and risk population management (p = .048). Our findings indicated that Medicare ACO rewarding model seems to be advantageous for HS-ACOs regardless of the overall quality of care performance.

For more information on this article, contact Young-Rock Hong at

The authors declare no conflicts of interest.

Back to Top | Article Outline


Accountable Care Organizations (ACOs) emerged as an alternative to the Medicare fee-for-service payment model with the signing of the Affordable Care Act (ACA). Under the ACA, the Medicare Shared Savings Program (MSSP) provides economic incentives aligned with accountability for reducing costs and improving the quality of care for a defined Medicare population.1,2 Participating ACOs consist of physicians, hospitals, and other health providers that work together to achieve measurable quality improvements and reductions in spending growth.3,4 In return, if the ACOs meet a set of quality standards, providers can share in the generated savings up to 50% of the difference between the actual costs and the spending benchmarks set by the Center for Medicare and Medicaid Services (CMS).5

Despite early evidence that MSSP ACO initiative has achieved some success in improving the quality of care and reducing Medicare costs,6–8 recent work has argued that the ACO rewarding model may disproportionately affect low-spending organizations that have fewer options to cut unnecessary health services while improving or maintaining quality of care compared with organizations that have high-spending level at the start of the program.6,9–11 Initially, CMS set an ACO's benchmark based on 3-year historical spending. There have been concerns raised about the fairness of this benchmarking method; that ACOs with higher costs and overutilization have an advantage in meeting targets and sharing in cost reduction savings compared with ACOs that were already more efficient at providing low-cost health services.9

Research has examined ACO performance regarding national Medicare spending and quality of care improvements.6,8,9,12 However, little is known about the effect of an ACO's historical spending level on its performance. Understanding how different spending levels relate to generating savings and quality of care performance could suggest possible strategies for health service organizations to continue their participation or join the ACO program or seek other payment arrangements. A greater understanding of past MSSP-ACO performance year (PY) results can provide additional insight to support the future development of payment policy in the Medicare ACO programs. Therefore, we examined the effects of spending level on ACO financial and quality of care performance, comparing ACOs with spending above local averages (high-spending) with ACOs with spending below local averages (low-spending).

Back to Top | Article Outline



The primary motivation of physician groups or other healthcare entities to participate in an ACO is to obtain patients (i.e., resources) and to reduce related costs through a coordinated continuum of healthcare services (i.e., improved quality of care), thereby maximizing their savings. Empirical evidence has demonstrated that integrated healthcare systems could provide higher quality of care with lower overall costs.13,14 However, given that the ACO benchmark for total cost of care for assigned beneficiaries is determined by the ACO's historical spending,5 organizations with high-cost structure (considered as less efficient) may generate more savings with more opportunities to reduce utilization and costs compared with organizations with low-cost structure (efficient).9 In the first year of Pioneer ACO program, two-third of the savings were generated from the organizations that run or practice at the most expensive hospital and in the region that had the highest per capita Medicare spending in the United States.15 In addition, organizations that left the program commonly reported that the unsustainability of the reward model for organizations that already provide care at lower costs was their primary reason for withdrawal.15,16 Therefore, our hypothesis was that ACOs with a high-cost structure would have larger gains in shared savings and better quality of care performance compared with those with a low-cost structure.

Back to Top | Article Outline

Data and Study Design

We conducted a cross-sectional retrospective study of ACOs in the MSSP. The principal sources of data were the MSSP-ACO PY results and the ACO Public Use File (PUF) in 2014 and 2015. The MSSP-ACO PY results data are composed of CMS annual reports on the quality of care and financial performance of all participants in the MSSP, and the ACO PUF contains ACO-specific metrics, characteristics of assigned beneficiaries, and provider information.17 We merged these data using the ACO identification number and legal business name for each organization as the matching variables. We also used the ZIP codes of the ACOs' addresses from the list of shared savings program ACO file and, if missing, from information on the ACOs' websites. Using the ACOs' ZIP codes, we identified their local hospital referral regions (HRRs). Then, HRR-level data of the Medicare fee-for-service population for each ACO were derived from the CMS database.18,19

Back to Top | Article Outline

Study Sample

We used the 2014 MSSP data to identify ACOs that had joined the program in 2012–2014 and analyzed the 2015 performance data of study eligible ACOs. We identified 333 ACOs in the PY 2014, and 303 ACOs remained in the PY 2015 (n = 30 dropouts). Of those, we excluded ACOs that failed to meet or report their quality of care performance measures (n = 6). A total of 297 ACOs in the MSSP were included for the study; of those, 108 ACOs were defined as low-spending ACOs (LS-ACOs), and 189 ACOs were defined as HS ACOs. Table 1 presents the characteristics of MSSP ACOs by spending level.

Table 1

Table 1

Back to Top | Article Outline


Spending Level

We defined ACO's spending level according to whether their risk-adjusted historical 3-year spending per beneficiary was above (HS-ACO) or below (LS-ACO) the standardized, risk-adjusted 3-year Medicare spending per beneficiary in its HRR.

Back to Top | Article Outline

Financial Performance

The amount of generated savings per beneficiary (the difference between ACO benchmark and actual spending divided by the number of assigned beneficiaries) was used to measure financial performance and presented in 2015 U.S. dollars.

Back to Top | Article Outline

Quality of Care Performance

Center for Medicare and Medicaid Services provides the ACO 33 quality of care metrics across 4 domains, including patient experience, coordination of care and patient safety, preventive health, and at-risk population care. For most of the 33 measures, a higher score represents better performance on a scale of 0–100, but 8 measures for admissions, readmissions, and diabetes control have a reverse scoring that a lower score is indicative of better performance. We conducted an exploratory factor analysis (EFA) to see if all 33 ACO measures span the initial 4 quality of care domains and to capture and measure core dimensions of quality of care performance. This assessment validated quality of care performance domains and allowed the generation of composite scores to represent each captured domain. The raw quality of care performance scores was converted to percentile scores using ranking transformation to address non-normal distributions and to be compared on the same scale of 1–100 (reverse-scored where required). We calculated composite scores of each quality domain that were significantly loaded and an overall quality of care performance score.

Back to Top | Article Outline

Accountable Care Organizations Characteristics

Organizational characteristics included ACO size (sorted into quartiles based on the number of beneficiaries: small [<8,000], medium [8,000–12,499], medium-large [12,500–20,999], and large [>21,000]), health provider composition (number of primary care physicians, specialists, and non–physician providers per 1,000 assigned beneficiaries), and inclusion of hospital (including critical access hospital and teaching amendment hospital) or health clinic/center (including Federally Qualified Health Center [FQHC] and Rural Health Clinic [RHC]) participating in ACO. Beneficiary characteristics included proportions of beneficiaries aged 64 or younger, 65–74, 75–84, and 85 or older; female; non–white minorities; disabled; with end-stage renal disease (ESRD); and dually eligible for both Medicare and Medicaid. Measures in health services utilization included number of primary care services per 1,000 person-years by a physician, specialist, and non–physician provider; rate of short-term acute care readmissions within 30 days of discharge from a hospital per 1,000 discharges; and number of emergency department visits (outpatient and inpatient) per 1,000 person-years.

Back to Top | Article Outline

Other Covariates

Several other geographical and organizational characteristics were included to control for their potentially confounding effect on ACO performance: number of years enrolled in the MSSP, Advanced Payment ACO Model status, HRR fixed effects, rurality, number of postdischarge provider visits, skilled nursing discharges, computed tomography/magnetic resonance imaging events, primary care services provided by FQHC or RHC, and CMS Hierarchical Condition Category risk scores for patients who are disabled, with ESRD, aged 65 or older with dual eligibility, and aged 65 or older without dual eligibility.

Back to Top | Article Outline

Statistical Analyses

An EFA was conducted first to identify core quality of care dimensions among the ACO 33 quality of care measures for all study ACOs (N = 297). The principal component analysis was used to extract factors with a Promax rotation allowing derivation of correlated factors.20 Cronbach alpha was then used to test the internal consistency and reliability among items loaded in the same quality of care dimension. Descriptive statistics were used to evaluate characteristics of ACOs by spending level. Statistical significance was assessed using two-sample t-tests for continuous variables, chi-square tests for categorical variables, and Fisher exact test for dichotomous variables. To examine the associations between ACO spending level and financial and quality of care performance, we estimated multivariate generalized linear regression models controlling for other ACO characteristics and the covariates listed above. All analyses were performed using SPSS Complex Samples (version 24; IBM Corp, Armonk, NY) and SAS statistical software (version 9.4; SAS Institute Inc, Cary, NC). Two-tailed p < .05 was considered statistically significant. The institutional review board of our institution determined that this study was exempt from review (IRB 201700946).

Back to Top | Article Outline


Validity of the Quality Performance Measures

Exploratory factor analysis was executed to capture and measure core dimensions of quality of care performance among ACO 33 quality measures. On the basis of the factor loadings (eigenvalues ≥ 1.0 and factor loading ≥ 0.4),20 we removed measures that did not load significantly onto their intended domains and reduced the number of measures to 24 items across 5 constructs that can be interpreted as (1) patient experience/satisfaction, (2) routine checkup/follow-up, (3) preventive care services, (4) hospital management, and (5) risk population care management. The internal consistency reliabilities of all five constructs were excellent (range of α: 0.772–0.903; Table 2).21

Table 2-a

Table 2-a

Table 2-b

Table 2-b

Back to Top | Article Outline

Descriptive Analyses Results

Figure 1 shows scatter plots of unadjusted relationships between generated savings per beneficiary, average local Medicare spending, and overall quality performance score. Average generated savings for MSSP ACOs enrolled in 2012–2014 were $133.6 per beneficiary in 2015. As shown in Figure 1A, we observed a positive linear relationship between the difference of spending between ACO and its HRR average and the amount of generated savings (albeit low variability), indicating that ACOs that provided care at costs above average local spending were more likely to generate savings. ACOs with spending above the local average (1–60% spending more) scatter widely compared with those with spending below average (0–20% spending less). Further examination showed that the unadjusted average savings generated per beneficiary was significantly higher for HS-ACOs than for LS-ACOs ($256.4 vs. −$81.3, p < .001). However, the unadjusted average overall quality of care performance score was higher for LS-ACOs than for HS-ACOs (54.4 vs. 49.5, p < .001; Table 2).

Table 2 presents ACO quality of care performance measures in the five constructs identified from the EFA and their factor loadings. Quality of care performance in “Patient Experience/Satisfaction” (Factor 1) was higher for LS-ACOs than for HS-ACOs (composite score: 52.3 vs. 48.8); however, this difference did not reach statistical significance (p = .099). By and large, LS-ACOs better performed than HS-ACOs in “Preventive Care Services” (Factor 3; 54.3 vs. 47.7, p < .008) and “Hospitalization Management” (Factor 4; 61.4 vs. 43.1, p < .001). We did not observe statistically significant differences in “Routine Checkup/Follow-up” (Factor 2) and “Risk Population Care Management” (Factor 5).

Back to Top | Article Outline

Adjusted Analyses Results

Table 3 presents regression-adjusted ACO financial and quality of care performance. After adjusting for covariates, HS-ACOs had $609 (95% confidence interval [CI], $438.9–$780.7, p < .001) significantly greater generated savings per beneficiary compared with LS-ACOs. In quality performance, LS-ACOs had higher overall quality of care performance score than HS-ACOs (mean difference, 4.5; 95% CI, 1.7–7.3, p = .002) and the difference in Patient Experience/Satisfaction became significant (LS-ACOs vs. HS-ACOs, mean difference, 5.1; 95% CI, 0.2–9.9, p = .002). Consistent with the unadjusted results, LS-ACOs had better performance in Preventive Care Services (mean difference, 7.6; 95% CI, 2.4–12.8, p = .004) and Hospitalization Management (mean difference, 4.8; 95% CI, 2.0–7.6, p = .001) than HS-ACOs. However, we found that HS-ACOs had better performance than LS-ACOs in routine checkup/follow-up (mean difference, 14.0; 95% CI, 8.6–19.4, p < .001) and Risk Population Management (mean difference, 10.8; 95% CI, 4.7–16.9, p = .048).

Table 3

Table 3

Back to Top | Article Outline


Our study has several limitations that need to be noted. First, our results may not be generalizable to all ACOs across the country because the data were limited to the ACOs participating in the MSSP and did not include provider groups or organizations with commercial ACO arrangements. Although commercial insurers have implemented their own ACO programs and similar payment models,22 commercial ACOs are not required to follow the Medicare's ACO definitions and specifications; thus, effects of spending level for commercial ACOs on healthcare organization performance may differ. Second, CMS provides limited information on the characteristics of MSSP ACOs; thus, we were not able to examine or control for other key ACO characteristics (use of health information technology, care coordination, or case management) that could influence ACO performance. Last, the HRR-level Medicare spending to determine ACO spending level was derived based on the location of the ACO's headquarter. The estimations of baseline spending for ACOs that had their beneficiaries serving in distant areas may be overestimated or underestimated. For example, it is possible that a health provider serving in a remote location has higher or lower Medicare spending due to regional and patient population characteristics. Nevertheless, our results are still useful in understanding effects of spending level on ACO performance. Moreover, we used 3-year historical spending of each ACO with identical weights as applied by CMS for benchmarking to determine spending level and adjusted for HRR-fixed effects to minimize the potential for geographical differences in ACO performance.

Back to Top | Article Outline


Our analyses of the MSSP ACO data indicate that, although HS-ACOs had lower scores in overall quality of care measures, they were more likely than LS-ACOs to generate savings. After adjustment for ACO characteristics and beneficiary characteristics, HS-ACOs had generated savings more than twice as much as those of LS-ACOs. These findings suggest that, although health service providers that provide care at higher costs may be capable of constraining their spending for patient care in line with the intent of ACO initiative,1,2,5 the Medicare ACO rewarding model was advantageous for those providers regardless of their quality of care performance. Our study provides evidence that holding spending for attributed beneficiaries below a benchmark level (historical spending level) seems to be the only metric to determine whether an ACO generates savings, which links directly to receiving bonuses.

Our results are consistent with earlier studies9,23 that have suggested the use of historical spending average as benchmarks could disproportionately affect ACOs. For example, “already efficient” providers that had low-spending history had less of a chance to receive bonuses with relatively low-set benchmarks.9 Moreover, it is difficult for providers that had generated higher savings in previous contract period to achieve better performance in future years with updated baseline spending,23 which may cause an exodus of such providers from the ACO program due to problems in sustainability.

Our findings could support CMS efforts to incorporate regional Medicare spending data into adjusting or updating an ACO's benchmark.24 However, our results also call for additional adjustments in the ACO rewarding model to better reflect quality of care performance of an ACO. There is consistent evidence to suggest improved patient case mix including clinical and social factors in the ACO measure set.23,25,26 Similarly, we found that HS-ACOs had poorer performance in hospitalization management, which reflects higher admission and readmission rates than LS-ACOs. Although this may have resulted from the fact that HS-ACOs had greater proportions of beneficiaries who were older, nonwhite, or with ESRD, it appears that HS-ACOs' better performance in routine checkup and follow-up or risk population management has not resulted in better patient outcomes like readmissions. Adjusting variations in patient case mix or developing standardized risk adjustments that are designed for ACO program could be one way to address this.23,27 When compared with HS-ACOs, LS-ACOs stood out for patient experience/satisfaction and preventive care services, which are considered less costly quality of care performance. Taken together, our findings suggest the need for developing value-adjusted quality of care performance (e.g., different weighting methodology) and additional patient-centered quality measures28 (e.g., accounting for patient preference and perceived value of care) to reflect quality of care in ways relevant to actual performance and patient outcomes.

Back to Top | Article Outline


High-spending ACOs generated greater shared savings than LS-ACOs, whereas they had lower overall quality of care performance. Our evaluation suggests that the current ACO model may be successful in lowering Medicare costs but may not be sustainable for promoting better quality of patient care.

Back to Top | Article Outline


Accountable Care Organization programs have been proposed as a remedy to the fragmented nature of healthcare delivery in the United States. Because the MSSP ACO model gains traction in the United States, it is important to evaluate the progress and keep the momentum moving forward. This study emphasizes the importance of evaluating ACO performance to understand opportunities for better shaping the next CMS benchmarking methodology. Further rigorous study of the MSSP is especially warranted to support refinement of such benchmarking methodologies.

Our findings hold important implications for policy makers regarding additional adjustments and development of quality measures to better reflect ACO performance. It may be useful for CMS to consider including solely quality of care performance-based bonuses for efficient and quality providers or application of fixed amount of incentives with different weighting methodology for costly care performance in the MSSP ACO program.

Back to Top | Article Outline


1. Berwick DM. Launching accountable care organizations—The proposed rule for the medicare shared savings program. N Engl J Med. 2011;364(16):e32.
2. GovTrack. H.R. 3590 (111th): Patient Protection and Affordable Care Act. Published 2009. Accessed July 7, 2017.
3. McClellan M, McKethan AN, Lewis JL, Roski J, Fisher ES. A national strategy to put accountable care into practice. Health Aff. 2010;29(5):982–990.
4. Fisher ES, McClellan MB, Bertko J, et al. Fostering accountable health Care: Moving forward in Medicare. Health Aff. 2009;28(2):w219–w231.
5. Centers for Medicare and Medicaid Services. Accountable Care Organizations: What Providers Need to Know; 2016. Accessed May 24, 2017.
6. McWilliams JM, Hatfield LA, Chernew ME, Landon BE, Schwartz AL. Early performance of accountable care organizations in medicare. N Engl J Med. 2016;374(24):2357–2366.
7. Toussaint J, Milstein A, Shortell S. How the pioneer ACO model needs to change. JAMA. 2013;310(13):1341.
8. The Lewin Group. CMS Bundled Payments for Care Improvement (BPCI) Initiative Models 2-4 : Year 1 Evaluation & Monitoring Annual Report; 2015. Accessed February 10, 2017.
9. McWilliams JM. ACO payment models and the path to accountability. J Gen Intern Med. 2014;29(10):1328–1330.
10. McWilliams JM, Chernew ME, Landon BE, Schwartz AL. Performance differences in year 1 of pioneer accountable care organizations. N Engl J Med. 2015;372(20):1927–1936.
11. Schwartz AL, Chernew ME, Landon BE, McWilliams JM. Changes in low-value services in year 1 of the medicare pioneer accountable care organization program. JAMA Intern Med. 2015;175(11):1815.
12. Sommers BD, Kenney GM, Epstein AM. New evidence on the affordable care Act: Coverage impacts of early medicaid expansions. Health Aff. 2014;33(1):78–87.
13. Hwang W, Chang J, Laclair M, Paz H. Effects of integrated delivery system on cost and quality. Am J Manag Care. 2013;19(5):e175–e184.
14. Weeks WB, Gottlieb DJ, Nyweide DJ, et al. Higher health care quality and bigger savings found at large multispecialty medical groups. Health Aff. 2010;29(5):991–998.
15. Goldsmith J. Pioneer ACOs' Disappointing First Year: Health Affairs Blog. Published 2013. Accessed November 22, 2016.
16. Advisory Board. Cassels: Why Some Pioneer ACOs Are Dropping Out. Published 2013. Accessed November 22, 2016.
17. Centers for Medicare and Medicaid Services. Medicare Shared Savings Program Publicly Available ACO Data and ACO Performance Data Sources Maintained by CMS; 2017. Accessed March 10, 2017.
18. Centers for Medicare and Medicaid Services. Medicare Geographic Variation. Published 2017. Accessed March 10, 2017.
19. The Dartmouth Atlas of Health Care. Geographic Crosswalks and Research Files. Published 2017. Accessed April 13, 2017.
20. Beaumont R. An Introduction to Principal Component Analysis & Factor Analysis Using SPSS 19 and R (Psych Package): Virtual Classr; 2012. Accessed June 13, 2017.
21. Nunnally J, Bernstein I. Psychometric Theory. 3rd ed. New York, NY: McGraw-Hill; 1994.
22. Gandhi N, Weil R. The ACO Surprise; 2012. Accessed December 1, 2016.
23. Rose S, Zaslavsky AM, McWilliams JM. Variation in accountable care organization spending and sensitivity to risk adjustment: Implications for benchmarking. Health Aff. 2016;35(3):440–448.
24. Centers for Medicare and Medicaid Services. Final Medicare Shared Savings Program Rule (CMS-1644-F). Published 2016. Accessed November 14, 2016.
25. Albright BB, Lewis VA, Ross JS, Colla CH. Preventive care quality of Medicare Accountable Care Organizations. Med Care. 2016;54(3):326–335.
26. Lewis VA, Fraze T, Fisher ES, Shortell SM, Colla CH. ACOs serving high proportions of racial and ethnic minorities lag in quality performance. Health Aff. 2017;36(1):57–66.
27. Figueroa JF, Wang DE, Jha AK. Characteristics of hospitals receiving the largest penalties by US pay-for-performance programmes. BMJ Qual Saf. 2016;25(11):898–900.
28. Hong YR, Holcomb D, Ballard M, Schwartz L. Multigroup path analysis of the influence of healthcare quality, by different health insurance types. J Healthc Manag. 2017;62(2):93–105.
Back to Top | Article Outline

Authors' Biographies

Young-Rock Hong, MPH, is a PhD student in the Department of Health Services Research, Management and Policy in the College of Public health and Health Profession, University of Florida, Gainesville, FL. His primary research interests focus on designing and analysis of service quality measures in health services, integrating evaluated patient-centered outcomes into Value-Based Payment (VBP) models, and improvements in clinician burnout and non-clinical employee safety.

Frederick Kates, MBA, PhD, is an assistant professor in the Department of Health Services Research, Management and Policy at the University of Florida. He has research interests related to health policy, most recently outcome-based evaluation of service quality in Accountable Care Organizations (ACOs) and the impact of the Medicaid expansion.

Soon Ju Song, MA, is a PhD student in the Department of Health Services Research, Management and Policy in the College of Public health and Health Profession, University of Florida, Gainesville, FL. Her research interests are health communication between health providers and patients, and disparity in access and utilization of oral health care.

Nayoung Lee, BA, is an administrative assistant volunteer in the Department of Volunteer Program, Alachua County Library District, Gainesville, FL. She has a solid background in analytical methods having worked as a marketing analyst. Her research interests include consumer behavior and patient experience in health services delivery.

R. Paul Duncan, PhD, is the Malcom and Christine Randall Professor of Health Services Research, and the Senior of Associate Dean of the Graduate School at the University of Florida. His research interests are focused on access and utilization, with particular emphasis on the role of health insurance as a facilitator of health care use.

Nicole M. Marlow, PhD, MSPH, is an assistant professor in the Department of Health Services Research, Management and Policy at the University of Florida in Gainesville, FL. Dr. Marlow's primary research focus is on improving healthcare quality by providing evidence to support patient-provider communication needs, particularly for persons with disabilities. Her work involves the analysis of large health databases, which has resulted in multiple collaborations with interdisciplinary and translational science investigators.


Accountable Care Organization; Medicare shared savings program; value-based payment; quality of care delivery; fee-for-service

© 2018 National Association for Healthcare Quality