Secondary Logo

Journal Logo


Looking Under the Streetlight? A Framework for Differentiating Performance Measures by Level of Care in a Value-Based Payment Environment

Naessens, James M. ScD; Van Such, Monica B. MBA; Nesse, Robert E. MD; Dilling, James A.; Swensen, Stephen J. MD; Thompson, Kristine M. MD; Orlowski, Janis M. MD; Santrach, Paula J. MD

Author Information
doi: 10.1097/ACM.0000000000001654
  • Open


Since the inception of the Joint Commission’s ORYX initiative in 1997,1 which integrated the use of performance measures into the hospital accreditation process, our health care system has been inundated with performance measures. Yet, Institute of Medicine (IOM) reports continue to identify problems; these findings have spurred improvements in both system performance and health care quality.2,3 Still, in 2006, only 54.9% of patients received the recommended care,4 which led to the implementation of additional performance measures, primarily assessing processes of care.5–8 As a result, many regulatory agencies, payers, and providers encounter duplicative and custom measures that are difficult to obtain and replicate and may not be linked to better patient outcomes.

Using reporting and assessment measures that are based on easily obtainable data is like “looking under the streetlight”9; that is, it is assessing health care quality and value by using only easy-to-find data, presumably in an attempt to reduce the burden of data collection on providers and payers. For the same reason of ease of use, health care claims and other administrative data are being used as performance measures, which has led to performance metrics being impacted by coding and billing practice issues. Many current quality report cards and assessments, then, do not reflect the breadth or complexity of many referral center practices, which are those institutions that provide advanced diagnosis and treatment specialists and facilities (usually academic centers and multispecialty group practices), after referral from primary or secondary care practices.

The National Quality Strategy, mandated by the Patient Protection and Affordable Care Act, established three aims for our health care system—better care, healthy people/healthy communities, and affordable care.10 Population health efforts, however, tend to focus on addressing high-frequency or common problems that are applicable to primary/community care along with developing performance measures to assess that care. At the opposite end of the illness spectrum is complex care. Complex care treats diseases that require subspecialty care or illnesses that are esoteric and/or difficult to diagnose and treat. This clinical complexity makes it difficult to assess the quality of the care provided using performance measures designed for common conditions. Measures that track the outcomes of episodes of care for specific individual conditions do not generally account for the impact of comorbidities on patient outcomes. With the dynamic shift to assess not only the quality but also the cost and value of care, as demonstrated by the Centers for Medicare and Medicaid Services (CMS) Accountable Care Organization and Value Based Purchasing (VBP) programs, performance measures may need to differ across levels of care and incorporate additional risk adjustment to eliminate referral bias. Referral or selection bias represents the reality that patients with more complex problems (even with the same underlying condition) usually represent a higher proportion of the patients at referral centers than at community practices. Risk adjustment is a method to adjust payment or measurement to account for some of these differences in patient mix across providers.

In this article, we highlight the differences between population health efforts and referral care and address issues related to value measurement and performance assessment in each of these domains.

Levels of Care

Health care services can be categorized into three tiers in a pyramid structure—primary/community care, secondary care, and complex care. This pyramid is dynamic, with most of the population accessing community care for acute illnesses, wellness support, and preventive services. Secondary care is needed when acute or chronic illnesses or traumas require a specialist consultation or concentrated resources for a specific disease or body system, such as a particular treatment, procedure, or alternative therapy (e.g., primary joint replacement, general surgery, cancer treatment). This level of care includes “focused factories.”11 At the top of the pyramid is complex care. Complex care, which is generally provided at tertiary and quaternary referral centers by multiple specialists, focuses on diagnostic mystery, treatment of rare or unusual diseases, previous failed therapies, and advanced diseases with no treatment options at other centers. It follows a “solution shop” model.12 While the focus of complex care is generally highly technical and specialized in nature, there is a misconception that only patients with severe, complex, or uncommon health problems require this type of care. Patients with multiple comorbidities also may require specialized, integrated care. Although the conditions themselves may be straightforward, managing them simultaneously may require all the knowledge, skill, and resources of an integrated care team or subspecialist.13

Table 1 summarizes the definitions, goals of care, and types of providers at each level of this pyramid.

Table 1
Table 1:
Characteristics of the Three Levels of Medical Care

Defining and Measuring Complexity

While managing a complex patient generally requires greater clinician effort and entails factors other than medical decision making, no standard definition for the term “complex patient” exists. A Veterans Affairs working group defined health care complexity as requiring challenging clinical decision making and care processes that are not routine or standard.14 Others have used proxies like the number of unique diagnoses15; domains that characterize complexity (e.g., chronic conditions, functional status, health care use)16; and factors such as medical decision making, care coordination, and socioeconomic status17 to identify complex care.

Identifying performance measures that accurately capture medical complexity remains a challenge. Many current quality reporting and improvement initiatives do not adequately distinguish the differences between the delivery of community care and that of complex care; hence, risk profiles do not capture the complexity of the patient.18 In addition, standard quality measures are often developed in study populations that exclude complex patients.19 Yet, in the era of pay-for-performance, measures that account for clinical complexity are needed.

In a review, a technical panel of experts found that VBP programs generally focus on a narrow set of measures and “estimated that a small fraction (less than 20 percent) of all care that is delivered by providers is addressed by performance measures in VBP programs.”20 A 2014 analysis by Kaiser Health News found that about half of academic medical centers (143 of 292) have been financially penalized for high rates of hospital-acquired conditions. Further analysis revealed that penalties were levied against 32% of the hospitals with the sickest patients, while only 12% of the hospitals with the least complex patients received penalties. Sicker patients and more complex cases are likely to have more complications, and complex procedures performed at academic medical centers, such as organ transplants and invasive cancer surgeries, are more likely to result in adverse events.21

Many of the current measures that compare providers treating complex patients versus those not treating these patients are victims of Simpson’s paradox.22,23 Simpson’s paradox (also known as the Yule–Simpson effect) explains how overall performance can show that one provider who actually performs poorer than a second provider with both common and complex patients can appear to perform better overall because of the relative proportions of common and complex patients treated by each provider.24,25 Using an indirect standardization (or regression adjustment) approach to account for case mix, severity, and comorbidity adjustments does not actually remove the differences when directly comparing provider profiles.26

Differences in patient panels then may lead to community or single-specialty hospitals dominating quality rating lists, like the recently released Medicare Overall Hospital Quality Star Ratings. A review by Kaiser Health News indicated that 102 hospitals received the top rating of five stars; yet, few of those hospitals are considered to be the nation’s best by private ratings sources such as U.S. News & World Report or as the most elite within the medical profession. Five stars were awarded to relatively unknown hospitals and to at least 40 hospitals that specialize in just a few types of surgery, such as knee replacements.27

Examples of the Effects of Referral Bias and Complex Care on Quality Measures

If complex patients were randomly distributed across the health care centers being measured, we would expect high variability in performance measures. The overall picture, though, would still be accurate. However, if these complex patients were concentrated at a few referral centers, as they are now, they could have a great impact on the performance measures of those centers, resulting in unadjusted referral bias.

A popular performance measure among patients with diabetes is a composite measure of blood glucose (HbA1c < 8), cholesterol (LDL < 100), and blood pressure (SBP/DBP < 140/80), as well as smoking cessation advice and daily aspirin prescription. This measure appears to be a very good indication of diabetes control in a primary care population but not in patients who are referred to endocrinologists.28 Increasing complexity is associated with poorer performance on diabetes metrics.29 For example, the subset of patients requiring specialty care usually are those with advanced or difficult-to-control disease, multiple medications and comorbidities, or unique situations, such as patients requiring chronic steroids or members of the transplant population. In these cases, patient-specific goals that differ from standard goals may be in the patient’s best interest. Additional risk adjustment then is needed to capture the greater management efforts or more intense interventions needed to obtain results similar to those in primary care.

Potential referral bias is not restricted to ambulatory care measures. Among surgical patients, the focus on select complications, including infections and the Agency for Healthcare Research and Quality patient safety indicators (PSIs), has increased.30 PSI-15 (accidental puncture/laceration) is one example where referral bias could potentially affect both the perception of the quality of care provided by colorectal surgeons and their reimbursement. It currently makes up almost 50% of the PSI-90 score, which is a composite of eight complication measures. PSI-90 accounts for 35% of a hospital’s overall quality score, which could lead to penalties for hospital-acquired conditions, and for 30% of a hospital’s overall score in the VBP program.31 Patients with complex abdominal surgeries or with prior abdominal surgeries are more vulnerable to PSI-15 injuries, and these surgeries tend to be concentrated at a few referral centers. While this issue appears to be a simple definition problem, coding variability, institutional coding policies, and the lack of an opportunity to indicate prior surgery in administrative databases influence the appropriate identification of an accidental puncture/laceration and limit potential risk adjustment.

Another example of potential referral bias, affecting colon surgery, occurs in the measurement of colon surgical site infection rates, which are evaluated as part of the VBP program. The Centers for Disease Control and Prevention National Healthcare Safety Network changed its surgical hierarchy for classifying procedure types in 2013.32 Prior to the change, patients who had colon procedures and either small bowel or rectal procedures were counted in the latter two categories, which have higher expected infection rates. After the change, these patients were counted only in the publicly reported colon surgery category; however, the expected infection rates and the standardized infection ratio were still based on historical data captured using the original categories. For most institutions, there was little impact. However, for at least one referral center with an active practice in treating inflammatory bowel disease, the 10% increase in colon surgery cases with additional small bowel or rectal procedures tripled the number of observed infections, with no corresponding increase in expected infection rates, leading to unadjusted referral bias.

Performance measures for nonsurgical hospital patients are similarly affected by referral bias. In 2014, CMS began publicly reporting 30-day mortality and 30-day readmission rates for ischemic stroke patients, despite concerns that the measures were not adequately risk-adjusted. No valid measure of initial stroke severity was incorporated into these calculations, even though research indicates that initial stroke severity, as indexed by the National Institutes of Health Stroke Scale, is a dominant predictor of mortality in ischemic strokes.33

Operationalizing Value Measurement and Payment for Specialty Care

Most health care needs are relatively straightforward and involve issues that are appropriately directed to primary/community care. Specialty care is needed when the issues become more complex and exceed the ability or capacity of primary care. A patient encounter for specialty care can take three distinct forms: (a) episodic consultation (e.g., standard advice and treatment); (b) diagnosis and treatment as part of a well-defined episode (i.e., “focused factory”); or (c) diagnosis and/or treatment involving uncertainty of approach, uncertainty of time frame, or patient complexity (i.e., “solution shop”) (see Figure 1). Patients’ health care needs are dynamic; they move back and forth between the levels of care based on their medical needs. The ultimate goal of specialty care is treatment and the return of the patient to the appropriate level of care or, if necessary, continuing specialty care to enable the patient to maintain optimum health. This movement between the levels of care contributes to the challenge of obtaining accurate performance measures. As patients move between the levels of care, the expected intensity of care changes, as do the resources needed to provide care; both performance measures and reimbursement schemes need to reflect this change in intensity.

Figure 1
Figure 1:
Framework for differentiating quality measures by level of care (primary/community care, secondary care, complex care), including definitions, expectations, and reimbursement models.

Most existing quality and performance measures appropriately focus on primary care and population health; however, these metrics may not be relevant and/or adequately risk-adjusted to reflect the breadth and intensity of the complex care provided at many referral centers. This “one size fits all” approach of using the same quality and performance measures for all levels of care needs to be revamped to provide relevant information about the value of complex care provided.

Recommendations for Differentiating Between Levels of Care in Performance Measurement

While several publications have called for changes to the way quality is measured,34–38 a workable solution has yet to be identified. CMS has taken steps to incorporate value measures into payment models for hospitals, physicians, home health care, and bundled care. The announcement by the Department of Health and Human Services that they were going to increase the proportion of traditional Medicare payments tied to quality and value to 85% by 2016 and 90% by 2019, coupled with the move to the Medicare Shared Savings Program and the advent of alternative payment models like the Bundled Payments for Care Improvement initiative, increases the need to accommodate the complex care offered by referral centers in value measurement.39 To provide a level playing field for referral centers, we propose the following policy changes, operational actions, and new model development. Table 2 summarizes the implementation strategy, implementation partners, and potential barriers for each of these recommendations.

Table 2
Table 2:
Recommendations for Differentiating Performance Measures by Level of Care

Policy changes

Match performance measures to levels of care.

To be useful to patients and consumers and to adequately assess performance, different measures should be applied to different levels of care and reported in a way that compares care at like institutions. In addition, in Crossing the Quality Chasm: A New Health System for the 21st Century,3 the IOM identified six domains that are pivotal to improving health care system performance—care should be safe, effective, patient-centered, timely, efficient, and equitable. We recommend that performance measures align with the level of the care and encompass these domains. Examples of metrics that meet these criteria are listed in Table 3.

Table 3
Table 3:
Performance Measures Across the Institute of Medicine Quality Domains by Level of Care

Move away from global or composite comparisons of providers and institutions.

In many cases, provider groups and health care centers have “institutes” or other entities in which specialty resources are focused on a specific diagnostic group of conditions (e.g., a cancer center). Most health care costs are also related to a small group of high-cost, high-prevalence conditions. Focusing on these conditions and reporting performance measures for specific conditions and procedures (e.g., cancer care, elective major surgeries, congestive heart failure) will allow for the differentiation of value to emerge, will provide a more valid and reliable basis for evaluating complex care, and should be more useful for consumers. The development of clinical interinstitutional registries (e.g., cardiac surgery, neonatology, transplant) has followed this model, and the recent release of seven core measure sets by CMS and America’s Health Insurance Plans is another step in the right direction.40

Align quality/outcome measures with appropriate reimbursement models.

Each level of care has different goals and expectations and requires a matching reimbursement model (see Figure 1) with appropriate value measures. For example, bringing down a diabetic patient’s HbA1c from 11.0 to 10.0 (still elevated) may have more “value” than lowering it from 8.5 to 7.5 (the measure goal).

Operational actions

Develop operational definitions to categorize level of care.

An agreed-upon operational definition for complex care using existing data sources does not exist. In addition, although not all specialty patients are complex, complex patients are more concentrated among specialty practices than primary care practices. To address this patient distribution bias, following our framework for measuring performance (see Figure 1), standard specialty care (secondary care) could be incorporated into primary/community care payments (global cost value-based payment), and well-defined, high-risk episodes could be addressed with risk-adjusted bundled services. Care that involves a higher degree of medical uncertainty, however, will require a more flexible approach that provides incentives that support high-value providers and allows patients to access and receive needed care (outlier reimbursement model).

Identify referral centers of excellence that consistently deliver high-quality care.

To identify these centers, we must first define the scope of services they provide and performance measures for standard care, using evidence-based medicine and practices. Then, we must identify what constitutes an outlier episode of care that falls outside the standard scope of services, requiring a more complex level of care and reimbursement model. Finally, we must establish, codify, and develop performance measures (e.g., mortality rate, complication rate, patient-reported outcomes) to identify referral centers of excellence that provide complex care. Birkmeyer and colleagues41 showed that referral centers that have higher volumes of complex surgeries also have improved outcomes.

New model development: Identify the characteristics of patients who require complex care and standardize performance measures that incorporate those definitions

Referral centers must identify the characteristics that distinguish their patient populations that require complex care and thus fall outside the standard care process. Once these populations have been identified, referral centers could analyze these groups to determine commonalities that tend to trigger complex care. Metrics targeting reductions in these triggers or that include risk adjustments that account for them then could be developed. For example, a recent study of adult cardiac surgery patients demonstrated a change in the cost curve at the 75th percentile, indicating a change in the complexity of care. More complex care was required for patients in the top 25th percentile.42


While quality measurement in health care has been a catalyst for performance improvement and payment reform, current metrics focus on primary care and population health. Complex care is not well represented in these efforts. Including referral centers, which provide highly specialized complex care and episodic procedural treatments, in population-focused measurement will not provide an accurate representation of the quality of the care provided. Without a change in measurement domains to account for the increased risk in caring for more complex patients, physicians and hospitals will likely reassess their willingness to take on such complex cases for fear of damaging their reputations and reducing their reimbursements.

In this article, we provided examples that highlight the measurement problems that referral centers face when they are evaluated using primary care/population health measures as well as recommendations to address these shortcomings. Our proposed approach to quality measurement and suggested reimbursement schemes will require a shift from the current norm of using population-based quality measures to assess care at all levels. This shift is necessary to continue improving health care quality and value. Referral centers, like those in academic medicine, should take a lead role in furthering this approach.


1. History of the Joint Commission. Accessed January 18, 2017.
2. Kohn LT, Corrigan JM, Donaldson MS. To Err Is Human: Building a Safer Health System. 2000.Washington, DC: National Academies Press.
3. Institute of Medicine. Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. 2001.Washington, DC: National Academies Press.
4. Asch SM, Kerr EA, Keesey J, et al. Who is at greatest risk for receiving poor-quality health care? N Engl J Med. 2006;354:11471156.
5. National Quality Forum. MAP pre-rulemaking report: 2014 recommendations on measures for more than 20 federal programs. Published January 2014. Accessed January 18, 2017.
6. National Quality Forum. Quality Positioning System. Accessed January 18, 2017.
7. Agency for Healthcare Research and Quality. National Quality Measures Clearinghouse. Accessed January 18, 2017.
8. Agency for Healthcare Research and Quality. U.S. Department of Health & Human Services. Measures inventory. Accessed January 18, 2017.
9. Freedman DH. Why scientific studies are so often wrong: The streetlight effect. Discover. July–August 2010. Accessed January 18, 2017.
10. Agency for Healthcare Research and Quality. National quality strategy. Accessed January 18, 2017.
11. Skinner W. The focused factory. Harv Bus Rev. 1974;52:113121.
12. Christensen CM, Grossman JH, Hwang J. The Innovator’s Prescription: A Disruptive Solution for Health Care. 2009.New York, NY: McGraw-Hill.
13. Flegel K. Tertiary hospitals must provide general care. CMAJ. 2015;187:235.
14. Weiss KB. Managing complexity in chronic care: An overview of the VA state-of-the-art (SOTA) conference. J Gen Intern Med. 2007;22:374378.
15. McCrum ML, Lipsitz SR, Berry WR, Jha AK, Gawande AA. Beyond volume: Does hospital complexity matter? An analysis of inpatient surgical mortality in the United States. Med Care. 2014;52:235242.
16. Simon TD, Mahant S, Cohen E. Pediatric hospital medicine and children with medical complexity: Past, present, and future. Curr Probl Pediatr Adolesc Health Care. 2012;42:113119.
17. Grant RW, Ashburner JM, Hong CS, et al. Defining patient complexity from the primary care physician’s perspective: A cohort study. Ann Intern Med. 2011;155:797804.
18. Mechanic RE. Mandatory Medicare bundled payment—Is it ready for prime time? N Engl J Med. 2015;373:12911293.
19. Turner BJ, Cuttler L. The complexity of measuring clinical complexity. Ann Intern Med. 2011;155:851852.
20. Damberg CL, Sorbero ME, Lovejoy SL, Martsolf GR, Raaen L, Mandel D. Measuring success in health care value-based purchasing programs: Findings from an environmental scan, literature review, and expert panel discussions. Rand Health Q. 2014;4:9.
21. Rau J. Medicare cuts payments to 721 hospitals with highest rates of infections, injuries. Kaiser Health News. December 18, 2014. Accessed January 19, 2017.
22. Simpson EH. The interpretation of interaction in contingency tables. J R Stat Soc B Methodol. 1951;13:238241.
23. Yule GU. On some points relating to vital statistics, more especially statistics of occupational mortality. J R Statist Soc. 1934;97:184.
24. Marang-van de Mheen PJ, Shojania KG. Simpson’s paradox: How performance measurement can fail even with perfect risk adjustment. BMJ Qual Saf. 2014;23:701705.
25. Manktelow BN, Evans TA, Draper ES. Differences in case-mix can influence the comparison of standardised mortality ratios even with optimal risk adjustment: An analysis of data from paediatric intensive care. BMJ Qual Saf. 2014;23:782788.
26. Shahian DM, Normand SL. Comparison of “risk-adjusted” hospital outcomes. Circulation. 2008;117:19551963.
27. Rau J. Many well-known hospitals fail to score 5 stars in Medicare’s new ratings. Kaiser Health News. July 27, 2016. Accessed January 19, 2017.
28. Nyman MA, Cabanela RL, Liesinger JT, Santrach PJ, Naessens JM. Inclusion of short-term care patients affects the perceived performance of specialists: A retrospective cohort study. BMC Health Serv Res. 2015;15:99.
29. Meduru P, Helmer D, Rajan M, Tseng CL, Pogach L, Sambamoorthi U. Chronic illness with complexity: Implications for performance measurement of optimal glycemic control. J Gen Intern Med. 2007;22(suppl 3):408418.
30. Agency for Healthcare Research and Quality. Patient safety indicators overview. Accessed January 19, 2017.
31. Rajaram R, Barnard C, Bilimoria KY. Concerns about using the patient safety indicator-90 composite in pay-for-performance programs. JAMA. 2015;313:897898.
32. Centers for Disease Control and Prevention. National Healthcare Safety Network (NHSN). Accessed January 19, 2017.
33. Fonarow GC, Alberts MJ, Broderick JP, et al. Stroke outcomes measures must be appropriately risk adjusted to ensure quality care of patients: A presidential advisory from the American Heart Association/American Stroke Association. Stroke. 2014;45:15891601.
34. Berenson RA, Pronovost PJ, Krumholz HM. Achieving the Potential of Health Care Performance Measures. 2013.Princeton, NJ: Robert Wood Johnson Foundation.
35. Chassin MR, Loeb JM, Schmaltz SP, Wachter RM. Accountability measures—Using measurement to promote quality improvement. N Engl J Med. 2010;363:683688.
36. Conway PH, Mostashari F, Clancy C. The future of quality measurement for improvement and accountability. JAMA. 2013;309:22152216.
37. Panzer RJ, Gitomer RS, Greene WH, Webster PR, Landry KR, Riccobono CA. Increasing demands for quality measurement. JAMA. 2013;310:19711980.
38. Pronovost PJ, Miller M, Wachter RM. The GAAP in quality measurement and reporting. JAMA. 2007;298:18001802.
39. Anderson GF, Davis K, Guterman S. Medicare payment reform: Aligning incentives for better care. Issue Brief (Commonw Fund). 2015;20:112.
40. Centers for Medicare and Medicaid Services. Quality measures. Core measures. Accessed January 19, 2017.
41. Birkmeyer JD, Siewers AE, Finlayson EV, et al. Hospital volume and surgical mortality in the United States. N Engl J Med. 2002;346:11281137.
42. Cook D, Thompson JE, Habermann EB, et al. From “solution shop” model to “focused factory” in hospital surgery: Increasing care value and predictability. Health Aff (Millwood). 2014;33:746755.
Copyright © 2017 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Association of American Medical Colleges.