Brokering Compromise to Achieve Consensus

The current transformation of the health care system comes with the increased use of performance measures for various applications, including value-based payment. Although there may be different perspectives on the appropriate approaches to shift from payment for individual services to payment for value, it is unlikely that the pace of change will slow. Given this environment, it is more important than ever that performance measures used in these applications represent the best combination of measurement science, achieving compromise across various stakeholders’ interests and adhering to a process to reach consensus that is flexible and responsive to the needs of the health care system.

The National Quality Forum (NQF) was formed in 1999 in response to a recommendation from the Presidential Advisory Commission on Consumer Protection and Quality in the Health Care Industry. The Commission called for the development of a public-private partnership that would help standardize measures across the health care system. Before the start of NQF, few performance measures for health care quality existed, and the role of consumers and purchasers in quality measurement was limited. A central tenet of the NQF structure and history is the importance placed on ensuring equal weight for the consumer and purchaser voice in the process of both determining priorities for the health system and endorsing performance measures that will drive overall health system improvement. In an environment that is moving toward payment for value, it is even more essential that there is balance among stakeholders making endorsement decisions, because these decisions have the potential to influence provider payment, health plan network design, consumer choice, and other important applications. For the thousands who have participated in NQF endorsement work, the interplay between stakeholders—especially patients and clinicians—can be a remarkable experience. As measurement science experts, we are often humbled by the power of the patient voice at our tables.

The measurement field is evolving rapidly, and the use of measures for payment introduces additional complexity in the process of building consensus and compromise among divergent interests. The rigor of the NQF endorsement process is intended to assure that only evidence-based, valid, and reliable measures that should be useful for improvement receive endorsement. Over the years, the NQF has been simultaneously criticized for being overly rigorous and rejecting measures as well as not being rigorous enough when reviewing candidate performance measures that have the potential to affect provider payment. In an article published in the Clinical Journal of the American Society of Nephrology, Fishbane and Wish (¹) offer a fairly critical view of the NQF endorsement process for a dialysis readmission measure that will be used in a dialysis payment program. Fishbane and Wish (¹) also raise several issues with the technical expert panel convened by the Centers for Medicare and Medicaid Services (CMS), which are outside of the scope of the NQF consensus process. We will not offer a point-counterpoint on the specific issues raised. Instead, we hope to offer some insights on the path toward consensus, the results of the intensive consensus–building process that led to the final compromise that was not included in the article by Fishbane and Wish (¹), and opportunities for improvement that the evaluation of the dialysis readmissions measure raised.

The Path toward Consensus

The term consensus is defined as “general agreement, but not necessarily unanimity, and includes a process for attempting to resolve objections by interested parties, as long as all comments have been fairly considered, each objector is advised of the disposition of his or her objection(s) and the reasons why, and the consensus body members are given an opportunity to change their votes after reviewing the comments” (²).

The article by Fishbane and Wish (¹) raises many important points but also, omits some of the specific aspects of the NQF process that led to endorsement of the modified dialysis readmission measure. As a consensus–based standard setting organization, the NQF’s processes are intended to reflect the five key attributes—openness, balance of interest, due process, appeals, and consensus. The NQF uses deliberative consensus process to address these key attributes of a consensus-based organization. At the first stages of the NQF consensus development process, measures are submitted by measure developers for review and assigned to a topical standing committee (e.g., Renal Standing Committee and Admissions and Readmissions Standing Committee). Our volunteer standing committees have diverse representation, including measurement experts, providers, physicians, health plans, patients, and purchasers. The standing committees evaluate submitted measures by using four evaluation criteria: importance to measure and report (including evidence for the measure focus), scientific acceptability of the measure properties (including reliability and validity), usability and use, and feasibility. In the standing committee review of this dialysis readmissions measure, the measure received moderate to high ratings for the dialysis readmission measure for importance to measure and report and scientific acceptability of the measure properties—the two criteria at the top of the evaluation hierarchy. However, concerns regarding the attribution methodology (i.e., the entity that will be measured) dominated the standing committee’s discussion, and the committee did not reach consensus on the measure.

To ensure broad stakeholder input into the deliberations, measures are posted for public comment, and the standing committee adjudicates responses. The NQF members are also invited to provide input by voting on the measures; however, often, the NQF members reserve their comments and votes for the issues that are most directly related to their interests. In the review of this measure, the portion of the membership that submitted comments and votes was heavily representative of dialysis providers, and the input across stakeholder groups was limited. Concerns were raised about attribution of readmissions to dialysis facilities within the first 3 days posthospital discharge, adequacy of clinical variables included for risk adjustment, the lack of timely access to data to drive improvement and intended use of the measure, and potential unintended consequences of the measure on the basis of the current measure specifications.

The Consensus Standards Approval Committee (CSAC) is charged by the NQF Board of Directors to review these multiple inputs, including the report and recommendations from the topical standing committee, the comments received during the comment period, and the votes received. The CSAC along with the NQF Board of Directors has a consumer and purchaser membership majority defined by the NQF’s bylaws to ensure that their voice is heard in measure endorsement decisions. The CSAC weighed all of the information and feedback around the measure, because there was no clear consensus from the Standing Committee. In particular, the CSAC considered the supportive Standing Committee votes on scientific acceptability, noted the importance of shared accountability for readmissions across multiple settings of care, and ultimately, recommended the measure for endorsement.

Notably, the CSAC deliberations did not focus in any way on the approval of the measure for any specific CMS payment program, as Fishbane and Wish (¹) suggest, because it is outside of the scope of the CSAC. It is important to note that the NQF–convened Measure Application Partnership (MAP), an expert multistakeholder group, separate from the endorsement process and charged with reviewing measures for specific CMS programs had previously evaluated the measure during the 2012–2013 prerulemaking for the ESRD Quality Improvement Program. During its review, the MAP supported the direction of the dialysis readmission measure for the program and noted that additional development and risk adjustment were needed. The MAP has repeatedly called for shared accountability to improve readmissions and the need to expand readmission measurement beyond hospitals to other facilities. However, we recognize that many clinicians and providers do not support the concept of accountability for care outside their direct control, such as treatment provided outside of the facility or practice. As the Institute of Medicine has noted, there are significant challenges to this “explicit departure” from prior approaches to accountability (³). In response to these challenges, the NQF recently launched new work to develop approaches for attribution of care processes that require shared accountability.

As a final step, endorsed measures are open for a final 30-day appeals period. During this appeals period, the Renal Physicians Association and others appealed the measure, largely on procedural grounds. The CSAC reviewed the appeals and the consensus process that was used in the endorsement decision. The CSAC noted that the process followed in the review and endorsement of this measure was consistent with the approved process for measures in which consensus of the Standing Committee and the membership is not reached. The CSAC acknowledged the appellant’s concerns but remained largely supportive of the endorsement of the measure. The Executive Committee of the NQF Board reviewed the appeal and the CSAC deliberations in March of 2015 and asked the appellants and the developer to undergo an intensive consensus–building process to identify the key areas of concern and a path forward to reaching a compromise before making a final decision on the appeal.

There was no stunning reversal, as Fishbane and Wish (¹) suggest, but rather, an intensive consensus–building process initiated by the Executive Committee of the Board that required significant investment by the NQF staff, the appellant, and the CMS along with their contractors. This intensive consensus–building effort was undertaken from March to June of 2015 and focused on building trust among the parties, identifying how the measure specifications can evolve to address the methodologic concerns raised by the renal community, and balancing the needs of other stakeholders to advance readmissions measurement for this important patient population.

In May of 2015, the NQF convened the appellants, the developer, and the CMS to discuss a range of issues, and together, the parties agreed on several immediate and future changes to the measure specifications as well as additional support for providers as the measure is implemented to allow them to engage in more effective quality improvement. Concerns were raised about the inclusion of readmissions within a few days after discharge when patients may not yet have been seen in the dialysis facility. After extensive consensus–building discussions led by the NQF, the following compromises were agreed on by the appellants and the developer and reviewed by the Executive Committee of the NQF Board in June of 2015.

• The CMS will exclude from the measure numerator and denominator all index discharges resulting in readmissions occurring within the first 3 days after discharge from an acute care hospital. The CMS also will develop a plan for monitoring the effect of excluding those 16% of readmissions occurring within 3 days of hospital discharge.
• Within 1 year, the CMS will work with the appellants to identify and incorporate, if appropriate, additional risk adjustment factors.
• Within 1 year, the CMS will identify a mechanism, if possible, by which to provide facilities with more updated information on their crude raw readmission rates.

The appellants and the CMS along with their measure developer contractor worked together to address many of the major concerns raised during the evaluation process and provide a clear pathway for additional work to be completed within 1 year. The results of this intensive effort were not noted by Fishbane and Wish (¹), occurring after the article was submitted.

Opportunities for Improvement

The NQF strives to improve the quality and performance of the health care system. We believe that these objectives apply to our processes as well. The NQF has a responsibility to continuously improve and be responsive to the changing needs of the health care system. At the end of the day, there are no perfect measures and no perfect processes. We take our role and the public’s trust in our processes very seriously.

The work of consensus is not an exact science and likely, never will be; the NQF is committed to continuously improving the consensus process and its measure evaluation criteria by seeking feedback and responding to stakeholder needs. Because of these commitments, many changes to various processes and procedures within the consensus development process have been implemented over the years.

In response to concerns regarding the consensus process that led to endorsement of the Hospital-Wide All Cause Readmission measures in 2012, the NQF Board approved a task force that reviewed and recommended enhancements for defining and achieving consensus. The NQF has since enacted many of their recommendations, including establishing a process for measures that fail to reach consensus. Many of these recommended processes were implemented, but the experience raised additional issues and opportunities for learning.

The measure evaluation for this dialysis readmission measure offers many important opportunities for learning. First, trust and compromise are critical elements to achieving consensus. Although our intensive consensus–building efforts led to a reasonable compromise accepted by all parties, we should intervene earlier in the process when there are deeply divided stakeholder groups to find opportunities for dialogue, trust building, and compromise. The NQF will work with our Renal Standing Committee as well as a new Renal Member Network to reach across stakeholder groups in an open and transparent way to forge consensus and build trust. As the stakes are raised on measurement, the NQF needs to bring groups with differing perspectives together, mediate discussions, and work toward a reasonable path forward.

Second, we need to recognize the importance of different stakeholder views. Often, complex scientific questions, such as attribution as raised by Fishbane and Wish (¹), and others, such as appropriate risk adjustment methods, are deeply rooted in the values of the stakeholder. To make meaningful progress on these important quality issues, we must take the time to more fully engage with all stakeholders in our work to understand various perspectives and interests. Attribution is especially difficult for measures like readmissions, where no one provider can be held solely accountable for improvement but in which many providers have a role.

The measurement community needs to develop feedback loops to track for both the intended and unintended consequences of measurement to better understand real and perceived concerns about measures as they are implemented. Although we acknowledge the role of potential unintended consequences in our evaluation criteria, the lack of a consistent way to garner feedback from the field is a major blind spot for all of us. We would very much like to partner with the renal community to better understand and track the effect of the readmission measures and other renal measures on patients and providers.

Finally, the most important lesson for all of us is humility. We can only do better when we learn from each other and work collaboratively to find better solutions to issues in our evolving health care system.

Disclosures

None.