Quality is never an accident, it is always the result of intelligent effort.
John Ruskin, 1819-19001
In 2000, the Institute of Medicine published a landmark document, Crossing the Quality Chasm, which launched a national focus on improving quality and safety for patients.2 In 2010, the US Congress passed the Affordable Care Act, which mandated that Medicare begin to incorporate quality scores into payments.3 Accordingly, Medicare currently incorporates factors such as hospital readmission rates, preventable hospital-acquired condition rates, and overall patient experience measures in public reports about quality of care and in payment adjustments to hospitals.4 , 5 It is now expected that by 2018, more than 50% of Medicare payments will be disbursed through alternative payment models that incorporate measures of quality.6 In addition, performance measures (PMs) are increasingly being incorporated into qualified clinical data registries to help programs, hospitals, and hospital systems measure, track, benchmark, and improve processes in order to improve meaningful patient outcomes and to report quality data to Medicare. For example, the American Heart Association and the American College of Cardiology registries include PMs related to referral to cardiac rehabilitation (CR),7 , 8 as well as other process and outcomes measures linked to class IA and IB recommendations in clinical practice guidelines.
In response to these trends, in fall of 2014, the American Association of Cardiovascular and Pulmonary Rehabilitation (AACVPR) board of directors charged the AACVPR's Quality of Care Committee (QCC) to create uniform standards to evaluate program quality. This committee is composed of a diverse group of AACVPR members that includes physicians, nurses, exercise physiologists, psychologists, dietitians, respiratory therapists, and physical therapists. Together, the QCC worked on a multiyear initiative to develop and validate new PMs for use in program certification, with a focus on developing outcomes measures that are linked to meaningful patient outcomes that can be influenced by cardiac or pulmonary rehabilitation. The ultimate goal of this project was to provide the tools and standards that programs would need to ensure high-quality clinical care and ultimately improve patient outcomes. As pulmonary rehabilitation (PR) already had 2 validated PMs that were endorsed by the National Quality Forum,9 , 10 this effort primarily focused on CR PMs but also included developing 1 additional PM for PR programs.
In the fall of 2014, the QCC began a deliberate and systematic approach to choosing, writing, and validating new PMs. The QCC followed a distinct 3-step process that included a modified Delphi approach that included structured, purposeful, and repeated cycles of QCC and AACVPR membership involvement to ensure stake holder engagement.11 First, we surveyed national AACVPR leadership and AACVPR members working as CR medical directors and program directors to identify and rank various potential PM topics. Next, we assessed the face validity of the drafted PMs in a second national survey. Finally, the inter- and intrarater reliability of each PM was assessed by abstracting patient charts at programs throughout the United States. As a quality improvement effort, institutional review board approval was not required for the first 2 steps because individual patient data were not included, and all surveys were anonymous. For the third step, institutional review board permission at both Helen Hayes Hospital in Haverstraw, New York (project planning and implementation), and at Baystate Medical Center in Springfield, Massachusetts (data collection and analysis), was obtained. In both cases, the project was determined to be exempt because it did not meet the federal definition of human subjects' research.
In an effort to identify key areas for PM development, the QCC utilized the core components of both cardiac and pulmonary rehabilitation12 , 13 in addition to several other potential topic areas. A survey regarding these topic areas was then sent to all current program directors and medical directors, as well as current and former AACVPR leaders, fellows, board members, and state affiliate board members. We surveyed professionals in both cardiac and pulmonary rehabilitation but they responded only to questions directly related to their area(s) of focus and specialization. The survey asked respondents to rate each of the potential topics on 8 characteristics (Table 1), utilizing a Likert scale ranked from 1 to 4, using the following response scale: strongly disagree, disagree, agree, and strongly agree. A free-text comment was also included for each topic area. The survey was distributed in December 2014 and used www.surveymonkey.com as the electronic platform. Participation was voluntary, no incentive was provided, and only 1 survey request was sent to each potential respondent. Participants also answered several basic questions about themselves and their roles, experience, and responsibilities. To avoid mixing responses and increase transparency, the members of the QCC also completed the survey and these results were recorded separately.
After selection of the highest priority PM topics, 1 member of the QCC drafted each PM with input from other members of the QCC and an ACCVPR experts group, as needed. Initial drafts were reviewed by the QCC and were modified for clarity and content. To assess face validity of the proposed measures, a second survey was then sent to all current AACVPR board members, AACVPR committee members, and state affiliate presidents to establish face validity of the PMs using similar methods as the first survey, except that Likert scale questions were now ranked from 1 to 5 with an additional middle ranking of 3 (neither agree nor disagree). Participants were asked to rate each PM according to both the 8 ideal characteristics of a PM (Table 1), as well as clarity of the PM language and definitions. Each PM had a free-text section for comments. Results of this survey were discussed by the QCC and needed adaptations in language and definitions were made by the PM authors.
In the third step of this project, we assessed the feasibility and reliability of the PMs using structured chart abstraction, as described previously.14 We first solicited volunteer programs from the AACVPR program directory, each of which participated in the project to the extent feasible. However, because of different populations and resources available, only a few programs were able to abstract every PM. Each participating program then identified 3 staff members to participate in the abstraction process for the PMs they volunteered for. One staff member (site coordinator) identified 35 patient charts through purposeful sampling where about 30 of the charts qualified for the measure denominator and 5 of the charts did not, which were then placed in random order.
Next, the site coordinator then provided this list of 35 patients to the 2 other staff members, who reviewed each patient chart to see whether the patient met the PM. First, they reviewed the denominator (does the patient have the condition?), then the exclusion (is there an exclusion present?), and finally whether the patient met the numerator (was the PM met?). They recorded the time needed to complete the chart abstraction and any feasibility issues encountered. All answers were entered into REDCap, a secure online research application that allows multiple users to enter data simultaneously from different locations.15 At no point were names or other identifying information entered into REDCap. Then, 1 to 2 wk later, the 2 abstractors again reviewed the same 35 patients following the same procedures as previously noted, so that each individual chart was reviewed 4 times for each PM.
For the prioritization survey and the face validity survey, we averaged the 8 Likert-scale scores of each domain (Table 1) into a single score for each PM. As an alternate measure of priority, we then ranked each PM within each of the 8 domains from highest score to lowest score. Performance measures were given 1 point for each individual domain score within the top 5. These points were then summed across domains with a summary score range between 0 and 8. In both surveys, comments were summarized, discussed, and needed changes were made to the PMs.
For the reliability assessment phase, all 4 chart abstractions were assembled on the same patient chart for each PM to facilitate calculation of inter and intrarater reliability. Time required for chart abstraction was averaged across each PM for both the first and second chart abstraction. We used Cohen κ statistic to assess agreement at each step of the PM, including agreement at the denominator, exclusions, and numerator levels. We calculated agreement as a combined group. We considered a κ value of ≥0.8 to be excellent agreement, 0.61 to 0.79 to be good agreement, 0.41 to 0.60 to be fair agreement, and ≤0.4 to be poor/unacceptable agreement. All calculations were performed on JMP version 12.0.1 (SAS Institute) and Microsoft Excel (2010).
The prioritization survey for both CR and PR was sent to 1562 professionals, of which 302 (19.3%) completed the survey. Of these 302 completers, 214 (70.9%) and 88 (29.1%) were CR and PR professionals, respectively. The majority of participants were program directors with master's degrees in either nursing or exercise physiology, who had more than 15 y of experience (Table 2).
For CR, of the 14 potential PMs, blood pressure control, improvement in functional capacity, diabetes self-management, psychosocial risk, and abstinence from tobacco use had the highest number of domains ranked in the top 5 and generally had some of the highest overall summary scores (see Supplemental Digital Content 1 Appendix, Table 1, available at: http://links.lww.com/JCRP/A82). Notably, question number 8, which assessed the ability of the PM to distinguish between a high- and low-quality program, consistently scored the lowest of all domains across all PMs and was therefore reported separately (see Supplemental Digital Content 1 Appendix, Tables 1 and 2, available at: http://links.lww.com/JCRP/A82). Quality of Care Committee scores generally agreed with the primary survey, differing in ranking appropriate utilization of CR as a top priority. After discussion and incorporation of survey comments, the QCC decided not to pursue diabetes self-management or appropriate utilization of CR because both were felt to be out of the locus of control of an individual CR program and would require significant physician and hospital coordination. It was decided to make tobacco cessation a process measure rather than an outcome measure, because of the consistently reported feeling that tobacco cessation was difficult to accurately measure and modify in CR. Among the potential topics in the psychosocial risk category, the QCC decided to pursue depression as the first PM, given that it is an important comorbidity,16–18 is treatable in CR,19–22 and is strongly proven to be associated with mortality risks.23–25
Of the 12 topics in the PR survey, improvement in sensation of dyspnea, improvement in functional capacity, improvement in health-related quality of life, and appropriate oxygen utilization had the highest mean scores and number of domains ranking in the top 5 (see Supplemental Digital Content 1 Appendix, Table 2, available at: http://links.lww.com/JCRP/A82). These rankings validated the importance of the 2 previously developed PMs (improvement in health-related quality of life and improvement in functional capacity). After incorporation of comments, the QCC decided to further develop the improvement in sensation of dyspnea PM, because improvement in dyspnea is the primary patient symptom addressed by PR. Although appropriate oxygen utilization scored strongly as a topic for measure consideration, the QCC felt that there was not yet sufficient peer-reviewed evidence to require uniform testing, prescription, and evaluation of oxygen adherence across all PR programs, so further development on this topic was suspended.
After drafting the initial PMs, we then sent the face validity survey to 144 CR and PR leadership individuals, with 82 (56.9%) completing the survey. Participants' characteristics were generally similar to the prioritization survey but this time represented a mixture of both CR and PR professionals (Table 2). Survey results are found in Table 3 and show that, nearly uniformly, the PMs scored between 4 and 5, indicating that participants agreed or strongly agreed on nearly all 8 domains as well as across PM language and definitions. Based upon comments submitted, additional minor modifications in language and definitions were made to each PM.
For feasibility and reliability testing, we recruited 16 sites (both CR and PR) across the United States to participate in abstracting up to 4 PMs at each site, with 12 sites completing at least 1 PM abstraction. The number of sites for each PM is shown in Table 4, as well as the number of charts reviewed for each PM at each step of validation. The number of patients who met both the denominator and exclusion definitions for the tobacco cessation measure was small (52 patients) whereas 286 and 253 patients were evaluated for the blood pressure and functional capacity PMs numerator definitions, respectively. In general, both intrarater and interrater agreement was excellent (κ≥ 0.8) or good (κ = 0.61-0.79) for most of the PMs. Abstractors struggled with operationalizing the exclusion definitions for both the depression and dyspnea PMs but otherwise had excellent intrarater and interrater reliability results. However, the tobacco cessation PM numerator showed low inter- and intrarater reliability and had the slowest abstractions times. After discussion with the QCC and chart abstractors, the QCC decided that the depression and dyspnea measure inconsistencies could be addressed with better PM definitions, but that the tobacco cessation PM should be simplified and retested with a larger eligible population in the future. Current PMs (including the 2 previously validated pulmonary PMs), with their data definitions, algorithm, and frequently asked questions, can be found in the online Appendix (see Supplemental Digital Content 1, available at: http://links.lww.com/JCRP/A82) or on the AACVPR website.26
Using a methodical approach for identification and validation of new PMs, we introduced 5 new PMs for use among both CR and PR programs across the United States. These were identified through significant engagement with stakeholders including AACVPR membership, AACVPR national and regional leaders, and the QCC. Development was an iterative process with modifications made at each step to improve and clarify each PM on the basis of the feedback received. Overall, results suggest that these PMs should have strong AACVPR member buy-in, are feasible to implement, and, when applied to programs across the United States, should perform adequately to distinguish high- and low-quality programs. Such information will be valuable to payers, referring clinicians, and to the programs themselves as they will give them objective feedback on their performance and should stimulate changes in clinical practice and improvements in quality of care.
The first PM within cardiopulmonary rehabilitation was originally developed in 2007 and focused on CR referral.27 This PM was validated and endorsed by the National Quality Forum, updated in 2010, and is currently undergoing an additional update.14 , 28 Similarly, the 2 PR measures were initially endorsed by the National Quality Forum in January 2011.9 , 10 Now, with the addition of the PM for dyspnea in PR as described in this article, PMs for the top 3 major outcomes of PR that have strong scientific evidence are developed and validated. In addition, although structure and process measures for CR programs were initially described in 2007,27 this is the first time any PM for CR programs has focused primarily on outcomes and been tested and validated. In the future, we anticipate that additional PMs will be developed and existing PMs will be updated and modified as new information becomes available. We also anticipate assessing how effective these PMs are in improving the actual quality of CR and PR services.
One of the major impacts of these PMs will be their use in AACVPR program certification and data registries29 , 30 to assess CR and PR program quality. Beginning with the 2018 program certification cycle, programs will be required to submit data on these new PMs as part of the application for AACVPR certification. However, currently, no standards have been set at which a program would be certified or otherwise denied certification, although some data regarding expected mean exercise workloads in CR have recently been published.31 Consequently, 2 of the major goals for this first round of program certification are to establish baseline PM data and to ensure that these PMs are incorporated into the individual treatment plans of programs across the United States. Then, when such data are created, collected, and submitted, clear standards for program quality can be set and justified scientifically. To this end, both the cardiac and pulmonary registries are in the process of being modified to capture all the necessary elements to evaluate program performance on these PMs.
Ultimately, the goal of these PMs is to ensure that all CR and PR programs in the United States are meeting or exceeding high-quality standards for patient care and that certification through the AACVPR is a marker of true program quality. Over time, we anticipate that these PMs will be used in quality improvement projects and will help standardize and improve care.
These data should be viewed in light of several limitations. First, there was a low response rate to the first prioritization survey, which suggests that these results may not be fully generalizable because they represent a select group of respondents. However, we still had 302 participants and survey responses were received from clinicians all over the United States. Second, during the validation, only a limited number of programs participated and chart abstractions varied substantially between PMs. This heterogeneity could mean that PMs might perform differently when used in settings other than the ones in which we tested them and have limited generalizability. This limitation may be overcome by future validation studies or as data about these PMs become available in AACVPR registries. Third, although the tobacco cessation PM had low reliability, this measure has now been simplified and is currently undergoing additional testing. This initial low performance is not a critical problem because, just like every other PM introduced here, it will not be immediately used to guide certification or reimbursement. This will allow the tobacco cessation measure to be retested before it is tied to any certification or financial outcomes.
We identified, drafted, modified, and validated 5 new PMs for use in cardiac and pulmonary rehabilitation programs across the United States. We anticipate that these PMs will be the beginning of a new focus on improving quality and consistency in cardiopulmonary rehabilitation care. Modifications will continue and new PMs will be validated. Ultimately, we believe that these PMs will help programs provide high-quality care and ultimately may be useful in future value-based purchasing agreements in the ever-changing landscape of US health care.
The authors acknowledge the members of the AACVPR Quality of Care Committee for their valuable contributions and insight while developing these performance measures. Members or past members of the QCC (in addition to all authors) included Ellen Aberegg, LD, MA, RD; Alison L. Bailey, MD; Bob Brown, MPH, MBA; Todd M. Brown, MD, MSPH; Dianne V. Jewell, PT, PhD; Steven Keteyian, PhD; Karen R. Lui, BSN, MS; Michelle M. Milic, MD; Ray Squires, PhD; Mark Stout, MS; Randal Thomas, MD, MS; and Mollie Corbett (AACVPR staff). The authors thank the programs that performed the feasibility and reliability studies. These programs include members at Beaumont Health Center, Unity Cardiopulmonary Rehabilitation, John Muir Health, Genesis Medical Center, The Heart Hospital Baylor Plano, Saint Patrick Hospital, Harrington Heart & Vascular Institute, The Miriam Hospital, UC Davis Medical Center, Mayo Medical Center, Vernon Memorial Medical Center, and Mount Carmel Cardiac Rehabilitation.
2. Institute of Medicine. Crossing the Quality Chasm: A New Health System of the 21st Century. Washington, DC: The National Academies Press; 2001.
3. Patient Protection and Affordable Care Act. Washington, DC: US Government Printing Office; 2010. Pub L No. 111-148.
5. Atreya AR, Pack QR. Care transitions measure score and coronary revascularization related readmission: ready for primetime use? J Gen Intern Med. 2016;31(7):707–709.
6. Burwell SM. Setting value-based payment goals—HHS efforts to improve U.S. health care. N Engl J Med. 2015;372(10):897–899.
7. Ellrodt AG, Fonarow GC, Schwamm LH, et al Synthesizing lessons learned from get with the guidelines: the value of disease-based registries in improving quality and outcomes. Circulation. 2013;128(22):2447–2460.
8. Brown TM, Hernandez AF, Bittner V, et al Predictors of cardiac rehabilitation referral in coronary artery disease patients: findings from the American Heart Association's Get With The Guidelines Program. J Am Coll Cardiol. 2009;54(6):515–521.
9. National Quality Forum. 700: Health-Related Quality of Life Before and After Pulmonary Rehabilitation. Washington, DC: National Quality Forum. http://www.qualityforum.org/QPS/0700
. Accessed December 29, 2016.
10. National Quality Forum. 701: Functional Capacity Before and After Pulmonary Rehabilitation. Washington, DC: National Quality Forum. http://www.qualityforum.org/QPS/0701
. Accessed December 29, 2016.
11. Fitch K, Bernstein SJ, Aguilar MS, Burnand B. The RAND/UCLA Appropriateness Methods User's Manual. Santa Monica, CA: RAND; 2001.
12. Ries AL, Bauldoff GS, Carlin BW, et al Pulmonary rehabilitation: joint ACCP/AACVPR evidence-based clinical practice guideline. Chest. 2007;131(5 suppl):4S–42S.
13. Balady GJ, Williams MA, Ades PA, et al Core components of cardiac rehabilitation/secondary prevention programs: 2007 update: a scientific statement from the American Heart Association Exercise, Cardiac Rehabilitation, and Prevention Committee, the Council on Clinical Cardiology; the Councils on Cardiovascular Nursing, Epidemiology and Prevention, and Nutrition, Physical Activity, and Metabolism; and the American Association of Cardiovascular and Pulmonary Rehabilitation. J Cardiopulm Rehabil Prev. 2007;27(3):121–129.
14. Thomas RJ, Chiu JS, Goff DC Jr, et al Reliability
of abstracting performance measures: results of the cardiac rehabilitation referral and reliability
(CR3) project. J Cardiopulm Rehabil Prev. 2014;34(3):172–179.
16. Lichtman JH, Froelicher ES, Blumenthal JA, et al Depression as a risk factor for poor prognosis among patients with acute coronary syndrome: systematic review and recommendations: a scientific statement from the American Heart Association. Circulation. 2014;129(12):1350–1369.
17. Rozanski A, Blumenthal JA, Kaplan J. Impact of psychological factors on the pathogenesis of cardiovascular disease and implications for therapy. Circulation. 1999;99(16):2192–2217.
18. Frasure-Smith N, Lesperance F. Depression and cardiac risk: present status and future directions. Heart. 2010;96(3):173–176.
19. Milani RV, Lavie CJ. Impact of cardiac rehabilitation on depression and its associated mortality. Am J Med. 2007;120(9):799–806.
20. Vizza J, Neatrour DM, Felton PM, Ellsworth DL. Improvement in psychosocial functioning during an intensive cardiovascular lifestyle modification program. J Cardiopulm Rehabil Prev. 2007;27(6):376–383; quiz 384-375.
21. Josephson EA, Casey EC, Waechter D, Rosneck J, Hughes JW. Gender and depression symptoms in cardiac rehabilitation: women initially exhibit higher depression scores but experience more improvement. J Cardiopulm Rehabil. 2006;26(3):160–163.
22. Lavie CJ, Milani RV. Effects of cardiac rehabilitation and exercise training on exercise capacity, coronary risk factors, behavioral characteristics, and quality of life in women. Am J Cardiol. 1995;75(5):340–343.
23. Frasure-Smith N, Lesperance F, Talajic M. Depression following myocardial infarction. Impact on 6-month survival. JAMA. 1993;270(15):1819–1825.
24. Barth J, Schumacher M, Herrmann-Lingen C. Depression as a risk factor for mortality in patients with coronary heart disease: a meta-analysis. Psychosom Med. 2004;66(6):802–813.
25. Bush DE, Ziegelstein RC, Tayback M, et al Even minimal symptoms of depression increase mortality risk after acute myocardial infarction. Am J Cardiol. 2001;88(4):337–341.
26. American Association of Cardiovascular and Pulmonary Rehabilitation. Performance measure
resource page. http://www.aacvpr.org/PMresources
. Accessed November 21, 2016.
27. Thomas RJ, King M, Lui K, Oldridge N, Pina IL, Spertus J. AACVPR/ACC/AHA 2007 performance measures on cardiac rehabilitation for referral to and delivery of cardiac rehabilitation/secondary prevention services. Circulation. 2007;116(14):1611–1642.
28. Thomas RJ, King M, Lui K, Oldridge N, Pina IL, Spertus J. AACVPR/ACCF/AHA 2010 update: performance measures on cardiac rehabilitation for referral to cardiac rehabilitation/secondary prevention services: a report of the American Association of Cardiovascular and Pulmonary Rehabilitation and the American College of Cardiology Foundation/American Heart Association Task Force on Performance Measures (Writing Committee to Develop Clinical Performance Measures for Cardiac Rehabilitation). Circulation. 2010;122(13):1342–1350.
29. American Association for Cardiovascular and Pulmonary Rehabilitation. Outpatient data registries. https://www.aacvpr.org/Registry
. Accessed December 29, 2016.
31. Keteyian SJ, Kerrigan DJ, Ehrman JK, Brawner CA. Exercise training workloads upon exit from cardiac rehabilitation in men and women: the Henry Ford Hospital experience. J Cardiopulm Rehabil Prev. 2017;37(4):257–261.