Thomas, Randal J. MD, MS; Chiu, Jensen S. MHA; Goff, David C. Jr MD, PhD; King, Marjorie MD; Lahr, Brian MS; Lichtman, Steven W. EdD; Lui, Karen RN, MS; Pack, Quinn R. MD; Shahriary, Melanie BSN, RN
Cardiac rehabilitation (CR) and secondary prevention (SP) services are significantly associated with positive health outcomes in patients with cardiac disorders,1–7 yet only a minority of eligible patients ever participate in CR/SP.8–10 The American Association of Cardiovascular and Pulmonary Rehabilitation (AACVPR), the American College of Cardiology Foundation (ACCF), and the American Heart Association (AHA)11 have developed, and the National Quality Forum has endorsed, performance measures (PMs) for CR/SP referral to increase the delivery of CR/SP to appropriate patients (see Table 1).12–17 In addition, the Centers for Medicare & Medicaid Services has included these measures in the Physician Quality Reporting System and will begin reporting audits of these PMs in the outpatient setting in 2015.
Assessment of the reliability of data collection for performance measurement is an important step included in the ACCF/AHA methodology for the development and identification of high-value PMs.18,19 However, to our knowledge, no studies have been published that have evaluated the reliability of collecting CR/SP PMs. To address this need, and to respond to the National Quality Forum requirements to provide such data as part of their endorsement process, we carried out a multisite study, the Cardiac Rehabilitation Referral Reliability (CR3) Project, aimed at analyzing the reliability of abstracting the CR/SP PMs from inpatient and outpatient records.
Hospitals and outpatient cardiology practices in the United States were identified from the ACCF, AHA, and AACVPR databases and were invited to participate. We sought various hospitals and clinics, on the basis of different geographical locations, community sizes, and hospital/practice types/sizes (Figure 1). All 540 outpatient cardiology practices that were members of the ACCF outpatient quality and outcomes data registry (known as the PINNACLE network) as of October 1, 2011, were invited by e-mail to participate in the CR3 Project as outpatient sites. The PINNACLE Network helps cardiovascular teams achieve practice success through quality measurement, performance improvement, and peer-to-peer learning through an interactive community that connects practices across the country. In addition, an invitation to participate in the CR3 Project as an inpatient and/or an outpatient site was sent by e-mail to 2916 members of AACVPR, and targeted invitations were sent to 5 board members, 6 past presidents, and 11 committee chairs of the AACVPR, as well as to the CR/SP programs that were participating in the Wisconsin State Cardiac Rehabilitation Registry (70 centers) and the Montana State Cardiac Rehabilitation Registry (145 programs). Twenty-nine hospitals and 23 outpatient practices responded, expressing interest in participating in the project.
On the basis of available resources to carry out the CR3 Project, we initially planned to include a maximum of 12 sites in the project, with varied geographical locations and center characteristics. An additional site was added, since it was able to participate without the need for CR3 Project resources, resulting in a total of 7 inpatient and 6 outpatient practices that participated in the project. Inclusion criteria included a willingness to participate and ability to (1) provide a study coordinator and 2 separate chart abstractors, (2) complete the project within the specified timeline, and (3) obtain local institutional review board clearance to carry out the project in their setting. Once each hospital and practice completed and submitted their required data, they were sent a small incentive as a token of appreciation for their participation and submission of complete project data from their site ($200 gift card). Completed data were received from 7 hospitals and 6 outpatient cardiology practices.
For inpatient facilities, charts of patients who had an index hospitalization (ie, a hospitalization for a cardiac event that is a qualifying diagnosis or procedure for CR/SP) between August 1, 2009, and August 1, 2010, were eligible for review and inclusion. For outpatient centers, charts of patients who had an outpatient visit between August 1, 2009, and August 1, 2010, were eligible for review and inclusion. However, since the PM allows as long as 12 months for a patient to complete CR/SP following a qualifying cardiac event, chart abstraction included a search for a qualifying cardiac event between August 1, 2009, and August 1, 2010, along with a search of records for up to 12 months after the cardiac event, to search for documentation of CR/SP referral during that time period.
Study sites designated 1 study coordinator and 2 chart abstractors. Each study coordinator identified 35 patients from a consecutive sample of patients: 30 patients with an eligible diagnosis for CR/SP referral, and 5 without an eligible diagnosis for CR/SP (see later for additional details). The 2 abstractors at each site reviewed the same 35 patient records that had been selected from their site twice (once at baseline and again 1 week later). Abstractors had a range of experience reviewing charts, from less than 1 month to greater than 5 years.
Abstractors were blinded as to which patients in their sample had a qualifying diagnosis and which patients had exceptions for CR/SP. Only the site coordinator, who did not participate in the abstraction process, had access to this information. Patients considered to have qualifying events for CR/SP, as defined by the Centers for Medicare & Medicaid Services and therefore as specified in the PM, had 1 or more of the following: myocardial infarction, percutaneous coronary intervention, coronary artery bypass graft surgery, heart valve surgery, heart transplantation surgery, and chronic stable angina. Patients without a qualifying event, for the purpose of this abstraction project, were to have had documented 1 or more of the following diagnoses that are not currently considered by the Centers for Medicare & Medicaid Services to be a covered indication for CR/SP:
* For inpatient centers: atrial fibrillation, heart failure, or syncope during the index hospitalization period under review (with no documented qualifying events for CR during that same hospitalization).
* For outpatient centers: atypical chest pain, palpitations, or dyspnea during the 12 months before the index outpatient visit (with no documented qualifying events for CR referral during that same time period).
The CR3 Project workgroup created chart abstraction forms, site coordinator instructions, abstractor instructions, a frequently asked questions document, and site tracking forms to allow the study coordinator to track and report site-specific results for intra-abstractor (1 abstractor reviewing the chart 2 times) and interabstractor (2 abstractors reviewing 1 chart) reliability. The workgroup held a kickoff call with each center's study coordinator to train them before the start of the CR3 Project. Thereafter, the workgroup communicated weekly with site coordinators to address any questions or operational concerns that arose. The training of site coordinators was carried out during one or two 1-hour conference calls before starting the project. When coordinators had questions, they contacted the staff liaison to the CR3 working group directly by e-mail or telephone. New questions and their corresponding answers were communicated weekly to all site coordinators. The entire project took approximately 20 weeks to complete (October 2011 through February 2012).
The following definitions were developed for use in the study.
Eligible patients for CR/SP referral:
* Inpatient: a patient who survived the index hospitalization and who had a qualifying event/diagnosis for referral to CR/SP during the index hospitalization period under review.
* Outpatient: a patient who had a qualifying event/diagnosis for referral to CR/SP within the previous 12 months before the index outpatient visit.
Patients not eligible for CR/SP referral:
* Inpatient: a patient who had a cardiac event/diagnosis (atrial fibrillation, heart failure, or syncope for purposes of this study) during the index hospitalization period under review and no indication for CR/SP referral as specified in the PM.
* Outpatient: a patient who had a cardiac event/diagnosis (atypical chest pain, palpitations, or dyspnea for purposes of this study) during the 12 months before the index outpatient visit and no indication for CR/SP referral as specified in the PM.
* Inpatient: documentation in patient hospital medical records that the patient was referred to an outpatient CR/SP program.
* Outpatient: documentation in patient outpatient clinical medical records that the patient has been referred to an outpatient CR/SP program within 12 months after a qualifying event/diagnosis.
For purposes of this project, documentation in the medical record could include any of the following sources: hospital discharge summaries, office notes, clinical notes and medical records, orders (written/electronic), prescriptions (eg, contact information for CR/SP specialist), or other parts of the clinical record that documents patient information.
Because there are valid reasons why certain patients should not be referred to a CR/SP program, exceptions to the CR/SP measures are allowed. When a clinician is allowed to document exceptions, he or she is given the flexibility to decide whether or not to institute a given intervention/process depending upon the overall benefits and risks to the patient. Exceptions allow clinicians this flexibility without the threat of being “penalized” for not referring a patient to CR/SP. Without the presence of exceptions, potential negative unintended consequences could arise, such as forcing CR/SP on patients who are unstable. Furthermore, analysis of exception rates for quality improvement purposes allows providers and health systems to test the effects of process changes within their practices and communities that may facilitate CR/SP referral. Relatively few patients would be expected to qualify for an exception to CR/SP referral. Such exceptions would generally be limited to factors that may make CR/SP unsafe or ineffective, or lack of accessibility to a CR/SP program within a reasonable commuting distance.
Such exceptions would generally be limited to factors that may make CR/SP unsafe or ineffective, or that otherwise prohibit access to a CR/SP program. Examples of exceptions from referral to CR/SP include:
* Patient exceptions (eg, patient resides in a long-term nursing care facility)
* Medical exceptions (eg, presence of an acute medical condition that makes the patient unstable and unsafe for exercise training)
* System exceptions (eg, lack of an available CR/SP program within 60 minutes of travel time from the patient home)
Since the measures look only at whether patients were referred, not whether they enrolled, patient refusal was not considered to be an exception. If a health care provider recommended CR/SP referral to a patient, the patient refused the referral, and the provider documented the patient refusal, then that encounter was judged to have met the PM since the provider complied with the expectation to recommend referral to CR/SP.
Both the Cohen κ statistic and percent agreement were used to measure the intra- and interabstractor reliability for the following qualitative ratings: (1) documented eligibility for CR/SP referral, (2) exception documented for CR/SP referral, and (3) documentation of CR/SP referral. The κ statistic is a chance-corrected index of agreement ranging from −1 to 1, with κ < 0 representing observed agreement worse than that due to chance alone. We interpreted a κ greater than 0.75 as excellent, 0.40 to 0.75 as fair to good, and less than 0.40 as poor, following the guidelines of Fleiss et al.20 Unlike the κ statistic, percent agreement does not take into account the agreement occurring by chance but can be informative in situations for which the prevalence of a given response is very high or low and the interpretation based solely on the value of κ may be misleading. This phenomenon, known as the κ paradox,21,22 occurs when the observed proportion of agreement is high but the value of the κ statistic is low.
For brevity, intra-abstractor reliability is reported for only 1 of the 2 abstractors (arbitrarily designated “abstractor 1” at each site), and interabstractor reliability only for the initial set of ratings (ie, “time 1”). Stratifying on inpatient versus outpatient setting, reliability was analyzed (1) on the overall group with sites pooled together and (2) within sites and summarizing the site-specific results across the overall group. All analyses were performed using the SAS statistical software package (version 9.2, SAS Institute Inc, Cary, NC).
Characteristics of the 234 inpatients and 211 outpatients (total 445) included in the CR3 Project are shown in Table 2. Most patients from both inpatient and outpatient sites were male, white, and younger than 65 years. A total of 1746 chart reviews were performed for the CR3 Project (415 of the total 445 patient charts [93%] were reviewed as specified in the CR3 Project protocol, each 1 being reviewed 4 times [twice by each abstracter], while incomplete reporting of data resulted in 26 that were reviewed only 3 times each and 4 that were each reviewed only twice).
Participating centers represented various practice types and settings, including the following: rural, suburban, or urban area locations; teaching and nonteaching centers; and single specialty and multispecialty centers. One hospital was from the Pacific Northwest, 4 from the Midwest, 1 from the Northeast, and 1 from the Southeast. Three inpatient centers used paper medical records, 5 used electronic medical records, and 2 used both. Outpatient clinics in the CR3 Project were located throughout the Midwest and in the Southeastern part of the United States. Two outpatient clinics used paper medical records and 4 used electronic medical records, while none used both.
Site abstractors involved in the CR3 Project had varying degrees of experience with chart abstraction before participating in the project, with 54% of abstractors having 2 years of experience or less and 23% having less than 1 month of experience. Among the 13 inpatient and outpatient sites, the pair of abstractors had similar levels of experience at 11 sites. Excluding the 2 sites in which the pairs of abstractors had discordant levels of experience, we found that ratings of CR/SP eligibility, exceptions, and referral were not more reliable from abstractors having more than 2 years of experience. Interestingly, some of these ratings reflected more favorable reliability in abstractors having less than 2 years of experience (data not shown). In addition, we did not find a difference between the reliability of the first abstractions and the second abstractions, suggesting that there was no “learning effect” among abstractors. The mean ± SD time per chart abstraction, reported by abstractors, was 4.9 ± 3.2 minutes for inpatient abstractions and 6.8 ± 4.7 minutes for outpatient abstractions.
Inpatient sites (Table 3)
Intra-abstractor reliability analysis of pooled inpatient data demonstrated excellent repeatability for ratings of CR/SP eligibility (100% agreement; κ = 1.00), CR/SP exceptions (96% agreement; κ = 0.76), and CR/SP referral (98% agreement; κ = 0.95). On the basis of site-specific inpatient data, each of the three CR/SP items showed high percent agreement (≥90%) at all sites and excellent repeatability (κ ≥ 0.75) in most sites (100% of sites for patient eligibility, 67% for patient exceptions, and 80% for patient referral).
Pooled analysis of inpatient sites demonstrated excellent interabstractor reliability analysis for ratings of CR/SP eligibility (94% agreement; κ = 0.77) and CR/SP exceptions (97% agreement; κ = 0.79), and modest agreement between abstractors for rating CR/SP referral (86% agreement; κ = 0.70). Consistent with the pooled results, site-specific analyses demonstrated excellent interabstractor reliability (as measured by κ ≥ 0.75) in most inpatient sites for ratings of eligibility (71% of sites) and exceptions (67% of sites) but in less than half (40%) of sites for the rating of CR/SP referral.
Outpatient sites (Table 3)
Pooled analyses of the 6 outpatient sites demonstrated excellent intra-abstractor reliability for the 3 ratings of CR/SP eligibility, exceptions, and referral (agreement ≥95%; κ ≥ 0.88). From site-specific analysis of intra-abstractor reliability, percent agreement ≥90% was observed in all 6 sites for ratings of CR/SP eligibility and exceptions, and in all but 1 site for rating of CR/SP referral. Likewise, excellent repeatability (κ ≥ 0.75) was demonstrated in most outpatient sites (100% of sites for rating of eligibility, 67% for exceptions, and 67% for referral).
Regarding interabstractor reliability for outpatient sites, pooled analyses reflected excellent agreement between abstractors for ratings of both CR/SP eligibility (κ = 0.78) and CR/SP referral (κ = 0.80), and poor to fair agreement in rating patient exceptions for CR/SP referral (κ = 0.43). Similarly, according to site-specific results, excellent interabstractor reliability was observed in most (two-thirds) of the outpatient sites for rating CR/SP eligibility and in none of the sites for rating CR/SP exceptions. Interestingly, despite excellent interabstractor agreement for rating CR/SP referral obtained from pooled analysis, site-specific results varied considerably (range of κ across 6 sites, −0.07 to 1.00), with excellent reliability seen in only one-third of outpatient sites (and percent agreement less than 90% in half the sites).
This study demonstrates high reliability for assessing CR/SP eligibility, referral, and exceptions by using the CR/SP outpatient and inpatient PMs. Data abstraction of patient records was performed by abstractors with varying amounts of abstraction experience at various inpatient and outpatient centers, suggesting generalizability of our findings.
Reliability testing is 1 of 3 important steps in developing high value PMs, as outlined by the ACCF/AHA Task Force on PMs.19 The 3 steps include (1) construction of the measurement set, (2) assessment of feasibility and reliability of data collection, and (3) measurement of clinician performance. Construction of the CR/SP PM set has previously been reported.12–17
Our testing generally found high reliability for comparisons between abstractors for the 3 key components of the CR/SP PMs: patient eligibility for CR/SP, patient exceptions to CR/SP referral, and patient referral to CR/SP. We included 2 measures of reliability, each shedding important light on the reliability of PM abstraction: percent agreement and the κ statistic. “Percent agreement” is a helpful assessment of reliability, but given that more than 80% of patients in the study sample were eligible for CR/SP and more than 90% of patients were absent exceptions to CR/SP participation, the percent agreement may have been somewhat inflated, since by chance alone abstractors may have chosen the correct eligibility or exception status.
Conversely, the κ statistic performs best when there is nearly equal chance of study outcomes. When there is a high likelihood of 1 of the 2 outcomes, as in our study (high likelihood of CR/SP eligibility), the results of the κ analyses can underestimate true reliability because of a phenomenon known as the “kappa score paradox” in which there is high percent agreement, yet a low κ score.21,22 Indeed, we observed this paradox in some centers. The true reliability of abstracting our PMs most likely lies between the results from the 2 methods of assessment we used. Since the “percent agreement” method generally suggests very high reliability of the CR/SP measures and the κ statistic generally suggests moderate to high reliability, the true reliability of the CR/SP PM would appear overall to be high.
Data abstractors reported that data abstraction time was modest for the inpatient (4.9 minutes) and outpatient (6.8 minutes) CR/SP PMs, and minimal barriers to their abstraction activities. If the CR/SP PMs are included in sets of other PMs, such as the PM set for coronary artery bypass graft surgery, for example, it is likely that efficiencies of scale will result in less time being required for the CR/SP PM assessment.
We selected participating centers to reflect variation in the location, size, and type of centers. However, our study is based on the experience of a relatively small number of centers from around the United States that volunteered to be in the project and may not be representative of other centers from different regions.
Outpatient abstraction of the CR/SP PM data was more time-consuming and somewhat less reliable than the abstraction of inpatient data. This is explained in large part by the fact that the review of inpatient data is limited to the time of the patient index hospitalization (ie, the time of the cardiac event that qualified them for CR/SP). Review of outpatient data is broader, including a review of records for up to 12 months previous to the outpatient visit and also a review of records for up to 12 month after the outpatient visit, because of the fact that patients are eligible for CR/SP for up to 12 months following their qualifying cardiac event.
Health care provider education through effective communication channels is critically important to help providers understand and document appropriate exceptions to CR/SP referral, as well as the key components of CR/SP referral documentation: (1) that the patient has been referred to CR/SP, (2) that the patient has been given information and guidance to help them enroll in CR/SP, and (3) that the receiving CR/SP program has been sent patient information to expedite CR/SP enrollment).
Current practices and existing ACCF and AHA registries only require documentation that the patient has been referred to a CR/SP program. Published evidence suggests that the use of additional communication components, as specified in the measures, may increase the predictive validity of the measures.23 Going forward, with the advent of better data collection systems for CR/SP referral and the ability now to track CR/SP enrollment through the AACVPR Outpatient Cardiac Rehabilitation Registry, we expect to be able to test the hypothesis that this more detailed definition of CR/SP referral will increase enrollment in CR/SP. Furthermore, computerized decision support, made more widely available through efforts to enhance the meaningful use of electronic health records, may also provide value by increasing the ability to track and improve the appropriate utilization of CR/SP.
Reliability of CR/SP PM abstraction is high. Data abstractors reported minimal barriers to the abstraction process and required a relatively small amount of time per patient to carry out the abstractions. These results contribute to published evidence regarding the soundness and generalizability of the CR/SP PMs. Further work will need to be carried out to assess the impact of the CR/SP PMs on patient referral rates and patient outcomes.
Funding for statistical analysis was provided by the Mayo Clinic and Foundation. Funding for study site incentives was provided by the AACVPR, the ACCF, and the AHA.
1. Goel K, Lennon RJ, Tilbury RT, Squires RW, Thomas RJ. Impact of cardiac rehabilitation on mortality and cardiovascular events after percutaneous coronary intervention in the community. Circulation. 2011;123:2344–2352.
2. Hammill BG, Curtis LH, Schulman KA, Whellan DJ. Relationship between cardiac rehabilitation and long-term risks of death and myocardial infarction among elderly Medicare beneficiaries. Circulation. 2010;121:63–70.
3. Oldridge NB, Guyatt GH, Fischer ME, Rimm AA. Cardiac rehabilitation after myocardial infarction. Combined experience of randomized clinical trials. JAMA. 1988;260:945–950.
4. Suaya JA, Stason WB, Ades PA, Normand SL, Shepard DS. Cardiac rehabilitation and survival in older coronary patients. J Am Coll Cardiol. 2009;54:25–33.
5. Taylor RS, Unal B, Critchley JA, Capewell S. Mortality reductions in patients receiving exercise-based cardiac rehabilitation: how much can be attributed to cardiovascular risk factor improvements? Eur J Cardiovasc Prev Rehabil. 2006;13:369–374.
6. Williams MA, Ades PA, Hamm LF, et al. Clinical evidence for a health benefit from cardiac rehabilitation: an update. Am Heart J. 2006;152:835–841.
7. Witt BJ, Jacobsen SJ, Weston SA, et al. Cardiac rehabilitation after myocardial infarction in the community. J Am Coll Cardiol. 2004;44:988–996.
8. Centers for Disease Control and Prevention. Receipt of cardiac rehabilitation services among heart attack survivors—19 states and the District of Columbia, 2001. MMWR Morb Mortal Wkly Rep. 2003;52:1072–1075.
9. Suaya JA, Shepard DS, Normand SL, Ades PA, Prottas J, Stason WB. Use of cardiac rehabilitation by Medicare beneficiaries after myocardial infarction or coronary bypass surgery. Circulation. 2007;116:1653–1662.
10. Thomas RJ, Miller NH, Lamendola C, et al. National survey on gender differences in cardiac rehabilitation programs. Patient characteristics and enrollment patterns. J Cardiopulm Rehabil. 1996;16:402–412.
11. Fonarow GC, Abraham WT, Albert NM, et al. Association between performance measures and clinical outcomes for patients hospitalized with heart failure. JAMA. 2007;297:61–70.
12. Thomas RJ, King M, Lui K, Oldridge N, Pina IL, Spertus J. AACVPR/ACC/AHA 2007 performance measures on cardiac rehabilitation for referral to and delivery of cardiac rehabilitation/secondary prevention services. J Cardiopulm Rehabil Prev. 2007;27:260–290.
13. Thomas RJ, King M, Lui K, Oldridge N, Pina IL, Spertus J. AACVPR/ACC/AHA 2007 performance measures on cardiac rehabilitation for referral to and delivery of cardiac rehabilitation/secondary prevention services. Circulation. 2007;116:1611–1642.
14. Thomas RJ, King M, Lui K, Oldridge N, Pina IL, Spertus J. AACVPR/ACCF/AHA 2010 update: performance measures on cardiac rehabilitation for referral to cardiac rehabilitation/secondary prevention services. J Am Coll Cardiol. 2010;56:1159–1167.
15. Thomas RJ, King M, Lui K, Oldridge N, Pina IL, Spertus J. AACVPR/ACCF/AHA 2010 update: performance measures on cardiac rehabilitation for referral to cardiac rehabilitation/secondary prevention services. J Cardiopulm Rehabil Prev. 2010;30:279–288.
16. Thomas RJ, King M, Lui K, Oldridge N, Pina IL, Spertus J. AACVPR/ACCF/AHA 2010 update: performance measures on cardiac rehabilitation for referral to cardiac rehabilitation/secondary prevention services. Circulation. 2010;122:1342–1350.
17. Thomas RJ, King M, Lui K, et al. AACVPR/ACC/AHA 2007 performance measures on cardiac rehabilitation for referral to and delivery of cardiac rehabilitation/secondary prevention services. J Am Coll Cardiol. 2007;50:1400–1433.
18. Spertus JA, Bonow RO, Chan P, et al. ACCF/AHA new insights into the methodology of performance measurement. Circulation. 2010;122:2091–2106.
19. Spertus JA, Eagle KA, Krumholz HM, Mitchell KR, Normand SL. American College of Cardiology and American Heart Association methodology for the selection and creation of performance measures for quantifying the quality of cardiovascular care. J Am Coll Cardiol. 2005;45:1147–1156.
20. Fleiss JL, Levin B, Cho Park M. Statistical Methods for Rates and Proportions. Hoboken, NJ: Wiley Interscience; 1981:352.
21. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43:543–549.
22. Lantz CA, Nebenzahl E. Behavior and interpretation of the kappa statistic: resolution of the two paradoxes. J Clin Epidemiol. 1996;49:431–434.
23. Grace SL, Russell KL, Reid RD, et al. Effect of cardiac rehabilitation referral strategies on utilization rates: a prospective, controlled study. Arch Intern Med. 2011;171:235–241.
cardiac rehabilitation; quality improvement; referral; reliability testing