In 2002, the Centers for Medicare & Medicaid Services (CMS) partnered with the Agency for Healthcare Research and Quality to implement the first national, standardized survey of patients' perspectives of health care.1,2 CMS now reports this information for many acute care hospitals through its Hospital Compare program, which is designed to help patients make decisions about where to obtain care and incentivize providers to improve quality.3,4 These actions are reflective of shifting priorities in health care quality research emphasizing the greater importance of patient-centered outcomes.5 More recently, CMS began using these same data to augment payment to acute care hospitals through its Value-Based Purchasing (VBP) program.6 Within these same hospitals, inpatient surgical care accounts for approximately 40% of annual hospital and physician spending—a figure due in large part to the management of postoperative complications.7,8
Whether patients' perspectives of care correlate with surgical outcomes remains unclear. Contextually similar work in the surgical literature is limited to specific populations. There are numerous studies correlating objective outcome measures with patient-centered outcomes in patients with cancer.9–11 This work suggests a negative correlation between higher patients' quality of life and the incidence of postoperative complications. Beyond the surgical literature, evidence from a large statewide demonstration suggests that patients' perspectives of care can incentivize providers to improve quality in the primary care setting.12,13 It is unclear whether this information can service as a stimulus for quality improvement in the delivery of surgical care.
Using data from the Michigan Surgical Quality Collaborative (MSQC), we sought to characterize the relationship between hospitals' performance on the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey and risk-adjusted outcomes after major surgery. A better understanding of how patients' perspectives of care relate to objective surgical outcomes may align priorities for quality improvement and promote greater attention to patient-centered care.
Data Source and Study Population
We used data from the MSQC clinical registry to identify patients undergoing major general or vascular surgery within participating hospitals between 2008 and 2012. We restricted our population to those patients who underwent elective operations requiring an inpatient hospitalization of at least 24 hours. The MSQC is a provider-led quality improvement organization funded by Blue Cross and Blue Shield of Michigan. Data for this project used standard data definitions and collection protocols of the MSQC platform as previously described.14 All available variables were collected for this analysis including patient demographics, preoperative risk factors, laboratory values, perioperative factors, and 30-day postoperative morbidity and mortality. Additional hospital-level data (bed size, hospital occupancy, surgical volume, Council of Teaching Hospital status, pain management services, hospice services, and inpatient social work) were obtained from the American Hospital Association Hospital and Health System Data Resources a priori on the basis of plausible relationship with surgical care quality and patients' perspectives.
HCAHPS data are publicly available through the CMS Hospital Compare program and the first reporting occurred in March 2008.4 The scoring of HCAHPS data is complicated. Briefly, HCAHPS surveys are disseminated to persons recently discharged from an acute care hospital for a medical, surgical, or maternity-related admission. There are 10 measures related to patients' perspectives of care that are included in the HCAHPS survey (Table 1). Each measure is evaluated by 2 or 3 items that vary in the level of affirmation for a given statement (eg, Patients who reported that their doctors always “communicated well.”). The most affirmative items for each measure, referred to as “top-box” items, are used to calculate the Patient Experience of Care Domain for the Hospital VBP program.15
The Patient Experiences of Care Domain comprises 2 separate scores: a Base Score and a Consistency Score. The Base Score (80 points maximum) represents an achievement mark (performance relative to the national median) or an improvement mark, calculated by changes in HCAHPS scores between a baseline period (July 2009–March 2010) and a performance period (July 2011–March 2012). The Consistency Score (20 points maximum) is derived from hospitals' scores for their lowest performing HCAHPS measure. The sum of these scores forms the Patient Experiences of Care Domain with a maximum score of 100. We will subsequently reference this as the HCAHPS Total Score. This score augments 30% of the Hospital VBP Total Performance Score. The remaining 70% is derived from Clinical Processes of Care Domain.
The primary outcomes for this study were the incidence of 30-day in-hospital morbidity and mortality. We determined various in-hospital postoperative complications such as surgical site infection (superficial, deep, and organ space defined separately), deep venous thrombosis, urinary tract infection, acute renal failure, postoperative bleeding requiring transfusion, stroke, unplanned intubation, fascial dehiscence, prolonged mechanical ventilation over 48 hours, myocardial infarction, pneumonia, pulmonary embolism, sepsis, vascular graft loss, and renal insufficiency.
Pearson χ2 test, Student t test, and Wilcoxon signed-rank test were used to compare patient characteristics and unadjusted rates of postoperative outcomes. We compared continuous variables using Pearson correlation coefficient and Cohen κ statistic. Multilevel, mixed-effects logistic regression models were used to adjust hospitals' morbidity and mortality rates. Final models included patient-level risk factors such as age, race, body mass index, comorbid conditions, American Society of Anesthesiologists class, surgeon specialty, work Relative Value Units, and do-not-resuscitate status as fixed-effects. We accounted for clustering of outcomes within centers using a hospital identifier. Next, we adjusted our estimates for reliability using empirical Bayes techniques as has been previously described.16
We conducted a primary analysis to determine the relationship between hospitals' HCAHPS Total Score and adjusted rates of postoperative morbidity and mortality. We grouped hospitals by quintiles based on their HCAHPS Total Score to improve generalizability and ensure that hospitals' could not be identified, given that HCAHPS information is publicly available. For this analysis, we evaluated hospitals' adjusted morbidity and mortality rates from March 2008 (the start of HCAHPS public reporting) to March 2012 (the completion of the HCAHPS performance period) in aggregate. We then conducted a sensitivity analysis using hospitals' scores for 2 separate “top-box” items for overall quality. Items included patients rating the hospital 9 or 10 on a scale from 0 to 10 and patients stating that they would definitely recommend the hospital.
We then conducted a second analysis to investigate whether changes in morbidity and mortality rates within hospitals correlated with the HCAHPS Base Score, which reflects sustained achievement or improvement in patients' perspectives of care between the baseline period and the performance period. The baseline period concluded on March 31, 2010. We compared adjusted rates of morbidity and mortality within hospitals and for the total population before and after this date. We calculated absolute differences in outcome rates and reported results across quintiles of hospitals' HCAHPS Base Score.
All statistical analyses were performed using the Stata statistical software (version 12.1; College Station, TX). This study was approved by the University of Michigan institutional review board.
We identified 41,833 patients undergoing major inpatient operations and meeting inclusion criteria, with 32 participating hospitals. There were significant differences in patient demographics and comorbid disease burden between the hospitals stratified by quintiles of HCAHPS Total Score (Table 2). In general, the overall profile of patient comorbidities was not markedly different across quintiles of HCAHPS Total Score. Patients of nonwhite race were twice as common in the lowest HCAHPS Total Score hospitals when compared with those with the highest HCAHPS Total Score (P < 0.001). The distribution of procedure mix between major general and vascular operations did not vary across quintiles (P = 0.11). HCAHPS Total Score (range = 55) and Base Score (range = 48) were variable across hospitals. The most common general surgery procedure performed was colectomy (n = 12,181; 29%), and the most common vascular procedure was abdominal aortic aneurysm repair (n = 2948; 7%).
HCAHPS Total Score and Surgical Outcomes
Unadjusted morbidity and mortality rates for the entire patient population were 22.0% and 3.7%, respectively. Adjusted morbidity rates varied widely across hospitals, ranging from 13.6% to 28.6%. Adjusted mortality rates across hospitals ranged from 2.7% to 5.2%. When stratifying hospitals into quintiles by their HCAHPS Total Score, we observed no significant differences in risk-adjusted morbidity (Fig. 1). There were no significant differences in risk-adjusted mortality rates between hospitals with the lowest and highest HCAHPS Total Scores (24.5% vs 20.2%, P = 0.312). Furthermore, adjusted mortality rates for hospitals with the lowest HCAHPS Total Score (4.0%; 95% confidence interval, 3.2–5.1) were not statistically different from those in hospitals with the highest HCAHPS Total Score (3.3%; 95% confidence interval, 2.7–4.2). We did not observe a stepwise decrease or increase in adjusted outcome rates across quintiles of HCAHPS Total Score. When assessing the relationship between patients' perspectives of care and postoperative morbidity in a continuous fashion, we found no correlation (κ = 0.03, P = 0.155). In Figure 2, we observed a random association between hospitals ranks of HCAHPS Total Score and risk-adjusted morbidity rates. The 4 quadrants represent hospital performance as high or low for each domain—objective outcomes and patients' perspectives.
HCAHPS Base Score and Outcome Improvement
No statistically significant differences in adjusted mortality rates were observed within hospitals between the HCAHPS baseline and performance periods. Overall mortality for the entire population decreased from 3.8% to 3.5%, although this difference was not statistically significant (P = 0.200). In contrast, overall adjusted morbidity rates for the entire population dgoecreased from 23.3% to 20.8% (P < 0.001) between the HCAHPS baseline and performance periods. We observed statistically significant improvement in 13 of 32 hospitals (41%), and these centers were equally distributed across quintiles of HCAHPS Base Score. The absolute difference in adjusted morbidity rates was not statistically different between hospitals with the lowest 20% (−2.0%) and highest 20% of HCAHPS Base Scores (−3.4%, P = 0.192).
Hospital Characteristics, Patients' Perspectives, and Surgical Outcomes
Finally, we conducted a comparison of hospital characteristics between hospitals with high performance in both domains and those with low performance described in Figure 2. We found no significant differences in overall size, inpatients surgical volume, average occupancy, and overall annual admissions. We also found that all hospitals in both groups provided resources associated with improved patients' perspectives of care such as inpatient social work, pain management teams, and palliative care/hospice. All hospitals also reported disseminating information to the patient community regarding the quality and cost of hospital care.
In this study, we report adverse outcomes of patients undergoing major general or vascular inpatient surgery across Michigan hospitals stratified by their performance on the HCAHPS survey. With the incorporation of patients' perspectives of care into the Hospital VBP program, CMS places a growing incentive on patient-centered outcomes. We failed to observe a significant relationship between patients' perspectives of care and the incidence of 30-day postoperative morbidity and mortality within a statewide surgical collaborative. When comparing hospitals' adverse event rates between the CMS baseline and performance periods, we also failed to show significant correlation between outcome improvement and sustained or improved patients' perspectives of care. These results indicate that metrics of objective and subjective surgical quality measure equally important, but not necessarily overlapping, domains of the surgical experience.
Accompanying the Patient Protection and Affordable Care Act, the Patient Centered Outcomes Research Institute was founded to improve the availability and quality of information provided to health care consumers.17 With the development of the Patient Centered Outcomes Research Institute, a new era of comparative effectiveness research brought increasing focus on patient-centered outcomes research. Despite this, the utility of publically available patient-centered hospital rankings remains uncertain.18 Previous studies have demonstrated that hospital choice is less influenced by comparative quality metrics but more so by word of mouth and anecdotal evidence.19,20 In addition, the interpretation of quality metrics is highly variable, with age, literacy, and knowledge of health care having huge implications for choice.21 The importance of hospital choice for major operations obligates the surgical community to improve delivery and sharpen the focus of hospital quality metrics.
As a leader in outcome measurement and quality improvement, the field of surgery should continue to focus on improving objective outcomes in conjunction with patient-centered care. Although beyond the scope of this analysis, it is entirely possible that metrics for postoperative morbidity and patient-centered care will not correlate at the hospital level. Changes in care practice that influence postoperative morbidity and mortality may be exclusive from hospitals' cultural changes that improve patients' perspectives of care. Previous collaboratives such as the MSQC have demonstrated quality improvement through public reporting.22 When the Alliance Group of Madison, WI, publically released hospital quality rankings for hip and knee surgery, cardiac care, and obstetrics, significant improvements in quality were seen in those hospitals included in the report.23 Despite this, market share was unaffected by the move and there was no significant change in volume across hospitals. CMS indicates that the reporting of HCAHPS measures may improve outcomes.3 As surgeons navigate between their clinical responsibilities and hospital-wide pressures to improve patients' perspectives of care, it will be important for all parties involved to understand that improvements in one domain of patient may not augment another.
An important qualification of HCAHPS data as reported by the Hospital Compare program is that they are not unique to particular patient populations. These data can be available for particular demographics or wards of a hospital. However, that information is generally not publicly available is not used to augment reimbursement through the Hospital VBP program. Although one may argue that little can be inferred about surgical care from data that include patients admitted to medical and obstetrical wards, there are several considerations that refute this assertion. First, the HCAHPS survey assesses many domains that are likely not unique to surgical care (eg, hospital cleanliness and quietness). Second, hospitals are rewarded or penalized collectively for their HCAHPS scores. Surgical teams whose patients record high HCAHPS scores will see no benefit to this if the overall hospital scores in aggregate are low.15 Most important is the fact that the data used for this analysis are derived from CMS and serve to directly impact hospital reimbursement and inform patients' and providers as to the “quality” of a particular hospital. It will be important to recognize that quality takes many forms that may be exclusive and should be considered independently, given the current metrics.
This study has several important limitations. This is a retrospective study, and we are not able to assert causation from our findings. Given this study design, our risk adjustment models are thereby subject to bias from unmeasured confounding, although we have no reason to believe that this would be systematically different across different centers. It is plausible that HCAHPS Total and Base Scores may not truly reflect patients' perspectives of care during the study period. It is also possible that sample selection and survey-taker bias confound the interpretation of HCAHPS survey data. However, this information is still being used to augment hospital payments and is publicly available to inform patient hospital choice. The HCAHPS Base Score reflects sustained achievement or improvement, and we are unable to determine this through the available data. Finally, although this may provide an accurate portrayal of the state of Michigan, regional differences in health care opinions may limit the generalizability of our results.
The incidence of morbidity and mortality after major surgery does not correlate with patients' perspectives of care from the HCAHPS survey. These data call to question the use of HCAHPS scores to inform patient decision making and incentivize quality improvement. Further work is needed to appropriately inform efforts to improve patients' perspectives of care and surgical outcomes, as these efforts may be uniquely different in the overall approach.
1. Goldstein E, Fyock J. Reporting of CAHPS quality information to Medicare beneficiaries. Health Serv Res. 2001;36:477–488.
2. Hargraves JL, Hays RD, Cleary PD. Psychometric properties of the Consumer Assessment of Health Plans Study (CAHPS) 2.0 Adult Core Survey. Health Serv Res. 2003;38:1509–1527.
5. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307:1583–1584.
6. VanLare JM, Conway PH. Value-based purchasing—national programs to move from volume to value. N Engl J Med. 2012;367:292–295.
7. Birkmeyer JD, Gust C, Baser O, et al. Medicare payments for common inpatient procedures: implications for episode-based payment bundling. Health Serv Res. 2010;45:1783–1795.
8. Birkmeyer JD, Gust C, Dimick JB, et al. Hospital quality and the cost of inpatient surgery in the United States. Ann Surg. 2012;255:1–5.
9. Karanicolas PJ, Graham D, Gonen M, et al. Quality of life after gastrectomy for adenocarcinoma: a prospective cohort study. Ann Surg. 2013;257:1039–1046.
10. Lee ES, Lee MK, Kim SH, et al. Health-related quality of life in survivors with breast cancer 1 year after diagnosis compared with the general population: a prospective cohort study. Ann Surg. 2011;253:101–108.
11. Andersson J, Angenete E, Gellerstedt M, et al. Health-related quality of life after laparoscopic and open surgery for rectal cancer in a randomized trial. Br J Surg. 2013;100:941–949.
12. Rodriguez HP, von Glahn T, Elliott MN, et al. The effect of performance-based financial incentives on improving patient care experiences: a statewide evaluation. J Gen Intern Med. 2009;24:1281–1288.
13. Rodriguez HP, von Glahn T, Rogers WH, et al. Organizational and market influences on physician performance on patient experience measures. Health Serv Res. 2009;44:880–901.
14. Campbell DA. Accelerating the pace of surgical quality improvement: the power of hospital collaboration. Arch Surg (Chicago. 1960). 2010;145:985–991.
15. Centers for Medicare & Medicaid Services. HCAHPS Fact Sheet. Baltimore, MD: Centers for Medicare & Medicaid Services; 2012.
16. Dimick JB, Ghaferi AA, Osborne NH, et al. Reliability adjustment for reporting hospital outcomes with surgery. Ann Surg. 2012;255:703–707.
17. Methodology Committee of the Patient-Centered Outcomes Research I. Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective. JAMA. 2012;307:1636–1640.
18. Marshall MN, Shekelle PG, Leatherman S, et al. The public release of performance data: what do we expect to gain? A review of the evidence. JAMA. 2000;283:1866–1874.
19. Schwartz LM, Woloshin S, Birkmeyer JD. How do elderly patients decide where to go for major surgery? Telephone interview survey. BMJ. 2005;331:821.
20. Huppertz JW, Carlson JP. Consumers' use of HCAHPS ratings and word-of-mouth in hospital choice. Health Serv Res. 2010;45:1602–1613.
21. Zwijnenberg NC, Hendriks M, Damman OC, et al. Understanding and using comparative healthcare information; the effect of the amount of information and consumer characteristics and skills. BMC Med Inform Decis Mak. 2012;12:101.
22. Lamb GC, Smith MA, Weeks WB, et al. Publicly reported quality-of-care measures influenced Wisconsin physician groups to improve performance. Health Aff (Millwood). 2013;32:536–543.
23. Hibbard JH, Stockard J, Tusler M. Hospital performance reports: impact on quality, market share, and reputation. Health Aff (Millwood). 2005;24:1150–1160.
Keywords:© 2014 by Lippincott Williams & Wilkins.
HCAHPS survey; Hospital Compare; patients' perspectives; regional quality improvement; surgical outcomes