Background: Providing culturally competent care shows promise as a mechanism to reduce health care inequalities. Until the recent development of the Consumer Assessment of Healthcare Providers and Systems Cultural Competency Item Set (CAHPS-CC), no measures capturing patient-level experiences with culturally competent care have been suitable for broad-scale administration.
Methods: We performed confirmatory factor analysis and internal consistency reliability analysis of CAHPS-CC among patients with type 2 diabetes (n=600) receiving primary care in safety-net clinics. CAHPS-CC domains were also correlated with global physician ratings.
Results: A 7-factor model demonstrated satisfactory fit (χ2 231=484.34, P<0.0001) with significant factor loadings at P<0.05. Three domains showed excellent reliability—Doctor Communication-Positive Behaviors (α=0.82), Trust (α=0.77), and Doctor Communication-Health Promotion (α=0.72). Four domains showed inadequate reliability either among Spanish speakers or overall (overall reliabilities listed): Doctor Communication-Negative Behaviors (α=0.54), Equitable Treatment (α=0.69), Doctor Communication-Alternative Medicine (α=0.52), and Shared Decision-Making (α=0.51). CAHPS-CC domains were positively and significantly correlated with global physician rating.
Conclusions: Select CAHPS-CC domains are suitable for broad-scale administration among safety-net patients. Those domains may be used to target quality-improvement efforts focused on providing culturally competent care in safety-net settings.
*Division of General Internal Medicine, University of California, San Francisco, CA
†Division of General Medicine and Health Innovation Program, University of Wisconsin, Madison, WI
‡Center for AIDS Prevention Studies, University of California, San Francisco, CA
§Department of Health Services Administration, University of Alabama, Birmingham, AL
∥College of Arts and Sciences, University of Cincinnati College of Medicine, Cincinnati, OH
Supported by the Russell Sage Foundation and the Commonwealth Fund.
The authors declare no conflict of interest.
Reprints: Hilary K. Seligman, MD, MAS, Department of Medicine, Division of General Internal Medicine, University of California, San Francisco, Box 1364, San Francisco, CA 94143-1364 (e-mail: firstname.lastname@example.org).
Culturally competent care has been defined as “the ability of health care providers and institutions to deliver effective services to racially, ethnically, and culturally diverse patient populations.”1 Cultural competence is associated with better interpersonal processes of care2 and has been identified as a promising tool in combating racial/ethnic and socioeconomic inequalities in health care.3–6 Although the utility of culturally competent care has been extensively theorized,7–9 little empiric research links cultural competence to clinical outcomes.3,10,11 The relative paucity of outcomes research related to culturally competent care has made health plans reticent to develop comprehensive strategies to improve cultural competence.12 A significant barrier in linking culturally competent care to outcomes is the lack of quantitative, patient-centered culturally competent care measures that can be used in diverse settings. To date, culturally competent care research has measured physician knowledge and attitudes rather than patients’ assessments of the cultural competence of the care they receive.5,13 The Consumer Assessment of Healthcare Providers and Systems Cultural Competency Item Set (CAHPS-CC) captures patients’ assessments of the degree to which their care is culturally competent, reasoning that patient assessments will be more closely linked to clinical outcomes than physician assessments.14
CAHPS-CC is intended as a supplement to the CAHPS surveys. The Agency for Healthcare Research and Quality developed CAHPS as a patient-administered assessment of health care quality in diverse settings. Medicare managed care plans are required to report CAHPS data to the federal government as a quality of care metric. Low CAHPS scores have been associated with poor medication adherence,15 increased emergency room use,16 and lower utilization of colon cancer screening.17 Although CAHPS has been extensively validated among insured patients, it is less clear how it performs with ethnically diverse, uninsured, safety net, and other vulnerable populations.18,19 Adding a set of culturally competent care domains makes CAHPS more relevant to these populations.
CAHPS-CC was originally designed for, and psychometrically evaluated in an insured population.20 Given the high prevalence of uninsured and underinsured minority populations in the United States, we evaluated its performance in an uninsured/underinsured sample of patients with type 2 diabetes. Provision of culturally competent care may be particularly important among patients with chronic diseases such as diabetes, where treatment requires patient engagement, competent self-management, and an ongoing relationship between patient and clinician.
Study Design and Participants
CAHPS-CC was administered in participants’ preferred language (English or Spanish) as part of the Immigration, Culture, and Health Care (ICHC) study. ICHC was a cross-sectional study investigating factors that impact diabetes self-management and health outcomes in vulnerable populations. The study included African American, Spanish-speaking and English-speaking Mexican American, and non-Latino white adults with diabetes who received primary care in safety-net clinics in Chicago and the San Francisco Bay Area. Eligible patients were adults (age 18 y or older) with chart-confirmed type 2 diabetes. Exclusion criteria, assessed by trained interviewers before study enrollment, included cognitive impairment, active substance abuse, or psychosis severe enough to interfere with survey administration. The ICHC survey and CAHPS-CC component were administered during face-to-face interviews. The Institutional Review Boards of the University of California, San Francisco, the Cook County Health and Hospital System, and other participating clinics approved all study activities.
Design and Measures
CAHPS-CC is a multidimensional set of 26 questions designed to measure patients’ overall experience of their physician’s interpersonal and cultural competence as well as their experience of their physician’s office. As Weech-Maldonado et al20 describe in a companion manuscript, literature review and examination of cultural competence theory yielded 5 domains of cultural competence: (1) patient-provider communication; (2) respect for patient preferences/shared decision-making; (3) experiences leading to trust or distrust; (4) experiences of discrimination; and (5) language services. To develop the 26 survey items, Weech-Maldonado and colleagues gleaned relevant items from the existing CAHPS survey, from other cultural competency instruments in the literature and public domain, and from a Federal Register call for measures. Items then underwent extensive cognitive interviewing, field testing, and rigorous translation.20 Exploratory factor analysis of the original items—conducted on a sample of patients insured through the Medicaid program—suggested that CAHPS-CC had a first-order factor structure comprised of 8 domains: Doctor Communication-Positive Behaviors—taken directly from the existing CAHPS survey (5 items), Doctor Communication-Negative Behaviors (3 items), Trust (5 items), Equitable Treatment (2 items), Doctor Communication-Health Promotion (4 items), Doctor Communication-Alternative Medicine (2 items), Shared Decision-Making (2 items), and Access to Interpreter Services (3 items). Given the multidimensional, first-order factor structure, CAHPS-CC was thought to represent several separate domains of cultural competence, rather than measuring a single cultural competence construct.
We also used a measure of global physician rating to evaluate the validity of the CAHPS-CC adjusting for potential confounders. The global physician rating asks participants to rate their primary care doctor using any number from 0 to 10, where 0 is the “worst possible” doctor and 10 is the “best possible” doctor. This item is a widely reported patient satisfaction measure and is associated with health care quality21 and some clinical outcomes.22 Measured confounders included age (18–34, 35–64, or 65+ y), sex, race/ethnicity (non-Latino white, African American, or Mexican American), language (English or Spanish), self-reported health status (excellent/very good/good vs. fair/poor/very poor), and depressive symptoms. We considered scores ≥10 on the Patient Health Questionnaire-9 to be consistent with depressive symptoms.23 The Patient Health Questionnaire -9 is a 9-item scale that asks respondents how often over the previous 2 weeks they have been bothered by each of the 9 symptom clusters in the Diagnostic and Statistical Manual’s (Fourth edition) criteria for diagnosis of depression.
CAHPS-CC domain scores were calculated by linearly transforming items to a 0–100 scale (with a higher score representing more favorable assessments), and then averaging items within each domain. We performed a confirmatory factor analysis using a weighted least squares estimator with a mean and variance adjustment (Mplus 6.12, estimator WLSMV) to assess the validity of the hypothesized factor structure.24 We assessed global model fit using the χ2 test of exact model-data fit, the comparative fit index (CFI), root mean square error of approximation (RMSEA), and weighted root mean square residual (WRMR). Ideal values for these tests are as follows: χ2 test of exact model fit, P>0.05; CFI≥0.95; RMSEA≤0.06; and WRMR≤1.00. Because the χ2 test of exact model fit may be sensitive to trivial departures of model-data fit, we followed the recommendations of Hu and Bentler25 by a priori accepting the confirmatory factor analysis results if 2 of the 3 descriptive statistics (CFI, RMSEA, WRMR) met the desired fit criteria.
We used SAS 9.2 to measure the internal consistency reliability of the factors identified in the confirmatory factor analysis using a Cronbach α statistic.26,27 We considered a Cronbach α of 0.70 indicative of adequate reliability.28
We then evaluated the relative importance of the CAHPS-CC domains on perceptions of care by measuring the variance in global physician ratings attributable to culturally competent care. We trichotomized global physician rating (0–4, 5–8, 9–10) in accordance with CAHPS recommendations29 and previous studies of global physician rating in diverse populations.30 We calculated the adjusted pseudo-r 2 and odds ratios from an ordinal logistic regression model to determine the relative contribution of each of the CAHPS-CC domains to the global physician rating and adjusted for age, sex, race/ethnicity, language, self-reported health status, and depressive symptoms. These analyses were performed in SAS 9.2.
Of 782 eligible patients approached, 91% consented to participate in the study (n=711). For this analysis, we excluded 111 patients who could not identify a primary care provider, resulting in an analytic sample of 600 participants. The sample was predominantly of Mexican descent (53%), low-income (65% uninsured or Medicaid only), and middle aged (Table 1). Excluded participants were more likely to be of Mexican descent (77.5% vs. 53%; P<0.0001), young (mean age 50 vs. 55 y; P<0.001), and fluent in Spanish (36.9% vs. 25.3%; P=0.01).
Confirmatory Factor Analysis
We were unable to evaluate the Access to Interpreter Services domain because few participants reported using interpreter services (n=36). We therefore conducted a confirmatory factor analysis of the 7 remaining domains. Sample size, means, and SDs for each domain can be found in Table 2. The χ2 test of exact model-data fit rejected the null hypothesis of exact fit (χ2 209=450.92, P<0.0001). However, 2 of the 3 descriptive model fit statistics indicated acceptable approximate fit (CFI=0.96, WRMR=1.07, and RMSEA=0.04) suggesting that the 7-factor model performed well in our sample. Standardized factor loadings for all items were significant at P<0.001. One item exhibited a negative variance estimate, a common problem among factors with a small number of items. Respecifying the model by setting the residual variance of this item to a small positive value did not substantially change the results.31 The correlations among latent constructs exhibited a wide range from essentially 0 (r=0.004) to strongly correlated (r=0.77) (Table 3).
We used Cronbach α statistics to assess the internal consistency reliability of each factor (Table 2). Three of the 7 factors had adequate to excellent reliability (Doctor Communication-Positive Behaviors, α=0.82; Trust, α=0.77; and Doctor Communication-Health Promotion, α=0.72). Reliability was <0.70 for the Doctor Communication-Negative Behaviors, Equitable Treatment, Doctor Communication-Alternative Medicine, and Shared Decision-Making domains.
On the basis of our previous work, in which the performance of some scales measuring trust in health care differed in English-speaking and Spanish-speaking populations,32 we stratified the internal consistency reliability analysis by language (Table 2). The internal consistency reliability for Doctor Communication-Negative Behaviors and Equitable Treatment was substantially worse among Spanish speakers than English speakers (α=0.45 vs. α=0.57 and α=0.35 vs. α=0.75, respectively).
Associations with Global Physician Ratings
We examined the association of each of the 7 CAHPS-CC domains with global physician rating using an ordinal logistic regression adjusted for age, sex, race/ethnicity, language, self-reported health status, and depression (Table 4). The Doctor Communication-Positive Behaviors and Trust domains exhibited the largest associations with global physician ratings, with each 1 point increase in Doctor Communication-Positive Behaviors associated with an 8% [95% confidence interval (CI), 7%–10%] increase in odds of rating one’s physician favorably and each 1 point increase in Trust associated with a 7% (95% CI, 6%–8%) increase in odds of rating one’s physician favorably (Table 4).
The confirmatory factor analysis of CAHPS-CC demonstrated that a 7-factor model performed well in a sample of predominantly uninsured and underinsured low-income adults with diabetes. This finding is consistent with the factor analysis of CAHPS-CC conducted among nonelderly enrollees in 2 Medicaid managed care programs in California and New York.20 The robustness of the 7-factor model suggests that CAHPS-CC is able to capture multiple aspects, or domains, of the multidimensional concept of “cultural competency.”
All CAHPS-CC domains were significantly and positively associated with global physician rating, with a single-point increase in scores on 2 domains (Doctor Communication-Positive Behaviors and Trust) each explaining a 7%–8% change in odds of higher global physician rating. This finding provides evidence that the domains of culturally competent care measured by CAHPS-CC are related to patient experience of physician quality, a finding also supported by the work of Weech-Maldonado et al20 in an insured population. Our analysis may underestimate the magnitude of the effect of cultural competency (as measured by CAHPS-CC) on patient experience of physician quality, as much of the variance in global physician rating in our sample was at the higher end of the scale.
The Doctor Communication-Positive Behaviors, Trust, and Doctor Communication-Health Promotion domains of CAHPS-CC showed acceptable internal consistency reliability among English and Spanish speakers. However, other CAHPS-CC domains showed poor internal consistency reliability or differed in internal consistency reliability.32 Of the 4 domains with low reliability, 3 (Equitable Treatment, Doctor Communication-Alternative Medicine, and Shared Decision-Making) were 2-item scales. Cronbach α is dependent on scale length and may underestimate reliability for 2-item scales. Further research is necessary to more adequately understand the performance of these small scales in underinsured and uninsured populations, particularly as the internal consistency reliability appears stronger for these 2-item domains in insured populations.20 In addition, the internal consistency reliability of the Shared Decision-Making domain may have been affected by its small sample size (n=204). The skip pattern structure of the ICHC survey excluded all respondents who reported that in the last 12 months their doctor did not “tell you there was more than one choice for your treatment or health care.”
Unlike the findings of Weech-Maldonado and colleagues, we found relatively low internal consistency reliability for the Doctor Communication-Negative Behavior domain. The reasons for this discrepancy are unclear. Additional studies are needed to understand whether the low internal consistency reliability we observed is an isolated finding or related to the particular sample in which we conducted our study.
Carle and Weech-Maldonado used multiple group confirmatory factor analyses to examine differences in the performance of CAHPS-CC across language (English and Spanish).32 They found general support for the equivalence of the English and Spanish versions of CAHPS-CC for the Doctor Communication-Positive Behaviors, Doctor Communication-Negative Behaviors, Doctor Communication-Health Promotion, Equitable Treatment, and Trust domains. Observed measurement bias was generally small and did not substantively impact findings for any of the domains. In contrast, we found that the internal consistency reliability estimates for the Equitable Treatment, Doctor Communication-Negative Behaviors, and Doctor Communication-Alternative Medicine domains were substantially lower among Spanish speakers than among English speakers. The difference we observed in the internal consistency reliability estimates for the Equitable Treatment domain among Spanish speakers (α=0.35) and English speakers (α=0.75) is particularly striking. Differential findings by language may be due to translation, cultural orientation of Spanish-speaking participants, or another unique characteristic of this population. Spanish-speaking patients may experience difficulty distinguishing whether inequitable treatment in the health care setting stems from their language, race, or health insurance coverage. In these individuals, Equitable Treatment questions may more logically function as individual items rather than a scale.
Because many of the physicians at the institutions where our survey was conducted are fluent Spanish speakers and monolingual speakers of other languages were excluded, we were unable to evaluate the Access to Interpreter Services domain. However, given the impact of interpretation services on clinical outcomes,33,34 we believe it is important that a cultural competency assessment include assessment of access to interpreter services.35 Previous efforts to add interpreter access questions to CAHPS have also faltered because of insufficient sample size.36 Additional psychometric analysis of this domain should be conducted in samples specifically recruited to examine access to interpreter services.
This study is limited to an assessment of the internal consistency reliability of CAHPS-CC. Future studies using larger samples must assess other forms of reliability, including test-retest reliability and interrater reliability. In addition, we administered the survey orally because of limited literacy skills of many patients in our population. Because CAHPS is traditionally self-administered, further testing is necessary to determine the acceptability and reliability of self-administering CAHPS-CC domains in a safety-net population.
Although our sample was ethnically and linguistically diverse, other large population groups represented in the United States (eg, Latinos of non-Mexican descent, Asian Americans) were not included. Our results cannot be generalized to patients of racial, ethnic, or linguistic groups not represented in our sample. There were significant differences between our included and excluded participants by race/ethnicity, language, and age. However, these differences likely reflect the demographics of patients with access to primary care in the San Francisco Bay Area and Chicago.
In summary, our study supports the 7-factor model structure of CAHPS-CC and suggests that these factors are important contributors to patients’ overall assessments of provider quality. Internal consistency reliability estimates were high in the Doctor Communication-Positive Behavior, Trust, and Doctor Communication- Health Promotion domains, but lower for the other domains. Further research is necessary to understand the discrepancies in internal consistency reliability estimates in our sample compared with Weech-Maldonado and colleagues’ sample.
Culturally competent care is thought to be a critical component of patient satisfaction and high-quality care,37 yet few patient-centered measures of cultural competence exist.12 CAHPS-CC has been proposed to examine patients’ perceptions of the cultural competence of their care, investigate the relationship of culturally competent care to clinical outcomes, and to facilitate the development of cultural competence policies for health plans. It may also provide a metric for quality-improvement interventions. Our data support these applications in the safety-net setting for the 3 domains with acceptable internal consistency reliability: Doctor Communication-Positive Behavior, Trust, and Doctor Communication-Health Promotion. The hope is that broad-scale dissemination of CAHPS-CC with CAHPS will enable improvements in health care for racially and ethnically diverse patient populations, including members of the safety-net.
1. Bigby J Cross-Cultural Medicine. 2003 Philadelphia, PA American College of Physicians
2. Fernandez A, Schillinger D, Grumbach K, et al. Physician language ability and cultural competence. An exploratory study of communication with Spanish-speaking patients. J Gen Intern Med. 2004;19:167–174
3. Brach C, Fraser I. Can cultural competency reduce racial and ethnic health disparities? A review and conceptual model. Med Care Res Rev. 2000;57(suppl 1):181–217
4. Benkert R, Peters RM, Clark R, et al. Effects of perceived racism, cultural mistrust and trust in providers on satisfaction with care. J Natl Med Assoc. 2006;98:1532–1540
5. Paez KA, Allen JK, Beach MC, et al. Physician cultural competence and patient ratings of the patient-physician relationship. J Gen Intern Med. 2009;24:495–498
6. Smedley B, Stith A, Nelson A Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. 2003 Washington, DC The National Academies Press
7. Britton CV. American Academy of Pediatrics Committee on Pediatric W. Ensuring culturally effective pediatric care: implications for education and health policy. Pediatrics. 2004;114:1677–1685
8. Betancourt JR. Cultural competence—marginal or mainstream movement? New Engl J Med. 2004;351:953–955
9. Lavizzo-Mourey R, Mackenzie ER. Cultural competence: essential measurements of quality for managed care organizations. Ann Intern Med. 1996;124:919–921
10. Anderson LM, Scrimshaw SC, Fullilove MT, et al. Culturally competent healthcare systems. A systematic review. Am J Prev Med. 2003;24(suppl 3):68–79
11. Beach MC, Price EG, Gary TL, et al. Cultural competence: a systematic review of health care provider educational interventions. Med Care. 2005;43:356–373
12. Betancourt JR, Green AR, Carrillo JE, et al. Cultural competence and health care disparities: key perspectives and trends. Health Aff (Project Hope). 2005;24:499–505
13. Kutob RM, Senf JH, Harris JM Jr.. Teaching culturally effective diabetes care: results of a randomized controlled trial. Fam Med. 2009;41:167–174
14. Byrne BM. Factor analytic models: viewing the structure of an assessment instrument from three perspectives. J Pers Assess. 2005;85:17–32
15. Browne K, Roseman D, Shaller D, et al. Analysis & commentary. Measuring patient experience as a strategy for improving primary care. Health Aff (Project Hope). 2010;29:921–925
16. Gary TL, Maiese EM, Batts-Turner M, et al. Patient satisfaction, preventive services, and emergency room use among African-Americans with type 2 diabetes. Dis Manage. 2005;8:361–371
17. Carcaise-Edinboro P, Bradley CJ. Influence of patient-provider communication on colorectal cancer screening. Med Care. 2008;46:738–745
18. Marshall GN, Morales LS, Elliott M, et al. Confirmatory factor analysis of the Consumer Assessment of Health Plans Study (CAHPS) 1.0 core survey. Psychol Assess. 2001;13:216–229
19. Hargraves JL, Hays RD, Cleary PD. Psychometric properties of the Consumer Assessment of Health Plans Study (CAHPS) 2.0 adult core survey. Health Serv Res. 2003;38(6 pt 1):1509–1527
20. Weech-Maldonado R, Carle A, Weidmer B, et al. The consumer assessment of healthcare providers and systems (CAHPS) cultural competence (CC) item set. Med Care. 2012;50(suppl 2):S22–S31.
21. Glazer J, McGuire TG, Cao Z, et al. Using global ratings of health plans to improve the quality of health care. J Health Econ. 2008;27:1182–1195
22. Schneider EC, Zaslavsky AM, Landon BE, et al. National quality monitoring of Medicare health plans: the relationship between enrollees’ reports and the quality of clinical care. Med Care. 2001;39:1313–1325
23. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606–613
24. Muthén LK, Muthén BO MPlus Statistical Analysis with Latent Variables. 2010 Los Angeles, CA Muthen and Muthen
25. Hu L, Bentler P. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modeling. 1999;6:1–55
26. SAS 9.2 for Windows [Statistical Software]. 2008 Cary, NC SAS Institute
27. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334
28. Measurement theory in the social sciences. 1981 San Francisco, CA Freeman
29. Agency for Healthcare R, Quality. CAHPS Survey and Reporting Kit, 2008
30. Weech-Maldonado R, Elliott MN, Oluwole A, et al. Survey response style and differential use of CAHPS rating scales by Hispanics. Med Care. 2008;46:963–968
31. Chen F, Bollen KA, Paxton P, et al. Improper solutions in structural equation models: causes, consequences, and strategies. Sociol Methods Res. 2001;29:468–508
32. Carle AC, Weech-Maldonado R. Does the consumer assessment of healthcare providers and systems cultural competence survey provide equivalent measurement across English and Spanish versions? Med Care. 2012;50(suppl 2):S37–S41
33. Enguidanos ER, Rosen P. Language as a factor affecting follow-up compliance from the emergency department. J Emerg Med. 1997;15:9–12
34. Karliner LS, Jacobs EA, Chen AH, et al. Do professional interpreters improve clinical care for patients with limited English proficiency? A systematic review of the literature. Health Serv Res. 2007;42:727–754
35. Schenker Y, Lo B, Ettinger KM, et al. Navigating language barriers under difficult circumstances. Ann Intern Med. 2008;149:264–269
36. Rothman AA, Park H, Hays RD, et al. Can additional patient experience items improve the reliability of and add new domains to the CAHPS hospital survey? Health Serv Res. 2008;43:2201–2222
37. Lieu TA, Finkelstein JA, Lozano P, et al. Cultural competence policies and other predictors of asthma care quality for Medicaid-insured children. Pediatrics. 2004;114:e102–e110
This article has been cited
Keywords:© 2012 Lippincott Williams & Wilkins, Inc.
cultural competency; factor analysis; minority health