Development of and Field Test Results for the CAHPS PCMH Survey : Medical Care

Secondary Logo

Journal Logo

New CAHPS® Surveys

Development of and Field Test Results for the CAHPS PCMH Survey

Scholle, Sarah Hudson MPH, DrPH*; Vuong, Oanh MBA*; Ding, Lin PhD; Fry, Stephanie BA; Gallagher, Patricia PhD§; Brown, Julie A. BA; Hays, Ron D. PhD∥,¶; Cleary, Paul D. PhD#,**

Author Information
doi: 10.1097/MLR.0b013e3182610aba
  • Free

Abstract

There is a growing interest in organizing and providing medical care in a way that is consistent with the principles of the Patient-Centered Medical Home (PCMH).1 Key features of the PCMH include an orientation toward each patient’s needs, culture, values, and preferences; comprehensive, team-based care; coordination of care across all elements of the complex health care system; access to care; and a systems-based approach to quality and safety.2 In many states, medical practices are eligible for financial incentives for adopting the PCMH model and the federal government is participating in multipayer demonstrations and supporting efforts to deploy the model in community health centers.3 The National Committee for Quality Assurance’s (NCQA’s) PCMH recognition program is the most widely used method for qualifying practices for rewards in multipayer demonstrations.4 NCQA recently released updated standards for recognition of PCMHs.5

Several of the important aspects of PCMH care (eg, Whole-Person Orientation, Comprehensiveness) can best be assessed by asking patients about their care experiences and many have argued that patients’ reports about their experiences should be an integral part of assessments of the care provided by PCMHs.6,7 Currently, no patient survey assesses all of the key functions of PCMHs. The goal of this study was to develop survey questions to assess patient experiences that reflect key elements of the PCMH care model.

METHODS

Survey Development

NCQA and the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Consortium originally initiated independent efforts to develop patient survey questions for assessing PCMHs. After soliciting public input on priorities for topics that could be used to assess PCMH care and reviewing 6 existing surveys with NCQA’s stakeholder and survey expert panels, NCQA decided to use the CAHPS Clinician & Group (CG-CAHPS) 1.0 Survey as the foundation for a survey to assess PCMHs because it was already widely used, assesses several key functions of PCMHs, was the only patient experiences survey endorsed by the National Quality Forum, and is part of a family of survey instruments with comparable questions.

The CAHPS Consortium conducted focus groups to confirm the PCMH domains of interest, to learn more about what is important to patients receiving ambulatory care, and to identify the wording that patients use to talk about their experiences. The focus groups were conducted in both English and Spanish, with adult patients and parents of pediatric patients. We included both patients in medical home practices and those in primary care practices that are not categorized as medical homes. Patients with chronic conditions were represented in all focus groups. One group was composed entirely of parents of children with special health care needs. Four focus groups, with a total of 30 participants, were conducted in Boston, Seattle, Philadelphia, and a rural town in Minnesota.

Candidate PCMH items were cognitively tested with adult patients and parents of pediatric patients to ensure that they were consistently understood as intended and that survey respondents could report their experiences using the response options offered. There were 2 iterative rounds of testing in both English (total n=15) and Spanish (total n=14). In addition to the sites where the focus groups were held, cognitive interviews were also conducted in Florida, Georgia, and Maryland.

On the basis of the results of the focus groups, cognitive interviews, and stakeholder input, we developed a draft questionnaire that included 115 items about: Access, Communication, Coordination of Care, Comprehensiveness, Self-Management Support, Shared Decision-Making, Office Staff, and a global rating of the provider question and items related to eligibility and patient characteristics. We used items from the CG-CAHPS 1.0 Survey8 and other existing surveys as well as new items developed using CAHPS principles9 addressing the same content.

Field Test

Sites

Our field test was conducted in collaboration with the Massachusetts Health Quality Partners, a not-for-profit coalition of physicians, hospitals, health plans, purchasers, consumers, academics, and government agencies. Massachusetts Health Quality Partners recruited practices from health care networks, including a collaborative of safety net providers. Forty-three practices, including 10 sites serving adults and 33 serving children in the Boston area were recruited for the field test.

Sample

The sample comprised adult (age 18 or older) and pediatric (under age 18) primary care patients who had visited a study practice between July 16, 2009 and July 15, 2010. A parent or guardian was asked to complete the survey for eligible children. Patients were considered primary care patients if they received care from a physician, physician assistant, or nurse practitioner in Internal Medicine, Family Medicine, General Medicine, or Pediatrics.

Patients were assigned to sites using visit, enrollment, and site data provided by sites. All patients with an eligible visit to a site were equally likely to be sampled regardless of the number of visits, type of visit, or number of providers seen. Once the samples were drawn, patients were assigned to the provider with whom they had the most recent visit. No more than 1 patient per home address was sampled. Sampling started at sites with the smallest available sample frames to minimize the potential impact of cross-site deduplication on achieving the desired sample sizes. Sample sizes were selected that were expected to yield approximately 200 completed questionnaires per site. At practices with a majority of patients covered by Medicaid or other noncommercial payer (excluding Medicare), we sampled approximately 833 patients per site. Because we anticipated a higher response rate at practices with a majority of patients covered by commercial insurers or Medicare, we sampled approximately 667 patients per site in those practices. Twenty-three of the pediatric sites were supported by different funding than the rest of the sites; and for internal reasons their sample sizes varied as a function of number of clinicians at the practice.

Procedures

Survey Protocol

The initial questionnaire packets were mailed in November 2010 and replacement packets to nonresponders mailed in January 2011. Two mailed contacts were comparable with the data collection protocol for the CAHPS Hospital and Medicare surveys.10,11 No incentive was offered for survey completion.

Analyses

We counted any questionnaire as incomplete that had responses for less than half of the items that all respondents were eligible to answer. We used the American Association for Public Opinion Research formula RR112 for calculating response rates. We used forward cleaning to eliminate responses to items where skip instructions were not correctly followed and calculated the rate of missing data and distribution of valid responses for each item.

To identify groupings of items for multi-item composites, we conducted exploratory principle factor analyses. To identify the number of underlying dimensions, we examined multiple criteria including the eigenvalues of factors and the loadings of items on factors. After determining the number of dimensions, we performed an oblique (Promax) factor rotation. We imputed missing data in factor analyses using SAS PROC MI (SAS Version 9.2) and obtained maximum likelihood estimates of the covariance matrix using expectation-maximization algorithm (SAS PROC MI, SAS version 9.2).13 Once we made decisions about candidate composites, we calculated scores using proportional scoring and the summated rating method; that is, we calculated the mean of the responses to each item, after transforming each response to a 0–100 scale. We estimated item-scale correlations and the internal consistency reliability (Cronbach α) of multi-item scales or composites.14 As the new PCMH items were to be used with the CG-CAHPS 1.0 Survey, we analyzed the responses to all questions together. After reviewing factor analysis results, we tested the individual-level and practice-level reliabilities of different combinations of items.

For each item and composite we estimated practice-level reliability and the number of respondents required to achieve a reliability of 0.70.15 We examined practice-level reliability using the following formula:

where F is the F test of the variation among sites.

For the self-management questions, we examined the performance of survey items and potential composites for patients with and without a chronic condition. Presence of a chronic condition was determined using responses to a series of questions from the CG-CAHPS 1.0 Survey that asked whether an adult patient received care for a condition or problem that had lasted for at least 3 months or used prescription medicine to treat a condition or problem that had lasted for at least 3 months. The pediatric chronic condition identifiers require care for a physical, developmental, behavioral or emotional condition, or medication to treat it for at least 12 months.

RESULTS

Response Rates

We received 4875 completed questionnaires, for a response rate of 25% for adults and 26% for children (Table 1). Older patients and patients with more visits and chronic conditions were significantly more likely to respond to the adult survey, as were patients of physicians (as opposed to physician assistants or nurse practitioners) (P<0.001). Medicaid patients were less likely to respond than patients with commercial insurance or Medicare (P<0.0001). There was not a significant difference in the sex of responders and nonresponders (P=0.66). For the pediatric survey, response rates were significantly lower for children with Medicaid coverage, patients of a nurse practitioner, and children between the ages of 2 and 9 (P<0.001). Below, we describe the final determinations that were made for new PCMH items in the context of the CG-CAHPS 1.0 Survey for both adults (Table 2) and children (Table 3). Items that were omitted from the CAHPS PCMH survey are included in Appendix A.

T1-4
TABLE 1:
Characteristics of Adult Survey Respondents, Child Survey Respondents, and Children who were the Patients Asked About on the Child Surveys
T2-4
TABLE 2:
Results for the Adult Version of the CAHPS Clinician and Group Core Survey and New PCMH Items/Composites
T3-4
TABLE 3:
Results for the Child Version of the CAHPS Clinician and Group Core Survey and New PCMH Items/Composites

Access

The CG-CAHPS 1.0 Survey includes 5 questions about access to care. Two additional PCMH Access items were selected because of their salience in the PCMH model. One item addressing after-hours care (Got needed care on evenings, weekend, or holidays) had good practice-level reliability (0.73 for adult survey and 0.74 for pediatric survey) and correlated highly with the existing C&G Access items (r=0.46 for adult survey and r=0.50 for pediatric survey). The second item, about the number of days it took to get an appointment for urgent care, had lower practice-level reliability in the adult survey (0.60 vs. 0.89 for child survey). These 2 items were retained but not included in the Access composite. It was decided that these items did not improve the composite enough to justify including them in it. In addition, the days to urgent appointments’ question had a different response scale. Only a small number of respondents reported they had sought advice by email. The 3-item composite from the CAHPS Health Information Technology survey is recommended as a supplemental set of items where access to and use of e-mail for advice are more salient.16

Information About After-Hours Care

Two questions about a practice’s efforts to provide information about after-hours care and reminders between visits were selected. These items were initially chosen to assess other domains (Access and Communication, respectively). However, the factor analyses showed that the items did not group well with the original domains. The question about information related to after-hours care had a loading of −0.07 for the adult survey and −0.18 for the pediatric survey on the Access factor. The question about reminders between visits loaded 0.01 for the adult survey and 0.14 for the pediatric survey on the Communication factor. Also, they did not form a cohesive separate scale (Cronbach α=0.33 for adults and 0.20 for child). Their content is particularly germane to the PCMH, however, and they demonstrated good practice-level reliability (0.75 and 0.92).

Communication

The CG-CAHPS Survey includes 6 items about communication in the adult survey and 10 items in the child survey. We tested 6 new PCMH items related to communication; all of these loaded strongly on the same factor as the core CAHPS Communications items. In addition, new items developed to address “Whole-Person Orientation” also correlated strongly with the Communication items (eg, “Provider knew you as a person” correlated 0.65 with the CAHPS Core Communication composite). The site-level reliability for a composite based on the combined set of items was higher than the CAHPS core composite for both the adult and pediatric survey, but the practice-level reliability of the core items in this field test was much lower than what has been observed in other settings. For example, previous analyses of the CG-CAHPS 1.0 Survey showed reliability for the composite of 0.71, compared with 0.62 in this report, although the sample size in that study was much larger.8 Furthermore, the internal consistency reliability of the expanded set of Communication items was lower than the core Communication items for both the adult and pediatric survey. The internal consistency reliability estimate was 0.90 and 0.86 for the adult and pediatric core items, respectively, but only 0.88 and 0.86 for the composites that also included the additional PCMH items. Given concerns about survey length, we decided to retain the core Communication composite without any additional items.

Care Coordination

The CG-CAHPS 1.0 Survey includes 1 item in the care coordination domain—“Provider’s office followed up to give you results of blood test, x-ray, or other test.” Of the 7 other care coordination items tested, 2 were retained: “Provider seemed informed and up-to-date about care you got from specialists” and “Provider talked with you about your prescriptions.” These items were included because they addressed different aspects of coordination relevant to a PCMH practice. The psychometric results suggest that they do not form an internally consistent scale (Cronbach α=0.49 for adult and 0.50 for child). In addition, the item about coordination with specialty care had low practice-level reliability (0.09 in the adult survey).

Stakeholders placed a high priority on care coordination as an essential component of PCMH practices. The question about coordination with specialists (whether the provider was informed and up-to-date on care received from specialists) is especially important because of frequent reports of poor communication between primary care and specialty providers. This question has performed well in other surveys (eg, the CAHPS Health Plan survey), so we decided to retain it after making question wording and format revisions. In the field test version, this question was placed after the Access and Communication sections. A final round of cognitive interviews conducted during the field test showed that some respondents were confused about whether the “provider” was the focal provider in the first part of the survey or the specialist. On the basis of these cognitive interviews, we moved this question to immediately follow the rating of the provider and changed the wording to clarify the referent: “In the last 12 months, how often did the provider named in Question 1 seem informed and up-to-date about the care you got from specialists?” In addition, introductory text was added before subsequent questions that asked about the referent provider: “Please answer these questions about the provider named in Question 1 of this survey.”

Comprehensiveness and Whole-Person Orientation

We originally identified a set of items to assess “whole-person orientation,” however, analyses indicated that several of those items assessed aspects of communication already adequately captured in that communication composite. Other items were more correlated with items that assessed Comprehensiveness, so we developed a short composite of items assessing behavioral and/or emotional health needs for adults. This 3-item composite had good practice-level reliability (0.89) and internal consistency reliability (α=0.68) and it addresses an important but often overlooked domain of care.

The pediatric CG-CAHPS 1.0 Survey already includes 2 composites addressing development and prevention. For the child CAHPS PCMH, items related to development and prevention performed better than the behavioral health domains (Table 3). A new question about “computer and television screen time” was added to the existing pediatric CAHPS preventive care items.

Self-Management Support

We developed new PCMH items to address Self-Management Support for general health needs and for chronic conditions. Factor analysis showed that almost all items related to Self-Management Support loaded heavily on a single factor. For the adult survey, all items had a loading of 0.47 or higher except for 2 questions about setting goals and getting support of making changes to habits or lifestyle, which were not retained (loadings of 0.39 and 0.03). For the pediatric survey, all loadings were at least 0.46.

These items were administered to all respondents in the field test, but we were interested in whether some items would work better with a targeted sample of patients with chronic conditions. For the adult survey, the Self-Management Support items performed better with patients with self-reported chronic conditions, and some of the candidate items are less appropriate for a nonchronic condition population. We considered a composite targeted at individuals with chronic conditions but this would have required additional screening questions and a complex skip pattern. We chose instead to include items that were relevant to help maintain good health and improve health in the presence of chronic conditions. This composite includes 2 items: “Work with you to set specific goals for your health” and “Ask you if there are things that make it hard for you to take care of your health.” The composite had better psychometric properties in the adult survey than in the child version (α=0.62 and practice-level reliability=0.83 for the adult vs. 0.57 and 0.69 for the child version) but there was strong stakeholder support for including this composite in both surveys.

Shared Decision-Making

We tested a series of 3 items about decision-making for 2 different kinds of decisions: stopping or starting a medication and having a “surgery or procedure.” Just under half of adult respondents were eligible to answer the questions about medications; about one quarter of respondents screened into the questions about surgery or a procedure. Neither set of items had a site-level reliability of ≥0.70. However, because this topic is of critical importance to consumers, we included the items on medications decisions in the PCMH item set for adults and recommend that the responses to these items be combined to calculate a composite score. Because of the small proportion of children for whom this series of questions was relevant (about 25%) and the large number of items related to prevention and development, we did not recommend this composite as a supplement for the child survey.

Office Staff

This domain was not prioritized by our advisory groups as particularly important for the PCMH model, but the Office Staff composite is part of the CG-CAHPS 1.0 Survey. The composite had good reliability (α=0.85, practice-level reliability=0.91).

DISCUSSION

Using broad stakeholder input and both qualitative and quantitative testing, we developed a set of items that can be used to evaluate attributes of primary care that are particularly relevant for PCMHs. The CG-CAHPS 1.0 Survey with the new PCMH items assesses Access, Coordination, Shared Decision-Making, Self-Management Support, Comprehensiveness, and Information-Sharing.

For the adult survey, to assess Access, 2 new items were developed that can be used as separate items (ie, not part of the Access composite). Two new items that are not part of any composite were developed to access information provided by the practice. Two new questions about Coordination of Care can be used as separate items (ie, not part of a composite) with the CG-CAHPS 1.0 coordination question. A new 3-item composite about Comprehensiveness-Behavioral Health/Whole-Person Orientation, a 2-item composite about Self-Management Support, and a 3-item composite to assess Shared Decision-Making were developed. Comparable items were developed for the pediatric survey, although some of the Comprehensiveness items were not new. Subsequent field tests will assess modifications to existing care coordination items and possible new questions. We also will continue to assess different ways of assessing Shared Decision-Making.

Although the CG-CAHPS 1.0 Survey supplemented with these new PCMH questions is particularly germane to evaluating performance of practices that have adopted the PCMH model, the items are relevant for all primary care settings. Recommendations for survey sampling and data collection, and quality improvement efforts are available on the Agency for Healthcare Research and Quality website (https://www.cahps.ahrq.gov/Surveys-Guidance/Item-Sets/PCMH.aspx).

Input from diverse stakeholders shaped the new CAHPS PCMH content: a public comment process led by NCQA identified Communication, Shared Decision-Making, Care Coordination, and Self-Management Support among the top priorities for evaluating PCMH practices. Adding the recommended new PCMH items to the core CG-CAHPS Survey increases the survey length from 34 to 52 for the adult survey (55–66 items for the child survey). Previous research by the CAHPS Consortium suggests that the survey length should not affect survey response rates.17

Overall, the CG-CAHPS PCMH items and composites have sound psychometric properties. However, some items were included because of the salience to the PCMH model. For example, we chose to include the Shared Decision-Making composite in the adult survey despite internal consistency and practice-level reliability of <0.70. We anticipate that as practices focus more on involving patients in decision-making (one of the tenets of the PCMH model), more patients will report that they discussed starting or stopping a medication with their providers (thus increasing the eligible denominator sizes for the measure) and these items will better discriminate among practices that do and do not offer discussion of the benefits and drawbacks of medications. The CAHPS Consortium is continuing to do research on the PCMH survey and will reevaluate the items as more information becomes available.

Patient survey data now are considered an integral part of the evaluation of health plans and hospitals; they should also be included in the evaluation of care at the provider and practice level. Although the CAHPS PCMH survey was developed to evaluate care provided by PCMHs, the content is equally relevant for Accountable Care Organizations and health plans.

This study had several potential limitations. The response rates were lower than seen in some other surveys, although they are similar to response rates in some implementations of CAHPS surveys. Fielding during the winter holidays was necessary to stay on schedule and may have contributed to lower response rates. Also, limited resources and time available for the field test precluded the use of telephone follow-up which typically boosts response rates, particularly in sites serving large numbers of Medicaid patients.17–19 Nevertheless, a low response rate, while potentially biasing the distribution of responses, should not have a major effect on analyses of associations, the primary focus of these analyses.

Another potential limitation is that the field test was only conducted in the Boston area. Thus, the results may not generalize to other areas. However, we think that although the distribution of responses and practice-level reliability may be different in other areas, the associations among items are less likely to vary across geographic regions.

Patient-centeredness is a critical domain of health care quality. Despite its potential limitations, this study provides support for the reliability and validity of the CAHPS PCMH survey. This survey can be used to evaluate important aspects of primary care that are consistent with the PCMH model of care and with other efforts to reform the health care delivery system.

ACKNOWLEDGMENT

The authors thank Carla Zema for her help in coordinating and guiding the instrument development process.

APPENDIX A. Items Tested But Not Included In CG-CAHPS Patient-centered Medical Home Survey

T4-4
Table:
No title available.

REFERENCES

1. Joint Principles of the Patient-Centered Medical Home. 2007. Available at: http://www.aafp.org/online/etc/medialib/aafp_org/documents/policy/fed/jointprinciplespcmh0207.Par.0001.File.tmp/022107medicalhome.pdf. Accessed April 9, 2012
2. What is a PCMH? 2012. Available at: http://pcmh.ahrq.gov/portal/server.pt/community/pcmh__home/1483/what_is_pcmh_. Accessed June 18, 2012
3. Pilots & Demonstrations in the United States. Available at: http://www.pcpcc.net/pcpcc-pilot-projects. Accessed August 15, 2010
4. Bitton A, Martin C, Landon BE. A nationwide survey of patient-centered medical home demonstration projects. J Gen Intern Med. 2010;25:584–592
5. NCQA’s patient-centered medical home (PCMH) 2011 Standards. Available at: http://www.ncqa.org/tabid/631/Default.aspx. Accessed December 2, 2011
6. Berenson RA, Hammons T, Gans DN, et al. A house is not a home: keeping patients at the center of practice redesign. Health Aff. 2008;27:1219–1230
7. O’Malley AS, Peikes D, Ginsburg PB Qualifying a Physician Practice as a Medical Home Policy Perspective. 2008 Washington, DC Center for Studying Health System Change
8. Dyer N, Sorra JS, Smith SA, et al. Psychometric properties of the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician and Group survey. Submitted for publication
9. Crofton C, Lubalin JS, Darby C. Foreword. Med Care. 1999;37:MS1–MS9
10. Goldstein E, Cleary PD, Langwell KM, et al. Medicare managed care CAHPS: a tool for performance improvement. Health Care Finan Rev. 2001;22:101–107
11. Keller S, O’Malley AJ, Hays RD, et al. Methods used to streamline the CAHPS® hospital survey. Health Serv Res. 2005;40:2057–2077
12. American Association for Public Opinion Research. Standard Definitions: Final Dispositions for Case Codes and Outcome Rates for Surveys. 20043rd ed Lenexa, KS AAPOR
13. Dempster AP, Laird NM, Rubin DB. Maximum likelihood from incomplete data via the EM algorithm. J Royal Stat Soc. 1977;39:1–22
14. Nunnally JC Psychometric Theory. 19943rd ed New York McGraw-Hill
15. Clark E. Spearman-Brown formula applied to ratings of personality traits. J Educ Psychol. 1935;26:552–555
16. McInnes DK, Brown JA, Hays RD, et al. Development and Assessment of CAHPS® Questions to Assess the Impact of Health Information Technology on Patient Experiences with Care. Submitted for publication 2012
17. Gallagher PM, Fowler FJ Jr. Notes from the field: experiments in influencing response rates from Medicaid enrollees. Proceedings. 2000 Data Quality Section, Alexandria, VA American Statistical Association:971–976
18. Brown JA, Nederend SE, Hays RD, et al. Special issues in assessing care of Medicaid recipients. Med Care. 1999;37:MS79–MS88
19. Gallagher PM, Fowler FJJ, Stringfellow VL. The nature of nonresponse in a Medicaid survey: causes and consequences. J Off Stat. 2005;21:73–87
Keywords:

patient-centered medical homes; PCMH; patient reports about care; health care quality; patient experiences survey; CAHPS

© 2012 Lippincott Williams & Wilkins, Inc.