Giving Voice to the Vulnerable

The Development of a CAHPS Nursing Home Survey Measuring Family Members’ Experiences

Abstract

Background: 

To complement a nursing home resident survey, the team developed a survey asking family members about their experiences with nursing homes. Although a family member does not receive care directly from a nursing home, their experiences at the nursing home and with staff can contribute to understanding nursing home quality.

Objectives: 

To describe how the nursing home family member instrument was developed, refined, tested, and finalized.

Research Design: 

The team developed a draft survey using information from a literature review, 12 focus groups with family members involved in choosing a nursing home for someone, review of nursing home surveys, and expert/stakeholder input. The survey went through 2 rounds of cognitive interviews (n=54) and revisions and was fielded in 15 nursing homes. Data from the pilot survey (n=885) were subjected to psychometric analyses to evaluate the measurement properties of items as well as the reliability and validity of the resulting composites. On the basis of these analyses and input from experts, the survey was finalized.

Results: 

Focus groups and experts provided input into discerning important indicators of quality, although in some cases family members were not the best sources of information. Cognitive testing refined the survey and eliminated some of the proxy items. The field test analysis and input from experts eliminated 10 items. The final survey included 21 items organized into 4 composites.

Conclusions: 

This survey measures family members’ experiences of nursing home care, and the results contribute to the understanding of quality of care in nursing homes.

The Consumer Assessment of Healthcare Providers and Systems (CAHPS^®) is a comprehensive and evolving family of survey instruments designed to capture and report various aspects of consumers’ experiences with care, such as access to care and experiences with clinicians. In addition to developing surveys that capture reliable and valid data, CAHPS surveys seek to measure what consumers consider important and for which consumers are the best or only source of information.¹ CAHPS survey data are used for public reporting of health care quality information to inform consumer choice and provider quality improvement initiatives and, more recently, as input into pay-for-performance awards to hospitals.

The development of a survey that measures the experiences of family members of nursing home residents appears to represent a departure from the CAHPS surveys because CAHPS principles of survey development discourage proxy response. Research has shown that when proxy responses are used instead of consumer responses, biased results are produced.^2–8 In general, agreement between proxies and consumers is higher for observable events such as ambulation and quality of speech, and lower for events internal to the patient such as depression and general feelings of wellbeing. These findings would suggest that the perspectives of nursing home residents are the best measure of the quality of care and quality of life in a nursing home. However, approximately two thirds of nursing homes’ residents have some degree of cognitive impairment as measured by the Cognitive Performance Scale, and the extent of this cognitive impairment prevents a substantial portion of nursing home residents from assessing the quality of care provided to them.⁹ To avoid proxy response, while at the same time providing assessments of quality for residents too cognitively impaired to respond, the objective of the family member survey was to ask about nursing home experiences that could be directly experienced by the resident’s family or caretakers. Thus, the Agency for Healthcare Research and Quality (AHRQ) and the Centers for Medicare and Medicaid Services (CMS) sponsored the development of the CAHPS Nursing Home Survey: Family Member (family member survey) to complement the existing CAHPS Nursing Home Survey: Resident Survey.¹⁰ Although the family members do not receive care from the nursing home, their experiences at the nursing home and with staff are important in describing the quality of the nursing home. Family members often help choose the nursing home and continue to be involved with their family member after entry, participating in care planning and making decisions on behalf of the resident. Like the CAHPS Nursing Home Resident Surveys, the family-reported experiences of care complement other, publicly reported data such as health deficiencies, staffing, and other quality indicators (eg, percent of residents with restraints), and thus contribute to providing a comprehensive view of quality care.¹¹

The survey development process followed CAHPS principles: asking questions for which the family member is the best, if not the only, source of information regarding the quality of care in a nursing home; asking respondents to report on actual experiences, rather than perceptions of care provided to residents; and, finally, including topics that stakeholders, particularly family members, have identified as being important. One challenge in developing a survey that adheres to these principles is the need to address variation in the degree of engagement exhibited by residents’ families. Some family members rarely see nursing home residents; perhaps as infrequently as once a year.¹² Other residents either have no family members or have family members who have legal guardians make decisions on the resident’s behalf. In such situations, the guardian is the person best positioned to answer the survey questions, and so we developed a survey process that includes a means of determining who would be the most appropriate respondent.

This paper describes the procedures that were used, the issues that were identified, and the bases on which candidate questions were developed, eliminated, retained, or revised to finalize the CAHPS Nursing Home Surveys: Family Member Instrument.

METHODS

The development of the Nursing Home Family Survey followed a process similar to that used for all other CAHPS survey development projects: formative research to inform the content of the draft survey, cognitive interviews and an expert review of the draft survey, a field test of the draft survey, psychometric analysis of the field test data, further revisions based on that analysis, expert review of these revisions, and production of the final version of the survey (Fig. 1). Findings from the Nursing Home Resident Survey also informed the development of the family member survey.

FIGURE 1: CAHPS survey development process.

Formative Research

The formative research included the original literature review from resident survey study, literature review updates, responses to the call for measures in the Federal Register, and focus groups.^13–14

The team conducted focus groups with family members to identify aspects of the nursing home experience that are most important to them. Focus groups illuminate the concepts most important to consumers, thereby contributing to defining quality of care. They also help to confirm findings from the literature review, determine additional constructs,¹⁵ and identify the language consumers use to describe quality of care. Twelve focus groups were conducted in 2005 with participants who either already had placed a family member into a nursing home or were considering placing a family member into one. Groups included a mix of men and women of different race, ethnicity, and educational attainment.¹⁴

Cognitive Testing and Expert Review

The team cognitively tested the survey with 27 participants in June 2005. Cognitive testing illuminates whether respondents understand survey items consistently with their intended meaning.¹⁶ Cognitive testing identifies imprecise wording, poor ordering of items, and mismatches between survey content, and respondent knowledge and literacy, thus minimizing respondent cognitive burden. After this initial testing round, a panel of experts provided guidance and confirmed which items were valuable and measureable indicators of quality of care. The experts included representatives from the nursing home industry, regulators, quality improvement organizations, consumer advocates (including nursing home ombudsman), providers, and long-term care researchers. The team revised the items and conducted additional testing (n=27) in June 2006. The instrument was revised again for the field test.

Field Test

The survey was field tested from October 2006 to January 2007 in Texas, using a convenience sample of 15 Texas nursing homes that varied in quality as indicated by Texas’s Long Term Care Quality Reporting System, derived from deficiency and quality measures from CMS. Researchers of our team had a history of working with the Texas State Long Term Care Ombudsman program that facilitated our working with these nursing homes on the field test. Five of the 15 participating nursing homes had Quality Reporting System scores below the statewide mean, 1 facility was at the mean, and 9 exceeded the mean.

We identified an eligible respondent as the person listed by the nursing home as the party responsible for a resident who was at the facility for ≥30 days. In addition to family members and friends, guardians, medical powers of attorney, and attorneys were considered to be eligible responsible parties. Some nursing homes had >1 responsible party listed. In those cases, we selected the one with the complete contact information, or if there was >1 with complete contact information, then we randomly selected one. On the basis of a power analysis of several sampling scenarios, we determined that we would need approximately 50 complete responses per facility to detect a moderate difference between facilities (effect size of 0.50 as per Cohen¹⁷). Assuming an 80% eligibility rate and a response rate of 50%, only facilities with at least 120 beds would yield samples of this size. To provide a margin of error in our sampling design, we increased this target to 150. We selected every resident in facilities with fewer than 150 beds, and for those with >150 beds, we drew a random sample of 150 residents.

The data collection process included mailing a questionnaire to respondents, mailing a reminder postcard 2 weeks later, mailing a second survey 2 weeks after the postcard, and calling nonrespondents 2 weeks after the second survey mailing to attempt computer-assisted telephone interviews.

Psychometric Analysis

To use all available data for the respondent-level factor analysis, a multiple imputation procedure (SAS/STAT version 9.2) was used to impute data missing due to structured item nonresponse (5 imputations were requested to reflect variability in estimates). This procedure calculates the maximum-likelihood estimates of the covariance matrix under the missing-at-random assumption.^18,19 The first correlation matrix produced by PROC MI was used as an input for the factor analysis models, and the analyses were repeated on each of the remaining 4 matrices to evaluate the similarity of conclusions based on the results of analyses for each imputation. Results from these subsequent analyses are only reported if they differ from the initial analysis.

Thirty-two survey substantive items were originally hypothesized as indicators of 5 composites: getting care quickly/availability of staff (5 items), quality of care from nurses and aides (8 items), communication with nurses and aides (4 items), communication with other staff and administrators (8 items), and the nursing home environment (7 items). To assess the validity of the hypothesized factor structure, we examined goodness-of-fit results from a confirmatory factor analysis (CFA) using the PROC CALIS in SAS/STAT software. As nontrivial amounts of the data were imputed, and imputed values tend not to be whole numbers, we determined that it would be best to treat the data as continuous instead of ordinal. Recent simulation studies indicate that rounding imputed values to make them integers can bias estimations.^20,21 We examined 3 indices of how well the data fit the hypothesized structure: the root mean square error of approximation (RMSEA),²² which describes how well the model fits the population covariance matrix; the normed-fit index (NFI), which compares the hypothesized model to a “worst case scenario” model where all composite items are uncorrelated; and the comparative fit index,²³ which is a variant of the NFI that takes into account sample size and performs well even with small samples.

The initial hypothesized factor structure did not fit the data (details provided in the Results section below), and thus we followed the CFA with an exploratory factor analyses using PROC FACTOR in SAS. These models used the principal factor method with squared multiple correlations as initial communality estimates and oblique rotation (promax) with Kaiser normalization. The number of factors was determined by Guttman’s weakest lower bound²⁴ (number of factors with eigenvalues >1) in conjunction with a scree plot of the eigenvalues²⁵ and examining the pattern of factor loadings upon rotation for simple structure²⁶ (ie, assessing the degree to which the number of factors extracted based on the first 2 criteria suggested a composite structure that was conceptually interpretable).

Multitrait, multi-item analyses provided supporting evidence for alternative factor structures.^27–29 This approach entails comparing the correlations of items with their composite total (correcting for overlap³⁰) to the correlations of items with competing composites. The “scaling success” statistic is one of a number of pieces of evidence that bears on the construct validity of the proposed composites—scaling success of 100% indicates that all items correlate more highly (at least 1 SE higher) with their own composites than with competing composites. The logic for this analysis as an evaluation of construct validity follows that laid out by Campbell and Fiske.³¹ In addition, to evaluate the relative importance of the items and composites in predicting the overall evaluations of the nursing homes, we examined the correlations between the composites and 3 global ratings of the nursing home: (1) ever unhappy about the care; (2) recommend the nursing home; and (3) rating of all of the care on a scale from 0 to 10.

At the respondent level, Cronbach α³² was used as an indicator of internal consistency reliability of the composites. We also calculated interunit reliability (IUR) for each item and composite. The IUR indicates the degree to which these measures discriminate across nursing homes, and is computed as the between-group variance minus the within-group variance over the between-group variance.^33,34 As IUR is sensitive to sample size, we also calculated the intraclass coefficient (ICC), which is the between-group variance minus the within-group variance over the total variance adjusted for the average number of respondents per nursing home.³⁵ We considered an IUR of 0.70 as the desired minimum for indicating good reliability.³⁶ Using the ICC and the Spearman-Brown prophecy formula,^37,38 and taking into consideration the item-level response rates because of skip instructions, we estimated the number of complete responses, as well as the total sample size, needed for a composite or item to yield an IUR of 0.70.

RESULTS

Focus Groups

The focus groups provided information on quality of care indicators important to family members. These included cleanliness, staff training, staffing levels, quality of interactions between staff and residents, quality of food, social engagement, and privacy (Table 1). Participants were interested in measures of both good and poor quality of care; so the team created items that measured both. Indicators of good quality included concepts such as how caring and approachable the staff is (I want to see how the staff treats the patient), treating the resident as a “real person,” and physical aspects of the nursing home, such as cleanliness, appearance, and size of rooms. Indicators of poor quality included concepts such as “no screaming, yelling, or crying [at the home].”

TABLE 1: Decisions About Quality of Care Constructs Emerging From Focus Groups

Participants also identified numerous concepts that family members could not have sufficient experience or knowledge with which to form a judgment, such as the quality of food, providing medications on time, and quality of clinical care. Ultimately, concepts that were considered for item development included those that respondents could observe or experience and a few proxy concepts.

Cognitive Testing and Expert Review

One challenge was to determine whether respondents had enough information or experience to make meaningful judgments. Candidate items included both obvious proxy items and items not initially perceived as proxy items (hereafter termed “hidden proxy”) (Table 2). Obvious proxy items were eliminated or revised. Formative research findings suggested measuring the taste and nutrition of food, an obvious proxy, but the team discarded measuring these concepts because family members would not be eating the food and would not be able to judge (and, in some cases, residents may be on a low-sodium diet, which may be perceived as poor tasting). The team developed and tested several hidden proxy items, for example, items asking whether the staff showed courtesy and respect, cared about the resident, or treated the resident with kindness. Cognitive testing participants were able to provide concrete examples of observing staff performing these behaviors.

TABLE 2: Obvious and Hidden Proxy Items

Other hidden proxy items did not fare as well. Cognitive testing participants could not provide meaningful answers about seeing staff encourage residents to participate in care decisions and encourage residents to be as independent as possible. These items were abandoned because respondents did not observe the behaviors, were not able to identify the behaviors, or, regarding encouraging residents to be independent, were not applicable because the resident was too frail as to be independent at all.

Formative research indicated that adequate staffing is critical to family members, but developing items that captured the construct while being observed by the family member was a challenge. To measure adequate staffing, the team developed items that measured unmet needs. Three items asked about whether the family member performed an activity (help with eating, drinking, and toileting) because no staff was available. An additional item was developed that asked how often the respondent was able to find a nurse or aide when he or she wanted one.

Even with these staff adequacy measures, stakeholders requested an item measuring overall nursing home staffing. The team decided to add the item “…did you feel there were enough nurses and aides?” although this item did not follow usual CAHPS principles. First, family members would not necessarily know if there were enough nurses and aides and thus would not be the best source of this information. Second, it was unclear if this was really a proxy for administrative staffing records. This item was placed after all of the substantive items to minimize its influence/order effect.

Field Test Analysis

The total sample size across all homes was 1471 with homes averaging 90 eligible respondents. The final response rate was 63% (n=885), 732 by mail and 153 by phone. More than half of the respondents were adult children of the resident (55%), 11% were siblings, 9% were spouses, 14% were other family members, and 11% were someone else. Most respondents (69%) visited their family member >20 times in the last 6 months; 11% visited 11–20 times; 10% visited 6–10 times; and 7% visited 2–5 times; only 3% visited ≤1 times (and their responses were excluded from further analysis).

Most respondents reported that their family members were long-stay: 85% of respondents had family members who lived in the nursing home for ≥6 months and 84% of the respondents expected their family member to live there permanently. Finally, 63% of respondents reported that their family member had a serious memory problem.

The CFA showed that the data did not fit the hypothesized 5 factor structure (CFI and NFI <0.90; RMSEA>0.10), and so the team identified an alternative structure and a revised set of composites. The exploratory factor analyses results also did not demonstrate a clear underlying factor structure for these 32 items. Accordingly, the team conducted iterative analyses that included additional factor analysis and multitrait multi-item analysis, along with input from the team, AHRQ, CMS, and the experts.

After reviewing the analytic results, the team agreed on a final set of 21 items organized into 4 composites (Table 3). The Cronbach coefficient α indicated a high level of internal consistency for all the 4 composites. The floor effects (the percentage of respondents at the lowest possible rating/score) were quite low (0%–<1%). Likewise, the ceiling effects (the percentage of respondents who gave the highest possible rating/score), were generally low (5%–14%) for 3 of the 4 composites. The “Basic Needs” composite had substantially higher ceiling effects, which is not unusual for a topic made up entirely of dichotomous items. The correlations between the composites and the global rating of the nursing home supported the concurrent validity of all composites. Composites 1, 2, and 3 have good discriminant validity with 100% scaling success. The fourth composite’s scaling success was lower but was retained because the team and stakeholders considered the items important to measure.

TABLE 3: Final Composites and Dispositions

Three out of 4 of the composites (2, 3, and 4) had high nursing-home–level reliabilities; that is, they discriminated across nursing homes. The basic needs composite had relatively low IUR because about two thirds of the respondents legitimately skipped the survey items based on their responses to screener questions. Further, all 3 items were dichotomous, and thus the variance in the composite is naturally limited.

Table 4 summarizes the statistical and stakeholder considerations that were taken into account in determining which of the field test items should be included in the final instrument. Ten of the 32 items were abandoned. In every case, these items had either low item-total correlation, were highly correlated with >1 composite, had low IUR, and/or contained content that was included elsewhere in the survey (Table 4). One item, (did you ever have a problem?) was changed to a global measure.

TABLE 4: Final Items and Dispositions

Surprisingly, the item “…did you feel there were enough nurses and aides in this nursing home?” had the highest IUR at 0.88, with an ICC of 0.12. Although this item was considered a nonstandard CAHPS item as described earlier, it was kept because of its ability to discriminate among nursing homes.

DISCUSSION

Unlike other CAHPS surveys, the CAHPS nursing home family member survey was developed to solicit information from respondents who do not directly receive care. This survey complements the resident survey and provides a way to address the care experience of the most vulnerable residents in nursing homes; those who are too cognitively impaired to respond for themselves.

The challenges inherent in creating such a survey are to include constructs important to consumers and to ask about concepts with which family member respondents have direct experience. Focus groups findings indicated that family members defined quality of care in some ways they would not have the relevant experience with which to make meaningful judgments. But the survey addressed constructs a family member could judge.

Cognitive testing identified issues such as interpreting items inconsistently, ambiguity of terms or items, lack of direct experience relating to the question, and redundant concepts in the eye of the respondents. Testing also identified items that were proxies for constructs for which nursing home residents would be the best source of information. Items that were deemed important by family members and stakeholders were refined through cognitive testing so that respondents could evaluate the concept based on their own experience(s).

All composites but “Meeting Basic Needs” had high levels of discrimination. Surprisingly, an atypical CAHPS item (how often respondent “felt” there were enough nurses and aides) provided the highest discriminatory value. This item may have effectively captured an important construct by filtering an objective measure through feelings.

The final family member survey, using formative research to develop the draft, cognitive testing to refine the items, psychometric analyses, and technical expert input, represents a well tested, valid, and reliable survey. Providers and health policy makers can use it to measure the quality of nursing home care from the family member’s perspective. Its mail or phone administration is an advantage—it is less expensive than the in-person administration required for the long-stay resident survey. Family members often have a large decision-making role, thus information obtained from those with sufficient experience with the home provides value for individuals and other family members who will make similar decisions.

These 3 CAHPS surveys—the family member survey, and the 2 resident surveys for long-stay resident, and the discharged short-stay resident—measure different perspectives of nursing homes.³⁹ The long-stay resident provides the perspective from residents able to respond about their care; the short-stay presents a view from persons receiving rehabilitation. This survey represents the experiences of those residents receiving the most days of care and who would otherwise be unable to voice their experiences.

ACKNOWLEDGMENT

The authors thank Dr Ron Hays for his suggestions that improved the quality of this manuscript.