The Psychological Impact of Cancer (PIC) Scale: development and comparative psychometric testing against the Mini-MAC Scale in UK and Australian cancer survivors

Abstract

Background: 

Clinicians and researchers make considerable use of both the Mental Adjustment to Cancer (MAC) Scale, and the shorter Mini-MAC, to measure psychological adjustment in cancer patients. The length of the scale is problematic when used clinically, and its psychometric properties have been criticized. This article presents two studies leading to the development of a novel scale, the Psychological Impact of Cancer (PIC) Scale, using items drawn from the MAC.

Methods: 

Study 1 used standard item-reduction techniques to shorten the Mini-MAC in a sample of 160 cancer patients of mixed diagnosis, recruited an average 46 days post-diagnosis. This resulted in a 12-item scale with a 4-factor structure, similar to that derived from a 2012 re-analysis of the Mini-MAC. Study 2 presents confirmatory factor analysis (CFA) of this new measure and tests its construct validity and test–retest reliability in a sample of 183 mixed cancer survivors.

Results: 

This study indicated that the shorter scale performed well on CFA indicators (RMSEA = .083; ECVI = .923; PNFI = .604; AGFI = .857) and tests of internal consistency (all >.623), and comparable concurrent validity with longer versions. The 4 factors were labeled cognitive distress, cognitive avoidance, emotional distress, and fighting spirit.

Conclusions: 

Given its shorter length and acceptable psychometrics, the PIC offers a useful clinical and research tool to assess the psychological impact of cancer. Psychometric properties of one subscale (fighting spirit) remain poor, but no worse than in the original scale; directions for further development of the scale are described.

1 Background

A cancer diagnosis often results in distress.^[1,2] Indeed, distress is such a pressing concern that recommendations and clinical practice guidance have been developed to advise oncology services in the screening, management, and intervention to ensure that distress is recognized and treated as efficiently as possible.^[3] Reliable and valid psychometric tools are needed to enable screening, to facilitate psychosocial treatment, and for research on the psychological impact of cancer.^[4]

One of the most commonly used measures in psychosocial oncology is the Mental Adjustment to Cancer (MAC) Scale^[5] (which is now also available in 20 languages (see www.ipos-society.org) and the shorter Mini-MAC.^[6] Although there is some debate as to the precise nature of the constructs measured (i.e., whether it assesses coping behaviors, cognitions, or some other underlying individual difference), the various subscales (of both the MAC and Mini-MAC) correlate with numerous psychosocial outcome measures,^[7–9] and are predictive of other important factors associated with illness such as information needs and information-seeking behaviours.^[10,11]

Watson and Homewood^[12] published one of the largest re-analyses of the full MAC scale to date (n = 1255), suggesting 2 higher order factors (positive and negative adjustment) as a more useful scoring algorithm compared with the 6 subscales in the original. Although there are some foreign-language validations of the shorter Mini-MAC—for example, Chinese, Greek, Italian, Korean, Norweigan, Portuguese, and Spanish^[13–19]—which demonstrate overall reasonable psychometric performance, there are few robust psychometric analyses of the original English-language format.^[20] Previous published applications of the scale have suggested that poor internal consistency of some subscales necessitates further scale development.^[15,17,20] Indeed, a number of authors suggest that when using the foreign-language translations of the scale, items perform psychometrically better when grouped according to alternative factor structures.^[13–20]

A recently published factor analysis of the English-language version of the Mini-MAC^[21] similarly suggested an alternative factor structure. Removal of 5 items yielded equivalent internal consistency and test–retest reliability, but markedly improved convergent validity with anxiety, depression, and quality of life. Furthermore, this analysis suggested that the 4 scales—cognitive distress, emotional distress, cognitive avoidance, and fighting spirit—were more parsimonious and were more easily interpreted in the context of modern psychological interventions. From a practical perspective, however, a 25-item scale is still lengthy and participant burden may reduce utility as both a clinical and research tool. Although there is a need to confirm this alternative factor structure, it is equally important that attempts are made to create as brief a measure as possible for maximized use in clinical practice.

2 Aims

This article comprises results from 2 studies. The first aim is to create a reliable, but much briefer measure of the psychological impact of cancer using items taken from the Mini-MAC. This was based on our previous work which had resulted in a revised 4-factor structure of the measure.^[21] The second study aimed to compare the psychometric properties of this newer, brief measure (The Psychological Impact of Cancer Scale) against the previously validated Mini-MAC scoring algorithms (both original 5 factor and revised 4 factor). This study pooled data from UK and Australian cancer patients for secondary statistical analysis.

3 Study 1

3.1 Method

This study was a secondary analysis of a previously published dataset in which an alternative 4-factor structure for the Mini-MAC was proposed.^[21] The same data were re-analyzed to create a new, brief measure of the psychological impact of cancer.

3.2 Recruitment and data collection

Following university and health service ethical approval, participants were recruited (by postal invitation from their cancer nurse specialist) from 3 clinical sites in North Wales, UK. Patients were excluded where life expectancy was <6 months, cancer diagnosis was recurrent, English language comprehension was poor, or where they were considered too distressed (as assessed by clinical care teams). Of 902 patients diagnosed, 554 (51%) met study entry criteria; of these, 160 participated (response rate = 35%). The sample comprised colorectal (n = 44), breast (n = 69), prostate (n = 19), and lung (n = 28) cancer patients recruited a mean of 46 days (SD = 24.8) post diagnosis. The sample included both male (n = 63) and female (n = 97) participants with a mean age of 64 years (SD = 9.97).

Informed consent was obtained using written information sheets. The Clinical Nurse Specialist (CNS) assessed patient eligibility using a standardized eligibility flowchart and subsequently distributed information packs, consent forms, and questionnaires via the post. Questionnaire packs comprised a brief sociodemographic questionnaire and the Mini-MAC.^[6]

3.3 Analysis

Using Hulbert-Williams et al's^[21] revised 4-factor structure of the Mini-MAC as a starting point, item-reduction techniques were used to create brief versions of each subscale.^[22] Our goal was to reduce the number of items in each subscale without causing too much deterioration in psychometric properties. Simulation studies show that it is difficult to cross-validate factor structures when <3 items are included in each subscale^[23]; therefore, 3 items were chosen to be the lower limit for each subscale. As the fighting spirit scale contained only 3 items, all were retained.

Some psychometricians recommend the use of Cronbach alpha as a reference criterion during item reduction.^[22] Robust criticisms have been leveled against coefficient alpha, in particular because it is biased by the number of items in the scale and must, therefore, be interpreted in the context of scale length.^[24] We therefore chose 2 pragmatic criteria. Items were removed so long as the mean inter-item correlation was not reduced by >.05 and a reduction no >.1 was seen in Cronbach alpha.

Analyses for both studies were conducted using SPSS v19 with AMOS add-on (IBM Corporation, Somers, NY).

3.4 Results

Through item reduction, a 12-item scale was developed:

• Cognitive Distress (9 items deleted): Three items cross-loaded with other factors and so were eliminated. Although this negatively impacted Cronbach alpha initially, this was improved through the elimination of 6 further items. The remaining 3 items had a Cronbach alpha of.733 (lower than the fuller scale, but still adequate) and mean inter-item of correlation of r = .493 (substantially improved from the full scale).
• Cognitive Avoidance (2 items deleted): Marginal improvement in Cronbach alpha was achieved through elimination of 1 item. Removing 1 further item caused a substantial improvement in mean inter-item correlation (r= .452), but caused a slight dip in Cronbach alpha (.709).
• Emotional Distress (2 items deleted): This was initially the weakest performing subscale. A substantially improved Cronbach alpha was achieved by eliminating 1 item, although this was attenuated slightly by the removal of a second item (.830). Overall, this resulted in an improved mean inter-item correlation for the scale (r = .619).
• Fighting Spirit: This subscale was not submitted to item-reduction, as it contained only 3 items.

With the exception of the Fighting Spirit subscale, this brief measure compares favorably with the 25-item version of the Mini-MAC from which it was developed.^[21] Only for 2 subscales is Cronbach alpha smaller, and even so it surpasses the .70 cut-off for adequacy recommended by Kline.^[25] Mean inter-item correlation is substantially improved for all subscales. Table 1 presents items included in the Psychological Impact of Cancer (PIC) Scale. As the measure deviated considerably from the full Mini-MAC, permission was given by the original authors to publish this as a new measure: the PIC Scale.

Table 1: Items included in the PIC (numbers indicate their respective question number in the original Mini-MAC).

4 Study 2

4.1 Method

Relevant data from 3 preexisting clinical studies^[10,11,21] were combined to produce a sample of sufficient size to power Structural Equation Modeling (SEM). Bentler and Chou^[26] recommend 5 participants per observed variable as the lower limit for confirmatory factor analysis.

4.1.1 Sample and procedure of data collection

An outline of the sampling strategy and procedure of data collection for each of the three samples are presented below (see also Table 2). Ethical approval was provided and participants were recruited using a fully informed procedure.

• Sample 1: A cross-sectional sample of adult cancer patients (breast, colorectal, lung and prostate) between 2 and 12 months post-diagnosis, was recruited by postal invitation from a regional cancer center in the UK (distributed directly by hospital cancer services). From 500 patients approached, 130 returned completed questionnaires (26% response rate); these were of mixed sex and had a mean age of 63.7 years (SD = 11.6). A small proportion (14.3%) had been diagnosed with palliative illness.
• Sample 2: Australian adults with primary diagnosis of lung cancer (mainly non-small cell and/or late stage disease) were recruited at first radiation oncology appointment. Eligible patients were identified in conjunction with the treating oncologists. Consenting patients were provided with study materials (in person, during routine clinical appointments) and followed up by phone if they had not mailed the questionnaire back within 2 weeks. Of 125 eligible patients, 73 returned completed questionnaires (58% response rate). The sample were mixed gender and had a mean age of 65.7 years (SD = 12.0). Participants were recruited at an early time-point from diagnosis: >50% were yet to commence treatment (42.5% with palliative intent). Fifty-nine participants also completed the Mini-MAC at a second time-point, 1 month later.
• Sample 3: Fifteen Australian adults with a primary diagnosis of lung cancer were recruited into a pilot study before data collection for sample 2; these were recruited in the same manner as sample 2, although they were not invited to participate in a follow-up time 2 questionnaire. This sample was of mixed sex and had a mean age 70.6 years (SD = 11.1).

Table 2: Overall demographic and clinical characteristics for the combined sample, including inferential tests comparing samples.

From each dataset we extracted Mini-MAC responses from all participants who had completed the questionnaire at the point of recruitment. Additionally, equivalent clinical and demographic variables were extracted for all 3 samples. From samples 1 and 2 we were able to match data on anxiety and depression (using the Hospital Anxiety and Depression Scales).^[27] Sample 1 also included data on Quality of Life (FACT-G),^[28] Perceived Stress (PSS-10^[29]), and Benefit Finding (Perceived Benefit Finding for Cancer Scale^[30]).

4.1.2 Analysis

Where <10% of data on the Mini-MAC were missing, this was replaced by expectation-maximization method; participants with higher proportions of missing data were excluded listwise. Three scoring algorithms were compared using confirmatory factor analysis: the original 5-factor Mini-MAC model (Hopelessness/Helplessness, Anxious Preoccupation, Cognitive Avoidance, Fighting Spirit, Fatalism)^[6]; the revised 4-factor Mini-MAC model (Cognitive Distress, Cognitive Avoidance, Fighting Spirit, Emotional Distress)^[21]; and the briefer PIC model derived in Study 1.

A confirmatory factor analysis (CFA) was conducted to compare the level of fit for each model. As our aim was to obtain the most parsimonious model capable of explaining the observed associations, we report Parsimonious Normed Fit Index (PNFI) and Adjusted Goodness of Fit Index (AGFI), which are adjusted for parsimony.^[28] We further report Expected Cross Validation Index (ECVI) and Root Mean Squared Error of Approximation (RMSEA), which involve estimation of population criteria, and thus better estimate the extent to which the tested models will be confirmed in other samples.^[31]

Cronbach alpha and inter-item correlation were calculated for each factor to allow for comparison of internal consistency. Test–retest reliability was analyzed by correlating baseline and 1-month follow-up responses for sample 2 participants only. Convergent validity was analyzed by correlating Mini-MAC scores with self-reported anxiety, depression, perceived stress, perceived benefit finding, and quality of life: based on the broad psychosocial oncology literature, scores more indicative of positive adjustment and psychological well-being were hypothesized to correlate with higher quality of life and benefit finding; and lower stress, anxiety and depression.

4.2 Results

4.2.1 Sample description

After exclusion of missing data, 183 participants remained in the study. There was an almost equal sex split. The sample was biased toward a greater proportion of lung cancer patients (55%). There was a good spread of participants at each disease stage and one-quarter of the sample were being treated with palliative intent. Mean age was 64.8 years (SD = 12; range 32–89), and mean time from diagnosis to recruitment was 163 days (SD = 115; range 30–577) (see Table 2).

Demographic and clinical characteristics were compared to ensure that these samples were suitable to be combined using Chi-Square and analysis of variance tests. Sample 1 included proportionally more females (mainly breast cancer patients), whereas Sample 2 recruited more males. There were no significant differences in participants’ age between samples. Sample 2 had a higher proportion of participants being treated with palliative intent, and, the 2 lung cancer samples (samples 2 and 3) had higher proportions at more advanced disease stage (χ² = 66.202, df = 2, P < .01). Mean time from diagnosis was significantly different between samples: Sample 1 participants were recruited, on average, later after diagnosis, although Sample 2 included a wider range of time interval. Although these differences indicate heterogeneity within our pooled sample, we are encouraged that the result is greater representativeness of broader cancer populations.

4.2.2 Confirmatory factor analysis

The results of the CFA and tests of internal reliability for each subscale are shown in Table 3.

Table 3: Results of the CFA, including internal consistency and mean inter-item correlations (n = 183).

Confirmatory Factor Analysis. For a well-fitting model, both RMSEA and ECVI (related statistics which estimate population parameters) should be small. For RMSEA, 0.05 is usually taken as indicating good fit, whereas 0.10 is suggestive of adequate fit: no similar cut-offs are well accepted for ECVI, although the general interpretation is that a smaller value indicates a better fit.^[32] Both the 5-factor Mini-MAC and PIC models demonstrate comparable RMSEA, with the 4-factor Mini-MAC performing less well. ECVI is considerably better for the PIC model, much better than both versions of the Mini-MAC. PNFI and AGFI should be closer to 1.00 for a well-fitting model; again, the PIC model is considerably better on both statistics than either of the longer scoring formats.

Internal Consistency. Both Cronbach alpha and mean inter-item correlation were examined as indicators of internal consistency as Cronbach alpha is known to be affected by the length of a scale.^[24]

Highest Cronbach alpha emerged for the 5-factor Mini-MAC, although this is not necessarily unexpected given the larger number of items in each subscale.^[33] The 4-factor Mini-MAC model scores equally high for cognitive distress and cognitive avoidance, but fighting spirit falls slightly below the .7 level usually considered good; emotional distress is considerably lower. The PIC performed slightly worse for cognitive distress but both cognitive avoidance and emotional distress demonstrate higher internal reliability than in the 4-factor Mini-MAC model. Although these are lower than the 5-factor Mini-MAC model, this is more likely a product of reduced item numbers rather than poorer performance of the scales per se. Again, we would draw readers’ attention to Streiner,^[34] Cortina,^[24] and Cronbach original exposition of the use of alpha^[35] to see why alpha tends to lend an appearance of acceptability to long scales, even where it is constructed from items which in truth share little variance. We suspect the length of the original Mini-MAC scale may be masking its low internal consistency.

Examining mean inter-item correlation, the 5-factor Mini-MAC structure performs well with the exception of the fatalism subscale. In all respects the PIC statistically out-performs the 4-factor Mini-MAC from which it was derived, in one case achieving a higher mean inter-item correlation than also any of the 5-factor Mini-MAC subscales.

4.2.3 Test-retest reliability and construct validity

Data from Sample 2 included a 1-month follow-up of data collection; these data were used to calculate test-retest reliability. Construct validity is assessed by testing correlation with a range of commonly used psychosocial outcome variables with data taken from Samples 1 and 3 (see Table 4).

Table 4: Summary of results from test-retest reliability analyses, and correlation with commonly used psychosocial outcome measures (sample size indicated in parentheses).

Time-lagged Spearman correlation coefficients were significant for all subscales, indicating good test-retest agreement, although effect sizes varied; for test-retest reliability, correlations >.7 are considered good, but those <.6 are considered weak and should be regarded with caution.^[25]Anxious preoccupation (r = .789) and fatalism (r = .889) within the original 5-factor Mini-MAC indicated highest test-retest reliability; these factors did not emerge in the 4-factor models and so direct comparison is not possible. The 5-factor structure also presents the scale with poorest test-retest reliability statistic—that for hopelessness/helplessness (r = .502)—although it may be that this construct is more sensitive to mood changes over time, and so retest stability may not be an appropriate psychometric dimension. The PIC model presents the most consistent set of test-retest reliabilities; in creating this brief measure, 3 subscales remain largely equivalent with regard to test-retest reliability, but cognitive distress decreased from r = .663 to.504. Although this falls below the .6 level of acceptability, this may not be a true reflection of weakness within the measure, but simply that the variable, like hopelessness/helplessness, itself is not stable over time.

Regarding concurrent validity, the PIC subscales are associated with many of the same psychological outcomes as the original Mini-MAC, and indeed the 4-factor Mini-MAC. Benefit-finding positively correlated with fighting spirit in each model. Each measure has at least 1 subscale that correlates positively with Quality of Life, although these subscales vary in construction because of the different factor solutions offered for each scoring algorithm. The PIC retains subscales which correlate with anxiety, depression, and perceived stress, although the strength of these relationships is slightly attenuated as one might expect when working with shorter psychometrics optimized for clinical utility. Based on these data, neither the Mini-MAC (5- or 4-factor) nor the PIC stands out as superior. What is perhaps more pertinent is that the pattern of significant correlation is largely identical for the full 5-factor and shorter 4-factor version of the Mini-MAC and the much briefer PIC scale: concurrent validity was not substantially affected by the item-reduction process.

5 Discussion

This article details the development and psychometric validation of a short measure of the psychological impact of cancer: the PIC Scale. This was developed using items previously forming the Mini-Mental Adjustment to Cancer Scale.^[6] Although a previous psychometric validation study of the Mini-MAC^[21] achieved somewhat improved psychometric properties, it was still a lengthy scale for clinical use (25 items) and confirmatory factor analysis was required. This study addressed both of these issues. First, by resulting in the construction of a briefer (12-item) measure of the psychological impact of cancer with a simplified four factor structure (fighting spirit, cognitive distress, cognitive avoidance, and emotional distress) as recommended by Hulbert-Williams et al.^[21] Second, it provided confirmatory psychometric analysis of the revised 4-factor Mini-MAC against the original scoring structure of the Mini-MAC.

Study 1 involved item-reduction analysis to create the PIC from the revised, 4-factor Mini-MAC. The original fighting spirit subscale only contained 3 items and so was excluded from item-reduction development. Lower Cronbach alphas for cognitive distress, cognitive avoidance, and emotional distress resulted from item reduction; however, these are still within acceptable ranges. The observed reduction in Cronbach alpha is likely a bi-product of including fewer items than a true indication of internal consistency.^[24,35] Analysis of mean inter-item correlation, an alternative indicator of internal consistency, supported this interpretation as item-reduction resulted in substantially improved mean inter-item correlation for all 3 subscales.

Pooled data from 3 studies assessing the Mini-MAC were then analyzed to compare the psychometric performance of the PIC against both the 25-item 4-factor version, and the original 29-item 5-factor version of the Mini-MAC. The results indicate that the PIC performs somewhat better than its longer counterparts on both confirmatory factor analysis indicators, and internal consistency. Convergent validity of the PIC is comparable with that of both scoring versions of the Mini-MAC here reported. These results indicate that although none of the scoring models are superior, the revised 4-factor Mini-MAC, and more-so the PIC, performs no poorer than the original 5-factor scoring model.

Alough test-retest reliability was slightly worse for the PIC, there is questionable utility of such a statistic for this particular measure. Psychological adjustment to cancer is complex^[13,36] and previous data show that psychological aspects are in constant flux through this period of stressor adaptation.^[37] It may be that lower test-retest reliability reflects temporal instability of underlying constructs rather than measurement error.

It was previously suggested that the 4-factor version of the Mini-MAC offered a conceptually improved subscale model, where it reorganized items group into variables that are more theoretically meaningful.^[21] On the basis of the data presented in this article, we maintain this position and concur that the 4 factors may hold more meaningful implementation as screening or outcome assessments in both the clinical setting and research use. Factor analysis did not offer clear confirmatory data for this 25-item model. Rather, these data suggest that the PIC, which contains these same 4 factors, offers a psychometrically improved measure than its longer counterpart. Indeed, on most criteria assessed, the PIC outperformed the original 5-factor scoring model of the Mini-MAC. We would encourage those seeking a measure of psychological adjustment to consider this version for their own use. In a move toward shorter and more parsimonious assessments in the psychosocial oncology setting, the PIC offers a valid and reliable assessment of various subtypes of psychological distress and adaptation observed within clinical settings.

Although this article offers a definitive comparison on the statistical utility of the measure, there are still areas for further development. Of the 4 components included, the fighting spirit subscale is the most weakly performing. As the only subscale phrased in terms of positive adjustment to cancer, the subscale seems to offer a unique construct not addressed by the remaining items. However, in both Hulbert-Williams et al,^[21] and this latest analysis, the reliability and validity data suggest scope for improvement. Further work exploring the importance of this construct within overall adjustment processes would be useful, as would a careful analysis of whether the performance of the subscale could be improved through revised wording of the question items. A more generic wording moving away from a focus on fighting spirit and toward a broader measure of positive adaptation and acceptance may be more useful and psychometrically valid.^[38]

These analyses suggest room for improvement with regard to convergent validity. It is important to consider, however, that with a few exceptions,^[39] attempts to validate the various versions of the Mini-MAC use other self-report measures of psychosocial outcome as the marker for convergent validity. Although we are not suggesting this is an inappropriate technique, we wish to highlight that such measures are themselves proxy indicators of well-being and clinically observable co-morbidities. It may, therefore, be appropriate to consider exploring convergent validity of this measure within a clinical setting where scores (and change over time) can be analyzed alongside real-world referral to, and treatment outcomes from, psychological services within the clinical oncology setting.

Although this study represents a useful advance on the science of measuring the psychological impact of cancer, it is not without flaws. As is often the case with survey research in psychosocial oncology,^[40] our response rates were somewhat disappointing; this is probably consequential of the impersonalized nature of postal recruitment in 2 of 4 included samples in this article. It is likely that with these lower response rates to the research invitation, the samples were relatively self-selecting and we hypothesize that this may have excluded those who were highly distressed who may perceive research participation as too additionally burdensome,^[41] a factor that we know to be problematic in broader clinical trial research.^[42] Although this is not a problem from a psychometrics point of view (there was still sufficient variance in scores to conduct the type of analysis intended), it would be interesting to explore whether the factor structure, and especially convergent validity, alter in a sample more highly distressed. Improving on previous validation work, we were able to gain a more varied sample of participants, including improved sex split, and higher proportions of more physically unwell cancer patients. This sample has the added value of including data from multiple countries, and therefore varied health-care settings (UK and Australia), although in non–English-speaking countries further validation is warranted.

In conclusion, from the original Mini-MAC items, a new, brief measure of the PIC was created. This initial analysis demonstrates that the PIC has good psychometric properties while being brief and conceptually easy to understand. Although confirmatory psychometric analysis of this new scale should be undertaken when data permit, the results from this study should provide some level of confidence for clinicians and researchers alike to use this tool to assess an individual's psychological response to cancer diagnosis and treatment.

6 Conflicts of interest statement

The authors declare no conflicts of interest.