Measuring Behavioral Interventions by Questionnaires and Prospective Diaries: An Example of Sunscreen Use

Abstract

Background.  

Diaries are used to measure behavior in intervention trials, despite the tendency of diaries to overestimate adherence.

Methods.  

We estimated the frequency of sunscreen application using prospective diaries and a questionnaire in a trial evaluating sunscreen use.

Results.  

Sunscreen application was reported more frequently in the diaries than in the questionnaire by the intervention group. This was supported by an increase in the weight of sunscreen used during the times participants were keeping diaries.

Conclusions.  

Diaries may be better as an aid to increase adherence than as a tool for its estimation. Use of diaries did not overestimate the frequency of application in the control group.

Measuring behavior of participants in epidemiologic studies continues to be a major challenge. Although for some behaviors there are biological measurements that correlate with exposure, such measurements are usually unsuitable for use in large-scale studies. We are thus forced to rely largely on self-reported behavior, accepting the limitations. Prospective diaries in which participants record their behavior have been widely used, for example to measure dietary intake or adherence in intervention trials. This is despite their demonstrated tendency to overestimate adherence with a range of different interventions. ^1–5 In addition to biased reporting, dietary studies have shown that the requirement to record a perceived “positive” behavior may result in alteration of that behavior. ⁶ Diaries have been used to record sun exposure and protection behaviors, but there have been few investigations of their accuracy in observational or intervention studies of sun protection. ^7,8 The aim of this analysis was to compare different measures of a purported cancer-preventive behavior (self-reported sunscreen use) among participants in a skin cancer prevention trial, half of whom were required to apply sunscreen daily.

Methods

The methods of the Nambour Trial of Skin Cancer Prevention have been described previously. ⁹ Briefly, this was a two-by-two factorial trial of sunscreen and beta-carotene for the prevention of skin cancer, carried out from 1992 to 1996. At baseline, 812 participants were randomized to the daily application of sunscreen (supplied by the study) to their face, head and neck, and hands and arms. Sunscreen was not supplied to the 809 participants in the control group, who were asked to use sunscreen at their usual discretionary rate. Sunscreen usage among the controls remained unchanged over the course of the trial. ¹⁰ We ascertained the amount of sunscreen used by participants in the intervention group by weighing participants’ sunscreen bottles approximately every 3 months. The average weight of sunscreen used per day was then calculated for 18 three-month periods and across the entire trial period.

All participants completed a questionnaire in 1996 in which they reported how many days each week and to which body sites they applied sunscreen. In addition, 300 randomly selected participants (150 of whom were in the treatment group) were requested to complete three 7-day Sun Diaries in which they recorded the sites and times of sunscreen application every day. Sun Diaries were completed in the autumn, winter, and summer of 1995. Nambour has a subtropical climate, with winter cooler than summer, but still sunny. At the beginning and end of each Sun Diary week, study staff weighed the sunscreen bottles used by the intervention group.

We divided the frequency of sunscreen use into five ordinal categories and used weighted kappas (κ_w) to compare the sunscreen use reported in the diaries and the questionnaires.

The average weight of sunscreen used in the three diary weeks and over the whole trial were grouped into quintiles for comparison with the questionnaire and diary data. Because of small numbers, quintiles one and two were subsequently combined, as were quintiles three and four.

Results

Questionnaires were sent to 1,464 (90%) of the trial participants, and 1,328 questionnaires (91%) were returned. Of the 300 participants selected to complete Sun Diaries, 202 people (67%) completed all three diaries and 36 (12%) completed no diaries.

Information from both the questionnaire and the diaries was available for 139 participants in the intervention group and 106 in the control group. The frequency of sunscreen application to the face was similar to the frequency of application to the hands and arms (κ_w = 0.8); hence data on facial applications were used for further analyses. The application frequency was averaged across all three diaries, as the difference in frequency among them was less than 5%.

Within the intervention group there was fairly good agreement between the sunscreen use reported in the diaries and in the questionnaire (κ_w = 0.6). However, 83% of participants reported applying sunscreen to the face on 5 or more days in the diary compared with 61% reporting this frequency in the questionnaire (Table 1). There was perfect agreement for only 45% of participants; the frequency of use calculated from the diaries was higher for 42% of people and lower for 14%. In contrast, the agreement in the control group was higher (κ_w = 0.78) with perfect agreement for 60% of the sample; 14% of participants reported a higher application frequency in the diaries than in the questionnaire, and 26% reported less frequent application.

Table 1: Comparison of the Frequency of Sunscreen Application (Days per Week) to the Face as Reported in the Sun Diaries and the Questionnaire

The discrepancy in the intervention group could have been due to overreporting of sunscreen use in the diaries or to an alteration in behavior during their completion. To explore this, we compared the weights of sunscreen used in the weeks the diaries were completed with those used in the 3-month periods containing the weeks of interest. Approximately twice as much sunscreen per day was used when the diaries were being completed compared with the periods encompassing these weeks (Table 2). If there had been no alteration in sunscreen use, approximately 8% of the total weight of sunscreen used in a 3-month period would have been used during the observed week. Instead, the median percentage of sunscreen used during the diary week was 17% in autumn, 12% in winter, and 16% in summer.

Table 2: Sunscreen Use during the Diary Weeks and the Relevent 3-Month Periods

There was a strong association between the average weight of sunscreen used (in both the diary weeks and during the whole trial) and the frequency of sunscreen applications, as reported in the questionnaire and the diary. Despite this, 44% of those categorized as low sunscreen users (based on the weight of sunscreen used when completing the diary) reported a high frequency of application in their diaries, compared with 26% reporting frequent application in the questionnaire (Table 3). Using the sunscreen weight during the entire trial for these comparisons resulted in a similar discrepancy.

Table 3: Association between Sunscreen Weight and Self-Reported Sunscreen Use

Among those who reported high sunscreen use at baseline, the median change in sunscreen use when the diaries were being completed was an increase of 1.2 gm/day. In contrast, those who were initially infrequent sunscreen users were more likely to increase their sunscreen use when under close observation (median change = +2.0 gm/day).

Discussion

In a number of trials, comparisons of self-reported diary data with objective biological measurements have demonstrated that people overreport their use of an intervention. ^11,12 Although overreporting did appear to have occurred, in the present study an additional problem was observed. During periods of more intense observation, participants in the intervention group actually altered the requested behavior (or possibly discarded sunscreen to falsify sunscreen weights). Thus, the estimates made through intermittent diaries could not be generalized beyond the limits of the observation.

The Hawthorne effect ¹³ appears to be responsible for the increased frequency of sunscreen use in the intervention group compared with the control group over the entire 5 years of the trial. These results provide clear evidence that this underlying Hawthorne effect is augmented when the intensity of observation is increased. This is consistent with studies of diet showing that the requirement to prospectively record food intake leads to marked modification of the usual diet. ⁶

Prospective diaries may not be useful as a tool for the estimation of adherence in intervention studies of sunscreen use. This may generalize to other lifestyle interventions. These results suggest that diaries could be considered as an adjunct to other methods used to maintain adherence, ¹⁴ although their longer-term effectiveness would need to be assessed for each specific intervention.