Is Measurement Error Altered by Participation in a Physical Activity Intervention?

Abstract

Purpose 

There is no “gold standard” measure for moderate to vigorous physical activity (MVPA); some error is inherent to self-report and device-based measures. Few studies have examined agreement between self-report and device-based measures in the intervention trial context or whether the difference between measures is influenced by intervention participation.

Methods 

MVPA was measured at baseline and after 6 months by Active Australia Survey (AAS) and by the GT1M accelerometer (≥1952 counts per minute) in the intervention (n = 135) and usual care control (n = 141) participants of a randomized trial targeting weight loss by MVPA increases and energy intake reductions in adults with type 2 diabetes. Agreement (for each group at each assessment) was examined using the Bland–Altman approach and regression-based modeling. Because the differences between MVPA measures varied with average values ([AAS + GT1M] / 2), they were examined as a percentage of average physical activity. t-tests were used to assess unadjusted group differences and changes over time. ANCOVA models tested intervention effects on measurement error at follow-up, adjusted for baseline.

Results 

Agreement worsened, and variability in the difference measures became greater, as the average amount of MVPA increased. Measurement error differed significantly between groups at follow-up (P = 0.010) but not at baseline (P = 0.157) and changed significantly within the intervention group (P = 0.001) but not the control group (P = 0.164). There was a statistically significant effect of the intervention on measurement error (P = 0.026).

Conclusions 

Measurement error of self-report relative to the accelerometer appeared to be affected by intervention. Because measurement error cannot be definitively attributed to self-report or accelerometer, it would be prudent to measure both in future studies.

Physical inactivity is a modifiable risk factor for chronic diseases including type 2 diabetes, cardiovascular disease, and some cancers. The development of effective, broad-reaching physical activity interventions is a population health priority, and to properly evaluate such interventions, valid and reliable measures are needed. There is no “gold standard” measure for moderate to vigorous physical activity (MVPA) (¹¹); interventions most commonly use self-report but increasingly incorporate objective monitoring. A systematic review (¹⁷) has shown that self-report typically has only low to moderate correlations with measures from monitoring devices, including accelerometers, and can yield much lower or higher estimates of physical activity than monitoring devices. The substantial disagreement between self-report and objective monitoring has implications for estimating intervention effects because there is reason to suspect that measurement error may be differential between intervention and control groups after intervention (²²).

A study of adolescent girls (²²) observed that measurement error (self-report relative to the accelerometer) was differential at follow-up but not at baseline and explored social desirabiltiy bias as a possible reason. Social desirability bias may cause interventions to appear more effective by self-report than they really were if, after intervention, participants who have received an intervention that involves intensive contact with intervention staff are more inclined to report their behavior in a socially desirable manner than controls who have had minimal or no contact with staff. Self-report may alternatively underestimate intervention effects, for example, if certain intervention components (e.g., self-monitoring) increase intervention participants’ awareness of their physical activity and thereby selectively improve their reporting accuracy (i.e., reduce their overreporting) (^23,26). Further, disagreement between self-report and accelerometer may be altered by interventions that increase physical activity because it is often reported to be worse at higher levels of physical activity than at lower levels (^2,9,21).

Whether measurement error is affected by intervention in adults, and the implications this has for trial outcomes remains to be established. In the context of a behaviorally based weight loss intervention trial (¹⁰), we examined the agreement between self-reported and accelerometer-measured MVPA at baseline and 6 months within the control and intervention groups. We further examined variation over time, differences between groups, and intervention effects on the difference between the measures of MVPA. We also examined the potential implications of any alteration in measurement error over the course of the intervention by comparing the trial’s intervention effects according to self-report and accelerometer.

METHODS

This study uses data from Living Well with Diabetes (LWWD), a randomized controlled trial of a telephone-delivered weight loss intervention targeting increased physical activity and reductions in energy intake. The study protocol has been reported previously (¹⁰). Recruitment of primary care practices commenced in October 2008 with participant recruitment occurring between February 2009 and April 2011. Data collection occurred at baseline and 6 months and is ongoing for 12, 18, and 24 months. Informed consent was obtained from all participants, and the University of Queensland Behavioural and Social Sciences Ethics Review Committee granted ethics approval for the trial.

Nine general practices were recruited in Logan, a diverse socioeconomic area approximately 30 km south of Brisbane, Australia. Within practices, 1407 eligible patients (i.e., diagnosed type 2 diabetes, age 20–75 yr, and having a listed telephone number) were identified using electronic medical records. Patients not excluded by GP screening for contraindications to unsupervised physical activity (n = 908) were posted study materials by the GP and were followed up by study staff to determine eligibility and to solicit informed consent unless they had declined further contact (n = 206). To be eligible, participants needed to be overweight or obese (body mass index [BMI] ≥ 25.0 kg·m⁻²) (²⁸) or inactive (<5 d·wk⁻¹ of ≥30 min of MVPA) (³) and could not report current cancer treatment other than endocrine therapy, prior or planned bariatric surgery, use of weight loss medications (e.g., Orlistat), or communication difficulties relevant to receiving the intervention. Of those reached by telephone and deemed eligible (n = 420), 302 (71.9%) agreed to participate, completed the baseline assessment and were randomized to receive either telephone counseling (n = 151) or usual care (n = 151). Randomization was by the minimization method (^1,19) using the MINIM program (www.sghms.ac.uk/depts/phs/guide/randser.htm), aiming to balance treatment groups across the following prognostic factors (without weighting for importance): gender, age (≥55 yr), BMI (≥40 kg·m⁻²), HbA_1c (≥8%), self-reported MVPA (≥150 min and ≥5 d·wk⁻¹), and diabetes management (insulin or combination therapy/traditional oral hypoglycemic medications/new diabetes agents, i.e., Exenatide or Sitagliptin/lifestyle alone).

Telephone Counseling Weight Loss Intervention

Intervention participants were assigned at least 14 calls over the intensive first 6 months of an 18-month telephone-delivered weight loss counseling intervention, described in detail elsewhere (¹⁰). Participants were provided with a detailed workbook, pedometer, a self-monitoring “tracker” to record daily physical activity and food intake and detailed feedback after each assessment that highlighted discrepancies between their reported behavior (including physical activity) and the study targets. The physical activity target was at least 210 min of planned, moderate-intensity physical activity per week (≥30 min·d⁻¹ every day). A further two to three resistance training sessions per week were also recommended.

Usual Care

After each assessment, participants in the control group received publically available brochures addressing health behaviors important for diabetes self-management (e.g., losing weight, alcohol consumption) and a letter thanking them for their participation. The letter included brief feedback summarizing their assessment results, including their physical activity levels but avoided any comparison with recommendations. The attention provided to the control group was designed to minimize attrition.

Data Collection and Processing

Self-reported physical activity.

Self-reported physical activity was assessed by computer-assisted telephone interview using the Active Australia Survey (AAS), an eight-item questionnaire that is used extensively in Australian research (⁴). Participants were asked to report the total time engaged in gardening, walking (for ≥10 min at a time), and moderate- and vigorous-intensity physical activities in the past week. Data were processed according to the AAS protocol (⁴), but without doubling of the duration of vigorous activity, to facilitate comparability with the accelerometer data. Total weekly MVPA was calculated as the summed durations of walking and moderate- and vigorous-intensity activities (excluding gardening), first truncating each activity at 840 min·wk⁻¹ and truncating total MVPA at 1680 min·wk⁻¹. In assessing the duration of MVPA, the AAS has test–retest reliability that is comparable with the US Behavioral Risk Factor Surveillance Survey and the International Physical Activity Questionnaire (⁹) and has acceptable agreement with these measures (⁷) and moderate correlation with accelerometer, ranging from 0.29 (95% confidence interval [CI] = 0.16–0.41) (²⁴) to 0.52 (95% CI = not reported) (⁸). Intraclass correlations reported for test–retest range from 0.32 (95% CI = 0.09–0.52) (¹⁶) to 0.64 (95% CI = 0.57–0.70) (⁹).

Accelerometer measurement of physical activity.

ActiGraph GT1M accelerometers (ActiGraph, LLC, Fort Walton Beach, FL), worn on elasticized belts around participants’ waists, positioned on the right midaxillary line, were used to monitor MVPA objectively. Participants were instructed to wear the accelerometers during all waking hours for seven consecutive days, except during water-based activities (e.g., showering, swimming). The accelerometers were set to collect data in 60-s epochs. Participants reported the times the accelerometer was donned and removed each day in a wearing log. Research staff identified nonwear periods by comparing participants’ log data with the precise times movement began and ceased, to overcome limitations in relying exclusively on wear time algorithms (²⁷) or self-report. Days were considered valid that had at least 10 h of wear and no implausibly high counts (≥20,000 counts per minute [cpm]). Accelerometer compliance was good, with almost all participants providing at least four valid days of wear (98.1% at baseline and 97.1% at 6 months). Mean ± SD wear time on valid days was 13.5 ± 1.7 h·d⁻¹ at baseline and very similar at 6 months (13.7 ± 1.7 h·d⁻¹). A program adapted from the National Cancer Institute was used to process and summarize the data (¹⁵). For each valid day, MVPA was identified as the number of minutes at or above the commonly used cut point of 1952 cpm (¹²). Average MVPA on valid days was multiplied by 7 to yield a weekly estimate. Sensitivity analyses examined alternative MVPA classifications: one of the lowest (≥574 cpm) and highest 3-MET cut points for MVPA (≥2743 cpm) (²⁵).

Anthropometric and demographic measures.

Anthropometric and demographic data were collected at baseline. BMI was determined from nurse-assessed height and weight; demographic information was collected via computer-assisted telephone interview by research staff blinded to study group allocation (¹⁰).

Statistical Analyses

Data analyses were performed in SPSS (IBM SPSS v.20; SPSS) and significance was set at P < 0.05. Analyses included the n = 272 participants with physical activity data from both self-report and accelerometer at both baseline and 6 months. Agreement of self-report with accelerometer MVPA was assessed using the Bland–Altman approach. Scatterplots depicting the differences between measures (AAS − GT1M) across the average value of the two measures ([AAS + GT1M] / 2) (⁵) were created separately for each group at each time point. As regression (^5,6) showed mean differences and variability both increased significantly (P < 0.05) with average values, and log transformation failed to resolve this issue, the scatterplots show mean differences and 95% limits of agreement (LoA) in terms of the regression equations. For this same reason, when examining changes, group differences and intervention effects on measurement error, the difference between the MVPA measures was examined as a percentage of average values (100 × [AAS − GT1M] / [AAS + GT1M] / 2). This outcome followed an approximately normal distribution. Unadjusted differences between groups in means (systematic error) were examined using in dependent-samples t-tests; Levene test was used to test differences between groups in variance (random error). Changes over time within groups were examined by paired t-tests.

ANCOVA models were used to examine intervention effects in terms of group differences in measurement error at follow-up, adjusted for baseline measurement error. To examine the potential effect of measurement error on results from the intervention, the trial’s intervention effects for MVPA by self-report and by accelerometer are reported. Although AAS and GT1M data were skewed, change scores (6 months − baseline) approximated normality. ANCOVA models examined change in MVPA, adjusted for baseline MVPA. From these models, group differences and the adjusted mean change within groups are reported, with 95% CI. All models showed minimal nonnormality and heteroscedasticity. Sensitivity analyses were conducted to examine whether the main study conclusions regarding changes, group differences, or intervention effects on measurement error were robust across a wide range of potential cut points for MVPA.

RESULTS

Participants.

Characteristics of study participants are presented in Table 1. Trial participants had a mean ± SD age of 58.3 ± 8.6 yr, were nearly all either overweight (26.2%) or obese (68.2%), mostly Caucasian (87.4%), and were more commonly male (56.3%) than female. Median duration of diabetes was 5 yr (25th, 75th percentile: 2, 10 yr). The trial had a high response rate (72% of those reached and eligible), and participants did not differ from nonparticipants on most variables (see Table, Supplemental Digital Content 1, https://links.lww.com/MSS/A245, Comparison of study participants with nonparticipants on demographic, health, and behavioral characteristics). However, participants had significantly higher BMI, shorter diabetes duration, more education, and were underrepresented by ex-smokers and overrepresented by never smokers. Loss to follow-up was low (12.6% intervention and 7.3% control) and was nondifferential. Most characteristics did not differ between those with complete (n = 272) and incomplete data (n = 30) (see Table, Supplemental Digital Content 2, https://links.lww.com/MSS/A246, Comparison of completers with those missing 6-month study outcomes); the only statistically significant differences were for use of insulin (P = 0.023) and for smoking status (P = 0.036), with those on insulin and current smokers tending to lack follow-up data.

TABLE 1: Baseline characteristics of study participants randomized to telephone counseling (n = 151) and usual care (n = 151).

Agreement of the AAS with GT1M at baseline and 6 months.

The Bland–Altman plots (Figs. 1A–D) show the agreement of self-reported MVPA (AAS) with accelerometer-measured MVPA (GT1M) at baseline in (A) intervention and (B) control groups and at 6 months in (C) intervention and (D) control groups. For each group at each time point, as the average value of the two measures ([AAS + GT1M] / 2) increased, the mean difference (AAS − GT1M) and the variability in the differences also increased significantly (all comparisons P < 0.05). This can be seen in all four plots in the slope of the lines for the mean differences and 95% LoA, which increase and widen, respectively, as average values increase. The mean differences and 95% LoA indicated substantial differences between self-report and accelerometer; however, the plot for the intervention group at follow-up showed smaller mean differences and narrower limits of agreement than the other plots did. For example, Fig. 1A shows that, for the intervention group at baseline, we would expect a mean difference of 104.1 min·wk⁻¹ (i.e., 68.1 + 0.6 × 60) with 95% LoA of −190.5 to 120.2 min·wk⁻¹ for those with 60 min·wk⁻¹ as their average MVPA ([AAS + GTIM] / 2) and a much larger mean difference (194.1 min·wk⁻¹) and wider LoA (−445.5 to 540.2 min·wk⁻¹) in those with much higher physical activity levels (average values of 210 min·wk⁻¹). By contrast, at follow-up in the intervention group, the mean differences and LoA were much smaller for a given amount of average MVPA, e.g., 19.9 min·wk⁻¹ (95% LoA: −187.7, 195.8 min·wk⁻¹) for those with average values of 60 min·wk⁻¹ and 49.9 min·wk⁻¹ (95% LoA: −292.7, 360.8 min·wk⁻¹) for those with average values of 210 min·wk⁻¹.

FIGURE 1: Bland–Altman plots of the difference between self-reported MVPA (Active Australia Survey [AAS]) and accelerometer (GT1M) across average values (mean of AAS and GT1M) at baseline for (A) intervention and (B) control and at 6 months for (C) intervention and (D) control.

Effect of the intervention on measurement error (difference between AAS and GT1M).

According to unadjusted findings, measurement error (as a percentage of average MVPA) showed no evidence of change between baseline and 6-month follow-up within the control group (P = 0.164), but it did change from approximately 19% lower to 10% higher within the intervention group (P = 0.001; Table 2). Although there was no evidence of a difference between groups in mean measurement error at baseline (P = 0.157), at follow-up, the mean measurement error differed significantly between groups (P = 0.010). The difference was such that, at follow-up, control group participants reported significantly less activity than was measured by the accelerometer (−24.2%; 95% CI = −43.7% to −4.7%), but intervention participants did not do so (10.3%; 95% CI = −6.7% to 27.3%). Coinciding with the differences in mean measurement error, at baseline, there was no evidence of a difference between groups in the variability in measurement error (as a percentage of average values; P = 0.497), although at follow-up, the variability in measurement error was significantly less for the intervention than for the control group (P = 0.037). ANCOVA models showed a significant intervention effect for measurement error. At follow-up, adjusted for baseline, the intervention and control groups differed significantly in measurement error (mean difference for intervention − control = 28.0% of average MPVA; 95% CI = 3.3–52.7; P = 0.026). As with the unadjusted results, the adjusted means showed the self-report to be significantly lower than accelerometer measures in the control group (−21.0% of average values; 95% CI = −38.2 to −3.9) but not in the intervention group (6.9% of average values; 95% CI = −10.7 to 24.6).

TABLE 2: Effect of the intervention on the difference in moderate to vigorous physical activity (MVPA) between the Active Australia Survey (AAS) and GT1M accelerometer as a percentage of average MVPA, using three different cut points for MVPA.

Sensitivity analyses.

Results of the sensitivity analyses (Table 2) showed that the definition of accelerometer MVPA had a dramatic effect on whether self-report tended to underestimate or overestimate relative to the accelerometer and some effect on study conclusions regarding whether measurement error varied by group, over time, or was affected by the intervention. The least discrepancy between self-report and accelerometer was seen in the main analysis. Self-report consistently significantly underestimated relative to the accelerometer when using the very low cut point for MVPA, by an unrealistic amount (i.e., up to seven times the average amount of MVPA). Self-report consistently significantly overestimated relative to the accelerometer when using the highest cut point for MVPA. Significant changes over time in the intervention group and significant differences between groups at follow-up (unadjusted) were replicated with both lower and higher MVPA cut points; however, significant control group changes in measurement error were seen with high and low cut points (but not the main analysis). The intervention effect on measurement error was significant in the main analysis and using the low cut point (P < 0.001) but not using the high cut point for MVPA (P = 0.083).

Intervention effects for physical activity by self-report and accelerometer.

Adjusted for baseline values, significant mean changes in MVPA (min·wk⁻¹) within the intervention group were slightly less according to the accelerometer (44.4; 95% CI = 25.0–63.9) than self-report (72.3; 95% CI = 42.0–102.6) (see Table, Supplemental Digital Content 3, https://links.lww.com/MSS/A247, Effect of the intervention on MVPA measured by the AAS and GT1M accelerometer). However, the nonsignificant changes within the control group tended to be less according to the accelerometer (1.5; 95% CI = −17.4 to 20.4) than by self-report (19.6; 95% CI = −10.1 to 49.2), resulting in intervention effects that appeared only slightly smaller for accelerometer-measured MVPA (42.9; 95% CI = 15.5–70.1; P = 0.002) than for self-reported MVPA (52.7; 95% CI = 10.4–95.1; P = 0.015).

DISCUSSION

Self-report has limited agreement with accelerometer physical activity (¹⁸). For trial results to be unbiased despite measurement error requires that the error remain constant over the study duration and be equal across treatment groups. This study observed a significant intervention effect on measurement error (of self-report relative to the accelerometer) in adults and provided some evidence that measurement error can vary significantly over the course of an intervention and is not equal for intervention and control groups at follow-up. The caution that measurement error relative to the accelerometer may not be constant over time and nondifferential by group is likely to apply across most cut points used to assess MVPA in the literature because both very low and very high cut points for MVPA replicated the changes over time within the intervention group and the group differences at follow-up (unadjusted) and the observed intervention effect on measurement error were significant both in the main analysis and with the low MVPA cut point.

This trial’s findings add to the few prior studies that have reported on potential changes between before and after the intervention in the correlation of self-report with device-based MVPA measures in adults (^14,20,22). Two studies have noted a weakening in correlation over the course of the intervention, within both treatment groups combined (^14,20). Another has observed a strengthening in correlation within the control group and no change in correlation for the intervention group (²²). However, correlations are not sensitive to shifts in systematic error. The studies suggest random error may vary over time during interventions, with the present study providing further evidence suggestive that systematic error (mean differences) also cannot safely be assumed to remain constant during intervention.

Mechanisms for variation in physical activity measurement error.

This study, and others comparing self-report MVPA with accelerometer (^2,7,21), have noted both systematic error (mean differences) and random error (limits of agreement) increase with average MVPA (average of self-report and accelerometer). This may contribute to the patterning of measurement error over the intervention, which was largely consistent with the patterning of physical activity, i.e., increases in the intervention group (but not in controls) and differences between groups at follow-up (but not baseline). Observations from the study regarding random error (seen in the limits of agreement and the variability in measurement error) were consistent with the idea that intervention participants may experience improved reporting accuracy after exposure to intervention, which included self-monitoring and education on what constitutes “moderate” and “vigorous” physical activity.

Issues with self-report measurement, such as misinterpretation of the survey questions, inaccurate recall, and/or social desirability bias, and issues with using accelerometers in free-living populations could both contribute to the measurement error observed in this study. The mean differences, although substantial, may or may not reflect underreporting or overreporting by participants because the direction of the mean differences was very sensitive to the choice of accelerometer data treatment and therefore could just as easily represent overdetection or underdetection of MVPA by the accelerometer. Despite their regular use as referent assessment methods in validity studies, accelerometers are not a gold standard for physical activity measurement. The proportional increase in measurement error with increasing amount of physical activity could arise from errors in self-report, if, for example, it is easier for inactive participants to recall and report infrequent, discrete bouts of physical activity than it is for active participants to recall physical activity that is completed often and which may be variable in terms of duration, frequency, and domain. The approach of applying cut points to define MVPA from accelerometer data could also explain the proportionality. Cut points are derived from equations that use accelerometer counts to predict energy expenditure (¹³) and do so very accurately for treadmill-based activities but poorly for activities of daily living (¹³). The imperfect sensitivity and specificity of any cut point for detecting each true minute of MPVA will naturally lead to an increase in the total amount of error (min) as the prevalence of true minutes of MVPA increases. The type of activity may also be relevant because the most active participants may also be the most likely to engage in activities that are not well captured by accelerometers (such as cycling and swimming).

Implications for physical activity interventions.

The evidence supporting that measurement error may be affected by intervention, may vary over time, and may be differential by treatment allocation has implications for comparing results from trials that use self-report exclusively with trials that use accelerometers. In the LWWD trial, physical activity changes over time within groups, and intervention effects were slightly larger when measured by self-report than by accelerometer, with an additional mean change of 18.1 min·wk⁻¹ for controls, 27.9 min·wk⁻¹ for intervention participants, and an additional 9.8 min·d⁻¹ for the intervention effect. Both measures led to the same conclusions regarding statistical significance; however, the discrepancy in effect size was sufficient to classify change as clinically relevant in the intervention group by self-report and not by accelerometer, using the LWWD a priori ≥60 min·wk⁻¹ definition. The only other trial to compare intervention effects obtained by self-report (7-d physical activity recall) with accelerometer (the RT3 triaxial monitor) for the same sample of study participants (²⁰) similarly found no difference to the statistical conclusions regarding intervention effects (i.e., both not significant) but tended to see intervention effects more in favor of the intervention group and larger changes within both intervention and control groups by self-report measures than by accelerometer measures. The results of physical activity trials may appear slightly more promising using when self-report than when using accelerometers, although with bias potentially arising from either measurement tool; it remains unknown whether results are exaggerated by self-report or understated by accelerometers.

Limitations.

It should be noted that the trial was not designed and powered to detect intervention effects on measurement error and resulted in effects with wide 95% CI. Further examination in larger trials may reveal more precise evidence regarding the extent to which measurement error is affected by intervention. The accelerometer monitoring period was scheduled without consideration of the AAS recall period; this is unlikely to affect conclusions regarding intervention effects on measurement error, as the scheduling was consistent over time and for both groups, but may have weakened the overall extent of agreement seen in this study. Importantly, because both intervention and control participants knew their activity was being monitored objectively, some biases in reporting may have been minimized in this study (e.g., such as those evidenced by sizeable self-reported improvements in unmonitored control groups). Thus, the extent of discrepancy of self-report against accelerometer in this study may be an understatement of the potential discrepancy between findings from interventions that use self-report (without objective monitoring) versus interventions that use accelerometers.

CONCLUSIONS

This trial showed evidence that measurement error of self-report (AAS) relative to the accelerometer was substantial and was affected by intervention, either directly or indirectly through the effect of intervention on the amount of physical activity. As a result, intervention trial outcomes were estimated as slightly stronger by self-report than by accelerometer. With errors in either or both assessment tools potentially contributing to these findings, it would be prudent for interventions to measure physical activity with both self-report and device-based measures.

This study was supported by grants from the Australian National Health and Medical Research Council (NHMRC) and the Diabetes Australia Research Trust. Authors were supported by Queensland Health core infrastructure funding (E.W.); a National Heart Foundation scholarship (L.W.); and, NHMRC Fellowships (E.E., N.O., and M.R.).

The authors thank the patients, general practitioners, and practice staff of the Greater Metro South Brisbane Medicare Local (Logan, Australia) and Diabetes Australia Queensland. The authors also thank the following project staff: Kym Spathonis, Charlotte Brakenridge, Erin Robson, Amy Chatwin, Jennifer Job, Fiona Heys, Natalie Doyle, Jodie Jetann, Ellen Baker, and Charani Kiriwandeniya.

No other sources of funding or conflicts of interest were declared by the authors.

The trial is registered with the Australian Clinical Trials Registry no. ACTRN12608000203358.

The results of the present study do not constitute endorsement by the American College of Sports Medicine.