Smith, Allan ‘Ben’; King, Madeleine; Butow, Phyllis; Grimison, Peter; Olver, Ian
To the Editor:
The recruitment of controls from the Internet is appealing, in that it is cheaper and faster than random digit dialing (RDD) or random sampling from the electoral roll. However, the psychosocial profile of Internet-panel controls may not be representative of the general population. We explored the representativeness of Internet-panel controls by comparing their scores on health-related quality of life and psychological distress with population norms.
We assessed a set of controls recruited in February 2011 from an Internet panel of 29,000 Australians as part of a study of testicular cancer survivors. Health-related quality of life was assessed using the SF-36v2, and psychological distress was assessed with the Depression Anxiety Stress Scales short-form. There are Australian population norms for these measures,1,2 which allowed us to compare the Internet-panel controls with the general population. We used one-sample t tests to compare norm-based scores of Internet-panel controls with age-adjusted Australian general population norms.2 We could not adjust for other demographic characteristics, as none was available for these normative data.
Two hundred ten Internet-panel members completed an online questionnaire for controls and received AU$4. Response rate could not be calculated as the survey was disabled once the recruitment target was reached. We excluded 16 respondents who completed the 104-item questionnaire in 5 minutes or less. The eAppendix Table (http://links.lww.com/EDE/A672) shows the demographic features of the 194 eligible participants.
Internet-panel controls had substantially worse scores than the general population for all domains of health-related quality of life (Table). The differences in physical aspects of health-related quality of life were generally small, while differences in mental aspects were larger. Internet-panel controls also reported higher levels of depression and anxiety than the general population, although differences were small (Table).
Our results are consistent with accumulating evidence that suggests Internet-panel respondents are systematically different from the general population. For example, Internet-panel respondents report significantly worse post-traumatic stress symptomatology and less social support than RDD respondents.4 One possible explanation for the low social support is that men who volunteer for Internet panels are more frequent Internet users.4 High-frequency Internet use in men is associated with increased social anxiety,5 which may lead to (or reflect) social isolation, reduced health-related quality of life, and psychological impairments.
The apparent impairments of Internet-panel respondents could also be a product of unreliable responses to rushed online questionnaires. One study found that Internet-panel respondents completed a survey faster than phone respondents.6 Repeated measures testing 3 months later found that the responses of participants who originally spent under four minutes completing the online questionnaire shifted 10% versus 5% in those who had taken six minutes or more.6 The same study found 80% of panelists preferred completing a less interesting survey for £2 than a more interesting survey with no reward.6 This casts doubt on the reliability of responses from Internet-panel participants who may hurry to maximize their rate of pay.
In summary, levels of health-related quality of life and psychological distress reported by Internet-panel controls may not be representative of the general population, possibly due to higher levels of social isolation or rushed responses to online surveys. Caution should be taken in recruiting controls through the Internet for studies of psychosocial outcomes.
ACKNOWLEDGMENTS
We thank Tim Luckett for assisting with study development, and the Australian and New Zealand Urogenital and Prostate Cancer Trials Group (G. Toner, M. Stockler, E. Hovey, H. Gurney, S. Turner, G. Hruby, M. Alam, J. Bucci, R. Eek, D. Thomson, B. Koczwara, G. Pranavan, N. Pavlakis, N. Singhal, and J. Stubbs) for assisting with study development, funding and recruitment.
Allan ‘Ben’ Smith
Psycho-oncology Co-operative Research Group
The University of Sydney
New South Wales, Australia
allan.smith@sydney.edu.au
Madeleine King
Psycho-oncology Co-operative Research Group
The University of Sydney
Australian New Zealand Urogenital and Prostate Cancer Trials Group
New South Wales, Australia
Phyllis Butow
Psycho-oncology Co-operative Research Group
The University of Sydney
New South Wales, Australia
Peter Grimison
Australian New Zealand Urogenital and Prostate Cancer Trials Group
Sydney Cancer Centre, Sydney
New South Wales, Australia
Ian Olver
Cancer Council Australia, Sydney
New South Wales, Australia
REFERENCES
1. Hawthorne G, Osborne RH, Taylor A, Sansoni J. The SF36 Version 2: critical analyses of population weights, scoring algorithms and population norms. Qual Life Res. 2007;16:661–673
2. Lovibond SH, Lovibond PF Manual for the Depression Anxiety Stress Scales. 19952nd ed Sydney, Australia Psychology Foundation
3. Cohen J. A power primer. Psychol Bull. 1992;112:155–159
4. Hines DA, Douglas EM, Mahmood S. The effects of survey administration on disclosure rates to sensitive items among men: a comparison of an Internet panel sample with a RDD telephone sample. Comput Human Behav. 2010;26:1327–1335
5. Mazalin D, Moore S. Internet use, identity development and social anxiety among young adults. Behaviour Change. 2004;21:90–102
6. Sparrow N. Developing reliable online polls. Int J Market Res. 2006;48:659–680
