Objectives: To develop methods to maximize the accuracy of reporting HIV risk behaviours in a general population survey. We assessed the feasibility of using a computer-assisted self-completion interview (CASI) in comparison with pen-and-paper self-completion interview (PAPI).
Design: A probability sample survey of residents aged 16–44 years in Britain, with alternate assignment of addresses to interview by CASI (462) or PAPI (439).
Methods: Personal interviews exploring demographic and sexual behaviour variables. Principal outcome measures were the impact of CASI in relation to PAPI on data quality and rates of reporting a range of behaviours.
Results: A total of 901 interviews were completed; 829 individuals were eligible for and accepted the self-completion module. Internal consistency of data items was greater with CASI than PAPI and item non-response was lower. Overall, there was no significant difference in rates of reporting between CASI and PAPI. The main effect for CASI compared with PAPI in a generalized estimating equation (GEE) analysis was an OR (95% CI) of 1.04 (0.92–1.17). Variables were also examined individually, including homosexual partnership (adjusted OR 1.26 95% CI 0.69–2.29), payment for sex (adjusted OR 0.68 95% CI 0.29–1.59), masturbation (adjusted OR 0.89 95% CI 0.66–1.22) and five or more partners in the past 5 years (OR 0.85 95% CI 0.61–1.19).
Conclusion: We found no evidence of a consistent effect of CASI on rates of reporting sexual HIV risk behaviours in this sample. CASI resulted in improvement in internal consistency and a reduction in missed questions.
From the aDepartment of Sexually Transmitted Diseases, Royal Free and University College Medical School, Mortimer Market Centre, off Capper Street, London WC1, UK; bNational Centre for Social Research (formerly Social and Community Planning Research), 35 Northampton Square, London EC1, UK; and cDepartment of Public Health and Policy, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1, UK.
Received: 16 December 1999;
revised: 20 July 2000; accepted: 1 September 2000.
Sponsorship: This study was funded by a grant from the Medical Research Council.
Correspondence to: Professor Anne M. Johnson, Department of Sexually Transmitted Diseases, Royal Free and University College Medical School, London WC1E 6AU. E-email: email@example.com
The paper is dedicated to the memory of Jane Wadsworth, who died in 1997 and was a grantholder.
The study was approved by the Joint UCL/UCH Committees on the Ethics of Human Research.
Sexual behaviour research is central to understanding HIV epidemiology, but the validity of parameter estimates depends on the ability of respondents to answer questions about sensitive behaviours accurately.
Since the surveys of the early 1990s [1–4], there have been important technological advances in survey methodology. In particular, computer-assisted self-completion interviews (CASI) allow respondents to key responses directly to questions displayed on a laptop computer screen. Studies comparing CASI with identical questions using pen-and-paper self-completion interviews (PAPI), have demonstrated the potential of CASI to improve the quality of data, and may increase the willingness of respondents to report sensitive behaviours [5,6].
The first British National Survey of Sexual Attitudes and Lifestyles (NATSSAL1) was carried out in 1990/1991 to inform HIV/AIDS projections and sexual health strategy [1,7,8]. In this paper, we report on a methodological experiment undertaken as part of a programme of feasibility work for a second National Survey of Sexual Attitudes and Lifestyles (NATSSAL2) being undertaken this year. We investigated whether the use of CASI was acceptable and feasible for the general population; and could obtain data of similar or improved quality to pen-and-paper methods. We measured the effect of CASI by comparing rates of reported behaviours in respondents assigned to CASI and PAPI. This was important, not only to assess whether CASI could reduce measurement error, but also to provide hypotheses concerning the magnitude of any method effect in assessing changes in behaviour between the 1990 and 1999 surveys.
A questionnaire was developed for a combination of face-to-face interviews with a trained interviewer, and self-completion of the most sensitive questions. The majority of questions were identical to those used in NATSSAL1 [7,8], although with some additions. Topics covered include age at first intercourse; homosexual and heterosexual experiences including numbers of partners; and sexual practices. Those with no sexual experience of any kind, and 16 and 17-year-olds with some heterosexual experience, but no heterosexual intercourse or homosexual experience reported in a screening face-to-face question, were ineligible for the self-completion module.
Answers to the face-to-face component of the questionnaire were keyed in directly by the interviewer. The self-completion component was developed both for CASI (90 questions) and for pen-and-paper (82 questions). Question wording was generally identical between the two formats. The additional eight questions in CASI were restricted to this method because they required complex routing.
A sample of 3360 addresses was selected from the small-user postcode address file for Britain, using a multi-stage stratified probability cluster design with over-sampling from Greater London.
Interviews were undertaken by the National Centre for Social Research between November 1997 and January 1998. Alternate households were assigned to either CASI or PAPI self-completion. A total of 3222 out of 3360 (95.9%) addresses were successfully screened for eligibility; residents aged 16–44 years were enumerated, and one was randomly selected to participate. Interviewers were present for self-completion interviews to provide assistance if required, but were not permitted to view the responses.
The study sample size was calculated to give 80% power to detect as significant (at the 5% level) a doubling due to CASI in the reporting of behaviours with a prevalence of 5%. We used the odds ratio (OR) of reporting behaviours in CASI compared with PAPI as the outcome measure. Initially, questions were considered separately. To test our primary hypothesis, that CASI may affect the reporting of behaviours broadly according to their sensitivity, we then grouped behaviours according to our perception of their sensitivity as high (e.g. reporting payment for sex), moderate (e.g. no heterosexual sex last year), or low (e.g. never donated blood). A CASI effect would be expected to result in OR above one for the highly sensitive behaviours, declining to around one for those of low sensitivity. This avoids multiple testing and has high power to detect a moderate CASI effect. We also used a generalized estimating equation (GEE)-based regression method  to provide a summary adjusted estimate of the effect of CASI across behaviours of moderate or high sensitivity and to test for a significant interaction between CASI and high question sensitivity.
As a secondary analysis we also examined the possibility of CASI effects arising only within four subgroups: men and women, London and outside. We tested for each question to see whether the CASI effect differed by either region or sex (i.e. significant interaction). We also considered behaviours according to their sensitivity as in the primary analysis, looking for a trend in the OR. Finally, we used GEE to test for a significant interaction between CASI and either sex or region.
Initial estimates of the effect of CASI compared with PAPI are reported after adjusting by logistic regression for demographic factors. The GEE analysis used an unstructured correlation matrix, but the features of the sampling (e.g. clustering) could not be incorporated. For the GEE analysis, only those questions asked of both sexes were included. The main effects for each behaviour and demographic terms (age, household size, sex, and location) were included in the GEE model. Because these factors determine the selection probabilities, all regression analysis was unweighted. All analyses were performed in stata release 6.
The household screening identified 1389 households with a selected eligible resident aged 16–44 years; 901 interviews were completed. The overall unweighted response rate was 61.2%, but after weighting for the over-sampling of London, the response rate was 64.7%. Response rates were calculated excluding those who did not speak English, or were sick or away from home during the fieldwork period (75 selected residents and estimated ineligibles in uncontacted households). Of the 901 people interviewed, 56 were ineligible for self-completion because they had no sexual experience. Sixteen of those who were eligible (1%) refused the self-completion module (eight in the CASI arm, eight in the PAPI arm) leaving 829 self-completion interviews on which our main results are based (including three who declined CASI but accepted PAPI instead). There was no difference in mean overall interview length between CASI (56 min) and PAPI (55 min).
No significant differences were observed between the groups assigned to CASI and PAPI in age, sex or marital status (data not shown).
We compared item non-response rates (defined as not answered, don't know and refusal) and the internal consistency of data between the two methods. Questions were found significantly more likely to have missing data using PAPI (P < 0.001 McNemar's test). In an analysis of 70 questions, 46 out of 70 (65.7%) in PAPI had at least one answer missing (range of non-response 0–14%) whereas in CASI only nine out of 70 (12.8%) questions had any missing data (range 0–1%).
In CASI, 24 consistency checks were built into the program, and queries to the respondent were raised on screen so that respondents could change the answer or suppress the query and move on. Other internal consistency checks were tested on the completed data. Sixty-two key consistency checks were analysed. The mean number of fails per record was 2.4 with PAPI and 1.8 with CASI, whereas similar proportions of PAPI and CASI respondents had no fails (18% PAPI and 16% CASI). Taking no fails and one fail together, the outcome was 29% PAPI and 59% CASI.
Table 1 shows the ORs of reporting a number of behaviours for CASI compared with PAPI, adjusting for age and the number of eligible adults in the household (grouped 1, 2, and 3+). The behaviours are grouped according to their presumed sensitivity.
There is no evidence that the summary OR are above one for the most sensitive questions, declining to around one for the least sensitive questions, as might be hypothesized if there is a CASI effect (P = 0.43, Mann–Whitney test comparing high with moderate sensitivity). The results of the GEE analysis are consistent. The observed main effect of CASI as an adjusted OR (95% confidence interval; CI) is 1.04 (0.92–1.17). No significant interaction was found between CASI and high question sensitivity. Equally, when each question is examined individually, there are no significant effects in the summary column.
There was no evidence of a pattern of declining OR with lower sensitivity in any of the four sex and region subgroups (Table 1). In the GEE analysis (detailed results not presented), we found no significant interaction between CASI and either sex or region (London or outside).
We examined heterogeneity in the OR for the CASI effect for individual questions by region and by sex. We found a significant difference (interaction) in the CASI effect by region for homosexual sex and some difference for heterosexual anal sex (P = 0.056), and a significant difference by sex for never a blood donor together with some difference for ever had a sexually transmitted disease diagnosis (P = 0.056). There is no consistency in the nature of the CASI effect found within subgroups for these questions. For example, the findings suggest that CASI may possibly lead to higher reporting of homosexual sex outside London, but lower reporting of heterosexual anal sex.
CASI was found to be highly acceptable. Compared with PAPI, there were no differences in unit response rates but there were substantial improvements in data quality. The CASI module was carefully developed so that respondents only used limited ranges of keys, so it needed no greater ‘computer literacy’ than that required to obtain cash from an automatic cashpoint machine. Qualitative research carried out to aid the design of the study showed that even those with no computer experience found CASI easy to use, appreciated the chance to correct mistakes and felt that the method enhanced confidentiality . The lower rates of item non-response of CASI reflect the programmed routing of questions. In PAPI, respondents need to follow written instructions, which they may misinterpret or misread, but in CASI once a question is answered the programme moves to the next relevant question.
Our primary hypothesis was that CASI might affect reporting rates according to the sensitivity of the question asked, but we found no evidence of such a general CASI effect either overall or within sex or location subgroups. The lack of consistency in the suggested effects for individual questions within subgroups is consistent with these effects arising by chance, given the multiple testing, but the possibility that they represent real effects cannot be excluded.
We are now undertaking a large-scale survey using the CASI methodology developed in this study. On the basis of our findings, any substantial changes in the reporting of behaviours that are evident in the new survey are unlikely to be in any large part attributable to a change in the method of data collection.
Our results contrast with the findings of Turner et al. , who reported significant CASI effects for the reporting of several sensitive behaviours. However, their sample was restricted to adolescent males, many from disadvantaged backgrounds, and the study used audio CASI to overcome potential literacy problems in this group. Method effects will be related to the degree of perceived social censure of particular behaviours, and these may vary between cultures and demographic groups. We have previously demonstrated the more tolerant attitudes to homosexuality and other sexual behaviours in Britain compared with the United States , which may partly explain the difference in method effect between studies.
The second National Survey of Sexual Attitudes and Lifestyles is being undertaken with funding from the Medical Research Council. We hope to interview 12 000 people in Britain aged 16–44 years, half of whom will also be invited to provide a urine sample for testing for genital Chlamydia trachomatis. The survey will provide up-dated estimates of the population prevalence of HIV risk behaviours, and allow an analysis of changes since 1990.
The authors would like to thank the interviewers from the National Centre for Social Research, and the participants for their essential contributions; Vern Farewell and Susan Purdon for statistical advice; and Sandy Gale for manuscript preparation. A more detailed version of this manuscript is available from the authors.
1. Johnson AM, Wadsworth J, Wellings K, Bradshaw S, Field J. Sexual lifestyles and HIV risk.
Nature 1992, 360: 410 -412.
2. ACSF investigators. AIDS and sexual behaviour in France.
Nature 1992, 360: 407 –409.
3. Cleland J, Ferry B. Sexual behaviour and AIDS in the developing world.
London: Taylor Francis; 1995.
4. Michael RT, Wadsworth J, Feinleib J, Johnson AM, Laumann EO, Wellings K. Private sexual behavior, public opinion, and public health policy related to sexually transmitted diseases: a US–British comparison.
Am J Public Health 1998, 88: 749 –754.
5. Weeks MF. Computer assisted survey information collection: a review of CASI methods and their implications for survey operations.
J Official Statistics 1992, 8: 445 –465.
6. Turner CF, Ku L, Rogers SM, Lindberg LD, Pleck JH, Sonenstein FL. Adolescent sexual behaviour, drug use and violence: increased reporting with computer survey technology.
Science 1998, 280: 867 –873.
7. Johnson AM, Wadsworth J, Wellings K, Field J. Sexual attitudes and lifestyles
. Oxford: Blackwell Scientific Press; 1994.
8. Wellings K, Field J, Johnson AM, Wadsworth J. Sexual behaviour in Britain.
London: Penguin; 1994.
9. Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes.
Biometrics 1986, 42: 121 –130.
10. Mitchell K, Wellings K, Elam G. NATSSAL 2 survey design and implementation report of the qualitative stage. Report to Medical Research Council
, June 1998. pp. 1–40.