Share this article on:

Prolonged Recruitment Efforts in Health Surveys: Effects on Response, Costs, and Potential Bias

Holle, Rolf*; Hochadel, Matthias; Reitmeir, Peter*; Meisinger, Christa; Wichmann, H -Erich†‡for the KORA Group

doi: 10.1097/01.ede.0000239731.86975.7f
Original Article

Background: In health surveys, considerable effort and expense are invested to achieve a high response proportion and thereby to reduce selection bias. We investigated the interrelation of recruitment efforts and expense with potential nonresponse bias based on data from a large health survey.

Methods: In a population-based health survey, a stratified sample of 6640 residents of the Augsburg (Germany) region was selected, of whom 4261 attended the main study between October 1999 and April 2001. A short telephone interview yielded additional information on nearly half of the nonparticipants. All recruitment contacts were documented, and expenses were estimated on the basis of unit costs. Different recruitment strategies were modeled retrospectively. We compared their cost savings as well as their influence on the response proportion and on prevalence estimates.

Results: The distribution of total contacting cost per individual was highly skewed with 50% of the total sum spent on 17% of the sample. Late responders showed many similarities with nonresponders; both included a higher percentage of people with impaired health and with greater behavioral health risks. We were able to identify recruitment strategies that may save up to 25% of the recruitment costs without significant shift in the parameter estimates. Data collected in the short nonresponder interview proved to be important to correct for possible nonresponse bias.

Conclusions: In general, prolonged recruitment efforts lead to a larger and more representative sample but at increasing marginal costs. Specific cost-saving recruitment strategies that do not enhance response bias can be suggested. Interviews of nonresponders are also useful.

From the *Institute of Health Economics and Health Care Management, GSF-National Research Center for Environment and Health, Neuherberg, Germany; the †Institute for Medical Information Processing, Biometry and Epidemiology, University of Munich, Munich, Germany; and the ‡Institute of Epidemiology, GSF-National Research Center for Environment and Health, Neuherberg, Germany.

Submitted 30 September 2005; accepted 30 May 2006.

The KORA research platform (KORA, Cooperative Research in the Region of Augsburg) was initiated and financed by the GSF-National Research Centre for Environment and Health, which is funded by the German Federal Ministry of Education and Research and by the State of Bavaria.

Correspondence: Rolf Holle, Institute of Health Economics and Health Care Management, GSF-National Research Center for Environment and Health, PO Box 11 29, Neuherberg, Germany D-85764. E-mail: holle@gsf.de.

Participation in surveys has generally declined during recent decades.1 The decision of a potential respondent to participate may depend both on recruitment efforts and on the respondent's personal characteristics. In general, a high response rate is believed to lower the risk of nonresponse bias and is thus considered an indicator of study quality. Recently, it has been argued that under certain assumptions, a higher response rate due to prolonged patient recruitment may even increase bias,2 although these model-based calculations lack empiric evidence.

There have been few empiric approaches to the investigation of nonresponse. Ideally, such studies would include exactly those data that are not available, ie, detailed information on nonparticipants. As a different approach, some authors have suggested a comparison of early and late responders with interpretation based on the extrapolation of observed trends.3,4 The rationale for such a comparison is that late responders would have been nonresponders with shorter recruitment. Under this approach, all study variables can be included in the analysis. However, late responders may represent only a specific subgroup of all nonresponders.

The value of increased recruitment efforts for avoiding nonresponse bias is still in doubt, whereas the effect of such efforts on study costs is evident if rarely investigated quantitatively. In this article, we describe a comparison of responders and a substantial subgroup of nonresponders as well as a comparison of early and late responders in a large German health survey. Our specific aim is to assess prolonged recruitment efforts for their cost-effectiveness with regard to achieving high response rates and low nonresponse bias.

Back to Top | Article Outline

METHODS

Study Population

The KORA S4 (Cooperative Health Research in the Region of Augsburg, Survey 4) is a population-based health survey conducted in the city of Augsburg, Germany, and 2 surrounding counties between October 1999 and April 2001. A total sample of 6640 subjects was drawn from the target population consisting of all German residents of the region who were born between July 1925 and June 1975. Overall, 4261 participants were interviewed in person about their health status, health-related lifestyle habits, and socioeconomic characteristics; they also underwent several physical examinations. More detailed information on study design has been reported elsewhere.5 The investigations were carried out in accordance with the Declaration of Helsinki as revised in 1996, including written informed consent of all participants. All study methods were approved by the ethics committee of the “Bayerische Landesärztekammer” Munich.

Back to Top | Article Outline

Recruitment Procedure

The recruitment procedure was different for Augsburg and the surrounding counties due to the limited availability of the local study centers. We confine ourselves to describing the procedure for the city of Augsburg. After sending an invitation letter, we attempted to make an appointment by repeated telephone calls. For persons who could not be reached after several attempts, we attempted in-person contacts at home. Persons unable or unwilling to participate were repeatedly contacted for up to 18 months unless they forbade further calls. The types, date, times, and results of all contacts were recorded in a database with information on more than 12,000 invitation and confirmation letters, 42,000 phone call attempts, and 1600 home visits for the whole sample. The final nonresponders were asked to answer a short questionnaire by phone or mail. This nonresponder questionnaire consisted of 13 questions, most of which were identical to those from the main interview.

Back to Top | Article Outline

Response Categories

Members of the total sample were classified as ineligible if they had died or moved away from the study region before they could have received the first letter or if they had insufficient command of the German language. Of the remaining sample of eligible participants, nonparticipants were classified into 4 categories. They were categorized as “too ill” if they did not participate due to physical impairment and as “too busy” if they explicitly stated a willingness to participate but were not able to make an appointment. The largest category was made up of those who did not want to participate due to lack of motivation (“refusal”). The remaining group of “never-reached” persons did not reply to a letter and could not be reached by phone or at home.

Those nonparticipants who took part in the nonresponder interview are called “short questionnaire responders”; those who did not are called “definite nonresponders.” All participants whose examination took place 3 months or more after the first invitation letter were “late responders,” whereas those examined within 3 months were “early responders.”

Back to Top | Article Outline

Recruitment Costs

To estimate the expenses for recruitment, we considered 3 sources of costs: personnel costs, telephone rates, and postal charges. Based on estimates by the recruitment team, we assumed 5 minutes for the preparation and postprocessing of every piece of mail and every phone call. The duration of the telephone calls was usually documented; otherwise, a value of 5 minutes for reached calls or of 1 minute for call attempts was imputed. Home visits were always conducted by 2 persons (one male, one female); the mean duration was 1 hour. Remuneration is calculated as 12 € per hour for students and 20 € per hour for other employees. Costs varied for postage (0.55–2.00 €) according to type of letter and for telephone (0.015–0.04 € per minute) according to time of day.

Back to Top | Article Outline

Analysis

To compare different groups of responders and nonresponders, we report crude rates and odds ratios (ORs). To account for the stratified sampling procedure, these are also adjusted for age and sex using multiple logistic regression modeling. We also calculated 95% confidence intervals (CIs). The calculations were performed using SAS software version 8.02 (SAS Institute, Cary, NC).

Back to Top | Article Outline

RESULTS

Participation

Of the 6640 members of the total sample, 260 (4%) were ineligible; most of these (especially in the younger age groups) had moved away. From the remaining 6380 persons, 4261 (67%) persons participated in the survey. The response proportion varied over strata, between 58% for elderly women (age 65–74 years) and 72% for younger women (age 35–44 years). Persons who could never be reached constitute 3% of the net sample, and another 3% were too ill to participate. Persons too busy comprised 5% of the net sample. The largest group was the 1410 persons (22%) who declined to participate without giving a specific reason.

Altogether, there were 1031 short questionnaire responders (49% of all nonparticipants). Among those too ill to participate and those too busy, substantially more answered the short questionnaire (65% and 66%, respectively) than among the refusers, in which the response rate was 46%.

Back to Top | Article Outline

Comparison of Response Groups

All further analyses and presented results are restricted to the Augsburg city subsample. We classified 29% of the participants in Augsburg city as late responders.

In Table 1, descriptive statistics are presented comparing the 4 response subgroups. Note that for definite nonresponders information is available for only some variables.

TABLE 1

TABLE 1

Table 2 presents 2 sets of comparisons based on the same variables. On average, the group of short questionnaire responders is slightly older than the participants, whereas late responders are 3.6 years younger than early responders. The sex distribution is similar across the 4 groups. After adjustment for sampling strata, many variables show differences between late and early responders that are similar to differences between short questionnaire responders and participants.

TABLE 2

TABLE 2

A more detailed comparison of late versus early responders was possible using the complete set of variables collected in the main study. Results were similar to those presented in Table 2. Late responders showed less favorable health behavior such as physical activity (adjusted OR = 0.64; 95% CI = 0.52–0.78) or participation in cancer screening in the past year (0.67; 0.53–0.85). Late responders also tended to be less healthy; for example, they were more likely to have systolic blood pressure >140 mm Hg (1.28; 0.99–1.66).

Back to Top | Article Outline

Recruitment Effort and Costs

The total cost for contacting the whole sample was approximately 185,000 €, 28 € per member of the total sample and 43 € per participant. The distribution of total contacting cost per individual was highly skewed with 50% of the total sum spent on 17% of the sample.

More detailed calculations are restricted to the Augsburg city subsample, which accounts for approximately half of the recruitment costs. Telephone calls were responsible for more than half of the total costs in Augsburg. On average, 4.3 phone calls (including attempted calls) were needed for each participant (median = 2). In comparison, for each nonparticipant, 9.9 phone call attempts (median = 6) were registered. Mailings and home contacts each accounted for approximately one fourth of the total expense. Overall, costs per participant decreased with age, from 58 € in the youngest age group (25–34 years) to 39 € in those age 55 to 64 years (Fig. 1).

FIGURE 1.

FIGURE 1.

To develop more cost-effective recruitment strategies, we assessed how many participants would have been lost and which costs would have been saved if a stricter stopping rule for recruitment had been applied. We show estimates of costs and participation rates had we stopped after first refusal (A), stopped after 10 attempts on the telephone (B), or not attempted home contacts (C).

These strategies would have resulted in a 6% (A and C) to 10% (B) lower response (Table 3). Estimated cost savings ranged between 16,000 and 25,000 € (ie, 20–30% of the recruitments cost for Augsburg city). The most cost-effective alternative would have been to abandon home visits, which was the most expensive mode of contact.

TABLE 3

TABLE 3

For each strategy, we compared lost and retained participants with respect to several baseline characteristics (Table 4). The subjects who would have been lost by not making home visits (strategy C) turned out to differ from the remaining participants with respect to nearly all factors. After adjustment for sampling strata, they were much younger but also more ill. As an example, 7.4% of lost participants reported a history of diabetes in contrast to 4.0% of the retained participants (adjusted OR = 2.7). The participants who would have been lost by the other strategies, however, appeared not to differ much from the remainder. Because the lost participants form only a small proportion of the sample, their influence on the overall estimates of prevalence would not be as dramatic as the figures suggest.

TABLE 4

TABLE 4

Back to Top | Article Outline

DISCUSSION

In a recent editorial, Stang6 listed 4 suggestions regarding how to study the potential of nonresponse bias in epidemiologic studies. Three of these methods (baseline characteristics from sampling frame, short questionnaire in random sample, early vs late responders) were applied in the KORA S4 survey, as presented here. The fourth method (the follow-up assessment of vital status) can be added in a few years.

Back to Top | Article Outline

Participation

The response rate in our study compares well with other recent health surveys in Germany and elsewhere. Compared with 3 earlier health surveys performed 5, 10, and 15 years ago in the same region and with a very similar study design, we observe a progressive decline in the response proportions from 79% in 1985 to 67% in 2000.

The participation of nonresponders in the short questionnaire was relatively high when compared with similar studies. Overall, short questionnaire responders differed from study participants in several aspects, and at least 2 patterns of nonresponse are apparent in the sample. One type is related to health problems and prevails among older subjects. Our results support the view that nonparticipants are less healthy, because they report worse subjective health and more myocardial infarction and diabetes. This implies that estimates of disease prevalence may be especially susceptible to nonresponse bias.

A second type of nonparticipation results from lack of time and is more common in younger people. The median age of the nonparticipants classified as “too busy” was 42 years, considerably below the 50 years of the total sample.

The interpretability of the results of the nonrespondent interview is limited in at least 2 ways. First, the assumption that the short questionnaire responders are representative of all nonparticipants need not be true. This can only be verified for age and sex (when there is no evidence against it) and for the reason given for nonparticipation. All categories of nonresponders except for those “not reached” contributed substantially to the short questionnaire survey. Second, the different mode of the nonrespondent interview (short questionnaire by phone or mail) and the main survey (personal interview and physical examination) may have biased the results.

A comparison of early versus late responders shows similar overall trends as for participants versus nonparticipants. These results were supported by analyses of variables that were available only for study participants. Therefore, possible response bias may be reinforced by recruitment strategies that allow only the inclusion of early responders.

Back to Top | Article Outline

Recruitment Efforts

Few studies have included contact cost in their analyses.3,7 Expenses for recruitment account for a substantial part of the overall study costs. Expenses for recruitment increase with decreasing age: more telephone calls were needed to reach younger subjects and, if reached, they were less willing to participate. The 3 strategies investigated in our study are confined to reducing the number of contacts, thereby saving costs and losing participants. Eliminating home visits would be the most cost-effective strategy (in terms of saved costs per lost participant), but the strategy also seems the most likely to induce bias. Therefore, a different strategy such as stopping after 10 futile telephone contacts may be more advantageous. Triplett8 suggests a cutoff point of 20 call attempts because further attempts yield less than 1% increase in the final response proportion. These strategies may help to save money, but these savings are in the range of only a few percent of the overall expenses of a health survey.

Back to Top | Article Outline

ACKNOWLEDGMENTS

The KORA group consists of H.-E. Wichmann (speaker), H. Löwel, C. Meisinger, T. Illig, R. Holle, J. John, and their coworkers, who are responsible for the design and conduct of the KORA studies. The authors appreciate the contribution of Klaus Papke and Bernhard Schwertner and their teams who were responsible for the field work, and last but not least, the authors thank all study participants for their voluntary cooperation.

Back to Top | Article Outline

REFERENCES

1. Hartge P. Raising response rates: getting to yes. Epidemiology. 1999;10:105–107.
2. Stang A, Jockel KH. Studies with low response proportions may be less biased than studies with high response proportions. Am J Epidemiol. 2004;159:204–210.
3. Hochstim JR. A critical comparison of three strategies of collecting data from households. J Am Stat Assoc. 1967;62:976–989.
4. Tennant A, Badley EM. A confidence interval approach to investigating non-response bias and monitoring response to postal questionnaires. J Epidemiol Community Health. 1991;45:81–85.
5. Holle R, Happich M, Löwel H, et al, for the MONICA/KORA Study Group. KORA—a research platform for population based health research. Das Gesundheitswesen. 2005;67:S19–S25.
6. Stang A. Nonresponse research—an underdeveloped field in epidemiology. Eur J Epidemiol. 2003;18:929–931.
7. Siemiatycki J, Campbell S. Nonresponse bias and early versus all responders in mail and telephone surveys. Am J Epidemiol. 1984;120:291–301.
8. Triplett T. What Is Gained From Additional Call Attempts and Refusal Conversion and What Are the Cost Implications? November 2002. Available at: http://mywebpages.comcast.net/ttriplett13/tncpap.pdf. Accessed March, 31, 2006.
© 2006 Lippincott Williams & Wilkins, Inc.