Skip Navigation LinksHome > July 2011 - Volume 29 - Issue 7 > Quota Sampling in Internet Research: Practical Issues
CIN: Computers, Informatics, Nursing:
doi: 10.1097/NCN.0b013e3181f9dc45
Continuing Education

Quota Sampling in Internet Research: Practical Issues

IM, EUN-OK PhD, MPH, RN, CNS, FAAN; CHEE, WONSHIK PhD

Free Access
Continued Education
Article Outline
Collapse Box

Author Information

Author Affiliations: School of Nursing (Dr Im) and Department of Mechanical Engineering (Dr Chee), The University of Texas at Austin.

This is part of a larger study funded by the National Institutes of Health (NIH/NINR/NIA) (R01NR008926).

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Corresponding author: Eun-Ok Im, PhD, MPH, RN, CNS, FAAN, School of Nursing, The University of Texas at Austin, 1700 Red River, Austin, TX 78701 (eim@mail.nur.utexas.edu).

Collapse Box

Abstract

Quota sampling has been suggested as a potentially good method for Internet-based research and has been used by several researchers working with Internet samples. However, very little is known about the issues or concerns in using a quota sampling method in Internet research. The purpose of this article was to present the practical issues using quota sampling in an Internet-based study. During the Internet study, the research team recorded all recruitment issues that arose and made written notes indicating the possible reasons for the problems. In addition, biweekly team discussions were conducted for which written records were kept. Overall, quota sampling was effective in ensuring that an adequate number of midlife women were recruited from the targeted ethnic groups. However, during the study process, we encountered the following practical issues using quota sampling: (1) difficulty reaching out to women in lower socioeconomic classes, (2) difficulty ensuring authenticity of participants' identities, (3) participants giving inconsistent answers for the screening questions versus the Internet survey questions, (4) potential problems with a question on socioeconomic status, (5) resentment toward the research project and/or researchers because of rejection, and (6) a longer time and more expense than anticipated.

With increasing use of the Internet in daily life, it is not surprising that the number of Internet studies has also increased tremendously in recent years. With this increase has come a corresponding increase in the criticism about potential selection bias in Internet studies because Internet users tend to be a select group: high-income, highly educated whites.1 Although recent demographic research indicated that significant changes are moving Internet populations in the direction of greater representation, Internet users still tend to be in this select group.1

To overcome the potential bias of having a select group of research participants in Internet-based studies, researchers have adopted diverse ways of sampling research participants via the Internet, including "pop-up" surveys, the volunteer panel method, and a hybrid sampling approach.2 In Internet studies, a modified systematic sampling method is often used because a probability sampling method is not possible in Internet-based studies. Currently, the only way to obtain a probability sample on the Internet is to contact potential participants via some conventional means (eg, by mail or phone) first and then ask potential participants to respond to a survey on the Internet.2 One of the modified systematic sampling methods is "pop-up" surveys that "pop-up" on the computer screen of every nth visitor to a Web site.3 Also, uncontrolled instrument distribution-a sampling method conducted by posting a survey on the Internet for anybody-is a common nonprobability sampling method used in Internet studies.4 Another nonprobability sampling method, used especially in marketing surveys, is the volunteer panel method, which involves assembling a group of individuals who have volunteered to participate in future surveys and recruiting them to the panel through some form of advertising.2,5 A hybrid sampling approach for Internet research that combines a large convenience sample with a probability sample was also examined by Schonlau and colleagues.2 Despite these efforts to experiment with diverse sampling methods, there is no clear consensus about what the best sampling method is.

In most modern Internet-based studies, quota sampling has been considered the standard sampling method.6 Quota sampling is a type of convenience sampling with an additional strategy that attempts to include subject types that are likely to be underrepresented in the convenience sample.7 The advantages of quota sampling are that (1) it is fast and relatively inexpensive to organize and (2) it does not require sophisticated skills or an inordinate amount of time or effort.8,9 However, quota sampling also has some disadvantages, including (1) it still does not represent the population as a whole, (2) it is impossible to assess the possible sampling error because it is still a nonrandom sampling method, and (3) the problem associated with nonresponse is concealed in quota sampling because the researcher may end up with a sample that underrepresents the portion of the population that is difficult to reach.7-9

The purpose of this article was to present the practical issues in using quota sampling that were identified in an Internet-based study. The purpose of the Internet-based study was to explore the menopausal symptom experience of multiethnic midlife women using an Internet survey. The study utilized a cross-sectional descriptive design, including a quantitative Internet survey and four ethnic-specific online forums. In this article, only the practical issues that arose in administering the Internet survey are presented because the focus of the discussion is on sampling methods that assume quantitative research methods are used. Then, based on the issues, we make suggestions for future Internet-based research.

Back to Top | Article Outline

SAMPLING PROCESS IN THE STUDY

Study Announcement

The study was announced in Internet communities/groups for midlife women (ICMWs) and other ethnic minorities (Hispanics, African Americans, and Asians). Because ethnic differences in the menopausal symptom experience were explored in this study, it was essential to reach targeted ethnic groups of midlife women in geographically dispersed areas. When ICMWs were searched for with Yahoo! and Google.com, 1 120 000 Web sites/pages were identified through Yahoo! and 1 370 000 through Google.com. Among the more than 2 million Web sites/pages for midlife women, there were more than 1 000 000 general ICMWs (not ethnic-specific), 101 000 Hispanic ICMWs, 297 000 African American ICMWs, and 269 000 Asian ICMWs. Each ICMW had up to 937 members. Considering the current statistics on the ethnicity of Internet users, we assumed that ethnic minority groups could be easily reached via the Internet when we approached them in ethnic-specific ICMWs. Thus, Internet communities for ethnic minorities (ICEMs) in the United States, including Internet communities/groups connected to churches, organizations, forums, healthcare centers, and professional groups, which were all ethnic-specific, were contacted and asked to announce the study. Previous studies reported that ethnic minorities were more successfully recruited through churches and ethnic-specific support/social groups with culturally specific memberships.10-12 When ICEMs in the United States were searched for with Google.com, the Web sites of 463 000 Hispanic, 2 110 000 African American, and 1 170 000 Asian American ICEMs were retrieved. These ICEMs had up to 406 members.

Back to Top | Article Outline
Sampling

Using a quota sampling method, 512 research participants were recruited for the Internet survey using multiple strategies in the various Internet settings described above. Because studies have shown that the menopausal symptom experience is influenced by menopausal status (MS) and socioeconomic status (SES), as well as ethnicity,13,14 the sampling method aimed to ensure a sufficient number of participants from each of four ethnic groups and to ensure MS and SES diversity. In this study, SES was determined based on women's self-reports on their degree of difficulty in paying for the basic necessities of life and categorized as (1) low (very hard to pay for basics), (2) middle (somewhat hard to pay for basics), and (3) high (not hard to pay for basics). Based on the answers to seven questions on menstruation, MS was determined and categorized as: (1) premenopausal, (2) early perimenopausal, (3) late perimenopausal, and (4) postmenopausal. We monitored ethnic, MS, and SES composition of the Internet survey participants as they were recruited.

Back to Top | Article Outline
Participants Recruited

In total, 1967 women visited the project Web site; only 512 of the visitors (26%) completed the Internet survey after they completed the screening process that checked the inclusion criteria and quota requirements. The quota size was set at 160 per ethnic group (oversampled to meet the target number of participants); the target total number of participants was 500; recruitment continued for 4 years. The 512 women recruited for the study included 160 whites (31%), 120 Hispanics (23%), 121 African Americans (24%), and 111 Asians (22%). The women's mean age was 49 (SD, 5.30) years. Thirty-three percent had a college degree, 44% did not have any difficulty in paying for basics, and 72% were employed. Twenty-eight percent were Protestants, and 63% were married. More details on the study can be found elsewhere.15

Back to Top | Article Outline

THE PRACTICAL ISSUES

Although quota sampling significantly contributed to the study described in this article in terms of accomplishing its primary goal-ethnic representation among the participants-we encountered several issues during the recruitment process that we did not expect. During the study, the research team recorded all recruitment issues that arose and made written notes indicating the possible reasons for the problems. Biweekly team discussions were conducted for which written records were kept. During this process, we identified the following practical issues using quota sampling: (1) difficulty reaching out to women in lower socioeconomic classes, (2) difficulty ensuring authenticity of participants' identities, (3) participants giving different answers for the screening questions versus the Internet survey questions, (4) potential problems with one question on SES, (5) resentment toward the research project and/or researchers because of rejection, and (6) a longer time and more expense than anticipated. In the following sections, we discuss each of these issues in detail.

Back to Top | Article Outline
Difficulty Reaching Out to Women in Lower Socioeconomic Classes

Because of the inherent nature of Internet populations (those having access to the Internet and the ability to use a computer), we expected that recruiting women from lower socioeconomic classes would be difficult. However, during the first 2 years of data collection, we were optimistic because the socioeconomic composition of the participants was not as lopsided as we expected. However, as the quota for each racial/ethnic group was filled, it became more difficult to recruit women from the lower socioeconomic classes. Indeed, by the end of the third year, all the quotas were filled except those for women from the lower socioeconomic classes, which were not filled until the end of the fourth year when data collection was completed. About 20% of the final sample was women from lower socioeconomic classes, which was actually much better than we expected.

To supplement the pool of participants with midlife women from lower socioeconomic classes, multiple strategies were adopted. First, ethnic-specific churches in low-income neighborhoods that were identified through the Internet were contacted and asked to announce the study. Then, libraries and community health clinics in low-income neighborhoods that served primarily women with low incomes were contacted and asked to announce the study. Notwithstanding these efforts, it clearly took more time to recruit women from the lower socioeconomic classes. One possible reason for the low response rate for women in the lower socioeconomic classes is that they were simply too busy to be involved in a research study. In addition, they tended not to have regular access to the Internet, and they were busy doing other chores, even when they were in community settings such as a public library. Interestingly, during the recruitment process, three informal leaders from ICMWs and ICEMs (two Hispanics and one Asian) volunteered to help recruit women from lower socioeconomic classes, which greatly facilitated the recruitment of women underrepresented on the Internet.

Back to Top | Article Outline
Difficulty Ensuring Authenticity of Participants' Identities

As frequently pointed out in the literature, Internet interactions render irrelevant markers such as race, sex, status, and age because Internet communications are typically based on non-face-to-face interactions.16 Consequently, ensuring authenticity has been a significant issue for Internet researchers.17 Indeed, in the Internet-based study presented in this article, participants were recruited on the Internet based on self-reports. Therefore, we had to trust that self-reported midlife women were actually midlife women because there was no mechanism to check the authenticity of their claims. However, several unauthentic participants were identified during the data collection process. For example, a woman who was rejected from the project Web site because the quota for her sociodemographic or ethnicity characteristics had already been filled tried to pose as another person by giving false answers to the screening questions; she actually succeeded in taking the Internet survey. She then posted a message in her Internet group/community describing how to answer the screening and Internet survey questions to qualify for the study and thus receive reimbursement for participation.

Back to Top | Article Outline
Potential Problems With a Question on Socioeconomic Status

In the process of recruiting Asian Americans and Hispanics with diverse SESs with predetermined quotas, we found that some potential participants who were referred to the project Web site by Internet communities/groups that served Asian American or Hispanic women with low incomes answered the question on SES differently than we expected. Most participants checked their SES as middle rather than low. As mentioned above, SES was based on women's self-reports of their degree of difficulty in paying for the basic necessities of life and categorized into (1) low (very hard to pay for basics), (2) middle (somewhat hard to pay for basics), and (3) high (not hard to pay for basics). Most of the women considered to have low SES by the Internet communities/groups checked "somewhat hard to pay for basics" (ie, middle) to the question on SES.

We believe that cultural beliefs, attitudes, and values concerning reputation and modesty in both Asian American and Hispanic ethnic groups may have influenced how participants identified their SES. Furthermore, because this was an Internet study that asked the participants to answer an SES question only one time (unless they later participated in an online forum later), there was no mechanism to verify if their actual situation placed them in low, middle, or high SES. Indeed, a recent study indicated that Hispanics, especially Spanish-speaking immigrants, tended to rely less on objective SES in determining their subjective SES than did those who were English speaking and United States-born and that the overall degree of congruence between objective SES and subjective SES was lower in low-income Mexican Americans than in other groups.18 In other words, while objective SES was a main determinant of subjective SES in all groups, the most disadvantaged groups tended to rank their SES based on different criteria than the more well off and relied less on objective SES. Furthermore, Hussain-Gambles and colleagues19 reported that older South Asians with low SES tended to be more mistrustful of researchers, which may have also influenced the participants' responses to the question on SES reported in the article.

Back to Top | Article Outline
Inconsistent Answers for the Screening Questions Versus the Internet Survey Questions

Because we checked the IP address of each visitor to the Internet survey site and automatically blocked participation by multiple persons from the same IP address, we thought that we had strict control over preventing unauthentic cases and minimizing potential threats to reliability. However, using quota sampling added an additional dimension to the study because some potential participants who were rejected because of the quota requirements returned to the Internet site and claimed a new identity with different SESs and MSs using a different IP address. We did not detect this problem initially because the women used an IP address different from what they used the first time they tried to take the survey. During data analysis, we subsequently discovered that some participants' answers for the screening questions were different from their answers for questions about SES and MS on the Internet survey. Indeed, as Coomber20 asserted, even when the desired sample is of Internet populations themselves, significant technical and operational problems remain in terms of how to ensure that the population targeted is, in fact, the population that responds. This is especially relevant because monetary incentives were used in this study, which we believe attracted more participants who were unauthentic.

Back to Top | Article Outline
Resentment Toward the Research Project and/or Researchers due to Rejection

In several instances, we found that women who were rejected from the Internet survey site (because they did not meet quota requirements) became upset. In one instance, a woman contacted the IT office of the university where the study was conducted and checked whether the study was actually being conducted there. Then, she posted a series of hostile messages about our study on the message boards of the Internet communities/groups with which she was affiliated. Because Internet interactions are non-face-to-face, the woman's response to the rejection based on the screening questions was more hostile than we imagined such a reaction would be. To prevent future problems, on the informed consent form on the project Web site that potential participants would read, we clarified in red font that participants could be rejected by the Internet survey site if the quota to which they belonged had already been filled.

Back to Top | Article Outline
A Longer Time and More Expense Than Anticipated

Quota sampling is known to be inexpensive and effective in reaching out to underrepresented groups of potential participants.7 Yet, compared with other conventional nonprobability sampling methods used in Internet studies, quota sampling required a longer time to recruit potential participants, thus making the project more expensive. As pointed out above, most of the quotas were filled by the end of the third year, but the quota for lower socioeconomic classes was not filled until the end of the fourth year. In other words, if we had not used a quota sampling method, the target number of participants (500) would have been recruited in a time frame much shorter than 4 years. Furthermore, additional computer programming was required to use quota sampling in the study. In addition to questions to determine which potential participants met the inclusion criteria, we added screening questions to the Internet survey to check whether the quota requirements were met, which was also an additional cost.

Back to Top | Article Outline

SUGGESTIONS FOR FUTURE RESEARCH

In the study discussed in this article, quota sampling was very effective in ensuring that an adequate number of midlife women were recruited from the four targeted ethnic groups; however, we also encountered some practical issues using quota sampling to recruit midlife women. Based on the issues discussed in this article, we conclude by offering the following suggestions for future Internet research. First, we suggest that researchers consider how significant quota sampling will be to their study. Despite the positive aspects of quota sampling (improved representation of samples), it also requires more time and effort. In the study presented in this article, quota sampling was critical to recruiting an adequate number of women from four targeted ethnic groups because ethnic differences in menopausal symptoms were the focus of the study. Thus, recruiting an adequate number of women from each of the four ethnic groups was essential. However, if quota sampling does not significantly contribute to the study's purpose, design, or plan, it would be unnecessary in terms of time and expense.

Second, we suggest that researchers consider adopting specific strategies to prevent unauthentic participants from being recruited. In the study described in this article, the monetary compensation that we provided to research participants may have induced some participants to be untruthful about their identities. Thus, using some kind of nonmonetary compensation would probably help prevent such instances. In the future, advances in Internet technologies will provide new ways to ensure the authenticity of individuals' identities in Internet interactions.17 However, until such technology is available, researchers should regularly update their knowledge and skills related to Internet interactions and technologies by subscribing to journals and attending conferences, tutorials, seminars, and panel discussions, to deal adequately and appropriately with authenticity issues in Internet research.

Third, questions about SES should be further examined. Although the question on SES that we used in this study was based on findings in the current literature, we believe that the question did not work well with Asian and Hispanic women because of the women's different perceptions of subjective SES, which may be due to cultural differences in values, beliefs, and attitudes related to SES. Further investigations on the cultural values, beliefs, and attitudes related to questions on SES in Internet research are essential for accurately measuring SES, especially among ethnic minorities.

Finally, we suggest the use of cultural/community consultants who would serve as informal leaders of the potential participants who are underrepresented on the Internet. Although we did not formally include cultural/community consultants in the study discussed in this article, three informal leaders from ICMWs and ICEMs who volunteered to help recruit women from the lower socioeconomic classes were extremely helpful. By using several cultural/community consultants, researchers could have access to potential participants who would not be easily reached by announcing the study through ICMWs and ICEMs only.

Back to Top | Article Outline

REFERENCES

1. Fox S. Generations online. http://www.pewinternet.org/. Accessed October 20, 2007.

2. Schonlau M, Fricker RD, Elliott MN. Conducting Research Surveys Via Email and the Web. Santa Monica, CA; Rand Corporation; 2002.

3. Malhota NK. Marketing Research. An Applied Orientation. International Edition. 3rd ed. London: Prentice Hall; 1999.

4. Im EO, Chee W. Issues in Internet research. Nurs Outlook. 2003;51(1):6-12.

5. Berrens RP, Bohara AK, Jenkins-Smith H, Silva C, Weimer DL. The advent of Internet surveys for political research: a comparison of telephone and Internet samples. Polit Anal. 2003;11(1):1-22.

6. River D. Fulfilling the promise of the Web. http://quirks.com/articles/a2000/200000205.aspx?searchID=14604140&sort=9. Accessed June 28, 2008.

7. Henry GT. Practical Sampling. Applied Social Research Methods Series, Volume 21. Newbury Park, CA: Sage; 1990.

8. Polit DF, Beck CT. Nursing Research. Principles and Methods. 7th ed. Philadelphia, PA: Lippincott; 2003.

9. Burns N, Grove SK. The Practice of Nursing Research: Conduct, Critique, & Utilization. 5th ed. Philadelphia, PA: WB Saunders; 2004.

10. Gavalier JS, Bonham-Leyba M, Castro CA, Harman SE. The Oklahoma Postmenopausal Women's Health Study: recruitment and characteristics of American Indian, Asian, African American, Hispanic, and Caucasian women. Alcohol Clin Exp Res. 1999;23:220.

11. Gilliss CL, Lee K, Gutierrez Y, et al. Recruitment and retention of healthy minority women into community-based longitudinal research. J Womens Health Gend Based Med. 2001;10(1):77-85.

12. Stoy DB, Curtis RC, Dameworth KS, et al. The successful recruitment of elderly African American subjects in a clinical trial: the CRISP experience. J Natl Med Assoc. 1995;87:280.

13. Ho SC, Chan SG, Yip YB, Chan SY, Sham A. Factors associated with menopausal symptoms reporting in Chinese midlife women. Maturitas. 2003;44:149-156.

14. Ramsey SD, Andersen MR, Etzioni R, et al. Quality of life in survivors of colorectal carcinoma. Cancer. 2000;88(6):1294-1303.

15. Im EO, Lee BI, Chee W, Brown A, Dormire S. A national internet survey on menopausal symptoms among four major ethnic groups in the U.S. West J Nursing Res. 2010;32(4):540-565.

16. Kollock P, Smith MA. Communities in cyberspace. In Smith MA, Peter K, eds. Communities in Cyberspace. London: Routledge; 1999:3-28.

17. Im EO, Chee W. Issues in Internet survey among cancer patients. Cancer Nurs. 2004;27(1):34-42.

18. Franzini L, Fernandez-Esquer ME. The association of subjective social status and health in low-income Mexican-origin individuals in Texas. Soc Sci Med. 2006;63(3):788-804.

19. Hussain-Gambles M, Leese B, Atkin K, Brown J, Mason S, Tovey P. Involving South Asian patients in clinical trials. Health Technol Assess. 2004;8(42):1-109.

20. Coomber R. Using the Internet for survey research. Sociol Res Online. 1997;2(2):1.1-8.1.

For more than 27 additional continuing education articles related to research, go to NursingCenter.com\CE.

Keywords:

Internet research; Practical issues; Quota sampling

© 2011 Lippincott Williams & Wilkins, Inc.

 

Login

Search for Similar Articles
You may search for similar articles that contain these same keywords or you may modify the keyword list to augment your search.

Featured Collections