Secondary Logo

Journal Logo

Supplement Article

Seeing Is Believing? Unique Capabilities of Internet-Only Studies as a Tool for Implementation Research on HIV Prevention for Men Who Have Sex With Men: A Review of Studies and Methodological Considerations

Grov, Christian PhD, MPHa,b; Westmoreland, Drew PhD, MSPHb; Rendina, H. Jonathon PhD, MPHc,d; Nash, Denis PhD, MPHb,e

Author Information
JAIDS Journal of Acquired Immune Deficiency Syndromes: December 2019 - Volume 82 - Issue - p S253-S260
doi: 10.1097/QAI.0000000000002217
  • Free

Abstract

INTRODUCTION

In the United States, men who have sex with men (MSM) are disproportionally impacted by HIV, especially younger MSM and MSM of color. MSM are 44 times more likely to contract HIV than other men and accounted for 67% of new infections in 2016 (82% of infections among men).1,2 In 2016, the CDC estimated that, if current trends in HIV infection go unabated, one-in-six MSM will acquire HIV in their lifetime, including one-in-two black MSM and one-in-four Latino MSM.3 These ongoing disparities, despite promising highly effective biomedical prevention methods [pre-exposure prophylaxis4 (PrEP) and treatment as prevention (TasP)5,6], highlight the continued need for novel strategies that will engage those who would benefit most from effective treatment and prevention.

In 2014, Grov et al7 published a comprehensive review cataloguing both MSMs' sexual behavior transitions in online environments from the 1990s through 2013, as well as researchers' efforts to use the internet to engage MSM in research, treatment, and prevention. Their piece noted how, in the 1990s and early 2000s, MSM used computer-based men-for-men websites to meet partners (eg, manhunt.net), and more recently, MSM have been transitioning to the use of geosocial networking apps installed on smartphones. In this article, we focus on research and events having occurred in the half decade since the Grov et al publication. In so doing, we use an implementation science8 lens as it can be applied to internet-mediated research. This includes a discussion of potential pitfalls, as well as areas for caution or further consideration. Of note, however, our foci in these articles are more on the United States and other Westernized countries. For more historical context, see also Chiasson et al,9 for their 2006 piece on the use of the internet for HIV research.

RECENT DEVELOPMENTS IN TECHNOLOGY AND HIV PREVENTION

Today, most US adults own smartphones. Back in 2011, 35% of American adults owned smartphones. By 2018, that number increased to 77%, with 95% ownership among adults under the age of 30 years.10 Geosocial networking apps on smartphones—that is, apps that allow you to interact with other users based on their proximity to you, namely Grindr and Scruff—and other forms of social media (Twitter, Facebook, and Instagram) have since established themselves as the most common mediums through which MSM meet sex partners. Grindr and Scruff have millions of daily users worldwide and have emerged as novel venues for sexual health research, recruitment, and engagement.7,11–17 Beyond benefitting the research community, these apps can be used to seamlessly deliver public health prevention messaging and HIV-prevention interventions.18

Second, biomedical prevention methods—PrEP, at-home rapid HIV testing, and TasP—have entered the lexicon and the proverbial HIV prevention toolbox (which includes behavioral prevention methods such as condoms, serostatus disclosure/serosorting). Although PrEP received FDA approval in 2012, we are only now beginning to see PrEP's potential to reduce HIV incidence at the population level.19–21 Likewise, it was not until 2015 and 2016 that Scruff and Grindr added “HIV-negative, on PrEP,” as a profile feature, and its users could check-off (and filter by).22,23

Also in 2012, the FDA approved OraQuick as an at-home self-administered rapid HIV antibody test, which can now be purchased over-the-counter at many pharmacies, or bought online for about $40-$50.24 In 2015, in an effort to scale up the distribution of home test kits and increase HIV status awareness, the NYC department of health launched a giveaway to distribute in-home tests to MSM at no charge, by mail. Participants (n = 1763) were recruited over the course of 23 days, through ads on dating apps and websites.18 The success of the project represented the potential for collaboration between health departments and geosocial networking apps for HIV prevention through the distribution of HIV-test kits to MSM.

Finally, and most recently, TasP—whereby health messaging explicitly states that HIV cannot be transmitted by someone who is virally suppressed due to antiretroviral medication—has increasingly been recognized as a biomedical prevention method.25 In 2016, the Prevention Access Campaign issued a consensus statement indicating that individuals with undetectable HIV viral loads present negligible risk for HIV transmission to their partners.26 There has since been consistent increasing support that HIV undetectable individuals do not transmit HIV,27 including endorsement by the CDC.28 In 2015 and 2016, Scruff and Grindr further added TasP as a profile feature that users could check off.22,23 Also in 2016, the Prevention Access Campaign launched its slogan “Undetectable = Untransmittable” (or U = U) to highlight that treatment-as-prevention works.

In the midst of this rapidly shifting technological landscape, researchers have continued to expand their use of online technology to engage MSM in a range of activities—these include identifying and enrolling participants for in person assessments,29–35 conducting fully online observational studies (ie, assessments),11,36–47 as well as delivering interventions entirely online.32,35,43,48–57 This has afforded researchers the opportunity to identify, recruit, and study MSM who may not live in urban centers or are otherwise geographically dispersed or isolated.7,11,45 One benefit of using technology to reach a more geographically diverse sample is to deliver at-home rapid HIV antibody tests and self-sampling kits for urethral and rectal sexually transmitted infections (STIs), to participants who may otherwise lack access to them.11,45 Furthermore, advanced technology has enhanced opportunities to quickly enroll online samples at relatively low cost58 and to recruit participants into in-person studies at lower cost than previous field-based recruitment efforts.34 In the next section, we discuss methodological considerations for conducting online studies (be they computer-based or smartphone-based).

METHODOLOGICAL CONSIDERATIONS

Researchers have expanded their employment of internet-mediated methods for the recruitment and engagement of key populations for HIV research, treatment, and prevention. However, with few exceptions (c.f. Bauermeister et al59), little has been published by way of “best practices” to ensure methodological rigor. In the next section, we provide brief overviews of important considerations for conducting online HIV prevention research and do so through an implementation science lens. Implementation science is a relatively new field,60 having emerged in the last 15 years. It is defined as “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice and, hence, to improve the quality and effectiveness of health services.”60,61 And, “implementation science, as a science, takes as part of its mission an explicit goal of developing generalizable knowledge that can be widely applied beyond the individual system under study.”8 Online studies are one of several approaches that can be used by implementation researchers to gather information on the uptake and engagement of individuals in HIV prevention interventions, as well as barriers and enablers to uptake and engagement. Importantly, because recruitment requires no more than a smartphone with an internet connection, they have the potential to reach participants who are outside the health care system more easily and quickly than community-based household or telephone surveys. This provides the critical ability to assess: (1) The extent to which access to the health care system is itself a barrier to effective uptake and engagement among populations that implementers intend to reach with an intervention; (2) Whether implementation strategies housed outside the health care system (eg, online) are effective at improving uptake and engagement (Figure 1).

FIGURE 1.
FIGURE 1.:
Summary of traditional outcomes research and implementation science research processes highlighting differences between internet-only research and in-person or hybrid research strategies. Select web-based strengths and challenges are emphasized for both traditional outcomes research and implementation science evaluations.

Sources of Recruitment

In their 2014 piece, Grov et al7 noted that much internet-based HIV prevention research enrolled participants through computer-based men-for-men sexual networking websites, and that there was a trending shift toward the use of geosocial networking apps on smartphones. Today, it is clear that use of geosocial networking apps as well as other forms of social media (eg, Twitter, Facebook, and Instagram) is ubiquitous among MSM and can be harnessed to engage potential participants. Scruff and Grindr support millions of daily users, whereas there are 2.4 billion Facebook users worldwide (half of all Americans use Facebook) and more than 1 billion people use Instagram each month.

Ads can be targeted to individuals based on their geolocation, and because Facebook owns Instagram, ads purchased on Facebook are currently cross-advertised on Instagram. For a period, Facebook advertisers were permitted to select demographic characteristics for whom they wanted their ads shown, but this feature was removed when concerns arose about the systematic exclusion of certain groups (eg, people of color) from ads for housing—a violation of the Fair Housing Act.62 Today, although Facebook does not allow advertisers to target ads directly toward gay and bisexual men, one can target ads to individuals who share likes and interests that are common among LGBT communities (eg, Lady Gaga, Ru Paul's Drag Race). Certainly, this can introduce some selection bias, as the likes and interests of LGBT communities are not homogenous.

Facebook is also host to numerous pages of groups who share common interests. For example, as of March 2019, “Gay Meme Nation” (a group devoted to gay memes) had more than 12,000 members and “PrEP facts” (a group devoted to the dissemination of PrEP) had over 21,000. However, in addition to Facebook, other social media platforms should be considered and leveraged especially considering differential usage among various age groups.63 For instance, Instagram meme account “Best of Grindr” boasts 1.4 million followers, and numerous accounts devoted to similar content have followings upward of 100,000 users. In other areas of health research, researchers are proposing to use social media monitoring of Twitter hashtags related to specific health topics to recruit participants for studies,64 and similar hashtag monitoring has proven feasible for gender minority research.65

There has also been recent growth in crowdsourced online panels (eg, Qualtrics, Amazon's MTurk) whereby researchers can prepopulate target demographics (eg, gender, and age), and panel members can choose to opt-in and complete surveys for some remuneration.66 Beymer et al67 compared samples of MSM obtained through MTurk, Qualtrics, and an HIV/STI clinic and found that the clinic-based sample demonstrated more demographic diversity and greater HIV-risk behaviors when compared with the online samples. However, the online samples were more likely than the clinic sample to correctly answer an attention-check question—a pattern that has been observed by other researchers.68

One of the primary strengths of all the aforementioned potential sources of recruitment lies in their ability to reach an exceptionally large number of individuals, quickly. However, like all recruitment strategies, consideration must also be given to any requirements related to representativeness. For example, individuals who do not use these platforms would not be represented, and methods designed to hone in on LGBT communities based on broad “likes,” and interests can inappropriately assume homogeneity among these communities, which is not the case.

Cross-Sectional and Longitudinal Retention Considerations

Fully online studies are well suited for cross-sectional research, whereby a limited commitment from participants is required to gather essential data to answer discrete research questions, or to take a “snapshot” of a given issue (eg, current prevalence of sexual behavior, substance use, and attitudes toward PrEP). Online studies are also well situated to follow participants for short periods (eg, daily or weekly online diaries).69–72 Some studies have managed to follow participants for several years. For example, the One Thousand Strong cohort study followed a sample of 1071 gay and bisexual men for a period of 3 years,11,45 whereby participants were identified through an existing marketing panel of LGBT adults. Participants in that study completed annual online surveys, as well as at-home self-administered rapid HIV testing (results submitted through photograph of the test paddle) and self-collected samples for urethral and rectal chlamydia and gonorrhea testing (samples returned to a study laboratory through mail). The study's retention (ie, 93% at 36-months) was likely enhanced by the use of brief videos that introduced and explained the content of each assessment, incentives, and by having enrolled a sample that was predisposed to completing online surveys (ie, through an existing panel). Presently, the NIH is supporting multiple large-scale epidemiological cohort studies that are following thousands of MSM, as well as transgender men and transgender women.73 In these cohorts, most participants were recruited through ads on geosocial networking apps to complete annual online assessments and self-collect at-home HIV testing.

However, retention is a concern for longitudinal online studies—rates range from 15% to 93%.11,45,74–78 Comparatively, some rates seem much lower than typical in-person study retention rates, but the very nature and logistics of an online study mean that standard retention rate goals should be re-evaluated to fit these study venues.76 Retention practices should also be tailored to meet the unique needs of online studies.75 Some have suggested that retention and participation in online studies, particularly longitudinal intervention studies, could be enhanced by face-to-face components such as video chatting.79 Of concern, however, would be that some participants many not feel comfortable with face-to-face chatting given that it will reduce anonymity.

Incentives

It is well documented that incentives improve the speed of recruitment, participation (response rates),80 completion,81–83 and longitudinal retention, and this remains true for fully online studies.76 However, given the convenience that the internet affords study participants (to complete questionnaires and specimen collection at a location of their choosing and often at a time of their choosing), incentives for online studies do not necessarily need to be commensurate with those with a face-to-face or in-person assessments.76,81,82,84–89 For example, Beymer et al67—in their study of gay and bisexual men on MTurk, Qualtrics, and in clinic-based settings—first offered MTurk participants $0.50 to complete an ∼10-minute survey. Only 3 MTurk participants responded in the first month of enrollment, so they increased the incentive to $1 and enrolled 264 participants in 4 days. They paid Qualtrics $6 per participant (of which participants themselves received $2–3) and enrolled 211 participants in 7 days. In comparison, they offered $10 gift cards to participants recruited face-to-face in clinic-based settings, and over a period of 9 months, 231 participants were enrolled. Enhancing participation and retention are both important considerations, however, so is representativeness. To the best of our knowledge, there is little published by way of examining how differential incentive rates may impact relative representativeness. One study, involving face-to-face and online assessments of MSM taking PrEP noted that 30.1% of participants included financial compensation as among the reasons they joined the study (only 7.8% said it was the sole reason). However, the study also found that white participants were more likely to indicate compensation as a motivation to join than participants of color (39.2% vs. 19.6%).90 Clearly, this is an important area for future research.

However, a word of caution, given the anonymity the internet can afford study participants and monetary incentives for participating in research, it is also important for researchers to put in place mechanisms to prevent repeat (duplicate) participation, thwart fraudulent participants, and spam bots.84 These mechanisms can include using CAPTCHA (completely automated public Turing test to tell computers and humans apart), as well as tracking IP addresses, using cookies, requiring unique email addresses, or requiring text message verification at a unique phone number.84 Other mechanisms include the use of attention-control checks and tracking time-to-survey completion.66 Of caution, however, there is some evidence that attention-control checks can introduce other kinds of bias, particularly with regard to false positives.91 For a review of recommendations to avoid fraudulent participants in online studies, see Teitcher et al.92

Confidentiality

As with face-to-face assessments, confidentially is equally important in online settings. The fact that participants can complete assessments in the privacy of their homes and on their personal computing devices has the potential to increase confidentiality; however, special considerations are also needed because there are multiple opportunities for a confidentiality breach, some of which cannot be controlled by researchers in online studies. This could include evidence of participation in a study by way of browser history, unencrypted survey data being viewed by a third party during its transmission from the participant to wherever data are being stored, and hacking of data stored in a cloud server. As with face-to-face studies, researchers should take care to ensure that participant identifiable information is kept separate from participant data itself, to encrypt and password protect files, and store files on password-protected computers.

Should the online study also include self-collection of biological samples (eg, at-home HIV testing and STI sampling), the process for directing materials to the participant should be discreet, and participants should be asked whether it is possible that a third party could intercept their mail (eg, parent, roommate, partner, and coworker). Alternately, materials could be directed to a self-service parcel delivery service (eg, Amazon Locker) where participants can pick up materials discreetly at their leisure—assuming such a service is available within a reasonable distance from the participant. Similarly, the process for returning materials to a study location or laboratory should not disclose the nature of the destination (eg, STI testing laboratory) or package contents. Participants should be instructed to discreetly and safely dispose of any materials after testing is complete (eg, HIV-test kit packaging). Digital communication with participants also has the potential to breach confidentiality. For example, the content of text messages might appear on their phone even if the device is locked, and thus, messages should be carefully worded to protect the privacy of participants.

Furthermore, given the recent attention to scandals involving users' data being harvested on social media and sold to third parties,93,94 it is also important to monitor how much end users (ie, potential participants) trust that any data they provide as part of a research study will be appropriately used, and that their confidentiality will be protected.95 Rendina and Mustanski,95 in a 2017 online study of over 11,000 MSM from across the United States, examined participant perspectives on the issues of trust, privacy, and data sharing in online and mobile research. They found that trust in online research was greater than trust in online and mobile platforms for personal use, such as social and sexual networking apps. Participants expressed the least concerns about privacy when such data were going to be shared anonymously with researchers and the most concern when these data were going to be sold anonymously to third parties. Participants were most willing to share information they disclose publicly within the app (eg, profile information on characteristics like age and height) and least willing to share information that could be collected by the app automatically (eg, GPS location or device usage information).

Representativeness

As we have noted, one of the greatest strengths of internet-based studies is their ability to reach a large number of geographically diverse individuals in a relatively short period. Subsequently, it becomes easier to reach narrowly defined subgroups and to enroll individuals who might not traditionally be represented or are otherwise more difficult to reach. That being said, representativeness and generalizability—both central features in implementation science—are likely to be limited due to bias in response rates and sample selection bias from online platform sources. For example, Instagram and Snapchat users tend to be younger than Facebook users, and many teens today are reticent to join or use Facebook, a platform their parents have been using for a decade. In fact, in 2018, Newsweek declared that “Facebook is officially for old people.”96 Meanwhile persons of color tend to be better represented on Twitter than on other platforms.97,98 Furthermore, in addition to representativeness of populations across varying platforms themselves, as previously noted, the representativeness of those responding on platforms must also be considered. As noted, methods designed to reach Facebook users based on popular interests among LGBT communities could systematically exclude those members who do not share those interests. Likewise, studies have also noted that features of recruitment materials themselves (ie, using ads that feature models who are persons of color) can impact racial differences in response rate (ie, participants of color are more likely to respond to ads where they are featured).99

It is important to remember, however, that population-level representativeness (eg, of all MSM) may not be the primary goal of HIV prevention research, which is often targeted to narrower and often more vulnerable groups of individuals. Moreover, it is not a prerequisite to impactful research and knowledge generation because the participants' most randomized trials in HIV prevention research are not representative of the target populations, and this does not need to compromise their potential usefulness as it relates to other populations and settings.100 Any given study's enrollment criteria (eg, must be sexually active and must report condomless sex) will automatically circumspect the study's ability to generalize, for example, all gay and bisexual men, which is rarely the goal in most HIV prevention research. Furthermore, because not all individuals have access to the internet, nor necessarily use social media or geosocial apps, fully online studies may not immediately generalize to the population who meets enrollment criteria but were otherwise not reached through online means.

CONCLUSION

The internet, be accessed through a smart phone, tablet, or computer, has established itself as a conduit through which researchers can identify potential participants to conduct HIV prevention research as well as to deliver interventions. As a result, it can be an exceptionally valuable medium through which to conduct implementation science research. Compared with studies relying on face-to-face recruitment, online studies afford researchers many advantages including geographic reach and the timely identification and enrollment of participants, and timely availability of data for analysis and dissemination, often with the expenditure of fewer resources. It can also reach populations that might be otherwise difficult to engage in face-to-face settings and thus could make results more relevant to real-world practice settings. The combined growth in popularity of geosocial networking and the forthcoming roll out of fifth generation (5G) high-speed mobile networks are likely to be essential to the future of HIV prevention research, as well as health research more broadly. As a greater number of devices become connected to the internet, this will open new opportunities for researchers to understand human behavior, beyond self-reported online surveys.

However, like any study design attribute, researchers must always evaluate the strengths of using the internet for HIV prevention research in light of its weaknesses—for example, the potential for participants to be distracted, difficulty ensuring unique and valid participants—and new challenges with regard to privacy and data security. New approaches, such as hybrid online and in person studies, where participants are recruited online and have one or more in-person encounters, may help advance our ability to conduct rigorous prospective studies and to collect more clinical and biological data. There is also a need for metadata and metaresearch around online studies to help document, evaluate, and inform best practices, including those that can maximize retention in longitudinal studies.

REFERENCES

1. Centers for Disease Control and Prevention. HIV and gay and bisexual men. 2018. Available at: https://www.cdc.gov/hiv/group/msm/index.html. Accessed October 17, 2018.
2. Centers for Disease Control and Prevention. HIV among gay and bisexual men. 2017. Available at: https://www.cdc.gov/nchhstp/newsroom/docs/factsheets/cdc-msm-508.pdf. Accessed November 21, 2017.
3. Centers for Disease Control and Prevention. Half of black gay men and a quarter of Latino gay men projected to be diagnosed within their lifetime. 2016. Available at: http://www.cdc.gov/nchhstp/newsroom/2016/croi-press-release-risk.html. Accessed April 26, 2016.
4. Fonner VA, Dalglish SL, Kennedy CE, et al. Effectiveness and safety of oral HIV preexposure prophylaxis for all populations. AIDS. 2016;30:1973–1983.
5. Cohen MS, McCauley M, Gamble TR. HIV treatment as prevention and HPTN 052. Curr Opin HIV AIDS. 2012;7:99–105.
6. Padian NS, McCoy SI, Karim SSA, et al. HIV prevention transformed: the new prevention research agenda. Lancet. 2011;378:269–278.
7. Grov C, Breslow AS, Newcomb ME, et al. Gay and bisexual men's use of the Internet: research from the 1990s through 2013. J Sex Res. 2014;51:390–409.
8. Bauer MS, Damschroder L, Hagedorn H, et al. An introduction to implementation science for the non-specialist. BMC Psychol. 2015;3:32.
9. Chiasson MA, Parsons JT, Tesoriero JM, et al. HIV behavioral research online. J Urban Health. 2006;83:73–85.
10. Pew Internet. Mobile Fact Sheet. 2018. Available at: http://www.pewinternet.org/fact-sheet/mobile/. Accessed January 9, 2018.
11. Grov C, Cain D, Whitfield TH, et al. Recruiting a US national sample of HIV-negative gay and bisexual men to complete at-home self-administered HIV/STI testing and surveys: challenges and opportunities. Sex Res Social Policy. 2016;13:1–21.
12. Pequegnat W, Rosser BS, Bowen AM, et al. Conducting internet-based HIV/STD prevention survey research: considerations in design and evaluation. AIDS Behav. 2007;11:505–521.
13. Newcomb ME, Mongrella MC, Weis B, et al. Partner disclosure of PrEP use and undetectable viral load on geosocial networking apps: frequency of disclosure and decisions about condomless sex. J Acquir Immune Defic Syndr. 2016;71:200–206.
14. Goedel WC, Duncan DT. Geosocial-networking app usage patterns of gay, bisexual, and other men who have sex with men: survey among users of Grindr, a mobile dating app. JMIR Public Health Surveill. 2015;1:e4.
15. Hambrick HR, Park SH, Goedel WC, et al. Rectal douching among men who have sex with men in Paris: implications for HIV/STI risk behaviors and rectal microbicide development. AIDS Behav. 2018;22:379–387.
16. Phillips G II, Magnus M, Kuo I, et al. Use of geosocial networking (GSN) mobile phone applications to find men for sex by men who have sex with men (MSM) in Washington, DC. AIDS Behav. 2014;18:1630–1637.
17. Phillips G II, Grov C, Mustanski B. Engagement in group sex among geosocial networking mobile application-using men who have sex with men. Sex Health. 2015;12:495–500.
18. Edelstein Z, Salcuni P, Khawja A, et al. Results from the HIV home test giveaway, New York City, 2015. Presentation given at the Annual meeting of the American Public Health Association; November 2, 2016; Denver, CO. Available at: https://apha.confex.com/apha/144am/meetingapp.cgi/Paper/353199#_blank. Accessed Febraury 20, 2019.
19. Chelsea and Westminster Hospital NHS Foundation Trust. 42% drop in new HIV diagnoses at 56 Dean Street in just 12 months. 2017. Available at: https://www.chelwest.nhs.uk/about-us/news/news-archive/2017/40-drop-in-new-hiv-diagnoses-at-56-dean-street-in-just-12-months. Accessed March 5, 2019.
20. Land E. San Francisco reaches new historic low number of HIV infections in 2017. 2018. Available at: https://betablog.org/san-francisco-reaches-new-historic-low-number-of-hiv-infections-in-2017/. Accessed March 5, 2019.
21. Ogaz D, Miltz AR, Saunders JB, et al. Preparing for PrEP in England: prevalence and incidence of HIV and bacterial STIs. Paper presented at: Conference on Retroviruses and Opportunistic Infections; March 4–7, 2019; Seattle, WA.
22. Tharrett M. Grindr adds new filters for HIV status, PrEP use. 2016. Available at: http://www.newnownext.com/grindr-hiv-prep-filter/11/2016/. Accessed Febraury 20, 2018.
23. Johnson J. On Scruff, coming out about HIV prevention just got a little bit easier. 2015. Available at: http://www.thebody.com/content/76671/on-scruff-coming-out-about-hiv-prevention-just-got.html. Accessed February 20, 2019.
24. International Association for Providers in AIDS care. 2017. Available at: https://www.iapac.org/uploads/IAPAC-ASLM-HIVST-FTC-Recommendations-012617.pdf. Accessed January 9, 2019.
25. Grov C, Rendina HJ, Patel VV, et al. Prevalence of and factors associated with the use of HIV serosorting and other biomedical prevention strategies among men who have sex with men in a US nationwide survey. AIDS Behav. 2018;22:2743–2755.
26. Prevention Access Campaign. What is undetectable = untransmittable? 2016. Available at: https://www.preventionaccess.org/undetectable. Accessed March 3, 2017.
27. Bavinton BR, Grinsztej B, Phanuphak N, et al. HIV treatment prevents HIV transmission in male serodiscordant couples in Australia, Thailand and Brazil. International AIDS Society; Paris, France; July 2017.
28. McCray E, Mermin JH. Dear colleague: information from the CDC's division of HIV/AIDS prevention. 2017. Available at: https://www.cdc.gov/hiv/library/dcl/dcl/092717.html. Accessed January 16, 2018.
29. Vial AC, Starks TJ, Parsons JT. Finding and recruiting the highest risk HIV-negative men who have sex with men. AIDS Educ Prev. 2014;26:56–67.
30. Hernandez-Romieu AC, Sullivan PS, Sanchez TH, et al. The comparability of men who have sex with men recruited from venue-time-space sampling and facebook: a cohort study. JMIR Res Protoc. 2014;3:e37.
31. Grov C, Rendina HJ, Parsons JT. Comparing three cohorts of MSM sampled from sex parties, bars/clubs, and Craigslist.org: implications for researchers and providers. AIDS Educ Prev. 2014;26:362–382.
32. Hightow-Weidman LB, Muessig KE, Pike EC, et al. HealthMpowerment.org: building community through a mobile-optimized, online health promotion intervention. Health Educ Behav. 2015;42:493–499.
33. Pachankis JE, Rendina HJ, Ventuneac A, et al. The role of maladaptive cognitions in hypersexuality among highly sexually active gay and bisexual men. Arch Sex Behav. 2014;43:669–683.
34. Vial AC, Starks TJ, Parsons JT. Relative efficiency of field and online strategies in the recruitment of HIV-positive men who have sex with men. AIDS Educ Prev. 2015;27:103–111.
35. Lelutiu-Weinberger C, Pachankis JE, Gamarel KE, et al. Feasibility, acceptability, and preliminary efficacy of a live-chat social media intervention to reduce HIV risk among young men who have sex with men. AIDS Behav. 2014;19:2014–1227.
36. Holloway IW, Rice E, Gibbs J, et al. Acceptability of smartphone application-based HIV prevention among young men who have sex with men. AIDS Behav. 2014;18:285–296.
37. Hirshfield S, Grov C, Parsons JT, et al. Social media use and HIV transmission risk behavior among ethnically diverse HIV-positive gay men: results of an online study in three states. Arch Sex Behav. 2015;44:1969–1978.
38. Grov C, Rendina HJ, Breslow AS, et al. Characteristics of men who have sex with men (MSM) who attend sex parties: results from a national online sample in the U.S. Sex Transm Infect. 2014;90:26–32.
39. Rendina HJ, Breslow AS, Grov C, et al. Interest in couples-based voluntary HIV counceling and testing in a national U.S. sample of gay and bisexual men: the role of demographic and HIV risk factors. Arch Sex Behav. 2014;43:149–159.
40. Rendina HJ, Jimenez RH, Grov C, et al. Patterns of lifetime and recent HIV testing among men who have sex with men in New York City who use Grindr. AIDS Behav. 2014;18:41–49.
41. Grov C, Rodriguez-Diaz CE, Ditmore MH, et al. What kinds of workshops do Internet-based male escorts want? Implications for prevention and health promotion. Sex Res Social Policy. 2014;11:176–185.
42. Mustanski B, Rendina HJ, Greene GJ, et al. Testing negative means I'm lucky, making good choices, or immune: diverse reactions to HIV test results are associated with risk behaviors. Ann Behav Med. 2014;48:371–383.
43. Mustanski B, Greene GJ, Ryan D, et al. Feasibility, acceptability, and initial efficacy of an online sexual health promotion program for LGBT youth: the Queer Sex Ed intervention. J Sex Res. 2015;52:220–230.
44. Oldenburg CE, Perez-Brumer AG, Hatzenbuehler ML, et al. State-level structural sexual stigma and HIV prevention in a national online sample of HIV-uninfected MSM in the United States. AIDS. 2015;29:837–845.
45. Grov C, Cain D, Rendina HJ, et al. Characteristics associated with urethral and rectal gonorrhea and chlamydia diagnoses in a U.S. national sample of gay and bisexual men: results from the One Thousand Strong panel. Sex Transm Dis. 2016;43:165–171.
46. Young SD, Belin TR, Klausner JD, et al. Methods for measuring diffusion of a social media-based health intervention. Soc Netw. 2015;4:41–46.
47. Chiu CJ, Young SD. The relationship between online social network use, sexual risk behaviors, and HIV sero-status among a sample of predominately African American and Latino men who have sex with men (MSM) social media users. AIDS Behav. 2015;19:98–105.
48. Young S, Holloway I, Jaganath D, et al. Project HOPE: online social network changes in an HIV prevention randomized controlled trial for African American and Latino men who have sex with men. Am J Public Health. 2014;104:1707.
49. Ybarra ML, DuBois LZ, Parsons JT, et al. Online focus groups as an HIV prevention program for gay, bisexual, and queer adolescent males. AIDS Educ Prev. 2014;26:554–564.
50. Hirshfield S, Downing MJ Jr, Parsons JT, et al. Developing a video-based eHealth intervention for HIV-positive gay, bisexual, and other men who have sex with men: study protocol for a randomized controlled trial. JMIR Res Protoc. 2016;5:e125.
51. Ybarra ML, Prescott TL, Philips GL, et al. Iteratively developing an mHealth HIV prevention program for sexual minority adolescent men. AIDS Behav. 2016;20:1157–1172.
52. Patel VV, Ginsburg Z, Golub SA, et al. Empowering with PrEP (E-PrEP), a peer-led social media–based intervention to facilitate HIV preexposure prophylaxis adoption among young Black and Latinx gay and bisexual men: protocol for a cluster randomized controlled trial. JMIR Res Protoc. 2018;7:e11375.
53. Bauermeister J, Muessig K, LeGrand S, et al. HIV and sexuality stigma reduction through engagement in online forums: results from the HealthMPowerment intervention. AIDS Behav. 2019;23:742–752.
54. Hightow-Weidman LB, LeGrand S, Muessig KE, et al. A randomized trial of an online risk reduction intervention for young black MSM. AIDS Behav. 2019;23:1166–1177.
55. Bauermeister JA, Pingel ES, Jadwin-Cakmak L, et al. Acceptability and preliminary efficacy of a tailored online HIV/STI testing intervention for young men who have sex with men: the Get Connected! program. AIDS Behav. 2015;19:1860–1874.
56. Bauermeister JA, Tingler RC, Demers M, et al. Development of a tailored HIV prevention intervention for single young men who have sex with men who meet partners online: protocol for the myDEx project. JMIR Res Protoc. 2017;6:e141.
57. Knight R, Karamouzian M, Salway T, et al. Online interventions to address HIV and other sexually transmitted and blood-borne infections among young gay, bisexual and other men who have sex with men: a systematic review. J Int AIDS Soc. 2017;20:e25017.
58. Holland CM, Ritchie ND, Du Bois SN. iTunes song-gifting is a low-cost, efficient recruitment tool to engage high-risk MSM in internet research. AIDS Behav. 2015;19:1914–1918.
59. Bauermeister JA, Pingel E, Zimmerman M, et al. Data quality in HIV/AIDS web-based surveys: handling invalid and suspicious data. Field Methods. 2012;24:272–291.
60. Glasgow RE, Vinson C, Chambers D, et al. National Institutes of Health approaches to dissemination and implementation science: current and future directions. Am J Public Health. 2012;102:1274–1281.
61. Eccles MP, Mittman BS. Welcome to implementation science. Implement Sci. 2006:1: 1.
62. Angwin J. Facebook to Temporarily Block Advertisers from Excluding Audiences by Race. 2017. Available at: https://www.propublica.org/article/facebook-to-temporarily-block-advertisers-from-excluding-audiences-by-race. Accessed Febraury 12, 2019.
63. Nelson KM, Pantalone DW, Carey MP. Sexual health education for adolescent males who are interested in sex with males: an investigation of experiences, preferences, and needs. J Adolesc Health. 2019;64:36–42.
64. Reuter K, Angyan P, Le N, et al. Monitoring twitter conversations for targeted recruitment in cancer trials in los angeles county: protocol for a mixed-methods pilot study. JMIR Res Protoc. 2018;7:e177.
65. Krueger EA, Young SD. Twitter: a novel tool for studying the health and social needs of transgender communities. JMIR Ment Health. 2015;2:pii: e16.
66. Lovett M, Bajaba S, Lovett M, et al. Data quality from crowdsourced surveys: a mixed method inquiry into perceptions of amazon's mechanical Turk masters. Appl Psychol. 2018;67:339–366.
67. Beymer MR, Holloway IW, Grov C. Comparing self-reported demographic and sexual behavioral factors among men who have sex with men recruited through mechanical Turk, Qualtrics, and a HIV/STI clinic-based sample: implications for researchers and providers. Arch Sex Behav. 2018;47:133–142.
68. Hauser DJ, Schwarz N. Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behav Res Methods. 2016;48:400–407.
69. Newcomb ME, Swann G, Mohr D, et al. Do diary studies cause behavior change? An examination of reactivity in sexual risk and substance use in young men who have sex with men. AIDS Behav. 2018;22:2284–2295.
70. Parsons JT, Rendina HJ, Grov C, et al. Accuracy of highly sexually active gay and bisexual men's predictions of their daily likelihood of anal sex and its relevance for intermittent event-driven HIV pre-exposure prophylaxis. J Acquir Immune Defic Syndr. 2015;68:449–455.
71. Eldahan AI, Pachankis JE, Jonathon Rendina H, et al. Daily minority stress and affect among gay and bisexual men: a 30-day diary study. J Affect Disord. 2016;190:828–835.
72. Rendina HJ, Ventuneac A, Mustanski B, et al. Prospective measurement of daily health behaviors: modeling temporal patterns in missing data, sexual behavior, and substance use in an online daily diary study of gay and bisexual men. AIDS Behav. 2016;20:1730–1743.
73. Department of Health and Human Services. RFA-AI-16-031 Limited Interaction Targeted Epidemiology (LITE) to Advance HIV Prevention (UG3/UH3). 2016. Available at: https://grants.nih.gov/grants/guide/rfa-files/rfa-ai-16-031.html. Accessed June 2016.
74. Salyers Bull S, Lloyd L, Rietmeijer C, et al. Recruitment and retention of an online sample for an HIV prevention intervention targeting men who have sex with men: the Smart Sex Quest Project. AIDS Care. 2004;16:931–943.
75. Bull SS, Vallejos D, Levine D, et al. Improving recruitment and retention for an online randomized controlled trial: experience from the Youthnet study. AIDS Care. 2008;20:887–893.
76. Khosropour CM, Sullivan PS. Predictors of retention in an online follow-up study of men who have sex with men. J Med Internet Res. 2011;13:e47.
77. Murray E, White IR, Varagunam M, et al. Attrition revisited: adherence and retention in a web-based alcohol trial. J Med Internet Res. 2013;15:e162.
78. Sauermann H, Roach M. Increasing web survey response rates in innovation research: an experimental study of static and dynamic contact design features. Res Policy. 2013;42:273–286.
79. Santarossa S, Kane D, Senn CY, et al. Exploring the role of in-person components for online health behavior change interventions: can a digital person-to-person component suffice? J Med Internet Res. 2018;20:e144.
80. Randell RL, Gulati AS, Cook SF, et al. Collecting biospecimens from an Internet-Based Prospective Cohort Study of inflammatory bowel disease (CCFA partners): a feasibility study. JMIR Res Protoc. 2016;5:e3.
81. El-Sadr WM, Donnell D, Beauchamp G, et al. Financial incentives for linkage to care and viral suppression among HIV-positive patients: a randomized clinical trial (HPTN 065). JAMA Intern Med. 2017;177:1083–1092.
82. Volpp KG, Troxel AB, Pauly MV, et al. A randomized, controlled trial of financial incentives for smoking cessation. N Engl J Med. 2009;360:699–709.
83. Giles EL, Robalino S, McColl E, et al. The effectiveness of financial incentives for health behaviour change: systematic review and meta-analysis. PLoS One. 2014;9:e90347.
84. Lynch HF, Joffe S, Thirumurthy H, et al. Association between financial incentives and participant deception about study eligibility. JAMA Netw Open. 2019;2:e187355.
85. Bernstein SL, Feldman J. Incentives to participate in clinical trials: practical and ethical considerations. Am J Emerg Med. 2015;33:1197–1200.
86. Shelus V, Taylor J, Greene E, et al. It's all in the timing: acceptability of a financial incentive intervention for linkage to HIV care in the HPTN 065 (TLC-Plus) study. PLoS One. 2018;13:e0191638.
87. Greene E, Pack A, Stanton J, et al. “It Makes You Feel Like Someone Cares” acceptability of a financial incentive intervention for HIV viral suppression in the HPTN 065 (TLC-Plus) study. PLoS One. 2017;12:e0170686.
88. Herman CW, Musich S, Lu C, et al. Effectiveness of an incentive-based online physical activity intervention on employee health status. J Occup Environ Med. 2006;48:889–895.
89. Wilson PM, Petticrew M, Calnan M, et al. Effects of a financial incentive on health researchers' response to an online survey: a randomized controlled trial. J Med Internet Res. 2010;12:e13.
90. D'Angelo AB, Lopez-Rios J, Flynn AWP, et al. What motivates gay and bisexual men to participate in PrEP-related research? Int J Sex Health. 2019;31:283–290.
91. Abbey JD, Meloy MG. Attention by design: using attention checks to detect inattentive respondents and improve data quality. J Operat Manag. 2017;53:63–70.
92. Teitcher JE, Bockting WO, Bauermeister JA, et al. Detecting, preventing, and responding to “fraudsters” in internet research: ethics and tradeoffs. J L Med Ethics. 2015;43:116–133.
93. LaForgia M, Confessore N, Dance GJX. Facebook rebuked for failing to disclose data-sharing deals. 2018. Available at: https://www.nytimes.com/2018/12/19/technology/facebook-data-privacy-criticism.html. Accessed March 20, 2019.
94. New York Times. Mark Zuckerber testimony: senators question Facebook's commitment to privacy. 2018. Available at: https://www.nytimes.com/2018/04/10/us/politics/mark-zuckerberg-testimony.html. Accessed March 20, 2019.
95. Rendina HJ, Mustanski B. Privacy, trust, and data sharing in web-based and mobile research: participant perspectives in a large nationwide sample of men who have sex with men in the United States. J Med Internet Res. 2018;20:e233.
96. Cuthbertson A. Facebook is officially for old people. 2018. Available at: https://www.newsweek.com/facebook-officially-old-people-803196. Accessed February 12, 2019.
97. Hargittai E, Litt E. The tweet smell of celebrity success: explaining variation in Twitter adoption among a diverse group of young adults. New Media Soc. 2011;13:824–842.
98. Bosker B. Why are African Americans more likely to join Twitter. 2011. Available at: https://www.huffingtonpost.com/2011/08/02/african-americans-twitter-use_n_916411.html. Accessed February 12, 2019.
99. Sullivan PS, Khosropour CM, Luisi N, et al. Bias in online recruitment and retention of racial and ethnic minority men who have sex with men. J Med Internet Res. 2011;13:e38.
100. Deaton A, Cartwright N. Reflections on randomized control trials. Soc Sci Med. 2018;210:86–90.
Keywords:

men who have sex with men; internet; smart phones; online cohorts

Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved.