Despite billions of research dollars dedicated to prevention of sexually acquired HIV and other sexually transmitted infections (STI) since HIV/AIDS first appeared, STI prevalence continues to increase in most developing countries.1 Prevention programs have historically focused on changing individual behaviors, but social, economic, and political environments in which populations negotiate condom use remained unchanged. There is growing consensus that STI/HIV prevention programs need to address the social structural context that influences an individual's behavior to bring about sustained behavior change.2 – 4 Addressing the larger social structural context surrounding STI/HIV is of particular importance among groups who experience inequality, discrimination, and exclusion from public life. For these groups, social change interventions have sought to create social cohesion and extend social networks, ensure community participation, challenge inequitable gender dynamics, promote humane policy, improve health services access, and encourage community mobilization for prevention.5 – 7
Sex workers have long been the focus of STI/HIV prevention efforts. Across cultures, they experience extreme social exclusion and discrimination, are often mobile or transient, confront unfavorable power dynamics, are deprived of health services and public social benefits, are prosecuted and imprisoned, and are denied basic rights of safety and respect.8 For this reason, the World Health Organization has recommended strategies that aim to empower sex workers in addressing STI/HIV prevention, including participatory planning and community mobilization.9 Community mobilization efforts among sex workers in Sonagachi, India provide the most compelling evidence to date that social structural change can improve health outcomes.10 – 12 Interim analyses from the more recent Avahan initiative in India have yielded promising results as well.13,14 Outside of India, there have been only a few studies with mobilizing or social change intervention components for STI/HIV prevention with sex workers,15 – 18 however, the data supporting these efforts to date are not decisive. Almost none of the interventions to date prospectively measured the social structural factors the interventions sought to change, leaving a gap in understanding the mechanisms through which these social change interventions work.
We conducted a multilevel combined social structural and clinical STI/HIV prevention intervention with sex workers in Brazil. Key intervention components included expanded clinical care for sexual health and strategies to create social cohesion, expand social networks and community partnerships, reduce stigma, and stimulate community mobilization. Using a prospective cohort design, we sought to determine whether the multicomponent Encontros intervention would decrease incidence of chlamydia and gonorrhea and improve condom use. We also examine whether participation in the intervention resulted in a more enabling social environment (increased cohesion and involvement in community life) for sex workers to practice safer sex.
MATERIALS AND METHODS
The study took place in Corumbá, located on Brazil's Western border with Bolivia. Home to 100,000 permanent inhabitants, Corumbá attracts approximately 75,000 fishing and ecotourists annually, giving rise to sex commerce catering to tourists and local residents alike. The Encontros intervention was designed to simultaneously fill the programmatic gaps for sex workers in Corumbá and modify the social environment around sex work and STI/HIV prevention in the community, creating a space for sex workers to access services and adopt safer sex behaviors.
Participants included 420 men, women, and transvestites who were 18 years or older, self-identifying sex workers, who spoke Portuguese or Spanish, and who did not plan on leaving the study area permanently in the month following recruitment. We used both convenience and snow ball sampling to reach as many sex workers in the area as possible. Sex worker peer educators recruited participants from local brothels, bars, motels, and other sex work establishments; they were instructed to approach all potential sex workers in each establishment. Additionally, sex workers were asked to tell their friends about the project. As a result, a number of participants presented spontaneously for participation. Participants received free transportation to the study site as well as condoms and small gifts at each visit.
Participation included an enrollment visit, 4 scheduled follow-up visits approximately 3, 6, 9, and 12 months subsequent to enrollment, and posttest counseling visits, approximately 1 month after enrollment and follow-up visits. Enrollment and follow-up visits included administration of a structured, interviewer-administered questionnaire, STI/HIV prevention counseling, a clinical exam, and collection of urine and blood samples for STI testing. Testing for chlamydia and gonorrhea was performed on urine specimens with polymerase chain reaction (PCR) Cobas Amplicor technology by Roche. Specimens were frozen and stored at −20°C at the study clinic before transport on dry ice to a Ministry of Health Reference Laboratory for processing according to manufacturer's instructions. During the clinical exam, specimens were collected for pap smears (if indicated) as well as wet mount. STI treatment was administered during clinical exams based on wet mount and syndromic management; PCR-diagnosed infections were treated at posttest counseling visits. Syphilis and HIV testing were offered through the municipal service. In this analysis, we report on chlamydia and gonorrhea (PCR) results only.
Intervention design was informed by review of international best practices and extensive formative research, including mapping sex work venues, stakeholder interviewers (brothel/bar owners, mototaxi drivers, public health officials), focus groups with sex workers in a variety of venues, and in-depth interviews with sex workers and clients. We also developed the intervention using the ecological model,19,20 combining strategies to engage sex workers on an individual level through participation in STI/HIV testing experiences, on an interpersonal level through peer education and counseling, and on a community level through outreach, workshops, and social activities. At the same time, we worked with city health officials to improve quality of and access to sexual health services. All project activities adhered to a human rights framework, aiming to reinforce the Brazilian national strategy of conducting HIV prevention with messages that destigmatize sex work and emphasize quality provision of services to all who seek care.21 Intervention activities based at the study clinic included extending clinic hours to provide expanded, holistic services for STI/HIV prevention, diagnosis, and treatment, as well as reproductive health services and intensive training to create a sex worker-friendly environment. Educators conducted outreach and distribution of condoms and educational materials designed by the Ministry of Health to reduce sex work-associated stigma.
Project activities in the community were designed with project participants to extend and strengthen collegial relationships through providing sex workers opportunities to engage in dialogue around sex work, discrimination, human rights, and prevention, specifically through workshops, trainings, and events. For example, sex workers participated in workshops from soap, candle, and chocolate making to theater and fashion design, all of which offered the sex workers an opportunity to talk about their work and themselves and strengthen their relationships. Once the workshops were underway, the sex workers requested a venue to showcase their new skills. As a result, the project sponsored “hot pink” parties, which were essentially cultural showcases for the sex workers that occurred at the city's cultural center, a public space where those attending included the general public, public officials, family members, and university students. The project staff ensured that performance messages focused on HIV prevention and stigma reduction; it was also hypothesized that the experience of celebrating culture and performance in a public space in and of itself would facilitate integration, social cohesion, and reduce stigma and discrimination. The project also sought to forge broad partnerships and links to the community to stimulate community recognition of sex workers as partners in health and human rights and strengthen sex workers' community identity (Table 1).22 All participants were invited to attend project-sponsored events and encouraged to come to the project office any time to talk to a counselor.
The research was approved by the Ethics Committee in Mato Grosso do Sul, Brazil, the Brazilian National Committee of Ethics in Research, the Institutional Review Board at the Population Council, and by the Committee for the Protection of Human Subjects at the University of California Berkeley.
Participation in the intervention was assessed by interviewer-administered questionnaire, including information regarding contact with peer educators and counselors; contact with project campaign and educational materials; participation in community cultural or social events; participation in project-sponsored workshops, educational activities, and organizations or associations; and adherence to scheduled visits. The 20 questionnaire items collectively measuring project participation were analyzed and pooled into scores using item response modeling, which provides a weighted ranking of items and participants.23,24 The item response modeling reliability estimate, which is akin to Cronbach's α, was 0.86, demonstrating good reliability. Continuous participation scores or ranks were then dichotomized into no/low or high participation at each visit, reflecting the overall level of participation in project activities in the preceding 3-month-interval, using an a priori theoretical cut point. No/low participation (herein called unexposed) describes those who attended scheduled appointments and who may have reported contact with an educator, but who participated very little or not at all in individual or community-focused project activities. The high participation group (referred to herein as exposed) attended scheduled appointments, sought additional contact with educators or counselors, and participated in project events, workshops, meetings, or organizations.
Study outcomes included diagnosis of incident chlamydia and/or gonorrhea and reported condom use in the 3-month interval following reported intervention exposure. Study participants responded to questions about condom use during vaginal and anal sex acts with their most recent new client, regular client, and nonpaying partner in the last 30 days. A summary variable for consistent condom use (condom use all of the time) during penetrative sex (vaginal and anal) with each partner type was created. We also measured social environmental factors hypothesized to enable protective behaviors, including perception of mutual aid, trust, and support among sex workers (social cohesion); participation in social networks; and access to and management of social and material resources (including purchasing of household and personal goods and access to services, such as banking and health insurance). Individual scale items (published elsewhere22) were summed into scores and then standardized.
We hypothesized that exposure to intervention activities would increase condom use and reduce incident STI as well as enhance perceptions of social cohesion, participation in social networks, and access to social and material resources. We determined the odds of incident STI (chlamydia and/or gonorrhea) and consistent condom use by comparing the exposed with the unexposed study population. In a previous analysis, using STI data from this data set, we examined the association of reported exposures at the end of 3-month intervals and infection status at the end of the same interval.25 In this analysis, we ensure time ordering of exposure and outcomes and thus a causal interpretation by using exposure data from the 3-month interval before that in which outcome measures were diagnosed or reported. In other words, participant outcomes in interval n are linked to exposure status in interval n-1. This approach limited our sample to those observations with complete exposure data from 2 consecutive follow-up visits. Overall, there were 424 complete exposure and outcome data points (or observation pairs with complete exposure data at n-1 and STI/condom use outcomes at n).
Because exposure was not randomly assigned and a sizeable proportion of the cohort was lost to follow-up (censored) or interrupted participation before study conclusion, selection bias needs to be accounted for. We utilized inverse probability weighting techniques26,27 to account for potential biases, as described in detail in a tutorial article on inverse probability weighting in this journal.25 The premise of treatment weights and generally the class of techniques called “marginal structural models” is to mimic randomization by using covariate information to weight each observation.27 Hypothetically speaking, to guarantee no selection bias or confounding in this design, one would need to randomly assign sex workers to exposed or unexposed status (or participation level) at every visit to ensure that the distribution of confounders between the 2 exposure groups is balanced at each visit. Using inverse probability weighting for exposure, we attempt to simulate this repeat randomization by multiplying each observation by the inverse probability of being in the reported exposure group, based on values of covariates, and past outcomes and exposures. We used this approach for both exposure and censoring weights, which were estimated using an automated model selection algorithm called DSA (deletion substitution algorithm) in the program R.28 Using weighted data, the effect of the intervention on all outcomes was estimated using generalized estimating equations with robust standard errors to provide a marginal estimate of intervention effect while accounting for the nonindependence of repeated measures on individuals.29 The resulting odds ratios (ORs) are interpreted as the average intervention effect across the entire study population, as if all participants were exposed compared with the counterfactual of having all participants unexposed.
We also determined the relationship of intervention exposure to reported levels of social environmental factors, including social cohesion, participation in networks, and access to resources, to assess whether participation effected perceptions of the social environment. Because participants reporting higher levels of social environmental factors (e.g., social cohesion) at enrollment had less opportunity to raise their reported social cohesion level compared with participants who began the intervention with lower reported cohesion, we also assessed the relationship of the intervention to cohesion and other social environmental measures among those with baseline values below the median. These analyses were performed using the same weighting approach as in the main analysis.
Of 474 potential participants screened for enrollment, 51 were ineligible and 3 chose not to participate. In total, 420 sex workers were enrolled in the study, including 385 (91.7%) women, 19 (4.5%) men, and 16 (3.8%) transvestites (Table 2). This gender distribution matches projections made during formative research, placing the proportion of females in the sex work population at approximately 90%. Most participants were Brazilian, with only 15 (3.6%) participants being Bolivian or Paraguayan. Mean age and years of schooling were 26.0 and 6.3, respectively, and >50% of the cohort had initiated sex work before the age of 19 years. At enrollment, participants had a mean of 6 new clients, 4 regular clients, and 1 nonpaying partner per month. Eighteen percent of the cohort had a prevalent gonorrhea or chlamydia infection at enrollment, and 73%, 53%, and 26% of participants were using condoms with their most recent new client, regular client, and nonpaying partner, respectively. Study subjects who were ever classified as exposed were more likely to be male or transvestite, Brazilian, work outside of the brothels, and to live with a family member as compared with study subjects always classified as unexposed (Table 2). Additionally, ever exposed participants reported fewer new clients and more regular clients than unexposed participants at enrollment.
In all, 329 participants (79%) returned for at least 1 follow-up visit following enrollment and baseline data collection. More than 60% remained through the third visit, but only 45% of the overall cohort completed all 5 visits. Reasons for loss to follow-up included leaving the practice of sex work (39%), leaving the study area (22%), withdrawal of consent (4%), and the remaining 35% were lost to follow-up with no recorded reason. Adherence was close to that expected given the extreme mobility of this population (particularly, mobility associated with the seasonal nature of this touristic area), and the frequent change of profession: >60% of the participants who did not complete the study left the cohort due to a documented change in profession or residence (both of which were eligibility criteria). Those who were lost to follow-up were more likely to be younger, from outside of the study area, and worked at a brothel (the brothel-based population tended to be younger and from outside of the area). Condom use and prevalent STI at baseline were not different for those who were lost to follow-up and those who remained in the study.
Overall, results indicate that exposure to the intervention was protective against incident STI and resulted in increased reporting of consistent condom use (Table 3). Inverse probability weighted estimates indicate that participation in the intervention was associated with a higher odds of consistent condom use with new clients (OR: 1.6, 95% confidence interval [CI]: 0.9–2.8), regular clients (OR: 1.9, 95% CI: 1.1–3.3), and nonpaying partners (OR: 1.5, 95% CI: 0.9–1.5). Only the association with regular clients was significant at α 0.05. The odds of an incident STI were reduced for the exposed group compared with unexposed participants (OR: 0.46, 95% CI: 0.2–1.3); this finding did not reach statistical significance despite a strong effect estimate.
As hypothesized, project exposure was associated with a significant increase (of 0.3 standard deviations) in participation in networks, both in the entire cohort and among those reporting below median participation in networks at enrollment (Table 4). Additionally, participants reported a significant increase in perceived social cohesion compared with nonparticipants among study subjects reporting below median perceived social cohesion at enrollment. There was no relationship between participation and access to social and material resources.
We found evidence that the Encontros intervention may be a successful intervention model, with all effect estimates going in the hypothesized direction. Sex workers exposed to the Encontros intervention were significantly more likely to use condoms consistently with regular clients. Analysis of condom use with new clients and nonpaying partners also demonstrated protective effects, significant at P = 0.1. Consistent condom use at enrollment with new clients was quite high (close to 80% in the ever-exposed group), making it more difficult to detect a difference due to the intervention. Odds of an incident STI were approximately 2 times lower among exposed participants compared with unexposed, although this association was not significant.
This analysis adds to the growing body of evidence generated in India and the Dominican Republic that multilevel interventions combining provision of services with community-based social change strategies can improve condom use and reduce STIs among sex workers, a population which almost universally experiences extreme social exclusion, poverty, and unfavorable gender power dynamics. The Sonagachi project in Calcutta, which included training peer outreach workers, formation of broad community partnerships, founding of literacy and loan service programs, and the establishment of a collective organization, has been held up as the best evidence to date that community organizing efforts and sex worker cohesion and empowerment are important factors in HIV prevention.10,11 Following successes of Sonagachi, the Gates Foundation funded the Avahan project across 6 states in India to increase intervention coverage for most at-risk populations, including a community-led structural intervention approach.30 Although evaluation of the program is still ongoing, initial analyses have noted a reduction in STI or HIV and increased condom use with clients.13,14 An environmental structural intervention that stimulated community-based solidarity for HIV/STI prevention and shifts in government regulation conducted among female sex workers in the Dominican Republic demonstrated significant reductions in STIs and increases in consistent condom use using a prepost design.15 What these successful projects all had in common was a commitment not only to involve community by including a peer outreach component or enhancing services, but to truly engage community in collective spaces, collective activities, collective planning (which necessitates flexibility), developing equitable partnerships (without which 1 mobilizing effort in South Africa failed),18 and often supporting organizational development for sex workers.
What previous studies with sex workers have largely failed to do is to pay adequate attention to pathway variables or community change mechanisms; to date almost no studies have prospectively monitored and measured change in social contextual factors that the social structural interventions aim to modify, including measurement of collective empowerment or mobilization. Effective prevention requires an understanding of mediation and pathways of effect, including empirical examination of the effects of social environmental change or mobilization on subsequent behaviors and infections. Processes of empowerment and social change are assumed to have taken place in the Sonagachi and Avahan interventions, and likely did, but remain unmeasured and unconfirmed. In this study, we found evidence that the intervention successfully led to a more supportive social environment. Increases in reported levels of perceived social cohesion and reported participation in networks among exposed participants has also been found in the 2 projects with sex workers that quantitatively measured change in social contextual factors.15,16 The IMAGE study among low income South African women also provided evidence that multicomponent HIV prevention projects can improve reported levels of social cohesion and social capital.31 Other studies have found associations with social factors, such as collective efficacy, and condom use behaviors when assessed cross sectionally among sex workers.22,32 Analysis to determine the extent to which social environmental factors are on the causal pathway from participation to improved outcomes will be the subject of future research and should be a priority in future mobilizing interventions.
This research was conducted using a prospective cohort. The ideal design to evaluate an intervention aiming to change community structures and environments would be a randomized community trial, which is generally prohibitively expensive and best reserved for interventions that first demonstrate positive results in an observational study. Randomizing individuals (or brothels/bars) to the study condition was not feasible in this context; social and community structures could not be selectively modified for some sex workers (or sex work establishments) and not others in the same community. As a result, we choose to follow a cohort in a single community with an internal comparison group, comparing the “less exposed” population in our cohort to the “more exposed” population. Notably, behavioral interventions are more commonly evaluated using comparison communities; however, evaluation of multilevel and social interventions may be less amenable to that study design, which would require a comparison community that is exchangeable or similar to the intervention community not only in terms of population characteristics but also in the social and structural factors that the intervention seeks to modify. Instead, the internal comparison ensures that subjects being compared experience the same community structure and environment, and it permits observation of how change unfolds over time. However, this design is particularly susceptible to selection bias and complex confounding. To address this issue, investigators in the area of AIDS treatment pioneered the weighting methods, which we used in this article (causal inference methods).33,34 These methods are now becoming standard in the epidemiologic literature and represent a cost-effective and valuable tool to improve utilization of observational data in STI/HIV prevention research.25
Limitations include a loss of power due to substantial attrition. The risk profile (condom use, STI at baseline) did not differ by attrition status. More than half of study subjects lost to follow-up early in the study were brothel-based sex workers, who were generally not from the study area, in town only during the fishing tourism season, and unlikely to leave the brothel to participate in community events. The mobile nature of the sex work profession presents a major challenge to prevention programming, particularly interventions seeking long-term social change, as meaningful change in social relationships, social norms, and community partnerships do not occur over night, much less changes in rates of HIV infection. In this study, despite the short time frame, we did observe improvements in safer sex behaviors, gains in social cohesion and networks, and reductions in STI infections, which we attribute to the intensive combination and implementation of activities. We addressed potential bias due to attrition (and mobility) using the causal inference methods described earlier, and our results remain promising; however, these effects may be attenuated outside of a well-run research study. Furthermore, the loss to follow-up also resulted in decreased power to detect intervention effects. The study was originally powered to detect an effect of the intervention when associating 3-month intervals of project participation with new STIs at the end of the same interval. This “same interval” analysis included a larger sample size and demonstrated a statistically significant reduced odds of STI infection among participants (OR: 0.49, 95% CI: 0.25–0.98).25 However, this same interval estimate mixes cross-sectional and causal (longitudinal) effects, as an infection could have been acquired any time during the 3 months of reported participation. We used a time-lag approach in the current analysis to ensure time ordering and a causal interpretation, which resulted in a larger effect estimate, but less power to detect an effect. Among the strengths of this intervention research, the strategies included project activities aimed at enabling positive change on an individual, peer, and community level, including modification of the social environment, which was measured at repeated intervals. Study findings include both biologic and behavioral end points, and we used statistical modeling to address potential confounding and biases due to self-selection and attrition. Finally, the intervention was implemented by a well-trained and engaged field team and supervised by a multidisciplinary advisory committee and included active participation of local sex workers and national sex worker advocates, the Brazilian Ministry of Health, local government, and partner non-governmental organizations.
Our findings add to the growing literature demonstrating that combined clinical and social interventions aiming to improve care and modify the social environment can affect biomedical and behavioral outcomes and indeed can change the social environment for vulnerable populations. Given the current state of the evidence, replicating this approach in additional geographic contexts beyond India and potentially in a gold standard community randomized trial is warranted to provide robust evidence of intervention efficacy. Furthermore, comprehensive investigation regarding the pathways of effect—how participation changes the social environment to improve health—will deepen our understanding of the community-level processes that shape behavior and inform future interventions.
1. World Health Organization. Global strategy for the prevention and control of sexually transmitted infections: 2006–2015. Key messages. Geneva, Switzerland: World Health Organization, 2006.
2. Coates TJ, Richter L, Caceres C. Behavioural strategies to reduce HIV transmission: How to make them work better. Lancet 2008; 372: 669–684.
3. Merzel C, D'Afflitti J. Reconsidering community-based health promotion: Promise, performance, and potential. Am J Public Health 2003; 93: 557–574.
4. Merson M, Padian N, Coates TJ, et al.. Combination HIV prevention. Lancet. 2008; 372: 1805–1806.
5. Beeker C, Guenther-Grey C, Raj A. Community empowerment paradigm drift and the primary prevention of HIV/AIDS. Soc Sci Med 1998; 46: 831–842.
6. Parker RG. Empowerment, community mobilization and social change in the face of HIV/AIDS. AIDS 1996; 10 (suppl 3): S27–S31.
7. UNAIDS. HIV-related stigma, discrimination and human rights violations: case studies of successful programmes. Geneva, Switzerland: UNAIDS, 2005.
8. Rekart ML. Sex-work harm reduction. Lancet. 2005; 366: 2123–2134.
8. World Health Organization. Toolkit for targeted HIV/AIDS prevention and care in sex work settings. Geneva, Switzerland: WHO, 2005.
10. Jana S, Basu I, Rotheram-Borus MJ, et al.. The Sonagachi Project: A sustainable community intervention program. AIDS Educ Prev 2004; 16: 405–414.
11. Jana S, Singh S. Beyond medical model of STD intervention—lessons from Sonagachi. Indian J Public Health 1995; 39: 125–131.
12. Basu I, Jana S, Rotheram-Borus MJ, et al.. HIV prevention among sex workers in India. J Acquir Immun Defic Syndr 2004; 36: 845–852.
13. Ramesh BM, Beattie TS, Shajy I, et al.. Changes in risk behaviours and prevalence of sexually transmitted infections following HIV preventive interventions among female sex workers in five districts in Karnataka state, south India. Sex Transm Infect 2010; 86 (suppl 1): i17–i24.
14. Reza-Paul S, Beattie T, Syed HU, et al.. Declines in risk behaviour and sexually transmitted infection prevalence following a community-led HIV preventive intervention among female sex workers in Mysore, India. Aids 2008; 22 (suppl 5): S91–S100.
15. Kerrigan D, Moreno L, Rosario S, et al.. Environmental-structural interventions to reduce HIV/STI risk among female sex workers in the Dominican Republic. Am J Public Health 2006; 96: 120–125.
16. Kerrigan D, Telles P, Torres H, et al.. Community development and HIV/STI-related vulnerability among female sex workers in Rio de Janeiro, Brazil. Health Educ Res 2008; 23: 137–145.
17. Campbell C, Mzaidume Z. Grassroots participation, peer education, and HIV prevention by sex workers in South Africa. Am J Public Health 2001; 91: 1978–1986.
18. Campbell C. ‘Letting them Die' Why HIV/AIDS Prevention Programmes Fail. Bloomington, IN: Indiana University Press, 2003.
19. Bronfenbrenner U. The Ecology of Human Development: Experiments by Nature and Design. Cambridge, MA: Harvard University Press, 1979.
20. Waldo CR, Coates TJ. Multiple levels of analysis and intervention in hiv prevention science: Exemplars and directions for new research. AIDS 2000; 14 (suppl 2): S18–S26.
21. Coordenação Nacional de DST e AIDS. Política Nacional de DST/AIDS: Princípios, Diretrizes e Estratégias. Brasília, Brazil: Ministério da Saúde, 1999.
22. Lippman SA, Donini A, Diaz J, et al.. Social-environmental factors and protective sexual behavior among sex workers: The Encontros intervention in Brazil. Am J Public Health 2010; 100 (suppl 1): S216–S223.
23. Wilson M. Constructing Measures: An Item Response Modeling Approach. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers, 2005.
24. Wilson M, Allen DD, Li JC. Improving measurement in health education and health behavior research using item response modeling: Introducing item response modeling. Health Educ Res 2006; 21 (suppl 1): i4–i18.
25. Lippman SA, Shade SB, Hubbard AE. Inverse probability weighting in STI/HIV prevention research: Analytic methods for evaluating social and community interventions. Sex Transm Dis 2010; 37: 512–518.
26. Hernan MA, Robins JM. Estimating causal effects from epidemiological data. J Epidemiol Community Health 2006; 60: 578–586.
27. Robins JM, Hernan MA, Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology 2000; 11: 550–560.
28. Sinisi SE, van der Laan MJ. Deletion/substitution/addition algorithm in learning with applications in genomics. Stat Appl Genet Mol Biol 2004; 3: Article 18.
29. Fitzmaurice G, Laird N, Ware J. Applied longitudinal analysis. Hoboken, NJ: Wiley-Interscience, 2004.
30. Laga M, Galavotti C, Sundararaman S, et al.. The importance of sex-worker interventions: The case of Avahan in India. Sex Transm Infect 2010; 86 (suppl 1): i6–i7.
31. Pronyk PM, Harpham T, Busza J, et al.. Can social capital be intentionally generated? A randomized trial from rural South Africa. Soc Sci Med 2008; 67: 1559–1570.
32. Blankenship KM, West BS, Kershaw TS, et al.. Power, community mobilization, and condom use practices among female sex workers in Andhra Pradesh, India. Aids 2008; 22 (suppl 5): S109–S116.
33. Hernan MA, Brumback B, Robins JM. Marginal structural models to estimate the causal effect of zidovudine on the survival of HIV-positive men. Epidemiology 2000; 11: 561–570.
34. Robins JM. The analysis of randomized and non-randomized AIDS treatment trials using a new approach to causal inference in longitudinal studies. In: Sechrest L, Freeman H, Mulley Aeds. Health Service Research Methodology: A Focus on AIDS. Washington, DC: US Public Health Service, National Center for Health Services Research, 1989: 113–159.