Evaluation design for large-scale HIV prevention programmes: the case of Avahan, the India AIDS initiative
Chandrasekaran, Padmaa; Dallabetta, Ginaa; Loo, Virginiab; Mills, Stephenc; Saidel, Tobid; Adhikary, Rajatashuvrae; Alary, Michelf,g; Lowndes, Catherine Mh,i; Boily, Marie-Claudej; Moore, Jamesa; for the Avahan Evaluation Partners
aBill and Melinda Gates Foundation, New Delhi, India
bIndependent consultant, formerly with the Bill and Melinda Gates Foundation, New Delhi, India
cFamily Health International, Vietnam
dIndependent consultant, formerly with Family Health International, New Delhi, India
eFamily Health International, New Delhi, India
fCentre Hospitalier affilié Universitaire de Québec, Québec, Canada
gUniversité Laval, Québec, Canada
hLondon School of Hygiene and Tropical Medicine, London, UK
iHealth Protection Agency, London, UK
jDepartment of Infectious Diseases Epidemiology, Imperial College, London, UK.
Correspondence to Padma Chandrasekaran, Avahan, the India AIDS Initiative, The Bill and Melinda Gates Foundation, Sanskrit Bhawan, Aruna Asaf Ali Marg, A-10 Qutab Institutional Area, New Delhi 110061, India. Tel: +91 11 4100 3100; fax: +91 11 4100 3101; e-mail: firstname.lastname@example.org
Background: Closing the HIV prevention gap to prevent HIV infections requires rapid, worldwide rollout of large-scale national programmes. Evaluating such programmes is challenging and complex, requiring clarity of evaluation purpose and evidential approaches substantively different to those employed for pilots and small programmes.
Objectives: This paper describes the evaluation design for the implementation phase of Avahan, the India AIDS initiative, a large HIV prevention programme funded by the Bill and Melinda Gates Foundation. Avahan, which began in December 2003, has a 10-year charter to impact the Indian epidemic and its response by implementing an HIV prevention programme targeting core and bridge groups in 83 districts of six Indian states, transferring the programme to the Government of India, and disseminating programme learning.
Methods: The foundation commissioned an external process to design Avahan's evaluation framework. An independent advisory group oversees and guides course corrections in the execution of this framework.
Results: Avahan's evaluation framework comprises: trend and synthetic analysis of data from core, bridge and household biobehavioural surveys in a subset of intervention districts, denominator estimates and programme monitoring from all intervention districts, and government's antenatal surveillance (two sites per district in all districts); bespoke transmission dynamics modelling to estimate infections averted (subset of districts); cost effectiveness studies (subset of districts). In addition, there are other knowledge-building and quality-monitoring activities.
Conclusion: Rather than a small set of monofocal outcome measures, scaled programmes require nuanced evaluations that approximate programmatic scale by collecting data with different levels of geographical scope, synthesize multiple data and methods to arrive at a composite picture, and can cope with continuous environmental and programme evolution.
The Global HIV Prevention Working Group and others have repeatedly issued calls for closing the HIV prevention gap using a combination of established intervention strategies [1,2]. Simultaneously there have also been calls for greater focus on evaluation better to inform country-level prevention scale-up . Bridging the prevention gap to reduce HIV infections requires rapid roll-out of large, comprehensive national HIV prevention programmes. The magnitude of resources such programmes entail will necessitate well-designed programme evaluations. The evaluation of large-scale public health programmes, particularly HIV prevention programmes, is challenging for many reasons including complex, diffuse pathways and a multiplicity of interventions [4,5].
This paper describes the evaluation design for the implementation phase of Avahan, the India AIDS initiative, a large multi-phase HIV prevention programme funded by the Bill and Melinda Gates Foundation . Other papers in the supplement present methodological and design strategies related to key components of the Avahan evaluation framework, results from a first round of surveys conducted among core and bridge populations as part of Avahan evaluation activities, and results from additional studies [7–20].
Avahan, the India AIDS initiative
In 2002–2003, limited data pointed to several sizeable and growing concentrated HIV epidemics in the south and north-east parts of India, largely driven by core and bridge group interactions [21–26]. Internationally, there were projections of a massive HIV epidemic in India [27,28]. The Indian national response had variable, but in general, low coverage of core groups, namely female sex workers (FSW), high-risk men who have sex with men (MSM)/transgender individuals and injecting drug users (IDU) [21,29,30]. Globally, there was evidence that intervening with core and bridge groups in early and concentrated epidemics translated into HIV reduction among general populations [27,31–48]. At the time there were few examples of large-scale HIV prevention interventions [1,33,37,38,46–48].
After consultation with the Government of India, the Bill and Melinda Gates Foundation funded Avahan, the India AIDS initiative, in December 2003 for its first 5-year term . By mid-2006, the foundation had in principle approved a second 5-year term, with formal funding approval subsequently in 2008. Over a 10-year period, Avahan's charter is to impact the Indian HIV epidemic and response by: implementing an HIV prevention programme with high coverage (over 80%, also referred to as saturated coverage here) of core groups and prevention interventions with bridge groups across the intervention areas; transferring the programme to the Government of India and other implementers in the country; and enabling replication of programme and best practices by other institutions by fostering and disseminating learning.
Avahan programmes are implemented in four southern (Tamil Nadu, Andhra Pradesh, Karnataka, Maharashtra) and two north-eastern states (Nagaland and Manipur) (combined total population of 300 million), which accounted for approximately 83% of the estimated HIV cases in India in 2002 . These states are heterogeneous with distinct languages and subcultures, different HIV epidemic drivers and phases, and a varying extent and history of previous HIV prevention interventions . The programme is built on two premises: (1) addressing core and bridge groups with appropriate interventions can reduce the number of new infections among the general population; and (2) both proximal and distal determinants of risk need to be addressed via a combination of structural interventions, community mobilization (participatory interventions) and prevention services and commodities .
Avahan intervention districts were selected in collaboration with respective state governments collectively to ensure saturated prevention coverage of core groups . This resulted in Avahan partners becoming sole core group programme implementers in 43 districts and joint players with government in 40 districts. For core groups, Avahan supports peer-led outreach and education, treatment of sexually transmitted infections (STI), referrals for HIV and tuberculosis testing and HIV care, free commodities supplies (condoms, needle/syringe exchange when appropriate), and initiatives to address structural and environmental barriers. In addition, in some districts Avahan also supports clinical services for core groups covered under government or other programmes (Fig. 1). Core group programmes are managed by seven lead implementing partners who subgrant to 134 local non-governmental organizations (NGO). These local NGO have implemented interventions to reach nearly 200 000 FSW, 60 000 high-risk MSM/transgender individuals and 20 000 IDU (Fig. 2).
Two lead partners address an estimated 5 million most at-risk men, including 2 million truckers, in high-risk venues in 100 towns and cities and across the national highways, respectively . Major intervention elements for men at risk include condom social marketing through the expansion of non-traditional retail outlets, STI treatment through project-owned clinics in truck stops and franchised clinics in towns and cities, and targeted behaviour change communications.
As a result of the perceived magnitude and potential growth of the Indian HIV epidemic at the time, Avahan's implementation grants were commissioned first, between December 2003 and March 2004 . An external team completed the evaluation design for Avahan's implementation phase by April 2004. One early decision was to commission a distinct set of evaluation partners to ensure some separation between implementation and evaluation. This separation is true for all states except Karnataka, where the implementing partner's grant scope includes some evaluation activities. By September 2004, evaluation grants were made. An advisory group of evaluation experts have since overseen the iterative refinement and execution of the evaluation framework.
Defining and measuring success
Success for Avahan during its implementation phase is demonstrating that it is possible to build quickly a scaled, quality programme for core and bridge groups across a large geography with complex and heterogeneous local environments. Although Avahan is not intended to prove experimentally that the package of prevention interventions result in a reduction of HIV among these populations and the general population, nevertheless this is a key expected outcome of the programme. Evaluation therefore requires both measuring scale and associated parameters achieved by the implementation, and capturing its impact on HIV among core, bridge and general populations. In addition, cost-effectiveness assessments are both useful evaluation outputs in their own right and major inputs into programme transfer.
Scale, coverage and quality questions
This requires answering questions around the scale, geographical footprint of the interventions and population coverage, service uptake by target populations, the quality of the programme and the speed with which these were attained. These questions are not only central to Avahan, but also provide major inputs into an assessment of the impact on the epidemic and cost effectiveness. Finally, they are critical for formative evaluation during programme roll-out.
HIV epidemic impact questions
This requires answering questions on whether (and by how much) Avahan succeeded in increasing the use of condoms and reducing the prevalence of STI among the various target populations, and reducing HIV incidence among the target and general populations.
Causal statements (with probability or strong plausibility) that could attribute changes in the epidemic to Avahan could only have been established with experimental approaches using randomization at individual or community level or quasi-experimental evaluation approaches using control groups [52–54]. These would have been impossible as a result of: a complex multiplayer environment with technically similar interventions at state and district level; diffusion within states and districts from other core, bridge and other HIV prevention programmes; ethical issues surrounding withholding of known interventions; diffuse causal chains applicable to health promotion and HIV because of its long incubation period; explicit intent to scale-up and consequent urgency and speed with which Avahan rolled out interventions; and Avahan's intent eventually to transfer programmes to the government, which precluded using government interventions as controls [5,6,52–56].
These considerations led to the decision that the focus of impact measurement would be on the combined efforts of all HIV prevention players in the district. It would therefore be appropriate to measure Avahan's overall contribution to HIV impact because attribution would be difficult without an experimental design. In addition, the geographical unit of impact evaluation activities would be on the entire district rather than Avahan sites within a district as a result of the intent to saturate coverage of core groups jointly, the fact that approximately half of the districts were joint districts, diffusion effects beyond direct beneficiaries in the districts, and some state-wide programme elements such as advocacy.
Accordingly, epidemic impact evaluation questions are: Have there been increases in condom use and a reduction in STI among core and bridge groups in the Avahan districts? Have there been reductions in HIV infections among core and bridge groups in Avahan districts? Have there been reductions in new HIV infections among the general population in Avahan districts (and states)? How much of the changes in the general population were caused by core and bridge group programmes implemented by all players? How much did Avahan contribute to these reductions?
Key questions that were identified in this area were: What was the cost per population reached with different services at various (geographical) levels (implementing NGO, district), over various time intervals and at multiple levels of scale (informed by the size of the cluster of populations addressed within a geographical area)? What was the cost effectiveness of the programme at appropriate geographical and scale levels as measured in terms of HIV infections averted among core, bridge and the general population? In addition what were the costs of the major components of the implementation (outreach, condom distribution, clinical services, etc.) in relation to their contribution to impact?
Figure 3 presents all the three categories of questions (scale and coverage, epidemic impact and cost effectiveness) in the form of a hierarchical flow chart.
The budget for monitoring and evaluation was set at or approximately 10% of the overall budget for the implementation phase. The size and scale of Avahan required that this budget be allocated in a way that ensured that the intensity and scope of evaluation activities were graduated across the 83 districts. A basic minimum set of evaluation activities thus takes place in all districts. Approximately one-third of the districts support more intensive activities in addition to this, whereas a much smaller subset support a fuller range of evaluation activities (Fig. 4). Table 1 shows how the evaluation questions map to methods and data collection activities.
Methods for questions on scale, coverage, service uptake and quality of programme
Three primary methods supported by data collection activities with differing geographical scope and led or implemented by different types of partners are used to inform these questions.
First, data triangulation of implementing partners' routine process monitoring systems, and target population denominators and their geographical distribution will yield internal estimates of the percentage of the populations who have availed themselves of various services, the intensity with which different services have been utilized and the speed with which the programme was rolled out. This method is applied to all the districts.
The second method uses responses on programme exposure along multiple dimensions among the target populations from district-wide surveys conducted by an Avahan evaluation partner in a subset of districts. These surveys are described more fully later in the paper.
The third method uses external quality monitoring of clinical services for core groups by programme officers from an STI capacity-building partner in a subset of districts. For the male client programmes retail condom availability and clinical services are monitored through internal mechanisms in a subset of districts.
Methods to answer epidemic impact questions
Avahan evaluation partners will deploy three main methods to assess the programme's contribution to the impact on the Indian HIV epidemic. These are described after an initial discussion of key considerations influencing methodological choices.
In 2003, major external HIV-related data collection activities in India comprised state-level behavioural surveys of at-risk and general population groups scheduled to be conducted by the government once in 5 years and the government's annual HIV sentinel surveillance system [57–59]. The behavioural surveys were not timed to coincide with the Avahan programme, were powered for state and not district level, and were thus not directly relevant for Avahan's purposes. The sentinel surveillance system aims to capture HIV prevalence among beneficiaries of a subset of core group interventions run by the government, attendees at government STI clinics, and attendees at government antenatal clinics (ANC; where in addition to HIV prevalence, age data were also captured). The data on the first two groups in the sentinel surveillance system were not relevant as the population in one case (STI clinic attendees) was not necessarily representative of the target populations, and in both cases the data are subject to all the biases of facility-based sampling.
The original evaluation design acknowledged the long-term strategic importance of the ANC HIV surveillance system but did not include methods that would use these data directly. The major challenges identified at the time included the low coverage of districts in the six high prevalence states, the lack of consistency in sites from which samples were collected year on year and possible representational biases of the populations attending government ANC . Other issues included the low sample size (400 per clinic site), the consequent limited power to detect changes at district level, given low prevalence levels and questions around data quality.
From 2004 onwards, however, the government has expanded ANC surveillance to include at least two consistent clinic sites in each district in the six Avahan states, and in 2006 it started gathering data on parity. Therefore by 2010, there will be a time series of age-segregated ANC data for 7 years across all the districts in six states. For a large majority of women in these states sexual debut coincides with marriage, and marriage and childbirth occurs relatively early . In the context of a core–bridge-driven epidemic, this suggests that trends in HIV prevalence among young women aged 15–24 years can be used as a proxy for trends in new HIV infections among the general population, as has been postulated elsewhere [61–63]. Although limited by the two caveats that HIV prevalence trends among 15–24 year olds do not explicitly capture changes in infections among men or indeed older groups, and incidence estimation by this method may yield overestimates of absolute changes in infections because it will ignore a substantial fraction of the population that is sexually inactive; overall such analysis will still be useful in indicating the direction if not the absolute numbers of general population infections.
This prospect, coupled with the need to create and transfer long-term and sustainable epidemic impact measurement systems to the government, led to the decision in late 2007 to explore methods that leveraged ANC data substantially. The utility and role of other routine data sources from voluntary counselling and testing and the prevention of mother-to-child transmission efforts in India will need to be examined periodically as these expanding programmes scale-up and stabilize.
There was significant interest in exploring the use of transmission dynamics modelling to estimate impact in terms of infections averted of FSW, client and high-risk MSM/transgender individual interventions, both as an end in itself and as input into cost-effectiveness assessments. Key considerations that resulted in the decision to fund the development of a custom model included the need to incorporate MSM transmission dynamics and multiple STI, the ability to use unbiased fitting procedures for the selection of model parameter ranges, and the ability to use automated fitting procedures given the number of districts to evaluate and the need to reflect parameter uncertainty adequately [64–70]. Accordingly, the development of a custom mathematical model for sexual transmission is one of the elements of the evaluation framework.
Analysis of data from cross-sectional surveys among target populations
Two rounds of cross-sectional surveys among target populations in the same subset of districts conducted by an evaluation partner will be used to assess changes in sexual and injecting drug use behaviour, condom use, and STI and HIV prevalence . Two implementing partners have run limited district-level behavioural-only surveys of their own, which provide additional input. Finally, although not directly comparable, the behavioural surveys run by the government also provide context on condom use, injecting behaviour in IDU and sexual behaviour in the target and general populations.
Mathematical transmission dynamics modelling of the counterfactual
The Avahan evaluation framework incorporates a tailor-made, deterministic transmission dynamics model of HIV and other STI that will be used within a Bayesian framework . First, available data will be used to define a plausible range of values for the different biological, behavioural and intervention parameters (the prior distribution). A large number of different parameter combinations will be sampled from these prior distributions in order to compare the model predictions with the observed HIV and STI outcomes measured in serial cross-sectional surveys among target populations (approximately one-third of Avahan districts) as well as among the general population (a small subset of districts). Only the subset of parameter sets that agree with the empirical data (posterior distribution) will be used to simulate outcomes in the presence and absence (control group) of the intervention in order to estimate the impact of the intervention with credibility intervals.
Primary model outputs will include HIV and STI prevalence and numbers of new HIV and STI infections averted at the district level in the target and general population over a specific time period with smaller or larger credibility intervals (depending on the extent of data available) for a little less than one-third of the Avahan districts, all in the southern states (e.g. districts with IDU-driven epidemics are not included). The model will aim to estimate the ‘contribution’ of Avahan to overall impact in districts where it is one of the players and the ‘attribution’ to Avahan where it is the only provider of interventions for target populations. In addition, the effectiveness of different components of the Avahan programme (primarily outreach and condom distribution, and STI treatment) will also be outputs of the model .
Analysis of age-segregated antenatal clinic surveillance time series
Multilevel, analytical and synthetic methods that can associate prevalence trends over time among 15–24-year-old ANC attendees (a good if partial proxy for incidence as mentioned earlier) with HIV prevention coverage will be deployed across all districts in a state . This will involve: a careful review of existing ANC data with cognizance of the data quality and integrity issues; characterization of all prevention coverage (e.g. geographical footprint, commodity distribution, service utilisation) not just Avahan programming; and potentially adjusting for district-level variations such as differences in epidemic maturity, recent antiretroviral therapy/prevention of mother-to-child transmission scale-up, etc. Avahan's contribution to any declines in prevalence among 15–24-year-old attendees will be a byproduct of this analysis. Further mathematical modelling based on transmission dynamics between core, bridge and general populations, fertility data, and ranges of possible condom use increase between core and bridge groups will be used to conduct sensitivity analysis and establish the theoretical plausibility of trickle-down effects from core and bridge group interventions to declines in prevalence among young ANC attendees .
Methods for cost-effectiveness questions
Answering the question related to the cost effectiveness of population reached with services requires triangulation of data collected on actual economic costs incurred at NGO, district and partner level against data from routine monitoring systems. This will be available for a large subset of districts.
Answering the questions related to cost effectiveness of impact and of different components in attaining impact will require triangulation of costing data against outputs generated by the modelling activities. This will be available for the subset of districts for which transmission dynamics modelling will generate outputs.
Overview of key data collection activities
All seven implementing partners working with core groups commissioned at least one formal, externally managed mapping and size estimation exercise to generate denominators of core populations at the start of programme roll-out in each of their respective districts. In brief, a combination of extensive geographical and social mapping in urban and peri-urban areas with repeated, intensive Delphi techniques and focus group discussions with key informants (target populations, influencers) was used to arrive at district-wide denominators [73–75]. In addition, the evaluation partner responsible for running cross-sectional surveys among the target population was also charged with independently generating and validating size estimates with alternative methods (multiplier, capture–recapture, multistage sampling) in the subset of districts .
Routine monitoring systems
The implementing partners' process monitoring systems collect data to report on a variety of routine indicators on a monthly basis . These indicators are generated at all district and implementing NGO levels. A critical subset of these indicators conforms to standardized definitions created by one of the evaluation partners. These include indicators related to: hard and ‘soft’ infrastructure (number of implementing NGO, numbers of drop-in centres, programme-owned and referral clinics, outreach workers and peer educators); geographical coverage (towns or subdistrict administrative units reached by peer educators); and population coverage and service uptake by the population (individuals met by peers and outreach workers, individuals who availed themselves of different clinical services, individuals receiving free condoms, number of condoms distributed).
Cross-sectional surveys among target populations
Repeat cross-sectional surveys (termed the integrated biological and behavioural assessment; IBBA) with the aim of measuring changes in behavioural, biological, programme exposure and sociodemographic characteristics over time among Avahan target populations are being performed in approximately one-third of the Avahan intervention districts . The first round covered 29 of the 83 Avahan districts. A total of 29 FSW survey groups, 17 male client groups, 11 high-risk MSM/transgender groups, one transgender group sampled across five districts, five IDU groups, and four trucker groups from four highway route categories was sampled in the first survey round [8–12]. District-wide probability sampling methods are used including conventional cluster sampling, time–location sampling or respondent-driven sampling, depending on the characteristics of the study populations. Behavioural parameters captured include the numbers and types of sexual partners, the number of sexual acts by partner type, reported condom use by partner type and injecting practices. Exposure parameters include exposure to outreach and behaviour change communications services, free condom distribution and the use of clinical services. Respondents are tested for STIs: syphilis serology, Neisseria gonorrhoeae, Chlamydia trachomatis, herpes simplex virus type 2 (HSV-2) serology (10% of samples) and HIV serology. Genital ulcers are tested for HSV-2, Treponema pallidum and Haemophilus ducreyi. Hepatitis B and C serology is also performed on IDU respondents.
Cross-sectional surveys among general population
General population household-based surveys (GPS) of both urban and rural populations are being performed to inform mathematical modelling [16,72]. A total of five districts, four in Karnataka and one in Andhra Pradesh, where the IBBA was also conducted, were identified. The GPS studies men and women in urban and rural areas using a stratified two-stage sampling method to select 6000 subjects randomly distributed equally between rural and urban areas and between men and women. The questionnaires are specifically designed to include questions relevant for epidemiological and modelling analyses and included questions on sociodemographic information, types of sexual partnerships, sexual behaviour, migration and STI history. Respondents are tested for STIs: syphilis serology, N. gonorrhoeae, C. trachomatis, HSV-2 serology and HIV serology. A second round of GPS will be carried out in all of the original GPS districts where the initial HIV prevalence was over 1%.
Other evaluation studies
To complement information obtained in the IBBA, special behaviour surveys (SBS) are conducted in high-risk MSM/transgender individuals and FSW after the IBBA has taken place. The SBS are carried out to validate and obtain more detailed sexual behavioural data for modelling purposes. The SBS employs two different collection methods: traditional face-to-face interviews supplemented by informal confidential voting interview on a subset of questions . The informal confidential voting interviews are interviewer-administered questionnaires that incorporate confidential self-completion methods by the respondent who is separated visually from the interviewer. SBS is being carried out in six districts for FSW and four districts for high-risk MSM/transgender individuals in the four southern states.
Polling booth surveys are administered on a subset of questions in all the GPS described above to validate proportions of the population reporting key high-risk sexual behaviours . Polling booth surveys are anonymous group interviews conducted with approximately 10 individuals separated from one another by a private booth. Because of the nature of the data collection, analysis is only at the aggregate level.
To assess reported condom use by men and exposure to messages, regular surveys performed every 6–12 months are conducted with men recruited from a subset of hotspots (randomly selected from all intervention sites in a state) who report sex worker contact in the previous 12 months .
Quality monitoring activities
At the Avahan-wide level, STI clinical service quality, the quality of condom outlets and the intensity of service engagement by target populations are monitored. STI services provided through project-supported clinics are monitored at the time of supervisory visits based on published standards [79,80]. Information is obtained through analysis of the routine reported clinic data, interviews and observation of the clinic providers and clinic record reviews. In the franchised clinics for men, mystery patient surveys are used to monitor quality by completing forms on the encounter after a consultation . Condom coverage surveys of condom outlets assessed adequate number for the size of the hot spot, visibility of promotional material and opening hours of the outlets . These quality monitoring efforts are ‘dipstick’ in nature covering a handful districts or NGO representative of all lead implementing partners/states in each assessment round. The districts or NGO vary in each subsequent assessment.
At the NGO level the intensity of service delivery (peer community member ratios, number of condoms distributed per community member, percentage of target population met monthly) and utilization (percentage of population seeking services in STI clinics) is routinely monitored and reported.
A compilation of data related to the HIV epidemic including intervention coverage for all 115 districts in the four southern states is compiled to provide a basis for interpreting district-level ANC trends and for extrapolating results from the intensive evaluation study districts to other Avahan intervention areas .
As part of a larger knowledge-building agenda to inform programme design and approaches, the foundation supports additional studies under Avahan in the documentation of community mobilization efforts, migration and mobility, and in STI treatment algorithm performance . These activities also contribute to the overall Avahan evaluation efforts. For example, questionnaires developed and utilized in the community mobilization research will help fine tune the survey instruments for further survey rounds, male migration data will help inform the parameters for the models as will the efficacy of STI treatment approaches under Avahan . In addition, the foundation under Avahan also supports research and documentation on community mobilization and structural interventions that systematically demonstrate intervention impact and identify the key components of successful intervention implementation .
District selection for evaluation
District selection for the various intensive evaluation activities followed a purposive approach. Funds were sufficient for conducting two rounds in the same district of at least one core group survey in one-third of the total number of districts during the implementation phase. A major criterion for IBBA district selection was that all states and all implementing partners be represented in the districts chosen. Within this context, districts were chosen based on the classification of sociocultural regions as a proxy for epidemic patterns, and based on the size of the main core group (FSW, IDU) population with a few exceptions . In addition, the capital region in each state was mandatorily included. Within this set, districts for male client IBBA were chosen to ensure that states and implementing partners were represented. Districts for high-risk MSM surveys were chosen based on the size of the MSM population from among the IBBA districts. Therefore, of the 83 Avahan districts, 29 are IBBA districts .
Funding constraints dictated that modelling outputs be restricted to the four southern states with primarily sexual transmission dynamics. GPS (conducted to inform modelling) were thus restricted to up to five districts in the south. These districts were chosen from within the IBBA districts to ensure a minimum set that would include ‘proof of concept’ districts (large female sex worker population, Avahan as sole provider, closed population with low migration and mobility), diverse scenarios (metropolitan, urban, rural), and diverse antenatal prevalence.
Basic costing data are being collected from all districts. Districts for intensive costing studies were, however, chosen so as to include mainly districts with sizeable core group populations where Avahan is the sole player in order to explore the economic cost of scaling up coverage better, and to ensure state and implementing partner representation. As Karnataka had few previous interventions before Avahan, all districts except one are ‘sole districts’ there, and consequently both costing and GPS studies have a larger presence in the state.
Implementing the evaluation design
As a result of an initial delay in making evaluation grants, negotiations on partnership dispositions between multinational partners, time to attain consensus on instrument design, and the complexity and sheer scale of the data collection activities, the bulk of the initial round of data collection by evaluation partners occurred during 2006, except for one state (Karnataka) where a large portion of the first round evaluation data collection took place during 2004–2005. By 2006, implementing partners had already attained ‘scale’ with approximately 80% of the eventual geographical footprint of services being established and 70% of the estimated target population receiving HIV prevention services . The first round of evaluation partner-led surveys for the majority of Avahan districts and target populations was thus not completed before programmes were well established. A second round of cross-sectional survey data collection is scheduled for 2009. Karnataka will complete the second round of data collection in 2008. For some districts in Karnataka, two rounds have been completed .
The ongoing evolution of the national programme further complicates the implementation, analysis and interpretation of the evaluation design. The National AIDS Control Organisation (NACO) launched the National AIDS Control Programme 3 (NACP-3) in 2007. NACP-3 has a funding commitment four times that of the previous programme, NACP-2, and aims to achieve high coverage with HIV prevention services of core groups across India . Another key aim is to ensure a single funder is responsible for all core group interventions in a district. As a consequence, there have already been some changes during 2007–2008 in the disposition of districts between Avahan and other players; these include some IBBA districts. In addition, the foundation expects to start transferring the funding, management and implementation of Avahan programmes in the vast majority of districts to the Government of India from 2010.
How accurately will Avahan's evaluation activities describe the programme?
The foundation has chosen to invest in a large portfolio of evaluation methods and data collection activities across varying geographical theatres to inform Avahan evaluation. Combined together, these will offer a complex picture with varying comprehensiveness of the programme's measures of success.
A clear assessment of programme scale-up, coverage, service uptake and quality at district, partner and state level of the Avahan programme is likely to emerge. As described above, routine monitoring data coupled with repeated size estimation data offer a progressive and comprehensive picture of target population coverage and service utilization as related to the denominator. Programme exposure data from the IBBA and service quality assessments offer additional external and internal measures of exposure and quality.
Detecting the outcome of interventions in terms of changes in sexual behaviour, condom use and STI prevalence may require more nuanced approaches. The delayed IBBA may lead to an underestimation of the true impact of interventions. In spite of this, it may be possible to arrive at adequacy statements around changes in sexual behaviour and condom use in both core and bridge populations by triangulating the following data: (1) external state-level historical baselines including the NACO BSS 2001; (2) individual district-level implementing partner surveys in two states; (3) two rounds of IBBA in the selected districts . Adequacy statements regarding biological outcomes among core and bridge groups may be limited because of low levels (except in a few districts) of STI detected in the first round of the IBBA and few earlier studies in India [22,52,54,86].
There are several opportunities for ‘natural experiments’ within the Avahan programme as a result of the large number of districts where cross-sectional surveys are carried out, the associated differences in timing of intervention start dates, and the availability of monitoring data for scale, coverage, quality and service uptake. These may allow the construction of analyses based on ‘dose-response’ and historical control, which may contribute to plausibility statements about Avahan interventions.
Given the long incubation period of HIV and the lack of reliable measures of new HIV infection, the full impact of prevention interventions can take years to detect, in terms of evidence of reduced HIV infections in core, bridge and general populations [61,87,88]. So whereas data on programme scale-up and changes in sexual behaviour among core and bridge populations may suggest eventual HIV impact; it may be harder to detect impact decisively in the evaluation timeframe over which Avahan will run before bulk transfer.
On the other hand, mathematical modelling is likely to be able to provide plausibility statements with credibility intervals around the number of HIV/STI cases averted at district level for a large subset of districts in the south, discern the relative contribution of the various technical interventions used in Avahan, and assess the contribution of Avahan's interventions to overall estimated averted HIV infections [72,89]. In addition, mathematical modelling will be used to generate sensitivity analyses around the levels of core and bridge group intervention effects required to enable changes in HIV trends among young ANC attendees [14,17].
The successful application of synthetic analysis of changes in HIV prevalence among 15–24 year age group ANC attendees and their association with extent, coverage and service utilization of all core group interventions will be dependent on two things. These include the power and size of the 15–24-year ANC subsets to detect changes for epidemiologically consistent subsets of districts and the quality and extent of the characterization of coverage data of all interventions, not just Avahan. If these issues can be suitably addressed it will be possible to provide additional evidence on the impact of all core group interventions, and as a corollary question, Avahan's contribution, subject of course to the limitations described earlier of using prevalence among 15–24 year age group ANC attendees as an incidence proxy .
Finally, there will certainly be useful measures of the cost effectiveness of Avahan in terms of the population reached or covered (cost per population reached per year for different types of services), in terms of cost per infection averted (in which infections averted is an output of the transmission dynamics model) and illustrative examples of resource allocation efficiency based on the contribution of different intervention components.
Avahan's evaluation data collection efforts are extensive and not replicable for national HIV programmes in which less resource-intensive efforts and routine data sources would be more appropriate. Nonetheless, Avahan's evaluation and knowledge-building efforts were also intended to develop an evidence base to inform HIV prevention practitioners and policy makers globally about approaches, costs, and the cost-effectiveness of scaled HIV prevention in concentrated epidemics, the costs relative to the impact of various intervention components, and approaches to evaluating large-scale interventions. The data being collected within Avahan combined with government data provide a data-rich source for these endeavours.
Globally there is a renewed focus on HIV prevention, with recommendations and associated funding for rolling out combination prevention interventions quickly with high population and geographical coverage. Avahan's continuing experience with impact evaluation offers important lessons for evaluating large-scale public health programmes that operate in the frequently messy, real world . Large-scale programmes must contend with continuously changing external environments that may impact on original programme design, thus necessitating adjustments to original evaluation elements, and creating some tension between the necessity of simply getting programmes and services out to beneficiaries and the desire for greater precision and rigour in evaluation. Evaluation frameworks for these programmes will need to examine and synthesize multiple datasets not all of which may be of the same provenance or quality.
Avahan evaluation partners
CHARME: Peter Vickerman, Lilani Kumaranayake, Charlotte Watts, Sushena Reza-Paul, Annie-Claude Labbé
Family Health International: Bitra George, Denis Jackson, Kathleen Kay, Robert Magnani, Motiur Rahman
University of Manitoba and Karnataka Health Promotion Trust: Stephen Moses, James Blanchard, Reynold Washington, B.M. Ramesh, Shajy Isac
Centre for Global Health Research, LKSI/KRC, St Michael's Hospital, University of Toronto: Paul Arora, Prabhat Jha, Shreelata Rao Seshadri, Catherine McLaughlin, Sema Sgaier, Rajesh Kumar, Raju Jotkar, Alice Easton, Ashleigh Sullivan, Nico Nagelkerke
Sponsorship: Support for this work was provided by the Bill and Melinda Gates Foundation. M.A. is a national researcher of the Fonds de la Recherche en Santé du Québec, Canada (grant no. 8722).
The views expressed herein are those of the authors and do not necessarily reflect the official policy or position of the Bill and Melinda Gates Foundation.
Conflicts of interest: None.
3. Bennett S, Boerma JT, Brugha R. Scaling up HIV/AIDS evaluation. Lancet 2006; 367:79–82.
4. Schachter J. Biologic versus behavioral endpoints – the duet continues. Sex Transm Dis 2000; 27:456–457.
5. Nebot M. Health promotion evaluation and the principle of prevention. J Epidemiol Commun Health 2006; 60:5–6.
7. Saidel T, Adhikary R, Mainkar M, Dale J, Loo V, Rahman M, et al
. Baseline integrated behavioural and biological assessment among most at-risk populations in six high-prevalence states of India: design and implementation challenges. AIDS 2008; 22(Suppl. 5):S17–S34.
8. Ramesh BM, Moses S, Washington R, Isac S, Mohapatra B, Mahagaonkar SB, et al
. Determinants of HIV prevalence among female sex workers in four south Indian states: analysis of cross-sectional surveys in twenty-three districts. AIDS 2008; 22(Suppl. 5):S35–S44.
9. Brahmam GNV, Kodavalla V, Rajkumar H, Rachakulla HK, Kallam S, Myakala SP, et al
. Sexual practices, HIV and sexually transmitted infections among self-identified men who have sex with men in four high HIV prevalence states of India. AIDS 2008; 22(Suppl. 5):S45–S57.
10. Mahanta J, Medhi GK, Paranjape RS, Kholi A, Roy N, Akoijam B, et al
. Injecting and sexual risk behaviours, sexually transmitted infections and HIV prevalence in injecting drug users in three states in India. AIDS 2008; 22(Suppl. 5):S59–S68.
11. Pandey A, Benara SK, Roy N, Sahu D, Thomas M, Joshi DK, et al
. Risk behaviour, sexually transmitted infections and HIV among long-distance truck drivers: a cross-sectional survey along national highways in India. AIDS 2008; 22(Suppl. 5):S81–S90.
12. Subramanian T, Gupte MD, Paranjape RS, Brahmam GNV, Ramakrishnan L, Adhikary R, et al
. HIV, sexually transmitted infections and sexual behaviour of male clients of female sex workers in Andhra Pradesh, Tamil Nadu and Maharashtra, India: results of a cross-sectional survey. AIDS 2008; 22(Suppl. 5):S69–S79.
13. Vadivoo S, Gupte MD, Adhikary R, Kohli A, Kangusamy B, Joshua V, et al
. Appropriateness and execution challenges of three formal size estimation methods for high-risk populations in India. AIDS 2008; 22(Suppl. 5):S137–S148.
14. Reza-Paul S, Beattie T, Syed HR, Venukumar KT, Venugopal MS, Fathima M, et al
. Declines in risk behaviour and sexually transmitted infection prevalence following a community-led HIV preventive intervention among female sex workers in Mysore, India. AIDS 2008; 22(Suppl. 5):S91–S100.
15. Moses S, Ramesh BM, Nagelkerke NJD, Khera A, Isac S, Bhattacharjee P, et al
. Impact of an intensive HIV prevention programme for female sex workers on HIV prevalence among antenatal clinic attenders in Karnataka state, south India: an ecological analysis. AIDS 2008; 22(Suppl. 5):S101–S108.
16. Munro HL, Pradeep BS, Jayachandran AA, Lowndes CM, Mohapatra B, Ramesh BM, et al
. Prevalence and determinants of HIV and sexually transmitted infections in a general population-based sample in Mysore district, Karnataka state, southern India. AIDS 2008; 22(Suppl. 5):S117–S125.
17. Boily MC, Pickles M, Vickerman P, Buzdugan R, Isac S, Deering K, et al
. Using mathematical modelling to investigate the plausibility of attributing observed antenatal clinic declines to a female sex worker intervention in Karnataka state, India. AIDS 2008; 22(Suppl. 5):S149–S164.
18. Blankenship KM, West BS, Kershaw TS, Biradavolu MR. Power, community mobilization, and condom use practices among female sex workers in Andhra Pradesh, India. AIDS 2008; 22(Suppl. 5):S109–S116.
19. Deering KN, Vickerman P, Moses S, Ramesh BM, Blanchard JF, Boily MC. The impact of out-migrants and out-migration on the HIV/AIDS epidemic: a case study from southwest India. AIDS 2008; 22(Suppl. 5):S165–S179.
20. Saggurti N, Verma RK, Jain A, RamaRao S, Anil Kumar K, Subbiah A, et al
. HIV risk behaviours among contracted and non-contracted male migrant workers in India: potential role of labour contractors and contractual systems in HIV prevention. AIDS 2008; 22(Suppl. 5):S127–S136.
21. Chandrasekaran P, Dallabetta G, Loo V, Rao S, Gayle H, Alexander A. Containing HIV/AIDS in India: the unfinished agenda. Lancet Infect Dis 2006; 6:508–521.
22. Hawkes S, Santhya KG. Diverse realities: sexually transmitted infections and HIV in India. Sex Transm Infect 2002; 78(Suppl. 1):i31–i39.
23. Gawande AV, Vasudeo ND, Zodpey SP, Khandait DW. Sexually transmitted infections in long distance truck drivers. J Commun Dis 2000; 32:212–215.
24. Gangakhedkar RR, Bentley ME, Divekar AD, Gadkari D, Mehendale SM, Shepherd ME, et al
. Spread of HIV infection in married monogamous women in India. JAMA 1997; 278:2090–2092.
25. Mehendale SM, Ghate MV, Kishore Kumar B, Sahay S, Gamble TR, Godbole SV, et al
. Low HIV-1 incidence among married serodiscordant couples in Pune, India. J Acquir Immune Defic Syndr 2006; 41:371–373.
26. Mehendale SM, Gupte N, Paranjape RS, Brahme RG, Kohli R, Joglekar N, et al
. Declining HIV incidence among patients attending sexually transmitted infection clinics in Pune, India. J Acquir Immune Defic Syndr 2007; 45:564–569.
27. Nagelkerke NJ, Jha P, de Vlas SJ, Korenromp EL, Moses S, Blanchard JF, et al
. Modelling HIV/AIDS epidemics in Botswana and India: impact of interventions to prevent transmission. Bull WHO 2002; 80:89–96.
29. Evaluation of HIV/AIDS targeted interventions in reduction of HIV transmission in five states in India
– draft national level report. UK: Options; 2003.
31. Ainsworth M, Beyrer C, Soucat A. AIDS and public policy: the lessons and challenges of “success” in Thailand. Health Policy 2003; 64:13–37.
32. Ghys PD, Diallo MO, Ettiegne-Traore V, Kale K, Tawil O, Carael M, et al
. Increase in condom use and decline in HIV and sexually transmitted diseases among female sex workers in Abidjan, Cote d'Ivoire, 1991–1998. AIDS 2002; 16:251–258.
33. Alary M, Mukenge-Tshibaka L, Bernier F, Geraldo N, Lowndes CM, Meda H, et al
. Decline in the prevalence of HIV and sexually transmitted diseases among female sex workers in Cotonou, Benin, 1993–1999. AIDS 2002; 16:463–470.
34. Merson MH, Dayton JM, O'Reilly K. Effectiveness of HIV prevention interventions in developing countries. AIDS 2000; 14(Suppl. 2):S68–S84.
35. Thomas JC, Tucker MJ. The development and use of the concept of a sexually transmitted disease core. J Infect Dis 1996; 174(Suppl. 2):S134–S143.
36. Foss AM, Hossain M, Vickerman PT, Watts CH. A systematic review of published evidence on intervention impact on condom use in sub-Saharan Africa and Asia. Sex Transm Infect 2007; 83:510–516.
37. Fung IC, Guinness L, Vickerman P, Watts C, Vannela G, Vadhvana J, et al
. Modelling the impact and cost-effectiveness of the HIV intervention programme amongst commercial sex workers in Ahmedabad, Gujarat, India. BMC Public Health 2007; 7:195.
38. Lowndes CM, Alary M, Labbe AC, Gnintoungbe C, Belleau M, Mukenge L, et al
. Interventions among male clients of female sex workers in Benin, West Africa: an essential component of targeted HIV preventive interventions. Sex Transm Infect 2007; 83:577–581.
39. Alary M, Lowndes CM. The central role of clients of female sex workers in the dynamics of heterosexual HIV transmission in sub-Saharan Africa. AIDS 2004; 18:945–947.
40. Herbst JH, Beeker C, Mathew A, McNally T, Passin WF, Kay LS, et al
. The effectiveness of individual-, group-, and community-level HIV behavioral risk-reduction interventions for adult men who have sex with men: a systematic review. Am J Prev Med 2007; 32(Suppl. 4):S38–S67.
41. Johnson WD, Hedges LV, Diaz RM. Interventions to modify sexual risk behaviors for preventing HIV infection in men who have sex with men
. Cochrane Database Syst Rev
42. Hoffmann O, Boler T, Dick B. Achieving the global goals on HIV among young people most at risk in developing countries: young sex workers, injecting drug users and men who have sex with men. WHO Tech Rep Ser 2006; 938:287–315, discussion 317–241.
43. Plummer FA, Nagelkerke NJ, Moses S, Ndinya-Achola JO, Bwayo J, Ngugi E. The importance of core groups in the epidemiology and control of HIV-1 infection. AIDS 1991; 5(Suppl. 1):S169–S176.
44. Hallett TB, Garnett GP, Mupamberiyi Z, Gregson S. Measuring effectiveness in community randomized trials of HIV prevention. Int J Epidemiol 2008; 37:77–87.
45. Ward H. Prevention strategies for sexually transmitted infections: importance of sexual network structure and epidemic phase. Sex Transm Infect 2007; 83(Suppl. 1):i43–i49.
46. Ainsworth M, Teokul W. Breaking the silence: setting realistic priorities for AIDS control in less-developed countries. Lancet 2000; 356:55–60.
47. Valdiserri RO, Ogden LL, McCray E. Accomplishments in HIV prevention science: implications for stemming the epidemic. Nat Med 2003; 9:881–886.
48. Stover J, Walker N, Garnett GP, Salomon JA, Stanecki KA, Ghys PD, et al
. Can we reverse the HIV/AIDS pandemic with an expanded response? Lancet 2002; 360:73–77.
49. National AIDS Control Organization (NACO). Annual Report 2002–2003, 2003–2004 (up to 31 July 2004)
. New Delhi, India: National AIDS Control Organization and Ministry of Health and Family Welfare; 2004.
50. Boerma JT, Weir SS. Integrating demographic and epidemiological approaches to research on HIV/AIDS: the proximate-determinants framework. J Infect Dis 2005; 191(Suppl. 1):S61–S67.
52. Habicht JP, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol 1999; 28:10–18.
53. Bryce J, Victora CG, Habicht JP, Vaughan JP, Black RE. The multi-country evaluation of the integrated management of childhood illness strategy: lessons for the evaluation of public health interventions. Am J Public Health 2004; 94:406–415.
54. Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health 2004; 94:400–405.
55. Susser M. Some principles in study design for preventing HIV transmission: rigor or reality. Am J Public Health 1996; 86:1713–1716.
56. Barreto ML. Efficacy, effectiveness, and the evaluation of public health interventions. J Epidemiol Commun Health 2005; 59:345–346.
57. National AIDS Control Organization (NACO) and the Ministry of Health and Family Welfare. National baseline behavioral surveillance survey (BSS) among general population, 2001
. New Delhi, India: National AIDS Control Organization (NACO) and the Ministry of Health and Family Welfare; 2002.
58. National AIDS Control Organization (NACO) and the Ministry of Health and Family Welfare. National baseline high risk and bridge population behavioral surveillance survey, 2001
. New Delhi, India: National AIDS Control Organization (NACO) and the Ministry of Health and Family Welfare; 2002.
60. International Institute for Population Studies and Ministry of Health and Family Welfare. Reproductive and child health: district level household survey (DLHS-2) 2002–4 – India
. New Delhi, India, 2006. http://mohfw.nic.in/dlhs_2002.htm
. Accessed: 12 July 2008.
61. Garnett GP, Gregson S, Stanecki KA. Criteria for detecting and understanding changes in the risk of HIV infection at a national level in generalised epidemics. Sex Transm Infect 2006; 82(Suppl. 1):i48–i51.
62. Zaba B, Boerma T, White R. Monitoring the AIDS epidemic using HIV prevalence data among young women attending antenatal clinics: prospects and problems. AIDS 2000; 14:1633–1645.
63. Kumar R, Jha P, Arora P, Mony P, Bhatia P, Millson P, et al
. Trends in HIV-1 in young adults in south India from 2000 to 2004: a prevalence study. Lancet 2006; 367:1164–1172.
64. Vickerman P, Terris-Prestholt F, Delany S, Kumaranayake L, Rees H, Watts C. Are targeted HIV prevention activities cost-effective in high prevalence settings? Results from a sexually transmitted infection treatment project for sex workers in Johannesburg, South Africa. Sex Transm Dis 2006; 33:S122–S132.
65. Vickerman P, Watts C, Delany S, Alary M, Rees H, Heise L. The importance of context: model projections on how microbicide impact could be affected by the underlying epidemiologic and behavioral situation in 2 African settings. Sex Transm Dis 2006; 33:397–405.
66. Boily MC, Lowndes C, Alary M. The impact of HIV epidemic phases on the effectiveness of core group interventions: insights from mathematical models. Sex Transm Infect 2002; 78(Suppl. 1):i78–i90.
67. White RG, Orroth KK, Korenromp EL, Bakker R, Wambura M, Sewankambo NK, et al
. Can population differences explain the contrasting results of the Mwanza, Rakai, and Masaka HIV/sexually transmitted disease intervention trials?: a modeling study. J Acquir Immune Defic Syndr 2004; 37:1500–1513.
68. Korenromp EL, de Vlass SJ, Nagelkerke NJ, Habbema JD. Estimating the magnitude of STD cofactor effects on HIV transmission: how well can it be done? Sex Transm Dis 2001; 28:613–621.
69. Korenromp EL, Van Vliet C, Grosskurth H, Gavyole A, Van der Ploeg CP, Fransen L, et al
. Model-based evaluation of single-round mass treatment of sexually transmitted diseases for HIV control in a rural African population. AIDS 2000; 14:573–593.
70. Brown T, Peerapatanapokin W. The Asian epidemic model: a process model for exploring HIV policy and programme alternatives in Asia. Sex Transm Infect 2004; 80(Suppl. 1):i19–i24.
72. Boily MC, Lowndes CM, Vickerman P, Kumaranayake L, Blanchard J, Moses S, et al
. Evaluating large-scale HIV prevention interventions: study design for an integrated mathematical modelling approach. Sex Transm Infect 2007; 83:582–589.
75. Blanchard JF, Bhattacharjee P, Kumaran S, Ramesh BM, Kumar NS, Washington RG, et al
. Concepts and strategies for scaling up focused prevention for sex workers in India. Sex Transm Infect 2008; 84(Suppl. 2):ii19–ii23.
76. Gregson S, Zhuwau T, Ndlovu J, Nyamukapa CA. Methods to reduce social desirability bias in sex surveys in low-development settings: experience in Zimbabwe. Sex Transm Dis 2002; 29:568–575.
77. Blanchard JF, Halli S, Ramesh BM, Bhattacharjee P, Washington RG, O'Neil J, et al
. Variability in the sexual structure in a rural Indian setting: implications for HIV prevention strategies. Sex Transm Infect 2007; 83(Suppl. 1):i30–i36.
78. Behavior Change Impact Survey (BCIS). Consistent condom use among male clients of female commercial sex workers in Andhra Pradesh, Karnataka, Maharashtra and Tamil Nadu
. Population Services International, New Delhi, India, 2006. Available at: http://www.psi.org.in/psi_research_report.pdf
. Accessed: 14 January 2008.
80. Steen R, Mogasale V, Wi T, Singh AK, Das A, Daly C, et al
. Pursuing scale and quality in STI interventions with sex workers: initial results from Avahan India AIDS initiative. Sex Transm Infect 2006; 82:381–385.
83. Centre for Global Health Research. Documenting the geographic variation in HIV-1, its determinants and intervention coverage in 115 districts in southern India
. Overview document. 23 November 2006. Available at: http://cghr.org/OD4.pdf
. Accessed: 15 May 2008.
84. Singh AK. People of India series, 1993–1998
. Anthropological survey of India, Ministry of Culture, Government of India. Available at: http://www.ansi.gov.in/people_india.htm
. Accessed: November 2008.
89. Williams JR, Foss AM, Vickerman P, Watts C, Ramesh BM, Reza-Paul S, et al
. What is the achievable effectiveness of the India AIDS initiative intervention among female sex workers under target coverage? Model projections from southern India. Sex Transm Infect 2006; 82:372–380.
90. Pettifor AE, MacPhail C, Bertozzi S, Rees HV. Challenge of evaluating a national HIV prevention programme: the case of LoveLife, South Africa. Sex Transm Infect 2007; 83(Suppl. 1):i70–i74.
bridge groups; core groups; HIV prevention; impact; India; large-scale; modelling; monitoring
© 2008 Lippincott Williams & Wilkins, Inc.
Highlight selected keywords in the article text.