Skip Navigation LinksHome > July 2008 - Volume 22 - Issue > How can we calculate the “E” in “CEA”?
AIDS:
doi: 10.1097/01.aids.0000327623.31856.45
Section I: Informing Scale-up and Resource Allocation

How can we calculate the “E” in “CEA”?

Bollinger, Lori A

Free Access
Article Outline
Collapse Box

Author Information

From the Futures Institute, Glastonbury, Connecticut, USA.

Correspondence to Lori A. Bollinger, Futures Institute, 41-A New London Turnpike, Glastonbury, CT 06033, USA. E-mail: LBollinger@FuturesInstitute.org

Collapse Box

Abstract

Because full funding for HIV/AIDS prevention interventions is unlikely to occur in the near future, it is essential that the resources available are spent in the most effective way possible. This paper presents a matrix of effectiveness coefficients for HIV/AIDS-related prevention interventions that can be used as an integral part of the coordinated strategic planning process currently underway by the World Bank and UNAIDS, as the interventions in the matrix are harmonized with the interventions in that process. Coefficients for four types of sexual behavior change (condom use, partner reduction, sexually transmitted infection treatment-seeking behavior, age at first sex) across three different risk groups (high, medium, low) are presented, along with their interquartile ranges. Results indicate that: (1) impacts seem greater when an intervention includes interpersonal contact, rather than targeting a more general audience; (2) although significant impacts are observed in the columns measuring changing condom use, other impacts are lower, and sometimes are actually (measured) zero; and (3) additional studies have evaluations of the number of sexual partners and have found a greater impact than previous studies. Although progress has been made in increasing the number of evaluation studies that can be utilized in this impact matrix, particularly in the area of youth interventions, there are still empty cells in which no studies report impacts. Finally, it is important to note that issues such as quality differences and synergies between programmes could have an effect on the impacts calculated for a particular strategic plan.

Back to Top | Article Outline

Introduction

Although funding for interventions to prevent the transmission of HIV/AIDS has continued to increase substantially over the past few years, a gap remains between all currently available resources and those that would be required to achieve universal access to HIV/AIDS prevention [1]. Because full funding is unlikely to occur in the near future, it is essential that the available resources are spent as effectively as possible.

As a result of the urgency to identify strategic ways to spend the increased funds, defining effective interventions has become more and more important. Early efforts to assess the effectiveness of HIV/AIDS-related prevention interventions found many gaps in the relevant evaluation literature [2,3]. Since then, however, the research base of correctly designed and executed evaluation studies has grown, and recent efforts to collate this information have found a substantially greater quantity of available research [4–10]. Although it is important to keep conducting high-quality evaluation research, planners can begin to evaluate the relative effectiveness of HIV/AIDS-related prevention interventions on the basis of findings to date.

In addition to publishing compilations of well-designed evaluation study results, it is important, however, to place these results within a strategic planning framework. Evaluating the effectiveness of prevention interventions in isolation, without using a context that allows for an examination of overall effects as well as interaction between effects, may result in faulty recommendations. In addition, the strategic planning process itself, based on situational analysis and incorporating key stakeholders in the decision-making process, provides a way to validate recommendations that might not be supported if made in isolation [11].

For the past few years, the United Nations Joint Program on HIV/AIDS (UNAIDS) has supported a strategic planning process to programme available funds effectively by coordinating the diverse elements of such processes, including: (1) performing situational analyses; (2) projecting HIV prevalence rates; (3) evaluating current spending by performing national AIDS spending assessments; and (4) estimating resources required to attain universal access to interventions, all at the national level. Over time, these disparate efforts have been harmonized so that output from one element of the process can be used as input to another element. Recently, the World Bank has assumed primary responsibility for coordinating these strategic planning efforts within the United Nations family at the national level through the AIDS Strategic and Action Plan (ASAP) initiative.

This paper presents a matrix of effectiveness coefficients for HIV/AIDS-related prevention interventions that can be used as an integral part of this strategic planning process. The intervention categories match the classifications used to estimate the resources required, described above. The National AIDS Spending Assessments (NASA) have also recently adopted these classifications in order to harmonize the strategic planning process further. The findings described in this matrix can thus also be used in conjunction with forthcoming results from the new NASA framework. Although the matrix can be used in isolation, with caveats, its primary purpose is for use in applying a resource allocation tool called ‘Goals’. The impact matrix forms the basis of this model that can be utilized along with other elements to assist in the national strategic planning process [12].

The results presented here are part of an ongoing effort that began in 1999 to develop a resource allocation tool to assist in the strategic planning process. The initial strategy was to develop a tool that national AIDS control programme managers could use; however, interviews with programme managers revealed that the real need for such a tool was at a more aggregate level, that is, at the strategic planning level [13]. In addition, although allocating resources as effectively as possible among prevention intervention options is important, other elements of an AIDS programme need to be considered during this allocation process. For example, the increasing availability of antiretroviral therapy (ART) treatment may have spillover effects on certain prevention interventions. It may both increase the demand for voluntary counselling and testing (VCT) and increase the need for VCT, as risky behavior may increase with use of ART (although the evidence on the effects of ART on sexual behavior is mixed) [14–17]. It is therefore important to evaluate the effectiveness of prevention interventions within an overall context or model that incorporates the interrelationship between various elements of a national AIDS control programme.

Back to Top | Article Outline

Methods

An extensive literature search was conducted to identify studies that evaluated HIV/AIDS-related prevention interventions in developing countries. This search included web searches of the National Library of Medicine Gateway and Popline databases; final reports and evaluations of projects from Horizons, AIDSCAP, Policy Project, and other projects; selective review of EMBASE, BIOSIS, SCISEARCH, and SIGLE; searches of web sites including UNAIDS and Department for International Development; hand searches of bibliographies of articles once they were identified; and author contact when appropriate.

The first round of the literature search was concluded in June 2001 [3]. The results from this initial search were examined closely and validated by a team at the World Health Organization during the summer of 2002. The validation consisted of a review of the literature contained in the matrix, including re-calculating all of the relevant coefficients to check for accuracy, as well as an independent literature search to identify additional studies for possible inclusion in the matrix.

A third round of updating took place as part of an effort led by the Working Group on Modeling and Infections Averted for the United States President's Emergency Plan for AIDS Relief (PEPFAR) during the summer of 2004 to project potential HIV infections averted. This updating included further literature search efforts to identify and include more evaluation studies published as of December 2003, following the process described above, as well as a validation of both the included studies and the coefficient calculations by the working group [18].

This paper presents the results of a fourth round of updating to December 2006, following the same methodology as the other literature searches. As noted earlier, the purpose of these estimates is to harmonize efforts among the various elements of national strategic planning processes. To this end, the matrix classifies effectiveness results into the same prevention intervention categories that NASA and UNAIDS-sponsored workshops use to estimate resources required for national-level HIV/AIDS programmes, shown in Table 1.

Table 1
Table 1
Image Tools

Because this impact matrix forms the basis for the ‘Goals’ resource allocation model, it can include only studies with certain endpoints that are used in the model equations: condom use; number of sexual partners; age at first sex; contact with sex workers; changes in risky sexual activity; or sexually transmitted infection (STI) treatment-seeking behavior.

To summarize, the following inclusion criteria are used in the literature searches: HIV/AIDS or STI prevention intervention; developing country setting; pre/post-intervention measures both provided; at least one useful endpoint reported.

Studies were excluded if published before 1981; otherwise, studies excluded from the matrix were disqualified mainly because they failed to consider endpoints of interest or lacked pre-intervention data.

It should be noted here that the ‘Goals’ model includes male circumcision as a possible prevention intervention, allowing for increasing coverage of a base male circumcision prevalence rate, with a resulting decrease in the rate of female–male HIV transmission. Because, however, no studies had evaluated any behavioral changes that may result from the counseling portion of a male circumcision intervention, it is not included (yet) as a line item in the impact matrix.

Table 2 displays the distribution of studies according to both research design and region. The included studies fall into several main research design categories: randomized trials (controlled/non-controlled); quasi-experimental; prospective cohort/longitudinal (with/without controls); cross-sectional and others (includes pre/post surveys, cluster, crossover, retrospective record review and case studies). Geographical coverage is wide, with studies from Asia, Latin America and the Caribbean, sub-Saharan Africa and a few from Eastern Europe.

Table 2
Table 2
Image Tools

As always, ‘the quality of the data is limited by the quality of the studies’ reviewed ([19], p. 1188). Some of the studies included suffer from methodological flaws, including: limited follow-up time (sometimes less than 6 months); lack of control group; non-random allocation of study subjects; high attrition (with subsequent loss of statistical power when comparing two groups); and various biases, including self-reporting and publication bias. A complete database of the literature used to calculate the impact coefficients, including for each study detailed descriptions of the intervention, study design, sample size, year of intervention, population, and results can be found at: http://www.policyproject.com/pubs/HIV-AIDSLiteratureDC.xls.

In the end, approximately 150 studies pertaining to the sexual transmission of HIV are included in the ‘goals’ impact matrix. The sexual transmission part of the impact matrix has three dimensions:

* Interventions (see Table 1 above for list of interventions);

* Risk groups:

* high risk: sex workers and their clients, men who have sex with men (MSM), and injecting drug users (IDU);

* medium risk: men and women with more than one sexual partner in the previous year (not including sex workers and their clients); and

* low risk: men and women with one partner only in the previous year.

* Key sexual behaviors:

* consistent condom use;

* STI treatment-seeking behavior;

* number of sexual partners; and

* age at first sex.

Note that not all interventions affect all risk groups or even all behaviors within a risk group. For example, one would not expect that an outreach programme for sex workers would have an impact on delaying age at first sex, nor would a school-based programme for youth be expected to affect the high-risk population significantly. When a particular cell is (qualitatively) judged to be irrelevant for either that particular risk group or that particular sexual behavior, it is shown as grey in the matrix tables below.

In order to be able to accumulate changes across a number of interventions and avoid problems with estimated condom use or STI treatment exceeding 100% in the model calculations, the impacts of interventions on condom use and STI treatment are calculated as percentage reductions in non-use. Impacts on the other two behaviors are calculated as the percentage reduction in the number of partners per year and the actual increase in number of years in the age at first sex. For example, if a study found that the proportion of respondents who always used condoms increased from 16.0 to 25.0%, then these statistics would first be changed to a non-use statistic (16% use becomes 84% non-use, 26% use becomes 74% non-use), and then the percentage reduction in consistent condom non-use is calculated:

Equation (Uncited)
Equation (Uncited)
Image Tools

In another example, if a study found that the mean number of a sex worker's clients in a previous week decreased from 20 before the intervention to 13 after the intervention, the reduction in the number of partners is a simple percentage calculation:

Equation (Uncited)
Equation (Uncited)
Image Tools

Equations for each of the intervention impacts are described fully in Appendix A.

Note that some of the coefficient calculations in the impact matrix have a qualitative component. For example, the model requires consistent condom use as a parameter, but some evaluations report condom use at last sex, rather than consistent condom use. After a detailed analysis of the correlation between consistent condom use and condom use at last sex, sponsored by the United States President's Emergency Plan for AIDS Relief Working Group on Modeling and Infections Averted, it was agreed to discount the impact of any outcome measuring condom use at last sex by one-third in order to approximate an outcome of consistent condom use.

Back to Top | Article Outline

Results

Table 3 presents the results for the calculation of the average impact of HIV/AIDS-related prevention interventions, whereas Table 4 shows the interquartile range of the impacts. If only one study was relevant for a particular cell in Table 4, only one number is entered, instead of a range.

Table 3
Table 3
Image Tools
Table 4
Table 4
Image Tools

Table 3 and Table 4 each include three dimensions: (1) the prevention interventions from Table 1, listed in the rows; (2) the types of sexual behavior changes that are measured, listed across the main column headings; and (3) the three risk groups, listed as subcolumns within each of the four sexual behavior change headings. Note that two interventions listed as target population groups are actually considered as separate risk groups in the ‘Goals’ model: IDU and MSM peer outreach interventions. To simplify the presentation, we listed the impacts of these interventions under the ‘high risk’ category, rather than specify five sets of risk group columns for each type of sexual behavior change.

If a cell is grey, it is presumed a priori that no studies would be available for that cell, while a cell is blank when no available studies measure an impact for that particular cell. Finally, a zero in a cell means that impacts were calculated, and that the value of the impact equals zero, i.e. there was no change in the relevant behavior.

Patterns in the columns reveal, first, that there is generally little information about the impact of prevention interventions on STI treatment-seeking behavior. Although a few studies do report an impact of an intervention on STI incidence or prevalence, the methodology of the ‘goals’ model requires that the outcome measure be STI treatment-seeking behavior, and so changes in incidence or prevalence could not be included in the calculations.

Another pattern to note is in the column measuring the increase in age at first sex. Although studies do exist that examine the impact of various interventions on increasing the age at first sex, including mass media, community mobilization, school-based programmes, and condom social marketing programmes, the impacts that are measured are quite small, ranging from an actual decrease of 0.3 years to a high of 0.11 years.

Examining results across the rows of the table makes clear that some interventions are more effective than others, at least when comparing the impact on condom use, the variable for which most interventions report evaluation data. Interventions with higher relative impacts include VCT, sex worker outreach programmes, workplace programmes, other peer education programmes, and public sector condom distribution interventions. Interventions with lower relative impacts include condom social marketing programmes as well as outreach programmes for MSM and IDU, whereas those interventions with the lowest relative impact include mass media and youth interventions. Not surprisingly, interventions providing STI treatment exhibit the highest impact on STI treatment-seeking behavior, although, as noted above, few interventions include this behavior in their evaluation design.

Interestingly, when comparing impacts on reducing the number of partners, the two relatively most effective interventions are both peer-oriented interventions: workplace programmes and other peer education. School-based programmes are also effective, resulting in a one-third decrease in the number of partners among youth. Other interventions, such as mass media, VCT, and condom social marketing programmes, seem to have a low impact on reducing the number of partners, including some zero impacts.

Back to Top | Article Outline

Discussion

Some generalizations can be drawn from the information in these matrices. First, impacts appear to be greater when an intervention involves interpersonal contact, rather than a more general target audience, although some of this may be due to the fact that the impact of peer interventions can be measured more directly than other interventions. For example, the impacts of peer counseling programmes, as well as VCT interventions, are higher than the impacts of interventions such as mass media or community mobilization. Second, although significant impacts are observed in the columns measuring changing condom use, impacts are much lower for other types of sexual behavior, such as the number of partners and age at first sex. Some of the interventions evaluating the impact on age at first sex had a (measured) zero impact. Third, since the last update, more studies have evaluated the impact of various interventions on the number of partners, and have found larger impacts than previously. This is quite encouraging, as previous evaluations found little or no impact, and partner reduction is thought to be an important component of controlling the HIV/AIDS epidemic.

Although progress has been made in increasing the number of evaluation studies included in this impact matrix, particularly in the area of youth interventions, there are still empty cells in which no impacts are reported. As noted earlier, the columns measuring STI treatment-seeking behavior reveal that changes in such behavior are not available from most evaluation studies, and the outcomes that the studies do report are not compatible with the model categories. As the presence of STIs facilitate HIV transmission, it would be especially useful to have compatible evaluation data to help identify the most effective interventions in this area.

Finally, it is important to note that certain measurement issues may influence the values of the coefficients that would have an impact on strategic planning efforts. For example, different impacts across studies may be a result of different levels of quality within interventions; a recent study found a wide variation in the quality of VCT services both within and across countries [20]. These variations may also have an impact on the effectiveness of the interventions. In addition, the existence of possible synergies between interventions would affect the impact coefficients, both across types of interventions and across organizations that implement similar interventions. When the impact matrix is applied via the ‘Goals’ model within the context of an overall strategic planning process, these measurement issues need to be considered when calculating the impact of the strategic plan on the number of HIV infections averted.

Sponsorship: Funding support was from the US Agency for International Development through Task Order 1 of the USAID/Health Policy Initiative under contract no. GPO-I-01-05-00040-00.

Conflicts of interest: None.

Back to Top | Article Outline

References

1. UNAIDS. Resource needs for an expanded response to AIDS in low- and middle-income countries. Geneva: UNAIDS, August 2005. Available at: http://data.unaids.org/publications/irc-pub06/resourceneedsreport_en.pdf. Accessed: March 2007.

2. Merson MH, Dayton JM, O'Reilly K. Effectiveness of HIV prevention interventions in developing countries. AIDS 2000; 14(Suppl. 2):S68–S84.

3. Bollinger L, Cooper-Arnold K, Stover J. Where are the gaps? The effects of HIV-prevention interventions on behavioral change. Stud Fam Plann 2004; 35:27–38.

4. Bertrand JT, O'Reilly K, Denison J, Anhang R, Sweat M. Systematic review of the effectiveness of mass communication programs to change HIV/AIDS-related behaviors in developing countries. Health Educ Res 2006; 21:567–597.

5. Wegbreit J, Bertozzi S, DeMaria LM, Padian NS. Effectiveness of HIV prevention strategies in resourse-poor countries: tailoring the intervention to the context. AIDS 2006; 20:1217–1235.

6. Johnson WD, Hedges LV, Diaz RM. Interventions to modify sexual risk behaviors for preventing HIV infection in men who have sex with men. Cochrane Database Syst Rev 2003; 1:CD001230.

7. Medley A, O'Reilly K, Sweat M. Systematic review of the efficacy of peer education programs in reducing HIV sexual risk behavior in developing countries. Presented at the 2006 International AIDS Conference. Toronto, Canada, 13–18 August 2006 [abstract no. ThpeD7661].

8. O'Reilly, K, Medley A, Denison J, Sweat M. The impact of abstinence-based interventions on sexual behavior in developing countries: a systematic review and meta-analysis. Presented at the 2006 International AIDS Conference. Toronto, Canada, 13–18 August 2006 [abstract no. THAX0301].

9. Denison, J, O'Reilly KR, Schmid GP, Kennedy CE, Strouse, D, Sweat MD. A systematic review of the impact of voluntary HIV counseling and testing (VCT) on risk behaviors in developing countries (1990–2005). Presented at the 2006 International AIDS Conference. Toronto, Canada, 13–18 August 2006 [abstract no. WEPE0395].

10. Vidanapathirana J, Abramson MJ, Forbes A, Fairley C. Mass media interventions for promoting HIV testing: Cochrane systematic review. Int J Epidemiol 2006; 35:233–234.

11. AIDS Strategy and Action Plan (ASAP) Service. HIV/AIDS strategic self-assessment tool. 2006. Available at: http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTHEALTHNUTRITIONANDPOPULATION/EXTHIVAIDS/0,contentMDK:20974772∼menuPK:2755553∼pagePK:210058∼piPK:210062∼theSitePK:376471,00.html. Accessed: March 2007.

12. Stover J, Bollinger L, Cooper-Arnold K. Goals model for estimating the effects of resource allocation decisions on the achievement of the goals of the HIV/AIDS strategic plan, Version 3. March 2003. Available at: http://www.futuresgroup.com/software/Goals/goals.pdf. Accessed: March 2007.

13. Stover J, Bollinger L. Resource allocation within HIV/AIDS programs. In: Forsythe, S., editor. State of the art: AIDS and economics. Washington, DC: POLICY Project, Futures Group; July 2002.

14. Creek TL, Alwano MG, Molosiwa RR, Roels TH, Kenyon TA, Mwasalla V, et al. Botswana's Tebelopele voluntary HIV counseling and testing network: use and client risk factors for HIV infection, 2000–2004. J Acquir Immune Defic Syndr 2006; 43:210–218.

15. Kozal MJ, Amico KR, Chiarella J, Cornman D, Fisher W, Fisher J, et al. A population-based and longitudinal study of sexual behavior and multidrug-resistant HIV among patients in clinical care. Med Gen Med 2006; 8:72.

16. Moatti JP, Prudhomme J, Traore DC, Juillet-Amari A, Akribi HA, Msellati P, Cote d'Ivoire HIV Drug Access Initiative Socio-Behavioural Evaluation Group. Access to antiretroviral treatment and sexual behaviours of HIV-infected patients aware of their serostatus in Cote d'Ivoire. AIDS 2003; 17(Suppl. 3):S69–S77.

17. Bateganya M, Colfax G, Shafer LA, Kityo C, Mugyenyi P, Serwadda D, et al. Antiretroviral therapy and sexual behavior: a comparative study between antiretroviral-naive and -experienced patients at an urban HIV/AIDS care and research center in Kampala, Uganda. AIDS Patient Care STDS 2005; 19:760–768.

18. Stover J, Bollinger L. Estimating infections averted by year one activities as described in the country operational plans of the President's Emergency Plan for AIDS Relief. Prepared for the Working Group on Modeling and Infections Averted. Washington DC: Constella Futures 2004.

19. Kunz R, Oxman AD. The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998; 317:1185–1190.

20. Marseille E, Dandona L, Marshall N, Gaist P, Bautista-Arredondo S, Rollins B, et al. HIV prevention costs and program scale: data from the PANCEA project in five low and middle-income countries. BMC Health Services Research 2007; 7:108.

Back to Top | Article Outline
Appendix A

1. Equation for reduction in condom non-use:

Reduction in condom non-uset = ([1−condom use0]−[1−condom uset]/[1−condom use0])

Reduction in condom non-use in year t is calculated as one minus the proportion of condom use in year 0 minus one minus the proportion of condom use in year t (postintervention) divided by one minus the proportion of condom use in year 0.

2. Equation for STI treatment: reduction in non-STI-treatment-seeking behavior:

Reduction in STI non-Tx-seeking behaviort = ([1−STI Tx0]−[1−STI Txt]/[1−STI Tx0])

Reduction in STI non-treatment-seeking behavior in year t is calculated as one minus the proportion seeking treatment in year 0 minus one minus the proportion seeking treatment in year t divided by one minus the proportion seeking treatment in year 0.

3. Equation for reduction in the number of partners:

Reduction in number of partnerst = number of partners0−number of partnerst

The reduction in the number of partners in year t for each risk group and sex is calculated as the number of partners in year 0 minus the number of partners in year t, postintervention.

4. Equations for changes in age at first sex. To calculate an expected age at first sex and the increase at age at first sex between preand postintervention:

Pre-intervention: calculated age at first sex = (proportion of students initiating sex before a given age)*(age at first sex) + (1 − proportion of students initiating sex before a given age)*(an average age at first sex from a Demographic and Health Survey [DHS])

Post-intervention: Calculated age at first sex = (proportion of students initiating sex before a given age)*(age at first sex) + (1 − proportion of students initiating sex before a given age)*(an average age at first sex from a DHS)

Increase in years of age at first sex: Postintervention expected age at first sex − pre-intervention expected age at first sex.

An example:

1991: 0.20*16 years + 0.80*20 years = 19.2 years

1992: 0.15*16 years + 0.85*20 years = 19.4 years

1993: 0.10*16 years + 0.90*20 years = 19.6 years

with an increase of 0.2 years for age at first sex between 1991/1992 and 1992/1993, where 0.20 = proportion of students initiating sex before 16 years in the year before the intervention; 0.15 = proportion of students initiating sex before 16 years in the first year after the intervention; 0.10 = proportion of students initiating sex before 16 years in the second year after the intervention.

0.80 = 1 − 0.20; 0.85 = 1 − 0.15; 0.90 = 1 − 0.10;

20 years = average year of age at first sex from the country's DHS.

Keywords:

communicable disease control/economics/methods; cost–benefit analysis; developing countries; disease transmission; HIV infections/prevention and control/transmission; sexual behavior

© 2008 Lippincott Williams & Wilkins, Inc.

Login

Search for Similar Articles
You may search for similar articles that contain these same keywords or you may modify the keyword list to augment your search.