Weighing the gold in the gold standard: challenges in HIV prevention research
Padian, Nancy Sa,b; McCoy, Sandra Ib; Balkus, Jennifer Ec; Wasserheit, Judith Nd
aSchool of Public Health, USA
bInstitute of Business and Economic Research, University of California, Berkeley, California, USA
cDepartment of Epidemiology, USA
dDepartment of Global Health, University of Washington, Seattle, Washington, USA.
Received 7 August, 2009
Revised 18 December, 2009
Accepted 22 December, 2009
Correspondence to Nancy S. Padian, MPH, PhD, Adjunct Professor, School of Public Health, Center of Evaluation for Global Action, University of California, Berkeley, F508 Haas Building, Berkeley, CA 94720-1922, USA. Tel: +1 510 926 1387; fax: +1 510 642 5018; e-mail: firstname.lastname@example.org
Objective(s): Few HIV prevention interventions have been evaluated in randomized controlled trials (RCTs). We examined design, implementation, and contextual considerations that may limit detection of a positive or adverse effect in HIV prevention trials.
Design: A systematic review of late phase RCTs for prevention of sexual transmission of HIV that randomly allocated intervention and comparison groups; evaluated interventions to prevent sexual transmission in nonpregnant populations; and reported HIV incidence as the primary or secondary outcome.
Methods: PubMed/MEDLINE, other electronic databases, and electronic conference proceedings of recent HIV/AIDS-related conferences were searched to identify published or unpublished trials meeting the inclusion criteria. Descriptive, methodological, and contextual factors were abstracted from each trial.
Results: The review included 37 HIV prevention RCTs reporting on 39 unique interventions. Only six RCTs, all evaluating biomedical interventions, demonstrated definitive effects on HIV incidence. Five of the six RCTs significantly reduced HIV infection: all three male circumcision trials, one trial of sexually transmitted infection treatment and care, and one vaccine trial. One microbicide trial of nonoxynol-9 gel produced adverse results. Lack of statistical power, poor adherence, and diluted versions of the intervention in comparison groups may have been important issues for the other trials that demonstrated ‘flat’ results.
Conclusion: Almost 90% of HIV prevention trials had ‘flat’ results, which may be attributable to trial design and/or implementation. The HIV prevention community must not only examine evidence from significant RCTs, but must also examine flat trials and address design and implementation issues that limit detection of an effect.
The global need for effective HIV prevention programs has never been more urgent. Although the number of people receiving antiretroviral drugs in low-income and middle-income countries increased 10-fold in the past 6 years, new infections in 2007 outpaced antiretroviral therapy uptake by a margin of five to two [1–5]. Randomized controlled trials (RCTs) are generally considered the gold standard to define the evidence base for HIV prevention programs and policies. However, only one in seven RCTs of interventions to prevent sexual transmission of HIV has shown efficacy [6–12]. In fact, the overwhelming majority of completed RCTs are ‘flat’ – unable to demonstrate either a positive or adverse effect.
Flat results may occur for three reasons. The underlying concept may be flawed; the concept may be sound, but the specific intervention approach may be ineffective; or, especially for interventions postulated to have only modest impact, aspects of the study design, implementation, or context may limit detection of a true effect. RCT results due to the first two situations have been critically important to advancing HIV prevention science [6,7,10,11]. The third squanders resources and costs lives. Our review addresses this last phenomenon.
We conducted a systematic literature review of phase IIb or III RCTs for prevention of sexual transmission of HIV. We searched PubMed/MEDLINE, EMBASE, the Cochrane Library, including the Cochrane Central Register of Controlled Trials (CENTRAL), and Web of Science for articles meeting our inclusion criteria as of 6 December 2009. There were no language restrictions to the search. We developed a customized search strategy for each database relying on the database's controlled vocabulary or index [e.g., medical subject headings (MeSHs)] or free text terms. In most cases, search strategies combined terms for HIV infection, incidence or hazard, prevention, and study design restrictions (randomized controlled designs). Appropriate MeSH or free text terms were also included to exclude trials that focused on the prevention of mother-to-child transmission of HIV. In PubMed/MEDLINE, we searched for clinical trials using Cochrane's ‘Highly Sensitive Search Strategy’ for identifying RCTs . Search strategies for each database are available from the authors.
We conducted a cited reference search with key articles, scanned reference lists of eligible articles and reviews, and searched the electronic conference proceedings of recent HIV/AIDS-related conferences [Conference on Retroviruses and Opportunistic Infections, International Society for Sexually Transmitted Disease (STD) Research annual meetings, and International AIDS Society annual meetings]. Clinicaltrials.gov was searched to identify ongoing or recently completed trials. We contacted study authors and reviewed secondary articles to obtain more information about the trials, as needed. We also searched reference lists of eligible articles and recent commentaries and reviews for potentially relevant trials and contacted colleagues for additional information not revealed in our search. The broad inclusion criteria and the resulting heterogeneity of interventions precluded the use of meta-analytic methods.
Eligible trials were those that used randomized controlled designs (individual or community); evaluated interventions focusing on sexual transmission of HIV in nonpregnant populations; and reported HIV incidence as the primary or secondary outcome. Trials that assessed HIV infection as part of an aggregate STD outcome (i.e., any incident STD at follow-up) were excluded [14–18]. We first examined the citations from the literature search to eliminate obviously ineligible studies (e.g., nonrandomized studies, studies without biological outcomes, inappropriate article types such as reviews or commentaries). Abstracts were specifically searched for mention of an intervention tested against a control intervention with biological outcomes. Report of any STD outcome in the abstract such as incident gonorrhea or chlamydia infections automatically warranted a full-length review of the article to determine whether HIV testing was performed. We then conducted a detailed manual review of full-length articles to determine eligibility.
All authors independently reviewed and abstracted data from each eligible article or abstract. Discrepancies among the authors were identified and resolved. In two cases, multiple interventions were described in a single article (i.e., multiple treatment arms with different interventions and a single control) [19,20]. In these instances, each intervention was counted separately as an appropriate comparison could be made to the control. We abstracted descriptive information about each trial and the most adjusted measure of effect on HIV infection and 95% confidence interval (CI), giving preference to intention-to-treat analyses. We computed the measure of effect using standard methods when only the risk or rate was presented stratified by study arm (e.g., incidence in the treatment group divided by the incidence in the comparison group) . We also abstracted the measure of effect in preplanned subgroup analyses among those studies for which we were aware that such analyses had been conducted.
We categorized trials by type of prevention interventions [i.e., behavioral, microfinance, diaphragm, microbicides, preexposure prophylaxis (PrEP), male circumcision, sexually transmitted infection (STI) treatment, and vaccines] and their effect on HIV incidence. ‘Positive’ intervention trials significantly reduced the risk of HIV infection in the intervention arm compared to the control arm, whereas ‘adverse’ trials significantly increased HIV risk. Interventions with ‘no effect’ were those in which no statistically significant effect on HIV acquisition or transmission was reported (positive or adverse); thus, the null hypothesis could not be rejected. Although we examined effect size and precision of the measures of effect [usually an incidence rate ratio (IRR) or hazard ratio], we considered statistical significance of the effect measure (α = 0.05) as the definitive measure of whether an intervention was positive, adverse, or flat (had no effect).
We also considered a broad set of methodological and contextual factors to identify specific issues in the design, conduct, or environment in which the trials were conducted that might have contributed to their results. We compared the projected and observed HIV incidence in the comparison arm (per 100 person-years or annual cumulative incidence) either given by the authors or computed using standard methods . The level of adherence achieved in both arms of the study was also abstracted. In general, adherence in the intervention arms was defined as attendance at intervention activities for behavioral trials, the proportion of sex acts in which the study product was used for diaphragm and microbicide trials, the proportion of prescribed doses completed for PrEP and for STI treatment or suppression trials, or the proportion of participants receiving all doses in vaccine trials. Adherence in the control arm was defined similarly for the placebo product or prevention intervention offered to the control group.
In all instances, the intervention arm examined the marginal benefit of the intervention over and above other prevention services in both the intervention and control arm. We assessed whether these services, often offered to both the intervention and control arms, exceeded the local community standard of care in terms of basic components of HIV prevention such as risk-reduction counseling or HIV education, provision of condoms and related counseling, and availability of STI treatment services or case management. Comparison arms in which all of these components were augmented beyond the local, existing standard of HIV prevention were classified as ‘exceptional’ prevention packages. Comparison arms in which at least one of these components was added or strengthened were classified as ‘enhanced’ prevention, whereas those with nothing above the local standard were classified as ‘standard of care.’ In addition, regardless of the intensity of the prevention services offered, in some cases the type of preventive services provided to the control arm overlapped with the primary intervention being tested, though in a diluted form [e.g., for behavioral interventions, improved voluntary counseling and testing (VCT) in the control group compared to enhanced VCT combined with other behavioral interventions in the intervention group]. Trials with control groups that overlapped with the main intervention in this way were highlighted as ‘overlapping’. Finally, to assess the impact of these improved prevention services, we evaluated whether risk behavior decreased in both arms of the study compared to baseline measurements.
The results of the literature search are presented in Fig. 1. We identified 3616 unique citations from electronic databases, of which 3555 were excluded based on title examination and 27 were excluded based on abstract-level review. Thirty-four full-length articles were reviewed in detail. Twenty-nine articles from the literature search met our inclusion criteria. Eight additional studies were identified through reference list examinations, reviews of conference abstracts, or published reviews or editorial articles [7,22–25]. Thus, 37 HIV prevention RCTs were included in this review reporting on 39 unique interventions to prevent sexual transmission of HIV (Table 1) [19,20,25–62]. Five studies included in the review were unpublished and obtained from conference presentations or other sources [20,25–28,63]. Note however, that one of these studies  has just been published. [Although the Regai Dzive Shiri trial was originally implemented as a community randomized controlled trial (CRCT) with evaluation in a longitudinal cohort, the analysis was re-conceptualized as a serial cross-sectional evaluation due to significant out-migration from trial communities [26,63]. We included this trial in the review, despite not reporting on HIV incidence (HIV prevalence was measured in study communities after the intervention was implemented) because of the size of the study and the important implications this study has for understanding the challenges in scaling up prevention packages.]. Here we summarize characteristics of the prevention trials and then discuss the most salient factors associated with RCTs demonstrating significant results and those with flat results, by type of intervention.
Characteristics of prevention intervention trials
The 37 trials were conducted over approximately 30 years from 1987 to 2009 at a rapidly accelerating pace with more than 80% being initiated since 1995 (Table 1). The majority were implemented partially or entirely in southern Africa (n = 32) and employed individually randomized designs (n = 26). Six RCTs included sites in the Americas [20,29–32,64], three enrolled participants in Asia [33–35], and one trial each involved populations in the Netherlands and Australia [31,32]. Nine CRCTs evaluated behavioral, microfinance, and STI treatment interventions and constituted more than half of the 16 RCTs in these categories [19,26,36–42]. The two remaining RCTs examined intervention impact on HIV transmission in discordant couples [28,43].
Most of the RCTs (n = 17) were conducted exclusively in women: 13 in female sex workers (FSWs) or women at high risk of HIV infection [33,34,44–53,64] and four in sexually active women in the general population [20,25,54,55]. Sixteen studies evaluated outcomes in both men and women, including three RCTs in adolescents [26,36,38,63]. Together the trials enrolled nearly 160 000 participants.
Trials demonstrating statistically significant effects
Six RCTs, all examining biomedical interventions, delivered definitive results on HIV infection (Table 2) [34,40,56–58,65]. Five of the six trials were positive; three RCTs of male circumcision (all of which assessed female-to-male HIV transmission) [56–58], one RCT of enhanced STI management , and one vaccine trial . In all three male circumcision trials, a substantial effect size was observed ranging from a 51 to 61% reduction in HIV incidence [56–58]. The trial of improved STI management in the Mwanza region of Tanzania demonstrated a 42% reduction in HIV incidence . The prime boost combination of ALVAC-HIV, a recombinant canarypox vectored vaccine, and AIDSVAX B/E, a recombinant gp120 vaccine, lowered the rate of HIV acquisition by 31% among Thai volunteers, though there was no reduction in postinfection HIV plasma viral load . One of the six significant RCTs, a microbicide trial of nonoxynol-9 gel, produced adverse results with a substantial effect size: a 50% increase in HIV incidence .
Statistical power was not a major issue in any of these six trials (Table 3) [19,20,25–61,66]. However, despite the large sample size (16 402) in the ALVAC/AIDSVAX vaccine trial , the low HIV incidence among this sample of the general population limited study power sufficiently so that both the strict intent-to-treat and per protocol analyses yielded a nonsignificant 26% reduction in HIV incidence and even the significant efficacy documented in the modified intent-to-treat analysis had such a wide CI that questions have been raised about the interpretability of the results. Five of the six studies were powered based on fairly close approximations of the observed HIV incidence in the comparison arm [40,56–58], and in the nonoxynol-9 gel trial, the observed incidence in the comparison arm was approximately double what had been anticipated (5% annually predicted versus 10.3 per 100 person-years observed) .
Similarly, poor adherence was not a key factor in most of the significant trials. Male circumcision is a one-time, directly administered procedure for which adherence issues are limited to refraining from intercourse during healing [7,8,11] and likewise, adherence was relatively high in the microbicide (70–72%) and vaccine (75–78%) trials [34,35]. In addition, the male circumcision and vaccine trials share the benefit of having directly observed interventions, substantially eliminating the problem of reporting bias. Although adherence was not reported in the STI treatment trial, it is likely to have been substantial because trial participants were symptomatic patients seeking STI care and health educators visited villages to encourage prompt treatment for symptomatic STIs.
In none of these trials did the control group receive a dilute version of the intervention being tested (i.e., no overlap). Male circumcision, nonoxynol-9 gel, and vaccine were not provided in any form to controls, and no additional STI training, drugs, or supervisory or educational visits were provided to the control communities in the Mwanza trial. However, in all of the interventions with the exception of the STI treatment trial , the control group received an enhanced or exceptional prevention package beyond the local standard of care.
Trials not demonstrating statistically significant effects
All seven behavioral interventions yielded flat results; however, in the Masaka, Uganda community randomized trial, significant protective effects were found in a subgroup analysis among sexually active women in analyses at the individual level (IRR = 0.41, 95% CI 0.19–0.89) [19,67]. Of the four studies that reported both projected and observed HIV incidence in the control arm, all but one observed incidence rates that were substantially lower than original projections. Only in Project EXPLORE did the projected HIV incidence in the control arm accurately reflect that observed in the trial . Thus, overall, limited power may have limited the ability to detect an effect.
A central issue common to six of these seven trials is that some form of the intervention overlaps with that provided to the control group, though in more dilute form. With one exception (Regai Dzive Shiri) [26,63], each of these studies offered some combination of risk-reduction counseling, condom promotion, and referral and treatment for STIs that exceeded the local standard of care. The effects of so-doing are apparent in the four trials that measured changes in risk behaviors, with all four noting declines in both arms [19,29,38,59].
The Regai Dzive Shiri youth intervention was a CRCT (see text under ‘Literature search’); no prevention services were offered to the comparison communities and uptake of the intervention was low (41%) among survey participants . Adherence was also poor in the Stepping Stones trial, with 44% of participants reporting attendance at 75% or more of the Stepping Stones sessions [26,38]. Adherence ranged between 70 and 75% for the other two trials for which adherence data were reported, Project Explore , and the Zimbabwean workplace intervention .
To date, there has only been one microfinance RCT that has examined HIV endpoints and there was no effect on HIV incidence in study communities . This may be due, in part, to the indirect nature of the intervention, which was directly offered to women microfinancers, whereas the effect on HIV was expected to diffuse to younger women in the populations, which might take years to manifest . The observed HIV incidence was greater than that projected, adherence was moderate (65%), and there was no change in behavior in either study arm .
Microbicide and diaphragm trials
Eleven of the 12 trials of microbicides and the latex diaphragm trial demonstrated flat results. However, of note, 0.5% PRO 2000/5 gel demonstrated a 33% reduction in incidence compared to placebo gel (hazard ratio = 0.7, 95% CI 0.5, 1.1, P = 0.10) in the intent-to-treat analysis and a 36% reduction compared to no gel (P = 0.04) in the per protocol analysis . However, in a larger study, PRO 2000 gel had no effect on HIV incidence . Although the overall estimates for the cellulose sulfate  and SAVVY (C31G) gel  trials were nonsignificant, subgroup analyses revealed significant adverse results in the interim and per protocol analyses of the cellulose sulfate trial and among women in the SAVVY trial with greater than median coital frequency and greater than median frequency of gel use.
Reduced power to detect an effect resulting from a lower than expected incidence of new infections was apparent in six of the seven flat microbicide trials that reported this information. Adherence ranged between 73 and 96% among those studies for which adherence was reported (the lower bound reported in the diaphragm trial), with only two achieving rates of at least 85% in the intervention arm [33,55]. In addition, results from the diaphragm trial indicated differential condom use over the course of the trial with an average of 53% in the intervention arm and 85% in the control . Although none of these studies offered a dilute, overlapping version of the intervention to controls, all of the studies offered exceptional prevention interventions to both arms. With one exception, risk-taking behavior decreased in both study arms in all of the trials that reported these data .
Only one trial of PrEP using antiretrovirals (ARVs) for HIV prevention has been completed . The observed annual HIV incidence was roughly half of what was anticipated, and premature closure of the trial markedly reduced study power. Adherence rates shy of 70% were observed in both arms . Enhanced prevention services were provided to controls and risk behavior was reduced in both study arms throughout the trial.
Male circumcision to prevent male-to-female transmission
A fourth circumcision trial examined HIV transmission to female partners of HIV-infected men who were enrolled in the circumcision RCT in Rakai, Uganda . Overall, no significant reduction in HIV incidence was observed, but HIV acquisition was increased in the subgroup of female partners of men who resumed sexual activity early before complete wound healing compared to those who delayed resumption of sexual activity (relative risk = 2.92, 95% CI 1.02–8.46, P = 0.06) .
Sexually transmitted infection treatment trials
Eight of the nine trials of STI treatment for HIV prevention delivered flat results, though one study found a significant effect on HIV incidence in a subgroup of men who attended program meetings (adjusted IRR = 0.48, 95% CI 0.24–0.98, P = 0.04) . Five RCTs evaluated various approaches to improved management of curable STIs, including two of syndromic STI management in the general population (both CRCTs) [19,42], two of periodic presumptive therapy (one a CRCT in the general population  and the other an individual RCT in FSWs ), and one individual RCT of intensive, microscopy-assisted STI screening in FSWs . The remaining three RCTs tested acyclovir suppressive therapy in both high-risk and low-risk populations [28,30,53].
Several factors probably contributed to the striking contrast between the positive results of the Mwanza STI treatment trial and the five flat RCTs that targeted curable STIs. Although low power, poor adherence, overlapping interventions, or other enhanced prevention services in the comparison communities did not appear to be problems in the Mwanza trial, at least one of these issues arose in each of the five flat trials. In addition, observed HIV incidence fell short of that projected in all four of the RCTs that reported this information [19,41,42,52] and adherence was a moderate 81% in the fifth trial . Perhaps more importantly, like the behavioral interventions, all of these trials offered enhanced or exceptional prevention services to controls, including improved STI services (constituting overlap with the primary intervention being tested) in three of the five RCTs [41,51,52]. Indeed, these control arm interventions were reflected in decreases in risk behaviors in both arms of all five RCTs. Finally, the Mwanza trial was implemented in an earlier phase HIV epidemic than was the case for the five flat trials of treatment of curable STIs, all of which were conducted in late-phase, generalized epidemics when genital herpes had largely replaced curable etiologies of genital ulcers, while rates of other curable STIs had fallen substantially in the general population.
Although epidemic phase was not a concern in the three acyclovir suppression trials, adherence was a challenge, with the proportion of participants reporting taking at least 90% of pills ranging from 51 to 73% in two of the three RCTs. In addition, exceptional HIV prevention services were available to controls with attendant reductions in risk behaviors in all trial arms [28,30,53]. However, growing data suggest that in these trials, it is likely that the 400 mg twice daily (b.i.d.) acyclovir regimen tested was not capable of extinguishing persistent immune activation or other biological mechanisms triggered by herpes simplex virus (HSV) infection, which could increase susceptibility to HIV [68,69].
Three vaccine trials yielded flat results [31,32,35,60]; however, subgroup analyses in the Step Study of the Merck Ad5 gag/pol/nef vaccine suggested significant adverse effects among uncircumcised men (hazard ratio = 3.8, 95% CI 1.5–9.3) and those with high preexisting Ad5 antibody titers (hazard ratio = 2.3, 95% CI 1.2–4.3) . However, the increased risk of HIV seroconversion associated with preexisting immunity to the Ad5 vector did not persist in an interim analysis of Step Study participants after unblinding, and it was not confirmed in analyses of the Phambili trial of the same candidate in South Africa [60,61,70]. Of note, vaccinations in the Phambili trial were stopped prematurely when the results of the Step trial became available and, as a result, the study accrued very limited numbers of HIV seroconverters prior to unblinding, limiting its power to assess this association.
HIV prevention can work, as demonstrated by successes in Thailand and Uganda and now the promise of male circumcision [56–58,71,72]. Yet in the face of what continues to be one of the most devastating pandemics, we have ever known, new, evidence-based approaches are urgently needed. In this context, the fact that almost 90% of RCTs of interventions for prevention of sexual transmission of HIV have delivered flat results demands careful analysis. This review revealed that the majority of flat RCTs are attributable, at least in part, to issues related to trial design and/or implementation. These issues must be addressed in future intervention research.
Compared to what?
A key issue in evaluating RCTs is the nature of the control group, and in particular, the intensity of prevention services offered to both study arms [73–76]. Enhanced or exceptional HIV prevention programs in the control arm that exceed the community standard of care (and may even include a diluted version of the primary intervention) are rarely sustainable after trial completion, and their intensity may dramatically reduce the ability to detect the effect of a new and effective intervention. Furthermore, in these cases, the results have little external validity because the comparison does not represent the effect of the intervention compared to the condition actually experienced by individuals in the community who could not avail themselves of the services offered to the control group.
Trials in which investigators or institutional review boards often feel obligated to provide controls with enhancements that are dilute versions of the intervention being tested (such as RCTs of behavioral or STI treatment interventions) are particularly vulnerable to flat results. In contrast, RCTs of new biomedical interventions such as vaccines, microbicides, or male circumcision rarely offer dilute forms of the intervention in the comparison group. This makes distinguishing an ineffective intervention from design issues more straightforward. Indeed, in all the trials that had a significant effect (positive or adverse), the control group did not receive a prevention package that resembled the main intervention.
The ethical issues of offering enhanced HIV prevention services in the comparison arm must be weighed against the ethical issues of lengthy and expensive prevention trials that provide the control group with an unsustainable level of prevention services that does not reflect community standards. Further, such trials may jeopardize our ability to identify and offer participants and at-risk individuals around the world additional effective HIV prevention options. Stepped-wedge designs are one approach that, when appropriate, may help address this ethical dilemma . Similarly, a reliable incidence assay might obviate the need for prospective follow-up of a control group, and thus render decisions about what to provide them over the course of the study irrelevant.
We also examined changes in risk behavior over the course of the trial in the intervention and control arms and found that in most cases, risk-taking behavior was reduced in both. Some of this change may be attributable to enhanced prevention services offered in the trial. A ‘Hawthorne effect’, in which simply participating in the study and being followed produces a positive result, might also have contributed. For example, in a prospective cohort study of FSWs in Kenya, HIV-1 incidence declined 10-fold during 3 years of follow-up . This phenomenon further attenuates the ability to detect the marginal benefit of the new intervention, especially if it is postulated to have only a modest effect.
Power to detect an effect: incidence and epidemic phase
Much has been written about unexpectedly low incidence in trials [24,79,80], which results in low statistical power unless additional endpoints are obtained by extending trial duration and incurring increased costs. We observed that this was the case in 14 (64%) of the 22 flat trials reporting this information. It was also recently announced that the PrEP study in Botswana would be unable to evaluate efficacy given a lower than expected rate of new infections (note this study is in progress and so was not included in this review) . Lower than expected incidence could result from erroneous assumptions about incidence prior to beginning the study. Development of improved assays to detect incident HIV infection might improve estimates of incidence before the onset of a trial and reduce trial costs by providing a more accurate basis for sample size calculations and assisting in the more reliable selection of populations with sufficiently high HIV incidence to permit trials of shorter duration . Alternatively, unexpectedly low incidence could result from changes in risk behaviors during the trial (potentially due to prevention services offered or to a Hawthorne effect). In addition, unexpectedly low incidence could reflect changes in the nature of the local epidemic, driven either by unanticipated implementation of new interventions or by changes in the epidemic phase in which the RCT was conducted – due, in part, to the protracted time for RCT design, implementation and completion, which can take as long as a decade . In such trials, it is important for investigators to document and iteratively reassess their original assumptions.
Effectiveness versus efficacy: does adherence matter?
Another key issue is the importance of adherence in trials [8,54]. If trials assess effectiveness as opposed to efficacy, one might argue that adherence is an inherent part of the intervention and that if it is suboptimal during the trial, it does not bode well for general uptake outside the course of the trial wherein adherence is likely to be even worse than in the context of a study. Nevertheless, if the methods require high adherence, as is likely to be the case with microbicides, PrEP, and treatment regimens for genital herpes, critical priorities are to better define, measure, analyze, and routinely report adherence, as well as to identify concomitant behavioral interventions to increase adherence (as we currently do for male condoms) before products are incorporated into HIV prevention packages.
The way forward
It is clear from results of recent trials that, in the near-term, there will be no single ‘magic bullet’ for HIV prevention. Instead, the emphasis in prevention research is shifting to evaluation of combination prevention packages in which synergies among interventions with modest levels of effect might lead to substantial efficacy overall [82,83]. What types of evidence for potential efficacy should be used to select interventions for inclusion in these packages? In addition to approaches such as male circumcision with robust RCT evidence for significant effects, we might consider interventions that demonstrate efficacy among specific, relevant subgroups (e.g., men who attended program meetings in the STI intervention in Zimbabwe ). Interventions with significant secondary outcomes in RCTs (e.g., other STDs [3,84]) might also be candidates for evaluation in combination packages if we assume that these outcomes lie on the causal pathway to HIV infection .
Should we also consider interventions for which evidence from RCTs is mixed or absent, but for which observational data or modeling strongly suggest that both the underlying concept and the specific intervention are likely to deliver substantial protection? Most HIV prevention programs and policies are not currently based on RCT evidence. Public health agencies such as the Centers for Disease Control and Prevention (CDC), the Agency for Healthcare Research and Quality, the World Health Organization, and UNAIDS have long recognized the dilemma posed by relying solely on RCT data. Therefore, many develop guidelines [as established, for example, by the US Preventive Services Task Force (USPSTF)] that are based not only on RCTs, but also on observational data from well designed nonrandomized trials; cohort, case–control, or multiple time series studies; mathematical modeling; dramatic results from uncontrolled experiments or even expert opinion by explicitly using a systematic approach to rating the evidence from these different methods . Similarly, epidemiologists and public health practitioners rarely rely only on the evidence from controlled trials to infer causality between an exposure and an outcome. For example, when reviewing evidence from multiple sources to establish whether smoking was associated with lung cancer, Sir Bradford Hill  suggested that experimental evidence was only one of several considerations for causal inference. Other criteria included parameters that characterized the nature of the association between the independent and dependent variable, including the magnitude of the effect, consistency, temporality, specificity, and the biological plausibility of the association.
That said, a critical issue is that most RCTs reviewed here are highly managed, labor-intensive, and expensive, meeting the standards of regulatory bodies such as the US Food and Drug Administration (FDA). As alluded to above, before abandoning randomization, it is important to consider the entire universe of RCTs, including stepped-wedge designs, large simple trials, or trials in which intervention communities are compared to either what currently exists or wherein incidence comparisons are made with nonparticipants as determined by reliable assays (should they be developed). The need for a counterfactual to infer causality cannot be denied and may be especially important when such combination programs are rolled out and evaluated at-scale, particularly if they are designed based on uncertain efficacy.
RCTs will undoubtedly remain our gold standard in defining the evidence base for prevention programs and policies. However, to assess the purity of this gold standard, the HIV prevention science community must not only examine evidence from RCTs with significant outcomes (including from subgroups and secondary outcomes), but must also examine flat trials and address the design and implementation issues discussed above. In addition, we must acknowledge and explicitly define the role of other types of evidence in the development of HIV prevention recommendations.
Pure gold is a thing of great beauty and value, but lacks the strength and affordability that make alloys like steel so useful and durable. Similarly, well designed and executed RCTs are magnificent and invaluable cornerstones of HIV prevention policies and programs. However, before abandoning entire classes of potentially beneficial interventions, we must forge ‘alloys’ of data from RCTs, observational studies, and other lines of evidence, cautiously and explicitly titrating the use of less rigorous sources, and recognize that these ‘alloys’ are likely to offer the best guide to decide what to include in prevention packages, what to scale up, and wherein further research is warranted.
We begin by acknowledging the numerous investigators responsible for these studies. Each one we were able to contact, willingly shared data and opinions that contributed greatly.
N.S.P. was supported by a grant from the Bill and Melinda Gates Foundation (No. 21082) and J.N.W. was supported by a grant from the National Institute of Allergy and Infectious Diseases (RO1 AI083034).
J.N.W. delivered the initial results at the 2007 Conference on Retroviruses and Opportunistic Infections reviewing the results of several STI treatment trials to reduce the incidence of HIV. N.S.P. elaborated on this concept and reviewed the larger array of biomedical trials using this paradigm and introduced the importance of ‘flat trials’ in 2007 in a plenary for the International AIDS Society. J.N.W. presented a preliminary version of this paper in 2009 (co-authored by N.S.P. and S.I.M. and J.E.B.) at the International Society for STD Research.
1. Wasserheit JN. The STD-HIV connection from research to action: are we lost in translation?
In: 14th Conference on Retroviruses and Opportunistic Infections
. Los Angeles, California; 2007.
2. Padian NS. Biomedical HIV prevention: current status, challenges, and future directions. 4th Annual International AIDS Society
. Sydney; 2007.
3. Wasserheit JN. STI/HIV prevention research: the agony, the ecstasy, & the future. 18th International Society for Sexually Transmitted Disease Research
. London; 2009.
5. UNAIDS. Report on the Global AIDS Epidemic
. Geneva; 2008.
6. Institute of Medicine (IOM). Methodological Challenges in Biomedical HIV Prevention Trials
. Washington, D.C.: The National Academies Press; 2008.
7. Padian NS, Buve A, Balkus J, Serwadda D, Cates W Jr. Biomedical interventions to prevent HIV infection: evidence, challenges, and way forward. Lancet 2008; 372:585–599.
8. Weiss HA, Wasserheit JN, Barnabas RV, Hayes RJ, Abu-Raddad LJ. Persisting with prevention: the importance of adherence for HIV prevention. Emerg Themes Epidemiol 2008; 5:8.
9. Potts M, Halperin DT, Kirby D, Swidler A, Marseille E, Klausner JD, et al
. Public health. Reassessing HIV prevention. Science 2008; 320:749–750.
10. Grant RM, Hamer D, Hope T, Johnston R, Lange J, Lederman MM, et al
. Whither or wither microbicides? Science 2008; 321:532–534.
11. Lagakos SW, Gable AR. Challenges to HIV prevention: seeking effective measures in the absence of a vaccine. N Engl J Med 2008; 358:1543–1545.
12. Grosskurth H, Gray R, Hayes R, Mabey D, Wawer M. Control of sexually transmitted diseases for HIV-1 prevention: understanding the implications of the Mwanza and Rakai trials. Lancet 2000; 355:1981–1987.
13. The Cochrane Collaboration. Cochrane Handbook for systematic reviews of interventions version 5.0.1
[updated September 2008]. In: Higgins JPT, Green S, editors; 2008.
14. Kamb ML, Fishbein M, Douglas JM Jr, Rhodes F, Rogers J, Bolan G, et al
. Efficacy of risk-reduction counseling to prevent human immunodeficiency virus and sexually transmitted diseases: a randomized controlled trial. Project RESPECT Study Group. JAMA 1998; 280:1161–1167.
15. Hobfoll SE, Jackson AP, Lavin J, Johnson RJ, Schroder KE. Effects and generalizability of communally oriented HIV-AIDS prevention versus general health promotion groups for single, inner-city women in urban clinics. J Consult Clin Psychol 2002; 70:950–960.
16. Branson BM, Peterman TA, Cannon RO, Ransom R, Zaidi AA. Group counseling to prevent sexually transmitted disease and HIV: a randomized controlled trial. Sex Transm Dis 1998; 25:553–560.
17. Baker SA, Beadnell B, Stoner S, Morrison DM, Gordon J, Collier C, et al
. Skills training versus health education to prevent STDs/HIV in heterosexual women: a randomized controlled trial utilizing biological outcomes. AIDS Educ Prev 2003; 15:1–14.
18. Pequagnat W; Group NCHSPT. Results of the RCT to test the community popular opinion leader (C-POL) intervention in five countries
[abstract no. LBPE1166]. AIDS 2008 - XVII international AIDS Conference.
19. Kamali A, Quigley M, Nakiyingi J, Kinsman J, Kengeya-Kayondo J, Gopal R, et al
. Syndromic management of sexually-transmitted infections and behaviour change interventions on transmission of HIV-1 in rural Uganda: a community randomised trial. Lancet 2003; 361:645–652.
20. Karim SA, Coletti A, Richardson B, Ramjee G, Hoffman I, Chirenje M, et al. Safety and effectiveness of vaginal microbicides BufferGel and 0.5% PRO 2000/5 gel for the prevention of HIV infection in women: results of the HPTN 035 trial. 16th Conference on Retroviruses and Opportunistic Infections
. Montreal, Canada; 2009.
21. Rothman KJ, Greenland S. Modern epidemiology. 2nd ed. Philadelphia: Lippincott Williams & Wilkins; 1998.
22. Cohen MS. Preventing sexual transmission of HIV. Clin Infect Dis 2007; 45(Suppl 4):S287–S292.
23. Vermund SH, Allen KL, Karim QA. HIV-prevention science at a crossroads: advances in reducing sexual risk. Curr Opin HIV AIDS 2009; 4:266–273.
24. Gray RH. Methodologies for evaluating HIV prevention intervention (populations and epidemiologic settings). Curr Opin HIV AIDS 2009; 4:274–278.
25. Medical Research Council. Microbicide gel safe but not effective in preventing HIV infection. Press release
. Durban; 14 December 2009.
26. Cowan FM, Pascoe SJS, Langhaug LF, Dirawo J, Mavhu W, Chidiya S, et al. The Regai Dzive Shiri project: results of a cluster randomized trial of a multi-component HIV prevention intervention for rural Zimbabwean adolescents. 18th International Society for Sexually Transmitted Disease Research
. London; 2009.
27. Gray GE, Allen MA, Bekker L, Churchyard G, Mlisana K, Nchabeleng M, et al. Results from the Phambili (HVTN 503) study: a multicenter phase IIB test-of-concept study of the MRKad5 HIV-1 Gag/Pol/Nef vaccine in South Africa. AIDS Vaccine
. Cape Town; 2008.
28. Celum C, Wald A, Lingappa J, Magaret A, Wang R, Mugu N, et al
. Acyclovir and transmission of HIV-1 from persons infected with HIV-1 and HSV-2. N Engl J Med 2010; 362:427–439.
29. Koblin B, Chesney M, Coates T. Effects of a behavioural intervention to reduce acquisition of HIV infection among men who have sex with men: the EXPLORE randomised controlled study. Lancet 2004; 364:41–50.
30. Celum C, Wald A, Hughes J, Sanchez J, Reid S, Delany-Moretlwe S, et al
. Effect of aciclovir on HIV-1 acquisition in herpes simplex virus 2 seropositive women and men who have sex with men: a randomised, double-blind, placebo-controlled trial. Lancet 2008; 371:2109–2119.
31. Flynn NM, Forthal DN, Harro CD, Judson FN, Mayer KH, Para MF. Placebo-controlled phase 3 trial of a recombinant glycoprotein 120 vaccine to prevent HIV-1 infection. J Infect Dis 2005; 191:654–665.
32. Buchbinder SP, Mehrotra DV, Duerr A, Fitzgerald DW, Mogg R, Li D, et al
. Efficacy assessment of a cell-mediated immunity HIV-1 vaccine (the Step study): a double-blind, randomised, placebo-controlled, test-of-concept trial. Lancet 2008; 372:1881–1893.
33. Van Damme L, Govinden R, Mirembe FM, Guedou F, Solomon S, Becker ML, et al
. Lack of effectiveness of cellulose sulfate gel for the prevention of vaginal HIV transmission. N Engl J Med 2008; 359:463–472.
34. Van Damme L, Ramjee G, Alary M, Vuylsteke B, Chandeying V, Rees H, et al
. Effectiveness of COL-1492, a nonoxynol-9 vaginal gel, on HIV-1 transmission in female sex workers: a randomised controlled trial. Lancet 2002; 360:971–977.
36. Ross DA, Changalucha J, Obasi AI, Todd J, Plummer ML, Cleophas-Mazige B, et al
. Biological and behavioural impact of an adolescent sexual health intervention in Tanzania: a community-randomized trial. AIDS 2007; 21:1943–1955.
37. Corbett EL, Makamure B, Cheung YB, Dauya E, Matambo R, Bandason T, et al
. HIV incidence during a cluster-randomized trial of two strategies providing voluntary counselling and testing at the workplace, Zimbabwe. AIDS 2007; 21:483–489.
38. Jewkes R, Nduna M, Levin J, Jama N, Dunkle K, Puren A, Duvvury N. Impact of stepping stones on incidence of HIV and HSV-2 and sexual behaviour in rural South Africa: cluster randomised controlled trial. BMJ 2008; 337:a506.
39. Pronyk PM, Hargreaves JR, Kim JC, Morison LA, Phetla G, Watts C, et al
. Effect of a structural intervention for the prevention of intimate-partner violence and HIV in rural South Africa: a cluster randomised trial. Lancet 2006; 368:1973–1983.
40. Grosskurth H, Mosha F, Todd J, Mwijarubi E, Klokke A, Senkoro K, et al
. Impact of improved treatment of sexually transmitted diseases on HIV infection in rural Tanzania: randomised controlled trial. Lancet 1995; 346:530–536.
41. Wawer MJ, Sewankambo NK, Serwadda D, Quinn TC, Paxton LA, Kiwanuka N, et al
. Control of sexually transmitted diseases for AIDS prevention in Uganda: a randomised community trial. Rakai Project Study Group. Lancet 1999; 353:525–535.
42. Gregson S, Adamson S, Papaya S, Mundondo J, Nyamukapa CA, Mason PR, et al
. Impact and process evaluation of integrated community and clinic-based HIV-1 control: a cluster-randomised trial in eastern Zimbabwe. PLoS Med 2007; 4:e102.
43. Wawer MJ, Makumbi F, Kigozi G, Serwadda D, Watya S, Nalugoda F, et al
. Circumcision in HIV-infected men and its effect on HIV transmission to female partners in Rakai, Uganda: a randomised controlled trial. Lancet 2009; 374:229–237.
44. Kreiss J, Ngugi E, Holmes K, Ndinya-Achola J, Waiyaki P, Roberts PL, et al
. Efficacy of nonoxynol 9 contraceptive sponge use in preventing heterosexual acquisition of HIV in Nairobi prostitutes. JAMA 1992; 268:477–482.
45. Roddy RE, Zekeng L, Ryan KA, Tamoufe U, Weir SS, Wong EL. A controlled trial of nonoxynol 9 film to reduce male-to-female transmission of sexually transmitted diseases. N Engl J Med 1998; 339:504–510.
46. Richardson BA, Lavreys L, Martin HL Jr, Stevens CE, Ngugi E, Mandaliya K, et al
. Evaluation of a low-dose nonoxynol-9 gel for the prevention of sexually transmitted diseases: a randomized clinical trial. Sex Transm Dis 2001; 28:394–400.
47. Peterson L, Nanda K, Opoku BK, Ampofo WK, Owusu-Amoako M, Boakye AY, et al
. SAVVY (C31G) gel for prevention of HIV infection in women: a phase 3, double-blind, randomized, placebo-controlled trial in Ghana. PLoS One 2007; 2:e1312.
48. Halpern V, Ogunsola F, Obunge O, Wang CH, Onyejepu N, Oduyebo O, et al
. Effectiveness of cellulose sulfate vaginal gel for the prevention of HIV infection: results of a phase III trial in Nigeria. PLoS One 2008; 3:e3784.
49. Feldblum PJ, Adeiga A, Bakare R, Wevill S, Lendvay A, Obadaki F, et al
. SAVVY vaginal gel (C31G) for prevention of HIV infection: a randomized controlled trial in Nigeria. PLoS One 2008; 3:e1474.
50. Peterson L, Taylor D, Roddy R, Belai G, Phillips P, Nanda K, et al
. Tenofovir disoproxil fumarate for prevention of HIV infection in women: a phase 2, double-blind, randomized, placebo-controlled trial. PLoS Clin Trials 2007; 2:e27.
51. Ghys PD, Diallo MO, Ettiegne-Traore V, Satten GA, Anoma CK, Maurice C, et al
. Effect of interventions to control sexually transmitted disease on the incidence of HIV infection in female sex workers. AIDS 2001; 15:1421–1431.
52. Kaul R, Kimani J, Nagelkerke NJ, Fonck K, Ngugi EN, Keli F, et al
. Monthly antibiotic chemoprophylaxis and incidence of sexually transmitted infections and HIV-1 infection in Kenyan sex workers: a randomized controlled trial. JAMA 2004; 291:2555–2562.
53. Watson-Jones D, Weiss HA, Rusizoka M, Changalucha J, Baisley K, Mugeye K, et al
. Effect of herpes simplex suppression on incidence of HIV among women in Tanzania. N Engl J Med 2008; 358:1560–1571.
54. Padian NS, van der Straten A, Ramjee G, Chipato T, de Bruyn G, Blanchard K, et al
. Diaphragm and lubricant gel for prevention of HIV acquisition in southern African women: a randomised controlled trial. Lancet 2007; 370:251–261.
55. Skoler-Karpoff S, Ramjee G, Ahmed K, Altini L, Plagianos MG, Friedland B, et al
. Efficacy of Carraguard for prevention of HIV infection in women in South Africa: a randomised, double-blind, placebo-controlled trial. Lancet 2008; 372:1977–1987.
56. Gray RH, Kigozi G, Serwadda D, Makumbi F, Watya S, Nalugoda F, et al
. Male circumcision for HIV prevention in men in Rakai, Uganda: a randomised trial. Lancet 2007; 369:657–666.
57. Bailey RC, Moses S, Parker CB, Agot K, Maclean I, Krieger JN, et al
. Male circumcision for HIV prevention in young men in Kisumu, Kenya: a randomised controlled trial. Lancet 2007; 369:643–656.
58. Auvert B, Taljaard D, Lagarde E, Sobngwi-Tambekou J, Sitta R, Puren A. Randomized, controlled intervention trial of male circumcision for reduction of HIV infection risk: the ANRS 1265 trial. PLoS Med 2005; 2:e298.
59. Patterson TL, Mausbach B, Lozada R, Staines-Orozco H, Semple SJ, Fraga-Vallejo M, et al
. Efficacy of a brief behavioral intervention to promote condom use among female sex workers in Tijuana and Ciudad Juarez, Mexico. Am J Public Health 2008; 98:2051–2057.
60. Gray G, Allen M, Churchyard G, Bekker L, Nchabeleng M, Mlisana K, et al. A multicenter double-blind randomized placebo-controlled phase IIB test-of-concept study to evaluate the safety and efficacy of a 3-dose regimen of the Clade B-based Merck Adenovirus serotype 5 HIV-1 gag/pol/nef vaccine in HIV-1-uninfected adults in South Africa. AIDS Vaccine
. Paris; 2009.
61. Corey L, McElrath MJ, Kublin JG. Poststep modifications for research on HIV vaccines. AIDS 2009; 23:3–8.
62. Long-term evaluation of the MEMA kwa Vijana Adolescent Sexual Health Programme in Rural Mwanza, Tanzania: a Randomised controlled trial. Technical Briefing Paper: No. 7
63. Cowan FM, Pascoe SJ, Langhaug LF, Dirawo J, Chidiya S, Jaffar S, et al
. The Regai Dzive Shiri project: a cluster randomised controlled trial to determine the effectiveness of a multicomponent community-based HIV prevention intervention for rural youth in Zimbabwe – study design and baseline results. Trop Med Int Health 2008; 13:1235–1244.
64. Patterson TL, Orozovich P, Semple SJ, Orozco HSS, Fraga M, Amaro H, et al
. A sexual risk reduction intervention for female sex workers in Mexico: design and baseline characteristics. J HIV/AIDS Soc Services 2006; 5:115–137.
65. U.S. Military HIV Research Program. RV144 phase III HIV vaccine trial press release
. Rockville, MD; 2009.
66. Hargreaves JR, Bonell CP, Morison LA, Kim JC, Phetla G, Porter JD, et al
. Explaining continued high HIV prevalence in South Africa: socioeconomic factors, HIV incidence and sexual behaviour change among a rural cohort, 2001–2004. AIDS 2007; 21(Suppl 7):S39–S48.
67. Quigley MA, Kamali A, Kinsman J, Kamulegeya I, Nakiyingi Miiro J, Kiwuwa S, et al
. The impact of attending a behavioural intervention on HIV incidence in Masaka, Uganda. AIDS (Lond) 2004; 18:2055–2063.
68. Delany S, Mlaba N, Clayton T, Akpomiemie G, Capovilla A, Legoff J, et al
. Impact of aciclovir on genital and plasma HIV-1 RNA in HSV-2/HIV-1 co-infected women: a randomized placebo-controlled trial in South Africa. AIDS 2009; 23:461–469.
70. Buchbinder S. Clinical outcomes from the STEP study. AIDS Vaccine
. Paris 2009.
71. UNAIDS. Evaluation of the 100% Condom Programme in Thailand. UNAIDS case study
. Geneva; 2000.
72. Green EC, Halperin DT, Nantulya V, Hogle JA. Uganda's HIV prevention success: the role of sexual behavior change and the national response. AIDS Behav 2006; 10:335–346, discussion 347–350.
73. Potts M. Thinking about vaginal microbicide testing. Am J Public Health 2000; 90:188–190.
74. van de Wijgert J, Elias C, Ellertson C, McGrory E, Blanchard K, Friedland B, et al
. Condom promotion in microbicide trials. Am J Public Health 2000; 90:1153–1154, author reply 1156.
75. Lurie P, Wolfe SM. Ethics require the inclusion of condoms and counseling in anti-HIV microbicide trials. Am J Public Health 2000; 90:1154–1155, author reply 1156.
76. de Zoysa I, Elias CJ, Bentley ME. Microbicide research and ‘the investigator's dilemma’. Am J Public Health 2000; 90:1155, author reply 1156.
77. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol 2006; 6:54.
78. Baeten JM, Richardson BA, Martin HL Jr, Nyange PM, Lavreys L, Ngugi EN, et al
. Trends in HIV-1 incidence in a cohort of prostitutes in Kenya: implications for HIV-1 vaccine efficacy trials. J Acquir Immune Defic Syndr 2000; 24:458–464.
79. Halpern V, Obunge O, Ogunsola F, Otusanya S, Umo-Otong J, Wang CH, Mehta N. Interim data monitoring to enroll higher-risk participants in HIV prevention trials. BMC Med Res Methodol 2009; 9:44.
80. Kaul R, Kimani J, Nagelkerke NJ, Fonck K, Keli F, MacDonald KS, et al
. Reduced HIV risk-taking and low HIV incidence after enrollment and risk-reduction counseling in a sexually transmitted disease prevention trial in Nairobi, Kenya. J Acquir Immune Defic Syndr 2002; 30:69–72.
81. Centers for Disease Control and Prevention. Status of the Botswana TDF2 Study of Pre-Exposure Prophylaxis for HIV Prevention. CDC Fact Sheet
. Atlanta, Georgia; December 2009.
82. Merson M, Padian N, Coates TJ, Gupta GR, Bertozzi SM, Piot P, et al
. Combination HIV prevention. Lancet 2008; 372:1805–1806.
83. Piot P, Bartos M, Larson H, Zewdie D, Mane P. Coming to terms with complexity: a call to action for HIV prevention. Lancet 2008; 372:845–859.
84. Manhart LE, Holmes KK. Randomized controlled trials of individual-level, population-level, and multilevel interventions for preventing sexually transmitted infections: what has worked? J Infect Dis 2005; 191(Suppl 1):S7–24.
85. U.S. Preventive Services Task Force. Guide to Clinical Preventive Services
. 2nd ed. Baltimore: Williams & Wilkins; 1996.
86. Hill AB. The environment and disease: association or causation? Proc Royal Soc Med 1965; 58:295–300.
adherence; behavior circumcision; control group HIV infection; microbicide; PrEP prevention; sexually transmitted infections statistical power; vaccines
© 2010 Lippincott Williams & Wilkins, Inc.
What does "Remember me" mean?
By checking this box, you'll stay logged in until you logout. You'll get easier access to your articles, collections,
media, and all your other content, even if you close your browser or shut down your
To protect your most sensitive data and activities (like changing your password),
we'll ask you to re-enter your password when you access these services.
What if I'm on a computer that I share with others?
If you're using a public computer or you share this computer with others, we recommend
that you uncheck the "Remember me" box.
Highlight selected keywords in the article text.
Data is temporarily unavailable. Please try again soon.
Readers Of this Article Also Read