In HIV vaccine trials, participants potentially can unblind themselves if they obtain a positive antibody test result (typically an enzyme immunoassay or EIA) followed by a Western blot test, so participants are asked not to have an HIV test outside of the study. Although it was not possible to determine if participants unblinded themselves in the first phase 3 HIV vaccine clinical trial (VaxGen VAX004, Brisbane, CA), our goal was to look more closely at the issue of participant HIV testing outside of an HIV vaccine trial to be able to make suggestions for reducing outside testing in future trials. The objectives of this reanalysis of the VaxGen VAX004 data were to determine (1) the proportion of participants who were tested for HIV outside of the study (despite instructions not to do this) and reasons why, (2) demographic and risk factors associated with reported testing outside of the study, and (3) if outside testing was related to participant loss to follow up.
VAX004 (VaxGen) was a randomized, double-blind, placebo-controlled efficacy trial of an HIV vaccine (Bivalent rgp 120 HIV-1 subtype B) conducted at a total of 61 sites (United States: n = 57, Canada: n = 3; The Netherlands: n = 1). Two-thirds of participants were randomly assigned to the vaccine arm and 1 of 3 to the placebo arm of the study. Seven doses of the vaccine or placebo were administered over the 36-month trial (baseline, 1, 6, 12, 18, 24, and 30 months visits). HIV counseling, testing, and education about the trial were provided at all trial visits. Study education emphasized that the vaccine being tested may not be effective and the participant may have received a placebo. Participants were told that if they wanted to be tested for HIV in between the scheduled HIV testing every 6 months, they should request testing from the study sites and not to be tested outside of the study. It was explained that if tested outside the study, they could test positive by standard HIV screening tests and that the result could be a false-positive due to the vaccine or it could be a true HIV infection. Eligible participants included HIV-uninfected men who have sex with men (MSM) and who engaged in anal sex with a male sex partner during the previous 12 months and women at risk for HIV infection. Because of the small percent of women in the study (5.7%), only men were included in this analysis. For more information about the trial, please refer to Harro et al.1
The dependent variable for analyses in this study was a participant's response to the question “During the last year, have you been tested for HIV outside of this study?” asked at the 12-, 24-, and 36-month visits. Information on the type of tests received or whether the participant was unblinded to their study arm was not collected. Independent variables included demographic characteristics and attitudes, beliefs, risk behaviors, having 1 or more sexually transmitted infections (STIs), lost to follow up, and lost for other reasons (eg, HIV infection). Note that all variables, except lost to follow up and lost for other reasons, were collected at each 12-month visit. HIV risk variables were also collected at the 6-month visits.
For analysis purposes and to include all reported data, we combined the biannual with the annual responses. Specifically, for questions asked every 6 months, answers to the question at the 12-, 24-, or 36-month visits were combined with the answer for the previous visit (all variables were either binary or made binary). An aggregate variable was coded as 1 (yes) if any of the component variables indicated the behavior/condition of interest occurred.
Analyses were restricted to MSM who completed a survey at 1 or more of the 12-, 24-, or 36-month visits. For any visit where a participant was missing the outside testing response, data from that visit were removed from the analyses. All statistical tests were 2-sided, and all confidence intervals were based on a 5% level of significance. Analyses were performed in SAS version 220.127.116.11
We used factor analysis to determine if any independent variables possessed high intercorrelation and loaded highly to a latent factor. A factor analysis allowed us to clearly and efficiently evaluate the correlation between groups of predictors instead of parsing through a large matrix of correlation coefficients. We used a loading value threshold of 0.7 or greater to denote high intercorrelation. Amphetamine and hallucinogen use was the only group of predictors highly correlated and a binary composite variable of the 2 was created.
A generalized linear mixture model3 was used to evaluate associations with rates of testing outside of the trial. The choice of model was determined by the longitudinal structure of the study, binary outcome, hypothesis to study lost to follow up participants, and potential for bias associated with dropout. The generalized linear mixture model minimizes this bias by incorporating a binary covariate denoting whether a person dropped out of the study and interactions between this binary covariate and other predictors in the model. Therefore, a binary predictor for those participants lost to follow up was included in the model. In addition, we believed that the participants who did not complete the study for reasons other than lost to follow up (eg, the subject became HIV-1 infected or noncompliant) could bias the results of the study. Therefore, another binary predictor for those participants who dropped out for other reasons than lost to follow up was included (henceforth referred to as “lost for other reasons”).
Any predictor interaction with a P value less than 0.25 in univariable models was included in the multivariable model, in accordance with variable selection research.4 In addition, a recommendation is made to include interactions with the dropout indicators to remove any potential bias associated with attrition.3 Therefore, an interaction with a dropout indicator possessing a P value less than 0.25 was included in the final model.
A total of 5095 MSM enrolled in the study from June 1998 to October 1999. Of these, 4774 MSM attended at least 1 of the 12-, 24-, or 36-month visits (13 men who were indicated by nucleic acid testing to be HIV infected at baseline were excluded from the analysis) for a total of 13,116 visits. For 104 of the participants, no response was noted for the outside testing question at 290 (2.2%) of those visits, reducing the total number of visits to 12,826 and the total number of participants in this analysis to 4670. There were 4618 men at the 12-month visit, 4259 at the 24-month, and 3949 at the 36-month. A total of 330 of 4670 (7.1%) participants were lost to follow up and 721 of 4670 (15.4%) missed the last visit.
Testing Outside the Study
Despite counseling instructions not to, 16.9% (791 of 4670) of male participants reported being tested for HIV outside of the study. At the 12-, 24-, and 36-month visits, the total numbers of outside tests reported were 367 (mean = 1.2, SD = 0.61), 331 (mean = 1.2, SD = 0.61), and 316 (mean = 1.3, SD = 0.94), respectively. The top 2 reasons given for having an HIV test outside of the study were a medical provider request (28.1%) and a health insurance requirement (17.1%). More than 1 reason could be given for outside testing; the full list of reported reasons is presented in Table 1.
Demographic and Risk Factors Associated With Being Tested Outside of the Study
Univariable marginal regression models determined that the following variables met the criteria to be included in the multivariable model: race/ethnicity, site location, perceived study assignment, joined the trial to get some HIV protection, engaged in unprotected anal sex, used amphetamines or hallucinogens, had 1 or more STIs, and had ≥1 adverse events rated as “severe” or that lasted ≥3 days. Two additional variables, lost to follow up and lost for other reasons, were put into the model regardless of whether they met the criteria.
Generalized Linear Mixture Model
Table 2 shows the results of the mixture model. Model diagnostics suggested no evidence of a lack of fit5 (χ2 (9) = 12.25; P > 0.19). Increased odds of self-reported outside testing were associated with site location; those participants living in the southern, midwestern, and western United States compared with the northeastern United States; reporting 1 or more STI; joining the trial to get some HIV protection; and engaging in unprotected anal sex. Decreased odds of self-reported outside testing were associated with participants reporting they perceived they were assigned to the vaccine arm of the study or did not know their assignment compared with those who perceived they were in the placebo arm. There were also increased odds of self-reported outside testing associated with participants who were lost to follow up. Of the 791 MSM who ever had an outside test, 56 (7.1%) were lost to follow up. Of the 56, 13 (23.2%) were assigned to the placebo arm.
Of the interactions included in the model to control for dropout bias, only the interaction between lost to follow up and site location was associated with whether the participant obtained an HIV test outside the study (χ2 (4) = 13.25, P < 0.01), indicating the potential for bias in the lost to follow up or site location variable. Compared with international study sites, lost to follow up participants in all regions of the United States [midwestern (adjusted odds ratio, aOR = 0.24, confidence interval, CI: 0.09 to 0.66), southern (aOR = 0.18, CI: 0.07 to 0.46), western (aOR = 0.25, CI: 0.09 to 0.70), and northeastern United States (aOR = 0.19, CI: 0.05 to 0.71)] had lower odds of outside testing.
Nearly 17% of MSM in a randomized, double-blind, placebo-controlled HIV vaccine efficacy trial reported HIV testing outside of the study contrary to study instructions and the availability of nonscheduled testing provided by the trial sites. There were numerous reasons given for the outside testing; the most common being that a medical provider wanted the participant to have an HIV test. Self-reported outside testing was associated with several independent factors including joining the trial to get some protection against HIV infection.
The nearly 17% of MSM reporting outside testing in our study is between the 13% (n = 1267) of MSM in a vaccine trial preparedness study who said that they would be motivated to unblind6 and the 22% (n = 37) of participants in a study of 2 HIV vaccine trials (phase I and phase I/II) who reported that they were tested for HIV outside the study.7 In the present study, most of the outside tests resulted from reported reasons that were viewed as a requirement or request and interpreted as authoritative or compelling (eg, medical provider request). In the above study, reasons for reported outside testing included new relationships, travel outside the United States, a routine check on status, military testing, and immigration.7 These reasons and those reported in our study raise the possibility that often it may be difficult for participants to refuse an HIV test and hard to negotiate returning to the study site to get tested. On the other hand, some participants may also have wanted to know to what treatment arm they were assigned or confirm their guess; thus, if an opportunity presented itself, they may have agreed to the HIV test outside of the study.
Our study documents that several independent factors were associated with MSM vaccine trial participants reporting being tested for HIV outside of the study. For instance, joining the trial to get some HIV protection and perceived assignment to the placebo arm were variables both associated with reported outside testing. It may be that men who were motivated to join the trial to benefit from vaccine protection had a strong interest in determining their study arm assignment. Similarly, participants who perceived that they were assigned to the placebo arm may have sought to validate their perception. Participants in our study who engaged in the high-risk behavior of unprotected anal sex may have been more inclined to have an outside test, even if they did not intentionally seek it out, to determine if they tested positive. The reason participants who had reported having 1 or more STI in the last 6 months were more likely to report being tested outside of the study may be because guidelines for HIV testing recommended counseling, testing, and referral at several sites including sexually transmitted disease clinics.8
Outside testing can bias clinical trial results when participants learn that they are in the vaccine arm and increase their risk behavior (risk compensation),7,9 or when participants learn that they are in the placebo arm and drop out of the study becoming lost to follow up.10,11 We found that participants who were lost to follow up were more likely to have obtained or received a test outside of the trial than those who remained in the study. This finding suggests that some participants may drop out of the study after receiving an outside test where they believe that they determined their study arm assignment. It is of note, however, that most of the participants who ever had an outside test and were lost to follow up were assigned to the vaccine arm. This finding is contrary to the notion that participants who believe that they are assigned to the placebo arm drop out of the study and deserves further study.
Our study results must be interpreted in the context of several potential limitations. First and most importantly, given that participants were instructed at enrollment to not get tested outside of the study, then were asked periodically if they did receive outside testing, responses were likely influenced by social desirability bias, such that the real percentage of outside testing is likely higher than reported. Despite this limitation, the analysis was able to identify factors associated with participants who reported outside testing. Second, the study could not identify the cases in which testing was intentional or the cases where an attempt was made to decline a test that was required or offered. Third, although most data were collected every 6 months, the outside testing question was only asked every 12 months requiring some of the independent variable responses to be combined, thus decreasing the sensitivity of the analysis. Fourth, the analysis did not provide a direction of the association between outside testing and the independent variables. Moreover, the direction of association may be different for the different significant independent variables. For instance, joining the trial to get some HIV protection may have led to outside testing. On the other hand, outside testing may have led to loss to follow up. Fifth, the number of participants lost to follow up was small (about 7%), so the significant interaction between lost to follow up and site location should be investigated in future trials. Sixth, the sample size of minority MSM was also small requiring collapsing of minority categories. Finally, no women were included in the analysis. The strength of the study is that it provides the first assessment of the issue of HIV testing outside of the study in a phase III vaccine trial. This information may help future vaccine trial sponsors design within-trial counseling/education modules to decrease the likelihood of outside testing.
The frequency of outside testing is likely to increase in future large HIV vaccine trials as a result of the revised Centers for Disease Control and Prevention guidelines on HIV testing. The guidelines recommend that patients in all health care settings should be routinely tested unless the patient opts out after being notified of the intention to test.12 Because of the importance of keeping participants blinded to their treatment assignment and to avoid biases associated with potential differential risk behavior change and participant attrition across study arms,9-11 the high percentage of participants in this trial who received outside testing underscores the need to develop methods to reduce the likelihood of outside testing and possibly increasing the accessibility of requested testing within the trial. It has been suggested that those who are motivated to join the trial because of HIV protection can be candidates for additional counseling regarding side effects.13 This may change subgroup behavior and potentially bias results, so a better option may be improved counseling on outside testing for all participants. In addition, as was done in this study, monitoring whether a participant believes he or she received the placebo or treatment drug is recommended so that this information can be used as a covariate in the efficacy analysis14 to account for variability. Apart from the trial, it has been recommended by the Centers for Disease Control and Prevention that medical professionals consider that positive HIV test results might not mean infection if the person has been identified as an HIV vaccine trial participant and that, if possible, the person should be sent back to their trial site for confirmatory testing.12 This, of course, will only be effective if the trial participants inform their health care providers (or the health care provider asks) about their participation in an HIV vaccine trial. It is likely that providers will encounter patients who were vaccine trial participants, given the estimated total number of HIV trial participants enrolled in US-based trials since 1987 is more than 14,000 (estimated data provided by National Institute of Allergy and Infectious Diseases, July 2008). In the future, the potential for unblinding will vary with evolving vaccine design. Nevertheless, because HIV nucleic acid tests are negative in HIV noninfected vaccine recipients, should nucleic acid testing become the HIV confirmatory test of choice, fewer false positives will result. In the meantime, to reduce the number of false-positive antibody tests in HIV vaccine recipients, a comprehensive communication plan targeted to health care providers and public health professionals; intensive counseling for HIV vaccine trial participants about the need and reasons to refrain from outside testing; and the support of the health care, public health, and nongovernmental organization communities will be necessary.
The authors thank Global Solutions for Infectious Diseases, the exclusive licensee of the VAX003, and VAX004 clinical data for its support and Dawn Smith for valuable comments on an earlier version of this article. We gratefully acknowledge the following individuals for their contributions to the VAX004 study: Vladimir Popovic, Marta Ackers, Eleanor McLellan, Marcus Durham, Bill Heyward, Marc Gurwith, Patti Orozco-Cronin, Participants, Tim Mastro, and Alan Greenberg.
1. Harro CD, Judson FN, Gorse GJ, et al. Recruitment and baseline epidemiologic profile of participants in the first phase 3 HIV vaccine efficacy trial. J Acquir Immune Defic Syndr
2. SAS version 9.1.3 [package insert]. Cary, NC: SAS Institute Inc; 2004.
3. Fitzmaurice GM, Laird NM. Generalized linear mixture models for handling nonignorable dropouts in longitudinal studies. Biostatistics
4. Mickey RM, Greenland S. The impact of confounder selection criteria on effect estimation. Am J Epidemiol
5. Horton NJ, Bebchuk JD, Jones CL, et al. Goodness-of-fit for GEE: an example with mental health service utilization. Stat Med
6. Bartholow BN, MacQueen KM, Douglas JM Jr, et al. Assessment of the changing willingness to participate in phase III HIV vaccine trials among men who have sex with men. J Acquir Immune Defic Syndr Hum Retrovirol
7. Chesney MA, Chambers DB, Kahn JO. Risk behavior for HIV infection in participants in preventive HIV vaccine trials: a cautionary note. J Acquir Immune Defic Syndr Hum Retrovirol
8. Divine B, Greby S, Hunt K, et al. Revised guidelines for HIV counseling, testing and referral. MMWR Morb Mortal Wkly Rep
9. Hedlund J. Risky business: safety regulations, risk compensation, and individual behavior. Inj Prev
10. Melton GB, Levine RJ, Koocher GP, et al. Community consultation in socially sensitive research. Lessons from clinical trials of treatments for AIDS. Am Psychol
11. Karlowski TR, Chalmers TC, Frenkel LD, et al. Ascorbic acid for the common cold. A prophylactic and therapeutic trial. JAMA
12. Branson B, Handsfield H, Janssen R, et al. Revised recommendations for HIV testing of adults, adolescents, and pregnant women in health-care settings. MMWR Morb Mortal Wkly Rep
13. Johnson MO. Belief of vaccine receipt in HIV vaccine trials: further cautions. J Acquir Immune Defic Syndr
14. Swartzman LC, Burkell J. Expectations and the placebo effect in clinical drug trials: why we should not turn a blind eye to unblinding, and other cautionary notes. Clin Pharmacol Ther