Katzen, Lauren L. MPA*; Fernández-Romero, José A. MS*; Sarna, Avina MD MPH, PhD†; Murugavel, Kailapuri G. MSc, PhD‡; Gawarecki, Daniel MA*; Zydowsky, Thomas M. PhD*; Mensch, Barbara S. PhD*
Microbicide clinical trials seek to test products designed to reduce or prevent transmission of HIV and/or other sexually transmitted infections.1 As administration of vaginal and/or rectal products is participant dependent, one of the major challenges confronting these trials is the measurement of product adherence.2–6 The reliability and validity of self-reports of product use are questionable, given the explicit understanding that participants are expected to comply with study protocols.7 Social desirability, fear of being excluded from the trial, and/or embarrassment about not using the product may result in participants' overreporting of product use. Accurate measurement of adherence is particularly important for trials that fail to demonstrate efficacy, as primary analyses on the intent-to-treat population may be less likely to demonstrate efficacy if participants fail to use the product.7–9
Concerned with the quality of self-reported data, and in advance of the Carraguard® Phase 3 trial, Population Council scientists sought to develop an assay to objectively measure vaginal insertion of applicators. Although tests for the growth of lactobacilli on inserted applicators were 100% predictive of vaginal insertion, this method was not considered desirable because it was time consuming, and the percentage of women with culturable lactobacilli is low.10 Council scientists therefore developed an assay in which the tips of applicators are sprayed with an aqueous blue dye solution that adheres to vaginal mucus present on applicators after insertion, producing a grainy, streaky turquoise pattern. This is distinct from the smooth and shiny pattern that is present on noninserted applicators when the dye adheres to residual gel only. The pattern is especially visible in and around the “seam” or grooves of the applicator tip10 (Fig. 1).
This dye stain assay (DSA) was validated twice using single dose Microlax®-type low-density polyethylene applicators (LDPE; Norden Pac, Kalmar, Sweden) filled with 2.5% methyl cellulose gel; positive controls were provided by female study participants who were instructed to both insert the applicator and expel the gel into their vaginas.10,11 The first validation study used a 1% (wt/vol) Trypan Blue dye solution.10 The second study used a 0.05% (wt/vol) FD&C Blue No. 1 Granular Food Dye solution because of concerns about safety of the staff performing the DSA, as Trypan Blue is an irritant and potential carcinogen.11,12 For both studies, applicator tips were sprayed with dye solution to thoroughly coat the applicator (3 sprays within approximately 5 seconds); sprayed applicators were rinsed after 10 seconds and left to dry overnight. Stained applicators were visually inspected to determine whether they had been vaginally inserted. The first study reported that the DSA had 97.5% sensitivity (the probability that an inserted applicator was read as positive); specificity (the probability that an applicator that had not been inserted was read as negative) was not reported.10,13 The second study reported 97.5% sensitivity and 96% specificity.11 Similar results were reported when applicators were stored to a maximum of 4 months and/or when inserted applicators were stored in the same plastic bag along with noninserted applicators.10,11
Based on these validation studies, the DSA was deemed to be an effective tool to validate self-reported vaginal applicator use for the Carraguard® Phase 3 trial.10,11,14 Participants were instructed to use the gel (Carraguard® gel, Clean Chemical Sweden AB, Borlänge, Sweden) before each act of sex, and the DSA was used to determine how product use corresponded to sex acts; the number of inserted applicators according to the DSA was compared with the number of sex acts reported by participants. According to self-reports, study product was used during 96% of sex acts, but DSA results indicated that participants used the study product during only 42% of sex acts. Further, only 61% of returned, opened applicators were identified as vaginally inserted.14 In a 3-month follow-up placebo study in South Africa assessing the effect of interview mode on adherence self-reports, the DSA indicated that 52% of opened applicators had been vaginally inserted, resulting in a mean difference ranging between 2.3 and 4.1 inserted applicators per month when comparing self-reports to the DSA.15 Depending on month of participation and interview mode, 31% to 60% of follow-up study participants reported ever having sex without inserting the gel.15 In both the Carraguard® Phase 3 trial and interview mode study, the number of sex acts reported was greater than the number of applicators that tested positive for insertion; participants seem to have had sex without inserting the gel beforehand.14,15
Council researchers contended that overreporting of gel use was because of social desirability bias; the presumption was that the participants squeezed the contents of unused applicators out ex vivo before returning them to the clinic so as to appear adherent.14,16 Other researchers in the field speculated that the DSA was inadequately validated,16,17 and that it would work only with the LDPE applicators with which it was originally validated.18 Consequently, 2 DSA studies were conducted by other researchers in the field.18,19 The first study tested 2 modified DSAs on single-use LDPE applicators filled with 0.5% PRO2000 or matched placebo gel.19 Applicators were either sprayed with a 0.4% (wt/vol) Trypan Blue solution or immersed in a 0.4% (wt/vol) Trypan Blue water bath for 1 minute; researchers waited 5 to 10 minutes before rinsing. The study concluded that the DSA was accurate and useful for future microbicide trials.19 The second study tested single-use high density polypropylene applicators that were either filled with UC-781 (used once daily) or VivaGel® (used twice daily).18 A 0.05% (wt/vol) FD&C Blue Dye No. 1 solution was used. It is unclear, however, how long the dye remained on the applicators between spraying and rinsing; in previous studies, dye remained on applicators for 10 seconds before rinsing so as to ensure that dye solution that adheres to vaginal mucus remains visible after washing.10,11,18 Variability among evaluators was reported, with the sensitivity of the assay ranging between 81% and 95% for single-use applicators, and between 44% and 71% for applicators used twice daily. Researchers speculated that the DSA performed with lower sensitivity with applicators used twice daily due to gel remaining in the vagina from the previous insertion.18 Results focused on the lower than optimal performance with applicators used twice daily, ignoring the high sensitivity of the DSA with applicators used once daily.18 A recent study of 1% tenofovir gel used the DSA with single-use polypropylene applicators; high interobserver variability and a relatively high false-positive rate led researchers to conclude that the DSA is not effective for determining polypropylene applicator insertion.20 Table 3 provides a summary of all DSA validation experiments published to date.
Results from the recent CAPRISA 004 trial of tenofovir gel, bolstered by those from the iPrEX PrEP trial of Truvada® (Emtricitabine combined with tenofovir), provided proof of concept that antiretroviral drug (ARV)-based products used before and after sex reduce HIV incidence.9,21 Adherence to ARV-based products is particularly crucial because inconsistent use may lead to drug resistance.22 Both studies reported an association between product adherence and HIV protection, as well as variability between self-reports and more objective measures, such as pill and applicator counts, and drug levels in blood.9 As it is difficult to detect drug levels in blood with topically applied products owing to low systemic absorption, biomarkers of gel adherence, such as cervicovaginal lavage and measurement of drug levels in tissue, are currently being investigated.23 In the meantime, researchers continue to rely on self-reports and counts to measure product use. Considerable effort has been devoted to improving the reporting of adherence through counseling and/or innovative reporting tools, such as audio computer-assisted self-interviews (ACASI), interactive voice response surveys (IVRS), and short message service (SMS).8,9,15,24,25
The DSA is currently being used by the Population Council researchers to validate the self-reported use of the “universal placebo” hydroxyethylcellulose (HEC) gel26,27 in a study among sex workers in India. While training of the site staff on the implementation of the DSA, the pattern characteristic of inserted applicators was not clearly visible on applicators known to have been inserted. HEC that remained inside the applicator after insertion had leaked out of the applicators during storage; other publications have suggested that gel may interfere with the DSA.18,19 Before implementing the DSA in India, and to respond to criticism of the assay's accuracy, Population Council researchers sought to systematically and comprehensively validate the DSA by the following: (i) optimizing the assay and related procedures and (ii) establishing predictive values for the DSAs ability to identify vaginally inserted applicators. This article presents the results of these experiments and is the first to report all 4 predictive values for the DSA: sensitivity, specificity, and positive and negative predictive values.13
MATERIALS AND METHODS
Study applicators consisted of single-use, twist-off top, LDPE Microlax®-type applicators (Tectubes, Åstorp, Sweden). Applicators were filled with 7 mL of 2.7% HEC placebo gel (Clean Chemical Sweden AB, Borlänge, Sweden) and delivered approximately 3.5 mL of gel.
Participant Inserted Applicators and Negative Controls
Female sex workers of reproductive age (18–45 years) were enrolled in a placebo microbicide study in Southern India. All participants gave informed consent in accordance with established guidelines and ethical standards for experimentation with human subjects. At enrollment, participants were trained to insert HEC-filled applicators and inserted their first applicator in the presence of the study clinician. Applicators inserted at the clinic by 252 participants served as positive controls for the experiments; each participant provided a single applicator. Inserted applicators were plugged with the stick-end of a cotton swab to prevent leakage and stored at ambient room temperature (not higher than 25°C) in separate, labeled plastic bags for a maximum of 5 months. Laboratory technicians created negative controls by squeezing HEC gel out of applicators, ex vivo. Negative controls were plugged and stored under the same conditions as positive controls, but were stored for only a few hours before testing. Masked test kits were prepared by randomly coding and labeling positive and negative controls, which were equally distributed in each kit.
Optimization and Validation Experiments
The DSA involves spraying applicator tips with an aqueous solution of FD&C Blue No. 1 Granular Food Dye. Before DSA validation, a 3 × 3 factorial optimization experiment was performed that involved 54 positive controls stored for no more than 1 month before testing and 54 negative controls, to ascertain optimal dye concentration and staining time (i.e., amount of time dye remains on the applicator before rinsing). Spray bottles were filled with 0.01%, 0.05%, and 0.25% (wt/vol) dye solutions. Applicators were sprayed 3 times within approximately 5 seconds; laboratory technicians waited 5 seconds, 5 minutes, or 10 minutes before rinsing. After applicators dried, readers visually examined them for the presence of the characteristic pattern indicating that applicators were “positive” (vaginally inserted) or the absence of this pattern indicating that applicators were “negative” (not vaginally inserted) (Fig. 1).
Only after the optimization experiment was complete, 3 validation experiments were conducted; each validation experiment involved 66 positive controls and 66 negative controls. Readers' observations were scored and analyzed to determine sensitivity, specificity, negative predictive values (NPVs), and positive predictive values (PPVs).
* Experiment 1 was designed to validate the DSA. Applicators were sprayed, rinsed, and dried by the same individual who performed these steps for the optimization experiment, and 5 readers' observations were scored and analyzed. Positive controls had been stored for 1 to 2 months.
* Experiment 2 was designed to validate the assay for use with “older” applicators (median storage time of 4 months). Further, to determine repeatability, applicators were sprayed, rinsed, and dried by the same individual who performed these steps for the optimization and first validation experiment. Four readers' observations were scored and analyzed.
* Experiment 3 was designed to determine reproducibility. Applicators were sprayed, rinsed, and dried by an individual who did not perform these steps in the previous experiments. Five readers' observations were scored and analyzed; positive controls had been stored for 2 to 4 months.
In all, 4 to 6 laboratory and clinical staff members involved in the India study served as readers. Readers were trained in how to identify positive applicators by a Population Council staff member, who also served as a reader.
Each reader independently examined and evaluated the applicators. Readers reviewed results from the optimization experiment to improve identification of inserted applicators for the subsequent validation experiments. Readers' observations and scores from the validation experiments were not disclosed until all experiments were completed. One individual, who did not serve as a reader, created the masked test kits, recorded, and scored all observations. Observations were scored as “True Positive,” “True Negative,” “False Positive,” or “False Negative.”
Sample Size Calculations and Statistical Analysis
Sensitivity and PPVs were tested for the probability that an inserted applicator would be read as positive. Specificity and NPVs were tested for the probability that an applicator that was not inserted would be read as negative.13
Sample size calculations performed with SAS 9.2's Proc Power were based on an exact test of binomial proportions. They indicated that the analysis of 66 positive controls and 66 negative controls would provide evidence sufficient to reject the DSA if the sensitivity (or specificity) was no more than 90%. Friedman tests were performed with SAS 9.2's Proc Freq to detect between experiment differences in sensitivity, specificity, NPV, and PPV.
Based on the optimization experiment, the dye concentration of 0.05% (wt/vol) and staining time of 5 seconds were determined to be optimal (100% sensitivity and 91.7% specificity, on average). This combination of dye concentration and staining time was used for the 3 validation experiments (Table 1).
There were a total of 1848 possible applicator readings across all 3 validation experiments; 1703 (92.2%) applicator readings were correct. On average, the DSA performed with 90.6% sensitivity, 93.9% specificity, and had an NPV of 91.0% and a PPV of 93.8% (Table 2).
To validate the DSA (Experiment 1), 5 readers evaluated 132 applicators, for a total of 660 possible readings. Of these, 607 (92.0%) applicator readings were correct. On average, the DSA performed with 92.4% sensitivity and 91.5% specificity, and with an NPV of 92.4% and a PPV of 91.7%.
To validate the DSA for use with “older” applicators (Experiment 2), 4 readers evaluated 132 applicators, for a total of 528 possible readings. Of these, 493 (93.4%) applicator readings were correct, and on average, the DSA performed with 91.7% sensitivity and 95.1% specificity, and had an NPV of 92.1% and a PPV of 95.0%. To determine repeatability, applicators were sprayed, rinsed, and dried by the same individual who performed these steps for the optimization and first validation experiments.
To validate the reproducibility of the DSA (Experiment 3), applicators were sprayed, rinsed, and dried by an individual who did not perform these steps in the previous experiments, and 5 readers evaluated 132 applicators, for a total of 660 possible readings. Of these, 603 (91.4%) applicator readings were correct and, on average, the DSA performed with 87.6% sensitivity and 95.2% specificity, and had an NPV of 88.5% and a PPV of 94.8%.
No statistically significant differences between the 3 validation experiments were noted.
The DSA was optimized and successfully validated for use with single-use, LDPE applicators filled with HEC placebo gel. The combination of dye concentration and staining time determined to be optimal was 0.05% (wt/vol) and 5 seconds. Across these experiments, averages of sensitivity, specificity, NPV, and PPV were at least 90%, and no significant differences were found between the results of the 3 experiments. As there were no significant differences between the results of Experiments 1 and 2, the DSA performed well with applicators that were stored for up to 4 months and is repeatable. Because there were no significant differences between the results of Experiments 2 and 3, the DSA can be replicated.
To compare the results of previous experiments with the results of our experiments, sensitivity, specificity, PPVs, and NPVs were derived from data presented in other DSA validation publications, when available (Table 3). Based on these data, the DSA is reliable when used with a number of different gel and applicator combinations: LDPE applicators filled with either MC or HEC, PRO2000 (or matched placebo) and high-density polypropylene applicators filled with UC-781, if used once daily.
All DSA experiments conducted to date have sought to identify applicators that were vaginally inserted; researchers have not yet conducted similar DSA experiments with gel products/applicators designed for rectal use. Although the DSA can be easily replicated, as the evaluation of sprayed applicators relies on visual inspection, hands-on training and practice are needed to reduce subjectivity identified by the high degree of observer variability. The DSA cannot be defined as a biomarker28; although the DSA cannot provide information about whether the product was actually expelled into the vagina, the timing of gel use, or the amount of product used, a biomarker might not be able to provide that information either.2 Future studies should seek to validate the DSA for use with applicators inserted after sex to determine whether the presence of semen interferes with the DSA.
Previous studies have highlighted the simplicity of the DSA as a surrogate marker to measure gel adherence and the relatively low cost of implementing it within a large-scale trial. Accordingly, even if the use of a true biomarker becomes feasible, the DSA's low cost and relative ease might make it a useful tool for the ongoing monitoring of trials.11,21 Recent publications suggest that the DSA might be used as part of a triangulated approach to measuring adherence, so as to capture accurate adherence information as well as reduce the error introduced by any one particular method.1,17 We recommend including the DSA in future microbicide trials that involve vaginal gels to identify participants who have low adherence to dosing regimens. In doing so, we can develop strategies to improve adherence1,9 as well as explore the association between adherence and efficacy, which is especially important in trials with inconclusive results.10
1. Lagakos SW, Gable AR, eds. Committee on Methodological Challenges in Biomedical HIV Prevention Trials. Methodological Challenges in Biomedical HIV Prevention Trials. Washington, DC: Institute of Medicine of the National Academies, 2008.
2. Tolley EE, Harrison PF, Goetghebeur E, et al. Adherence and its measurement in phase 2/3 microbicide trials. AIDS Behav 2010; 14:1124–1136.
3. Feldblum P, Adeiga A, Bakare R, et al. SAVVY vaginal gel (C31G) for prevention of HIV infection: A randomized controlled trial in Nigeria. Plos ONE 2008; 3:e1474.
4. Peterson L, Nanda K, Opoku BK, et al. SAVVY (C31G) gel for prevention of HIV infection in women: A Phase 3, double-blind, randomized, placebo-controlled trial in Ghana. PLoS One 2007; 2:e1312.
5. Stirratt MJ, Gordon CM. Adherence to biomedical HIV prevention methods: Considerations drawn from HIV treatment adherence research. Curr HIV/AIDS Rep 2008; 5:186–192.
6. Stone A, Jiang S. Microbicides: Stopping HIV at the gate. Lancet 2006; 368:431–433.
7. Turner AN, De Kock AE, Meehan-Ritter A, et al. Many vaginal microbicide trial participants acknowledged they had misreported sensitive sexual behavior in face-to-face interviews. J Clin Epidemiol 2009; 62:759–765.
8. Tucker J, Foushee H, Black B, et al. Agreement between prospective interactive voice response self-monitoring and structured retrospective reports of drinking and contextual variables during natural resoluation attempts. J Stud Alcohol 2007; 68:538–542.
9. Karim QA, Karim SSA, Frohlich JA, et al. Effectiveness and safety of tenofovir gel, an antiretroviral microbicide, for the prevention of HIV infection in women. Science 2010; 329:1168–1174.
10. Wallace A, Thorn M, Maguire RA, et al. Assay for establishing whether microbicide applicators have been exposed to the vagina. Sex Transm Dis 2004; 31:465–468.
11. Wallace AR, Teitelbaum A, Wan L, et al. Determining the feasibility of utilizing the microbicide applicator compliance assay for use in clinical trials. Contraception 2007; 76:53–56.
12. IARC. IARC Monographs on the Evaluation of the Carcinogenic Risks to Humans, Overall Evaluations of Carcinogenicity: An Updating of IARC Monographs Volumes 1 to 42. In: Cancer IAfRo, ed. Lyon, France: World Health Organization, International Agency for Research on Cancer, 1987:73.
13. Spitalnic S. Test properties I: Sensitivity, specificity, and predictive values. Hosp Physician 2004; 40:27–31.
14. Skoler-Karpoff S, Ramjee G, Ahmed K, et al. Efficacy of Carraguard for prevention of HIV infection in women in South Africa: A randomised, double-blind, placebo-controlled trial. Lancet 2008; 372:1932–1933.
15. Mensch BS, Hewett PC, Abbott S, et al. Assessing the reporting of adherence and sexual activity in a simulated microbicide trial in South Africa: An interview mode experiment using a placebo gel. AIDS Behav 2011; 15:407–421.
16. van de Wijgert J, Jones H, Kilmarx PH. Vaginal microbicide adherence biomarkers should be validated. Lancet 2009; 373:721; author reply 721–722.
17. Pool R, Montgomery CM, Morar NS, et al. A mixed methods and triangulation model for increasing the accuracy of adherence and sexual behaviour data: The Microbicides Development Programme. PLoS One 2010; 5:e11600.
18. Austin MN, Rabe LK, Hillier SL. Limitations of the dye-based method for determining vaginal applicator use in microbicide trials. Sex Transm Dis 2009; 36:368–371.
19. Hogarty K, Kasowitz A, Herold B, et al. Assessment of adherence to product dosing in a pilot microbicide study. Sex Transm Dis 2007; 34:1000–1003.
20. Keller MJ, Madan RP, Torres NM, et al. A randomized trial to assess anti-hiv activity in female genital tract secretions and soluble mucuosal immunity following application of 1% tenofovir gel. PLoS One. 2011; 6:e16475.
21. Grant RM, Lama JR, Anderson PL, et al. Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. N Engl J Med 2010; 363:2587–2599.
22. Karim SSA, Baxter C. Anteretroviral prophylaxis for the prevention of HIV infection: Future implementation challenges. HIV Ther 2009; 3:3–6.
23. Nel AM, Coplan P, Smythe SC, et al. Pharmacokinetic assessment of dapivirine vaginal microbicide gel in healthy, HIV-negative women AIDS Res Hum Retroviruses 2010; 26:1181–1190.
24. Hays MA, Irsula B, McMullen SL, et al. A comparison of three daily coital diary designs and a phone-in regimen. Contraception 2001; 63:159–166.
25. Lester RT, Ritvo P, Mills EJ, et al. Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): A randomised trial. Lancet 2010; 376:1838–1845.
26. Schwartz JL, Mauck C, Lai JJ, et al. Fourteen-day safety and acceptability study of 6% cellulose sulfate gel: A randomized double-blind Phase I safety study. Contraception 2006; 74:133–140.
27. Tien D, Schnaare RL, Kang F, et al. In vitro and in vivo characterization of a potential universal placebo designed for use in vaginal microbicide clinical trials. AIDS Res Hum Retroviruses 2005; 21:845–853.
28. Mauck C. Biomarkers for evaluating vaginal microbicides and contraceptives: Discovery and early validation. Sex Transm Dis 2009; 36(suppl 3):S73–S75.
29. Govender S, Skoler S, Maguire R, et al. Evaluation of microbicide applicators to determine vaginal use in the CarraguardTM Phase 3 Clinical Trial. Microbicides; 2006; Cape Town, South Africa.