Moench, Thomas R. MD*; O'Hanlon, Deirdre E. PhD†; Cone, Richard A. PhD†
Vaginal microbicides are products being developed for women to protect themselves from sexually transmitted infections including human immunodeficiency virus (HIV). The majority of microbicide candidates studied thus far have been formulated as gels for vaginal application. Incomplete adherence to gel use reduces power to detect microbicide protection (or harm). Although most microbicide trials have relied on self-report, when self-report has been compared with an objective measure of adherence in HIV prevention and therapy studies or in other fields, participants have substantially overestimated adherence.1–6 Microbicide studies may be particularly likely to suffer from over reporting of adherence, since socially desirable responses appear to be particularly prevalent with interventions associated with sexual intercourse.7
Objective adherence measurements have been applied to HIV prevention trials in only a minority of published studies.1–3 One means to assess whether microbicide gels have been used is to recover used applicators from trial participants, and stain the applicator barrel with a dye that discloses vaginal fluid and gel adhering to the barrel after vaginal insertion. FD&C Blue 1 dye (FD&C)8 has been used for this purpose and appears to perform well when used with polyethylene applicators (Microlax, Norden Pac, Kalmar, Sweden).8–10 However, this method was shown10 to have low accuracy when applied to the polypropylene applicators used in nearly all recent microbicide gel trials, the HTI polypropylene syringe-type applicator (HTI Plastics, Lincoln, NE). Although the study confirmed the original reports of robust FD&C staining of inserted polyethylene Microlax applicators, it found poor sensitivity when the FD&C method was applied to polypropylene HTI applicators. Sensitivity was lower still (47%–77% among 3 observers) when applied to applicators inserted after multiple previous doses of gel. The authors concluded that the FD&C method was insufficiently accurate for use with HTI polypropylene applicators.
We undertook the current study to compare 4 alternative methods to test for vaginal insertion of HTI polypropylene applicators loaded with clear, colorless, and nonfluorescent hydroxyethylcellulose (HEC) placebo gel. We designed the study to assess performance with applicators inserted without prior gel doses, and applicators inserted in a series of 8 consecutive daily doses, the latter to assess the effect of prior daily gel doses on test sensitivity. Additionally, we tested the adaptability and accuracy of a medical event monitoring system (MEMS)5,11,12 in recording the time and date of applicator disposal events, to assess its potential to provide immediately downloadable adherence data at microbicide study follow-up visits.
Alcian blue (Alcian blue 8GX), potassium iodide, and iodine were from Sigma-Aldrich, St. Louis, MO. Syringe-type vaginal applicators with polypropylene barrels (HTI Plastics, Lincoln, NE) containing 2.5 g of the HEC “universal” placebo gel13,14 were provided by ReProtect, Inc., Baltimore, MD. MEMS caps (MEMS6 TrackCap 45 mm) were from Aardex Ltd., Zug, Switzerland. MEMS caps were adapted for use with 1-gallon polyethylene milk bottles (Fig. 1). MEMS cap data were read with an MEMS Reader (Fig. 1) and PowerView Version 3 software, both from Aardex Ltd.
A viewing box (Fig. 2) was constructed to examine applicators under ultraviolet (UV) illumination provided by two 385-nm 60 light-emitting diode (LED) array lamps (Shenzhen JiaJin Electronics Co, Ltd., Guangdong China), covered with 400-nm short-pass glass filters (70- × 1-mm UG-1, Schott-colored glass). Applicators were viewed through a 76- × 102-mm Tiffen UV Haze-2A filter to block transmission of reflected source illumination.
Study Participants and Clinical Procedures
Seventy-five participants were recruited under a protocol approved by the Homewood Institutional Review Board of The Johns Hopkins University. They were predominantly students and staff of the University, aged 18 to 44 years, in good general health, nonpregnant, without current symptoms of genital tract discharge or irritation, had not used vaginal medication within the past 7 days, and were at least 3 days from the end of their last menses. At the study site, and witnessed by study staff, participants vaginally inserted applicators and dispensed gel, and handled but did not vaginally insert “sham-inserted” applicators, that is, gripped the applicator barrel and dispensed the gel into a waste container. The study staff member documented the time and date of insertion and applied a label with a unique 3-digit randomly generated applicator identification number to the applicator. Participants were divided into 3 groups as follows:
Group 1: 10 participants provided a witnessed inserted applicator and a witnessed sham-inserted applicator for use in training of test readers (Readers).
Group 2: 50 participants provided a witnessed inserted and a witnessed sham-inserted applicator to be used to evaluate the tests.
Group 3: 15 participants provided 8 consecutive daily-inserted applicators and 2 sham-inserted applicators as follows: on both initial and final visits, participants provided a witnessed inserted and a witnessed sham-inserted applicator. During the 6 intervening days, participants were instructed to insert 1 applicator daily at home between the initial and final visits, dispose the applicators into the MEMS capped disposal container, and record the time and date of disposal on a study diary. These 8 consecutive daily-inserted applicators from each group 3 participant were obtained to evaluate methods for monitoring adherence in trials requiring daily applications of gel. Applicators inserted after previous gel use were compared with applicators inserted without previous gel insertion for signal intensity for the UV and Alcian blue methods and for test accuracy with all 4 methods.
Adherence Monitoring Methods
Applicators were identified, and their insertion status masked, by labeling with unique randomly generated numbers. Readers were masked to applicator insertion status, except the applicators from group 1 used for training. To investigate test-retest reliability, each Reader performed 2 readings of each method, with repeat readings separated by at least 3 days, and with the applicators in an order randomized for insertion status, but not altered from the original reading. The time required for each Reader to read the entire set of 250 applicators was recorded, not including staining procedures or racking time. In a semiquantitative assessment, Readers graded the perceived signal intensity as 0, 1, 2, or 3 for both the UV and Alcian blue methods on applicators that had been inserted on 8 consecutive days by 15 participants.
Test Methods and Rationale.
Applicators were evaluated for vaginal insertion by the following methods in the order listed: under ambient light without staining; under UV light without staining (based on the longstanding utility of UV for forensic detection of body fluids15); by microscopic detection of iodine stained cells transferred by applicator swabbing (based on a similar transfer method10 and on the long-standing use of iodine as a specific test for glycogen-bearing vaginal epithelial cells16); and after staining with Alcian blue17 (selected as an alternative to FD&C Blue 1 based on our pilot studies indicating improved staining intensity compared with that provided by FD&C Blue 1).
Ambient Light Inspection (Ambient).
Applicators were placed over support posts of a single-row rack that allowed inspection on all sides and viewed under laboratory overhead fluorescent lights. Applicators with visible adherent material (secretions and/or gel) were scored as positive.
UV Light Inspection.
Applicators were loaded into a tray holding 36 applicators. The tray was slid over rails into a viewing box (Fig. 2), providing illumination with UV light, and viewed in a darkened room. As the tray is moved through the viewing box, the applicators rotate on suspension rails, thus enabling rapid viewing of all portions of each applicator. Applicators with a streaked pattern of blue-green fluorescence (Fig. 3) were scored as positive.
Iodine Staining of Transferred Cells (Iodine).
Cells were transferred from applicators to the wells of Teflon-coated glass microscope slides, containing twelve 4-mm diameter bare glass wells (Erie Scientific, Portsmouth, NH), by swabbing approximately 5% of the surface area of the applicator with the end of a water-moistened swab tip and firmly pressing the end of the swab tip 3 times against the glass surface of a well. Wells were covered with Lugol's iodine solution (5% iodine, 10% potassium iodide in water) diluted 125-fold in water, to stain glycogen, and viewed 1 to 5 minutes later with a 10× objective under bright field illumination to detect glycogen-containing vaginal epithelial cells. Four or more iodine-stained cells per well were considered to indicate applicator insertion.
Alcian Blue Staining.
Applicators were placed on support posts of a single-row rack that allowed inspection on all sides. Applicators were sprayed on all sides with 0.1% Alcian blue in water, incubated 1 minute, inverted, dipped 3 times in a tap water bath, dried, and observed for blue staining of adherent secretions and/or gel. Applicators with a streaked pattern of blue staining were scored as positive.
Signal Intensity Assessment.
For the UV and the Alcian blue methods, a semiquantitative assessment of perceived signal intensity (0, 1, 2, or 3) was recorded for applicators that had been inserted on 8 consecutive days by the 15 group 3 participants (120 applicators).
Readers and Training.
Two Readers were undergraduate students, one with 4 months and another with 2 years of intermittent laboratory experience. The third Reader was a laboratory technician with 25 years of experience. Readers viewed the “training set” of inserted and noninserted applicators from group 1 with a method before reading group 2 and 3 applicators with this method. Applicator insertion status was identified to Readers, who were then instructed to view the training set to establish signal character and intensity thresholds to maximize accuracy in distinguishing inserted from sham-inserted applicators with each test method.
Medical Event Monitoring System.
MEMS caps were adapted to fit one-gallon polyethylene milk bottles serving as applicator disposal containers large enough to contain 2 months' worth of daily applicators, by attaching the threaded neck of a medication vial that matched the threads of the cap. The MEMS cap was removed to deposit each used applicator and then replaced. Sham-inserted applicators were deposited along with inserted applicators to assess possible cross contamination of inserted and sham-inserted applicators during storage. One bottle and MEMS cap were used at the study site to collect applicators from group 1, 2 bottles for the 100 group 2 applicators, and a separate bottle and MEMS cap were given to each participant in group 3 to collect applicators used at the study site and at home. Data stored in the MEMS caps were downloaded to a computer via an MEMS Reader device at the end of the study. The times and dates of applicator disposal events were compared with the time and date independently recorded by the study coordinator and by participants on diaries used to record time and date of applicators disposed of at home.
Sensitivity and specificity were calculated by standard formulae (Table 1). Chi square tests (2 sided with Yates continuity correction) were used for comparisons between tests, and included all data (2 readings from each of 3 Readers for each test). Test-retest reliability for each Reader was expressed as the percentage identity between the first and second readings.
Two hundred fifty applicators obtained as described earlier in the text were evaluated by the 4 test methods. The test order (Ambient, UV, Iodine, and Alcian blue) was chosen to avoid interferences by preceding tests. One hundred and seventy applicators were inserted, and 80 were sham-inserted (handled but not inserted).
Sensitivities and specificities are presented in Table 1, which lists values for individual Readers and the mean values for all 3 Readers. Because we observed substantially different sensitivities for applicators inserted without prior gel application compared with those inserted after one or more prior gel applications, sensitivity is reported for all inserted applicators, and separately for applicators inserted with and without prior gel applications.
Effect of Prior Gel Applications
The mean signal intensity score versus day of insertion is shown in Figure 4A. Signal intensity increased for both methods when gel had been applied previously (days 2–8 vs. 1). Similarly, Figure 4B shows that the sensitivity for the first insertion (day 1, no prior gel exposure) was 78% to 81% and increased for days 2 to 8 (∼90% on day 8).
Effect of Duration of Prior Laboratory Experience
Although all Readers had the same training in assessing insertion status with the tests, they differed in duration of prior laboratory experience. Table 1 shows that test accuracy did not increase with greater duration of prior laboratory experience.
Comparative Performance of Methods
The UV method had significantly higher sensitivity for all inserted applicators (P < 0.01 compared with Alcian blue, and P < 0.0001 against Ambient and Iodine). UV likewise had significantly higher sensitivity for applicators that followed prior gel insertions (P < 0.001 against Alcian blue, and P < 0.0001 against Ambient and Iodine). As with UV, the sensitivity of the other 3 methods was also significantly higher for applicators inserted after one or more prior insertions of gel compared with without prior gel insertion. Specificities were not significantly different for UV, Alcian, and Iodine (all ≥80%), but specificity was significantly less for Ambient (P < 0.0001). The mean (range) of test-retest reliability values for the 3 Readers (expressed as percent identity between first and second reading by each Reader) for UV, Ambient, Alcian blue, and Iodine respectively were 88% (range, 81–95), 86% (79–91), 68% (51–78), and 87% (80–91). MEMS cap data for witnessed date and time of applicator disposal events matched the written records by the study staff witness to within 2 minutes in all cases. The MEMS cap proved easy to adapt to containers suitable for storing ∼60 applicators (2 months worth of daily applicators).
Our results support the utility of adherence monitoring with MEMS cap documentation of the time and date of applicator disposal, and inspection of polypropylene applicators under UV light to determine whether returned applicators have been vaginally inserted. This approach would be highly suitable for monitoring studies using daily noncoital dosing, where the date each gel dose should be given is defined, and thus MEMS cap data could document the adherence rate, but also appears superior to previously published adherence assessment methods for intermittent dosing regimens using polypropylene applicators.
UV light inspection of applicators had the highest accuracy in correctly assessing applicator insertion status. In addition to its greater accuracy, the UV method does not require the preceding staining and rinse steps of Alcian blue, Iodine, or the previously reported FD&C method.1,8–10 This reduces labor, space, facility requirements, and dye waste disposal issues. Moreover, the UV reading step was also >40% faster than the reading step of the next fastest method (Alcian blue). Total time saved with UV would be much greater still since the UV method requires no stain and rinse steps and allows for rapid loading of applicators into inspection trays. The second most accurate method was Alcian blue. These results were superior to the results with the FD&C method reported by Austin et al,10 who reported low sensitivity and reproducibility of FD&C when applied to polypropylene HTI applicators, with particularly low accuracy after multiple prior gel insertions. These improved results might be because of superiority of the Alcian blue dye, different criteria used to determine insertion status than used previously, or the better defined dip-rinse method used here, which may have been easier to standardize and control than the previously reported spray-rinse method1,8–10 or other unknown factors. Notwithstanding the improved performance with the present dye method, our results show still better performance and practicality of the UV method compared with Alcian blue.
Test reading accuracy for UV (or other methods) did not improve with increasing Reader's duration of prior laboratory experience. This encourages us to believe that the UV method would be transferrable to study sites and staff without extensive prior laboratory work experience. However, we believe the training method we used (providing hands-on experience reading a set of known inserted and noninserted applicators, to allow Readers to set optimal thresholds) will likely be important to achieve good accuracy. Such a training set of witnessed applicators could be obtained on-site during participant screening sessions or other pilot activities, and it could be used for training, training refreshers, and quality assurance evaluations.
We investigated the effect of multiple gel applications on the method sensitivity and found substantially improved sensitivity with all methods after prior gel doses. This is in marked contrast to the unexplained severe fall-off in accuracy after multiple gel doses found using the FD&C method.10 Increased accuracy after prior insertions with the presently studied methods may have been because of gel from prior doses accumulating in the vagina, and hence, more gel, mucus, and cells retained on the surface of inserted applicators, increasing both signal intensity and test accuracy. Since gel from prior insertions will be routinely present in the vaginas of women adhering to daily insertion regimens, the higher sensitivities observed would therefore be expected. In contrast, coital regimens may result in reduced test sensitivity more similar to the applicators studied here without prior gel insertions due to the intermittency of coitus. However, we have pilot data indicating the accuracy of the UV method can be improved using more powerful UV light sources (data not shown).
Both the Ambient and Iodine methods were less accurate than UV or Alcian blue. The Iodine method may have underperformed because of the limited area (∼5%) of the applicators we sampled in order to avoid interference with the planned subsequent Alcian blue staining step. Nonetheless, although we believe the swab transfer of cells from a large portion of the applicator surface might improve test performance, microscopy of large numbers of samples would be tedious and time consuming, particularly compared to the UV method.
The MEMS cap data can be rapidly downloaded during follow-up visits. Adherence messages, and potentially more intensive monitoring such as home visits, could be promptly instituted when low rates of adherence are noted on MEMS cap data. UV staining would be useful to detect instances where women are disposing of applicators on schedule, but have not inserted them. This appears to have occurred in the Carraguard trial, where roughly half the applicators appeared to have been returned empty but apparently without having been inserted.1
Our study has important strengths. A relatively large number of applicators (250) were used and were studied with direct comparative assessments of multiple methods performed on the same applicators. The status of most applicators was witnessed as to inserted or sham-inserted, thus providing an effective gold standard. We studied applicators both with and without prior gel administrations, thus assessing whether accumulated vaginal gel altered the ability to detect insertion, as was found with the FD&C method.10 Thus, different gels will not likely influence the sensitivity of the UV method. Additional strengths included secure blinding to applicator status and the use of multiple Readers and repeat readings. We tested inserted and sham-inserted applicators that were comingled after disposal, to provide a rigorous test of the specificity of the tests in the presence of realistic opportunities for cross contamination.
A limitation of our study is that the performance of some or all of these tests might be dependent on the gel and applicator used, and should be validated for the materials to be used in a given trial. However, since HEC gel is nonfluorescent, the signal from the UV method results from fluorescence of vaginal secretions, not the HEC gel. Thus, the improved UV performance after prior gel deliveries likely results from residual gel increasing the amount of vaginal secretions that adhere to the applicator. Candidate gels that, like HEC, are nonfluorescent will likely provide the same UV sensitivity, whereas candidate gels that are fluorescent will likely further improve the sensitivity of the UV method. A second limitation is that some applicator insertions were unwitnessed. However, study participants appeared to have been highly motivated, and their participation was brief and minimally burdensome compared with the months to years of participation in a microbicide clinical trial. Therefore, we have confidence that most or all of these unwitnessed insertions did, in fact, take place. In support of this expectation is the finding that both the average intensities and the accuracies for the applicators inserted at home were closely comparable with those of the directly witnessed applicators on day 8, the last day of the study (Fig. 4). Furthermore, the rate of positive UV and Alcian blue tests on unwitnessed applicators was not lower than for witnessed applicators. Finally, performances of multiple assays on the same applicators may have caused interferences, for example, reducing sensitivity of the iodine method due to limited area (5%) that could be swabbed or slightly reduced sensitivity of Alcian blue because of clearing this swabbed area.
Our study indicates that UV inspection of applicators can efficiently detect whether HTI polypropylene applicators have been vaginally inserted. Although the lower sensitivity of all 4 methods with applicators inserted without prior gel applications may limit their accuracy for coital-based regimens, for daily dosing regimens, the high sensitivity and speed of the UV method is likely to provide a highly reliable assessment of adherence. Moreover, the MEMS cap monitoring method can identify poorly adherent participants in a timely manner and enable early interventions to improve their adherence.