Self-reported Human Papillomavirus Vaccination and Vaccine Effectiveness Among Men Who Have Sex with Men: A Quantitative Bias Analysis

Abstract

Background: 

Self-report of human papillomavirus (HPV) vaccination has ~80–90% sensitivity and ~75–85% specificity. We measured the effect of nondifferential exposure misclassification associated with self-reported vaccination on vaccine effectiveness (VE) estimates.

Methods: 

Between 2017–2019, we recruited sexually active gay, bisexual, and other men who have sex with men aged 16–30 years in Canada. VE was derived as 1−prevalence ratio × 100% for prevalent anal HPV infection comparing vaccinated (≥1 dose) to unvaccinated men using a multivariable modified Poisson regression. We conducted a multidimensional and probabilistic quantitative bias analysis to correct VE estimates.

Results: 

Bias-corrected VE estimates were relatively stable across sensitivity values but differed from the uncorrected estimate at lower values of specificity. The median adjusted VE was 27% (2.5–97.5th simulation interval = −5–49%) in the uncorrected analysis, increasing to 39% (2.5–97.5th simulation interval = 2–65%) in the bias-corrected analysis.

Conclusion: 

A large proportion of participants erroneously reporting HPV vaccination would be required to meaningfully change VE estimates.

INTRODUCTION

Observational studies are often necessary to measure vaccine effectiveness (VE) under real-world field conditions.¹ However, the validity of such studies depends on accurate ascertainment of vaccination status.² Information bias can result if the exposure is measured with error, whereby vaccinated participants are misclassified as unvaccinated or vice versa. One often-cited source of misclassification in VE studies is self-report. Prior simulation studies have shown that self-report can substantially bias effect estimates, with specificity having a greater impact than sensitivity, particularly under scenarios of low vaccine coverage.^3,4

Previously, we found that self-reported receipt of ≥1 dose of the human papillomavirus (HPV) vaccine was associated with a 27% lower anal prevalence of quadrivalent HPV vaccine-preventable types among gay, bisexual, and other men who have sex with men (MSM) aged 16–30 years in Canada.⁵ As expected, this real-world estimate was lower than the ≥1-dose efficacy of 49% against incident HPV detection reported in a clinical trial of young gay, bisexual, and other MSM aged 16–26 years who had limited sexual experience.⁶ This discrepancy is likely because most participants in our study were exposed to HPV before vaccination, along with differences in study design and outcomes.⁵ However, we wanted to quantify the amount of misclassification due to self-reported vaccination as another possible explanation.

Since 2015/16, several Canadian provinces have offered publicly funded HPV vaccine to men aged ≤26 years who self-identify as gay, bisexual, or other MSM according to a 2- or 3-dose schedule.⁷ However, most jurisdictions do not maintain registries for adult vaccinations,⁸ precluding validation of self-reported vaccination status. Self-report has moderate to high sensitivity (~80–90%) and specificity (~75–85%) for initiation of ≥1 dose of HPV vaccine in young adults (Table 1).^9–13 Only one study to our knowledge has specifically assessed accuracy of self-report among gay, bisexual, and other MSM.¹³ Forward et al. found sensitivity of 83.2% (95% CI = 78.4–87.3%) for self-reported ≥1-dose receipt of HPV vaccine among gay, bisexual, and other MSM and transwomen aged 18–26 years.¹³

We undertook a quantitative bias analysis to measure the effect of nondifferential exposure misclassification associated with self-reported HPV vaccination on our VE estimate.

METHODS

Recruitment and Data Collection

We recruited 2,449 sexually active gay, bisexual, and other MSM aged ≥16 years in Montréal, Toronto, and Vancouver, Canada. Men were recruited between February 2017 and August 2019 using respondent-driven sampling, a modified form of chain-referral sampling that allows for the adjustment of selection biases using sampling weights.^14,15 At enrollment, we asked men if they had ever received ≥1 dose of HPV vaccine in their lifetime using a computer-assisted, self-interview. Younger gay, bisexual, and other MSM aged 16–30 years (n = 1003) were invited to self-collect an anal specimen for HPV testing.⁵ We performed type-specific HPV-DNA genotyping using the PCR-based Roche Linear Array^®.¹⁶ All participants provided written informed consent. Research ethics boards at participating institutions granted study approval.

VE Analysis

We compared the baseline prevalence of ≥1 quadrivalent vaccine-preventable type (HPV-6/11/16/18) between vaccinated (≥1 dose) and unvaccinated gay, bisexual, and other MSM using a modified Poisson regression for binary outcomes with robust standard errors.^17,18 We calculated VE as (1–prevalence ratio) × 100%. We adjusted the multivariable model for age group, city, education, smoking, lifetime history of sexually transmitted infections, and number of condomless receptive anal sex encounters in the past 6 months. As there is no agreed-upon method for multivariable regression using respondent-driven sampling data and tools for weighted quantitative bias analysis have not been developed, we report statistics unweighted for respondent-driven sampling.^19,20

Quantitative Bias Analysis

We assumed that exposure misclassification was nondifferential (i.e., misclassification of vaccination status does not depend on anal HPV infection) on the basis that participants self-reported their vaccination status without knowledge of their HPV results (most infections in males are asymptomatic) and that HPV testing was performed on all participants, regardless of vaccination status. Under these assumptions, misclassification of a binary exposure will, on average, bias estimates toward the null.²¹

We conducted a quantitative bias analysis in two ways. First, we performed a multidimensional analysis using open-access Excel (Microsoft Corp., Redmond, WA) templates developed by Lash et al. (https://sites.google.com/site/biasanalysis/).²² We systematically varied sensitivity (Sn) and specificity (Sp) in 0.05-unit increments ranging from 0.75–1.0 and 0.7–1.0, respectively, according to the formulas $A=[(a-(a+b)\ast (1-Sp)]/[Sn-(1-Sp)]$, $C=[(c-(c+d)\ast (1-Sp)]/[Sn-(1-Sp)]$, $B=(a+b)-A$, and $D=(c+d)-C$, where lower case letters refer to cells of the original 2 × 2 table and upper case letters refer to cells of the bias-corrected table. Second, we performed a probabilistic analysis using the %sensmac macro developed by Fox et al.²³ (https://sites.google.com/site/biasanalysis/) in SAS (SAS Inc., Cary, NC) that was adapted to output prevalence ratios instead of odds ratios. Compared to the multidimensional analysis, this probabilistic approach accounts for both random and systematic error and gives greater weight to more plausible sensitivity and specificity values.^22,23 We modeled bias parameters as joint probabilities assuming a trapezoidal distribution for sensitivity (min = 0.75; mode = 0.8, 0.9; max = 0.95) and specificity (min = 0.7; mode = 0.75, 0.85; max = 0.9) with 10,000 simulations. These modes represent the lower and upper limits of sensitivity (80–90%) and specificity (75–85%) identified in prior literature (Table 1).^9–13 We calculated relative change as (VE_c/VE_u–1) × 100% where VE_c is the bias-corrected estimate and VE_u is the uncorrected estimate.

TABLE 1. - Summary of Sensitivity and Specificity Estimates for Self-reported Uptake of ≥1 Dose of Human Papillomavirus Vaccine in Young Adults Based on Published Literature.

Study	Setting	Population	Comparison	Sensitivity (95% CI)	Specificity (95% CI)
Rolnick et al. 9	USA, 2007	Females aged 18–26 years	Electronic medical records	91.2% (87.3–95.1%)	76.1% (70.2–82.1%)
Lewis et al. 10	USA, 2016	Males and females aged 14–29 years	Provider-verified vaccination records	87.0% (73.7–95.1%)	83.3% (71.5–91.7%)
Thomas et al. 11	USA, 2013–2015	Sexually experienced men aged 13–26 years	Electronic medical records and/or statewide immunization registry	Overall: 79.5% 14–18 years: 50.6% 19–21 years: 75.9% 22–26 years: 93.2%	Not reported
Oliveira et al. 12	USA, 2013–2018	Women aged 23–38 years undergoing cervical cancer screening	Provider-documented vaccination records	89% (82–94%)	80% (74–86%)
Forward et al. 13	USA, 2016–2018	MSM and trans women aged 18–26 years	Electronic medical records and/or statewide immunization registry	Overall: 83.2% (78.4–87.3%) 18–21 years: 79.1% (69.3–86.9%) 22–26 years: 85.1% (79.3–89.7%)	Not reported

CI, confidence interval; MSM, men who have sex with men.

RESULTS

Of the 608 participants aged 16–30 years who provided a valid anal specimen and had nonmissing data for HPV vaccination at baseline, 245 (40.3%) self-reported ≥1 HPV vaccine dose. Of those vaccinated, 61.8% received three doses. A total of 156 (25.7%) participants tested positive for ≥1 quadrivalent HPV vaccine-preventable type, including 54/245 (22.0%) vaccinated (≥1 dose) and 102/363 (28.1%) unvaccinated participants, corresponding to a crude VE of 22% (95% CI = −5% to 41%).

For the multidimensional quantitative bias analysis, bias-corrected VE estimates were relatively stable across a range of sensitivity values but began to differ from the uncorrected estimate at lower values of specificity (Figure). Holding specificity constant at 1.0 and varying sensitivity from 0.95–0.75, the bias-corrected VE estimate increased from 22% to 26% (relative change = 3–21%). Holding sensitivity constant at 1.0 and varying specificity from 0.9 to 0.7, the bias-corrected VE increased from 26% to 59% (relative change = 20–174%).

FIGURE.: Multidimensional bias analysis to correct for nondifferential exposure misclassification of self-reported HPV vaccination status among gay, bisexual, and other MSM aged 16–30 years, Canada, 2017–2019. MSM, men who have sex with men; HPV, human papillomavirus; Sn, sensitivity; Sp, specificity; VE, vaccine effectiveness.

For the probabilistic quantitative bias analysis, the uncorrected median VE against prevalent anal infection was 27% (2.5–97.5th simulation interval [SI] = −5% to 49%) after adjustment for potential confounders (Table 2). In the bias-corrected analysis, the median adjusted VE was 38% (2.5–97.5th SI = 18–61%) considering only systematic error (relative change = 41%) and 39% (2.5–97.5^th SI = 2–65%) considering both random and systematic error (relative change = 44%).

TABLE 2. - Probabilistic Quantitative Bias Analysis to Correct for Nondifferential Exposure Misclassification of Self-reported HPV Vaccination Status Among Gay, Bisexual, and Other MSM Aged 16–30 Years, Canada, 2017–2019.

Model	Median PR a (2.5–97.5th SI)		Median VE, b % (2.5-97.5th SI)
	Unadjusted	Adjusted c	Unadjusted	Adjusted c
Uncorrected analysis
Random error only	0.78 (0.56–1.09)	0.73 (0.51–1.05)	22 (−9 to 44)	27 (−5 to 49)
Bias-corrected analysis
Systematic error only	0.64 (0.41–0.85)	0.62 (0.39–0.82)	36 (15–59)	38 (18–61)
Random + systematic error	0.64 (0.37–1.01)	0.61 (0.35–0.98)	36 (−1 to 63)	39 (2–65)

HPV, human papillomavirus; MSM, men who have sex with men; PR, prevalence ratio; SI, simulation interval; VE, vaccine effectiveness.

^aPrevalence ratio comparing the prevalence of anal infection with quadrivalent HPV vaccine-preventable types between vaccinated (≥1 dose) and unvaccinated participants.

^bVE calculated as 1–PR × 100%.

^cAdjusted for potential confounders: age group, city, education, smoking, lifetime history of sexually transmitted infections, and number of condomless receptive anal sex encounters in the past 6 months.

DISCUSSION

We conducted a quantitative bias analysis to quantify the magnitude of nondifferential exposure misclassification associated with self-reported HPV vaccination. In the probabilistic analysis, our VE estimates increased from 27% to 39% when a misclassification error was considered. Although higher than our original estimate, this bias-corrected estimate remained lower than the 49% efficacy for ≥1 dose found in the clinical trial,⁶ suggesting that exposure misclassification cannot entirely explain this discrepancy. Rather, differences in population, including possible exposure to HPV before vaccination, outcomes (incidence vs. prevalence), and study design should be considered when generalizing clinical efficacy to real-world VE estimates.¹ Higher VE is anticipated in individuals who receive the full three-dose series and are immunologically naïve to HPV.⁶

Although we were unable to validate self-reported HPV vaccination, we have reason to believe that sensitivity and specificity would be high for our cohort. First, the circumstances surrounding HPV vaccination for GBM likely make it a salient event, minimizing recall error. In most Canadian provinces, gay, bisexual, and other MSM aged ≤26 years are eligible for publicly funded HPV vaccine but must disclose same-sex activity to their healthcare provider.⁷ For men aged >26 years, HPV vaccine can cost upwards of $600 for the full three-dose series.²⁴ Most participants in our study reported receiving their first dose of HPV vaccine within 2 years of study enrollment.⁵ Second, in repeated collection of HPV vaccination history, only about 2% of men who reported being vaccinated at enrollment said they were unvaccinated at subsequent time points, suggesting that false reports are rare.²⁵

There are limitations to this analysis. We did not consider other potential sources of bias, aside from confounding in the adjusted probabilistic models. We acknowledge that some men may have confused HPV vaccine with other recommended vaccines (e.g., hepatitis B). We dichotomized HPV vaccination as receipt of ≥1 dose versus none, even though it is recommended as a two- or three-dose series,²⁶ and misclassification is anticipated to be worse for recall of number of doses, although data are more limited.^10,11,13 While quantitative bias assessment methods for 2 × 2 tables could be easily extended to polytomous variables,²⁷ we are not aware that automated tools have been developed beyond binary exposures.^22,23 We assumed non-differential misclassification; however, we acknowledge that misclassification could be differential if GBM who are aware of prior infections are more likely to seek out or report vaccination and if prior infection is highly correlated with current infection (e.g., through sexual behaviors).²⁸

In conclusion, we found that bias-corrected estimates were further from the null when we corrected for nondifferential exposure misclassification, corresponding to higher VE, in this sexually active population of Canadian gay, bisexual, and other MSM. Consistent with prior simulation studies,^3,4 specificity had a greater impact on VE estimates but our results were relatively robust across a range of sensitivity values. A large proportion of gay, bisexual, and other MSM erroneously reporting HPV vaccine receipt would be required to meaningfully change our VE estimates. This provides reassurance that our original estimates of VE against anal HPV infection are conservative.

ACKNOWLEDGMENTS

The Engage Study is led by Principal Investigators in Toronto: Daniel Grace and Trevor Hart; Montreal: Joseph Cox and Gilles Lambert; and Vancouver: Jody Jollimore, Nathan Lachowsky, and David Moore. We would like to thank the Engage study participants, clinical and research staff, and Community Engagement Committee members, as well as our community partner agencies. Study data were collected and managed using REDCap electronic data capture tools hosted at St. Michael’s Hospital. We acknowledge the Applied Health Research Centre at St. Michael’s Hospital for data management. We would particularly like to thank Dr. François Coutlée and Ms. Julie Guenoun at CHUM at the Université de Montréal for their intellectual contributions and conducting the HPV laboratory testing, Dr. Marc Brisson at Université Laval for coordinating economic analyses related to this project, and Dr. Shayna Sparling at Toronto Metropolitan University for national site coordination. More information about the Engage Cohort Study can be found here: https://www.engage-men.ca/.