Observational studies are often necessary to measure vaccine effectiveness (VE) under real-world field conditions.1 However, the validity of such studies depends on accurate ascertainment of vaccination status.2 Information bias can result if the exposure is measured with error, whereby vaccinated participants are misclassified as unvaccinated or vice versa. One often-cited source of misclassification in VE studies is self-report. Prior simulation studies have shown that self-report can substantially bias effect estimates, with specificity having a greater impact than sensitivity, particularly under scenarios of low vaccine coverage.3,4
Previously, we found that self-reported receipt of ≥1 dose of the human papillomavirus (HPV) vaccine was associated with a 27% lower anal prevalence of quadrivalent HPV vaccine-preventable types among gay, bisexual, and other men who have sex with men (MSM) aged 16–30 years in Canada.5 As expected, this real-world estimate was lower than the ≥1-dose efficacy of 49% against incident HPV detection reported in a clinical trial of young gay, bisexual, and other MSM aged 16–26 years who had limited sexual experience.6 This discrepancy is likely because most participants in our study were exposed to HPV before vaccination, along with differences in study design and outcomes.5 However, we wanted to quantify the amount of misclassification due to self-reported vaccination as another possible explanation.
Since 2015/16, several Canadian provinces have offered publicly funded HPV vaccine to men aged ≤26 years who self-identify as gay, bisexual, or other MSM according to a 2- or 3-dose schedule.7 However, most jurisdictions do not maintain registries for adult vaccinations,8 precluding validation of self-reported vaccination status. Self-report has moderate to high sensitivity (~80–90%) and specificity (~75–85%) for initiation of ≥1 dose of HPV vaccine in young adults (Table 1).9–13 Only one study to our knowledge has specifically assessed accuracy of self-report among gay, bisexual, and other MSM.13 Forward et al. found sensitivity of 83.2% (95% CI = 78.4–87.3%) for self-reported ≥1-dose receipt of HPV vaccine among gay, bisexual, and other MSM and transwomen aged 18–26 years.13
We undertook a quantitative bias analysis to measure the effect of nondifferential exposure misclassification associated with self-reported HPV vaccination on our VE estimate.
Recruitment and Data Collection
We recruited 2,449 sexually active gay, bisexual, and other MSM aged ≥16 years in Montréal, Toronto, and Vancouver, Canada. Men were recruited between February 2017 and August 2019 using respondent-driven sampling, a modified form of chain-referral sampling that allows for the adjustment of selection biases using sampling weights.14,15 At enrollment, we asked men if they had ever received ≥1 dose of HPV vaccine in their lifetime using a computer-assisted, self-interview. Younger gay, bisexual, and other MSM aged 16–30 years (n = 1003) were invited to self-collect an anal specimen for HPV testing.5 We performed type-specific HPV-DNA genotyping using the PCR-based Roche Linear Array®.16 All participants provided written informed consent. Research ethics boards at participating institutions granted study approval.
We compared the baseline prevalence of ≥1 quadrivalent vaccine-preventable type (HPV-6/11/16/18) between vaccinated (≥1 dose) and unvaccinated gay, bisexual, and other MSM using a modified Poisson regression for binary outcomes with robust standard errors.17,18 We calculated VE as (1–prevalence ratio) × 100%. We adjusted the multivariable model for age group, city, education, smoking, lifetime history of sexually transmitted infections, and number of condomless receptive anal sex encounters in the past 6 months. As there is no agreed-upon method for multivariable regression using respondent-driven sampling data and tools for weighted quantitative bias analysis have not been developed, we report statistics unweighted for respondent-driven sampling.19,20
Quantitative Bias Analysis
We assumed that exposure misclassification was nondifferential (i.e., misclassification of vaccination status does not depend on anal HPV infection) on the basis that participants self-reported their vaccination status without knowledge of their HPV results (most infections in males are asymptomatic) and that HPV testing was performed on all participants, regardless of vaccination status. Under these assumptions, misclassification of a binary exposure will, on average, bias estimates toward the null.21
We conducted a quantitative bias analysis in two ways. First, we performed a multidimensional analysis using open-access Excel (Microsoft Corp., Redmond, WA) templates developed by Lash et al. (https://sites.google.com/site/biasanalysis/).22 We systematically varied sensitivity (Sn) and specificity (Sp) in 0.05-unit increments ranging from 0.75–1.0 and 0.7–1.0, respectively, according to the formulas , , , and , where lower case letters refer to cells of the original 2 × 2 table and upper case letters refer to cells of the bias-corrected table. Second, we performed a probabilistic analysis using the %sensmac macro developed by Fox et al.23 (https://sites.google.com/site/biasanalysis/) in SAS (SAS Inc., Cary, NC) that was adapted to output prevalence ratios instead of odds ratios. Compared to the multidimensional analysis, this probabilistic approach accounts for both random and systematic error and gives greater weight to more plausible sensitivity and specificity values.22,23 We modeled bias parameters as joint probabilities assuming a trapezoidal distribution for sensitivity (min = 0.75; mode = 0.8, 0.9; max = 0.95) and specificity (min = 0.7; mode = 0.75, 0.85; max = 0.9) with 10,000 simulations. These modes represent the lower and upper limits of sensitivity (80–90%) and specificity (75–85%) identified in prior literature (Table 1).9–13 We calculated relative change as (VEc/VEu–1) × 100% where VEc is the bias-corrected estimate and VEu is the uncorrected estimate.
TABLE 1. -
Summary of Sensitivity and Specificity Estimates for Self-reported Uptake of ≥1 Dose of Human Papillomavirus Vaccine in Young Adults Based on Published Literature.
Sensitivity (95% CI)
Specificity (95% CI)
|Rolnick et al.
||Females aged 18–26 years
||Electronic medical records
|Lewis et al.
||Males and females aged 14–29 years
||Provider-verified vaccination records
|Thomas et al.
||Sexually experienced men aged 13–26 years
||Electronic medical records and/or statewide immunization registry
14–18 years: 50.6%
19–21 years: 75.9%
22–26 years: 93.2%
|Oliveira et al.
||Women aged 23–38 years undergoing cervical cancer screening
||Provider-documented vaccination records
|Forward et al.
||MSM and trans women aged 18–26 years
||Electronic medical records and/or statewide immunization registry
CI, confidence interval; MSM, men who have sex with men.
Of the 608 participants aged 16–30 years who provided a valid anal specimen and had nonmissing data for HPV vaccination at baseline, 245 (40.3%) self-reported ≥1 HPV vaccine dose. Of those vaccinated, 61.8% received three doses. A total of 156 (25.7%) participants tested positive for ≥1 quadrivalent HPV vaccine-preventable type, including 54/245 (22.0%) vaccinated (≥1 dose) and 102/363 (28.1%) unvaccinated participants, corresponding to a crude VE of 22% (95% CI = −5% to 41%).
For the multidimensional quantitative bias analysis, bias-corrected VE estimates were relatively stable across a range of sensitivity values but began to differ from the uncorrected estimate at lower values of specificity (Figure). Holding specificity constant at 1.0 and varying sensitivity from 0.95–0.75, the bias-corrected VE estimate increased from 22% to 26% (relative change = 3–21%). Holding sensitivity constant at 1.0 and varying specificity from 0.9 to 0.7, the bias-corrected VE increased from 26% to 59% (relative change = 20–174%).
For the probabilistic quantitative bias analysis, the uncorrected median VE against prevalent anal infection was 27% (2.5–97.5th simulation interval [SI] = −5% to 49%) after adjustment for potential confounders (Table 2). In the bias-corrected analysis, the median adjusted VE was 38% (2.5–97.5th SI = 18–61%) considering only systematic error (relative change = 41%) and 39% (2.5–97.5th SI = 2–65%) considering both random and systematic error (relative change = 44%).
TABLE 2. -
Probabilistic Quantitative Bias Analysis to Correct for Nondifferential Exposure Misclassification of Self-reported HPV Vaccination Status Among Gay, Bisexual, and Other MSM Aged 16–30 Years, Canada, 2017–2019.
% (2.5-97.5th SI)
| Random error only
||22 (−9 to 44)
||27 (−5 to 49)
| Systematic error only
| Random + systematic error
||36 (−1 to 63)
HPV, human papillomavirus; MSM, men who have sex with men; PR, prevalence ratio; SI, simulation interval; VE, vaccine effectiveness.
aPrevalence ratio comparing the prevalence of anal infection with quadrivalent HPV vaccine-preventable types between vaccinated (≥1 dose) and unvaccinated participants.
bVE calculated as 1–PR × 100%.
cAdjusted for potential confounders: age group, city, education, smoking, lifetime history of sexually transmitted infections, and number of condomless receptive anal sex encounters in the past 6 months.
We conducted a quantitative bias analysis to quantify the magnitude of nondifferential exposure misclassification associated with self-reported HPV vaccination. In the probabilistic analysis, our VE estimates increased from 27% to 39% when a misclassification error was considered. Although higher than our original estimate, this bias-corrected estimate remained lower than the 49% efficacy for ≥1 dose found in the clinical trial,6 suggesting that exposure misclassification cannot entirely explain this discrepancy. Rather, differences in population, including possible exposure to HPV before vaccination, outcomes (incidence vs. prevalence), and study design should be considered when generalizing clinical efficacy to real-world VE estimates.1 Higher VE is anticipated in individuals who receive the full three-dose series and are immunologically naïve to HPV.6
Although we were unable to validate self-reported HPV vaccination, we have reason to believe that sensitivity and specificity would be high for our cohort. First, the circumstances surrounding HPV vaccination for GBM likely make it a salient event, minimizing recall error. In most Canadian provinces, gay, bisexual, and other MSM aged ≤26 years are eligible for publicly funded HPV vaccine but must disclose same-sex activity to their healthcare provider.7 For men aged >26 years, HPV vaccine can cost upwards of $600 for the full three-dose series.24 Most participants in our study reported receiving their first dose of HPV vaccine within 2 years of study enrollment.5 Second, in repeated collection of HPV vaccination history, only about 2% of men who reported being vaccinated at enrollment said they were unvaccinated at subsequent time points, suggesting that false reports are rare.25
There are limitations to this analysis. We did not consider other potential sources of bias, aside from confounding in the adjusted probabilistic models. We acknowledge that some men may have confused HPV vaccine with other recommended vaccines (e.g., hepatitis B). We dichotomized HPV vaccination as receipt of ≥1 dose versus none, even though it is recommended as a two- or three-dose series,26 and misclassification is anticipated to be worse for recall of number of doses, although data are more limited.10,11,13 While quantitative bias assessment methods for 2 × 2 tables could be easily extended to polytomous variables,27 we are not aware that automated tools have been developed beyond binary exposures.22,23 We assumed non-differential misclassification; however, we acknowledge that misclassification could be differential if GBM who are aware of prior infections are more likely to seek out or report vaccination and if prior infection is highly correlated with current infection (e.g., through sexual behaviors).28
In conclusion, we found that bias-corrected estimates were further from the null when we corrected for nondifferential exposure misclassification, corresponding to higher VE, in this sexually active population of Canadian gay, bisexual, and other MSM. Consistent with prior simulation studies,3,4 specificity had a greater impact on VE estimates but our results were relatively robust across a range of sensitivity values. A large proportion of gay, bisexual, and other MSM erroneously reporting HPV vaccine receipt would be required to meaningfully change our VE estimates. This provides reassurance that our original estimates of VE against anal HPV infection are conservative.
The Engage Study is led by Principal Investigators in Toronto: Daniel Grace and Trevor Hart; Montreal: Joseph Cox and Gilles Lambert; and Vancouver: Jody Jollimore, Nathan Lachowsky, and David Moore. We would like to thank the Engage study participants, clinical and research staff, and Community Engagement Committee members, as well as our community partner agencies. Study data were collected and managed using REDCap electronic data capture tools hosted at St. Michael’s Hospital. We acknowledge the Applied Health Research Centre at St. Michael’s Hospital for data management. We would particularly like to thank Dr. François Coutlée and Ms. Julie Guenoun at CHUM at the Université de Montréal for their intellectual contributions and conducting the HPV laboratory testing, Dr. Marc Brisson at Université Laval for coordinating economic analyses related to this project, and Dr. Shayna Sparling at Toronto Metropolitan University for national site coordination. More information about the Engage Cohort Study can be found here: https://www.engage-men.ca/.
1. Hanquet G, Valenciano M, Simondon F, Moren A. Vaccine effects and impact of vaccination programmes in post-licensure studies. Vaccine. 2013;31:5634–5642.
2. Orenstein WA, Bernier RH, Hinman AR. Assessing vaccine efficacy in the field. Further observations. Epidemiol Rev. 1988;10:212–241.
3. Jackson ML. Use of self-reported vaccination status can bias vaccine effectiveness estimates from test-negative studies. Vaccine X. 2018;1:100003.
4. De Smedt T, Merrall E, Macina D, Perez-Vilar S, Andrews N, Bollaerts K. Bias due to differential and non-differential disease- and exposure misclassification in studies of vaccine effectiveness. PLoS One. 2018;13:e0199180.
5. Chambers C, Deeks SL, Sutradhar R, et al.; Engage-HPV Study Team. Anal human papillomavirus prevalence among vaccinated and unvaccinated gay, bisexual, and other men who have sex with men in Canada. Sex Transm Dis. 2022;49:123–132.
6. Palefsky JM, Giuliano AR, Goldstone S, et al. HPV vaccine against anal HPV infection and anal intraepithelial neoplasia. N Engl J Med. 2011;365:1576–1585.
7. Goyette A, Yen GP, Racovitan V, Bhangu P, Kothari S, Franco EL. Evolution of public health human papillomavirus immunization programs in Canada. Curr Oncol. 2021;28:991–1007.
8. Wilson SE, Quach S, MacDonald SE, et al. Immunization information systems in Canada: Attributes, functionality, strengths and challenges. A Canadian Immunization Research Network study. Can J Public Health. 2017;107:e575–e582.
9. Rolnick SJ, Parker ED, Nordin JD, et al. Self-report compared to electronic medical record across eight adult vaccines: do results vary by demographic factors? Vaccine. 2013;31:3928–3935.
10. Lewis RM, Markowitz LE. Human papillomavirus vaccination coverage among females and males, National Health and Nutrition Examination Survey, United States, 2007-2016. Vaccine. 2018;36:2567–2573.
11. Thomas R, Higgins L, Ding L, Widdice LE, Chandler E, Kahn JA. Factors associated with HPV vaccine initiation, vaccine completion, and accuracy of self-reported vaccination status among 13- to 26-year-old men. Am J Mens Health. 2018;12:819–827.
12. Oliveira CR, Avni-Singer L, Badaro G, et al. Feasibility and accuracy of a computer-assisted self-interviewing instrument to ascertain prior immunization with human papillomavirus vaccine by self-report: Cross-sectional analysis. JMIR Med Inform. 2020;8:e16487.
13. Forward T, Meites E, Lin J, et al. Sensitivity of self-reported human papillomavirus vaccination history among 18- to 26-year-old men who have sex with men: Seattle, WA, 2016 to 2018. Sex Transm Dis. 2022;49:81–85.
14. Heckathorn DD. Respondent-driven sampling: a new approach to the study of hidden populations. Soc Probl. 1997;44:174–199.
15. World Health Organization. Introduction to HIV/AIDS and sexually transmitted infection surveillance. Module 4: Introduction to respondent-driven sampling. World Health Organization; 2013. Available at: https://apps.who.int/iris/handle/10665/116864
. Accessed October 16, 2020.
16. Coutlée F, Rouleau D, Petignat P, et al.; Canadian Women's HIV study Group. Enhanced detection and typing of human papillomavirus (HPV) DNA in anogenital samples with PGMY primers and the Linear array HPV genotyping test. J Clin Microbiol. 2006;44:1998–2006.
17. Spiegelman D, Hertzmark E. Easy SAS calculations for risk or prevalence ratios and differences. Am J Epidemiol. 2005;162:199–200.
18. Zou G. A modified poisson regression approach to prospective studies with binary data. Am J Epidemiol. 2004;159:702–706.
19. White RG, Hakim AJ, Salganik MJ, et al. Strengthening the reporting of observational studies in epidemiology for respondent-driven sampling studies: “STROBE-RDS” statement. J Clin Epidemiol. 2015;68:1463–1471.
20. Avery L, Rotondi N, McKnight C, Firestone M, Smylie J, Rotondi M. Unweighted regression models perform better than weighted regression techniques for respondent-driven sampling data: Results from a simulation study. BMC Med Res Methodol. 2019;19:202.
21. Rothman KJ, Greenland S, Lash TL. Validity in epidemiologic studies. Rothman KJ, Greenland S, Lash TL, eds. In: Modern Epidemiology. 3rd
ed. Lippincott Williams & Wilkins; 2008:128–147.
22. Lash TL, Fox MP, Fink AK. Applying Quantitative Bias Analysis to Epidemiologic Data. Springer; 2009.
23. Fox MP, Lash TL, Greenland S. A method to automate probabilistic sensitivity analyses of misclassified binary variables. Int J Epidemiol. 2005;34:1370–1376.
24. Toronto Public Health. Human papillomavirus (HPV) vaccine fact sheet. City of Toronto; 2021. Available at: https://www.toronto.ca/community-people/health-wellness-care/diseases-medications-vaccines/human-papillomavirus-hpv-vaccine/
. Accessed October 14, 2021.
25. Chambers C, Deeks SL, Sutradhar R, et al. Increases in human papillomavirus vaccine coverage over 12 months among a community-recruited cohort of gay, bisexual, and other men who have sex with men in Canada. Vaccine. 2022;40:3690–3700.
26. Public Health Agency of Canada. Canadian immunization guide: Part 4 - active vaccines. Ottawa: Government of Canada. 2018. Available at: https://www.canada.ca/en/public-health/services/canadian-immunization-guide.html
. Accessed October 14, 2021.
27. Greenland S, Lash TL. Bias analysis. Rothman KJ, Greenland S, Lash TL, eds. In: Modern Epidemiology. 3rd
ed. Lippincott, Williams & Wilkins; 2008:345–380.
28. van Smeden M, Lash TL, Groenwold RHH. Reflection on modern methods: Five myths about measurement error in epidemiological research. Int J Epidemiol. 2020;49:338–347.