Prediction of HIV Acquisition Among Men Who Have Sex With Men

Abstract

Objective: 

To develop and validate an easy-to-use prediction model for HIV acquisition among men who have sex with men (MSM).

Methods: 

We developed prediction models using medical records data from an STD clinic (2001–2008) and validated these models using data from the control arm of Project Explore, an HIV prevention trial (1999–2003).

Results: 

Of 1903 MSM who tested for HIV more than once in the development sample, 101 acquired HIV over 6.7 years of follow-up. Annual HIV incidence was 2.57% (95% confidence interval [CI]: 2.09%, 3.12%). During 4 years of follow-up of 2081 Project Explore control arm participants, 144 acquired HIV for an incidence of 2.32% (95% CI: 1.96%, 2.73%). A prediction model that included variables indicating use of methamphetamine or inhaled nitrites in the prior 6 months, unprotected anal intercourse with a partner of positive or unknown HIV status in the prior year, ≥10 male sex partners in the prior year, and current diagnosis or history of bacterial sexually transmitted infection was well calibrated overall (expected-observed ratio = 1.01; 95% CI: 0.97, 1.05) and had modest discriminatory accuracy at 1 year (area under the receiver-operator characteristic curve = 0.67; 95% CI: 0.60, 0.75) and at 4 years (area under the receiver-operator characteristic curve = 0.66; 95% CI: 0.61, 0.71). Over 4 years, cumulative incidence ranged from 3.9% to 14.3% for groups of men defined by the prediction model.

Conclusions: 

A new risk score was predictive of HIV acquisition and could assist providers in counseling MSM and in targeting intensified prevention to MSM at greatest risk for HIV infection. Its accuracy requires further evaluation.

In Brief

A validated prediction model for human immunodeficiency virus acquisition among men who have sex with men may prove useful in counseling men who have sex with men and in prioritizing prevention efforts to men who have sex with men at highest risk.

Men who have sex with men (MSM) continue to comprise the greatest number of HIV infections in the US and many other nations.^1–3 However, HIV risk is not evenly distributed among all MSM,^4–8 and efforts to prevent HIV might be more effectively employed if they stratified men based on validated risk criteria.

Clinical prediction models are an efficient means of identifying individuals at high risk for a medical problem who might benefit from treatment or intensified prevention efforts. Scientists have created models to predict the risk of developing coronary heart disease,⁹ diabetes,¹⁰ breast cancer,^11,12 lung cancer,¹³ and stroke among persons with atrial fibrillation,¹⁴ and findings from these models have contributed to the development of treatment and prevention guidelines for these diseases.^15–20

Several studies have identified characteristics, behaviors, and contexts that associated with HIV acquisition,^4–8 but few studies have used these factors to predict the probability of future HIV acquisition.^21–23 To our knowledge, no studies have created prediction models that are specific to the MSM population. Prediction models may help clinicians better counsel clients about their HIV risk, provide appropriate referrals to further services, make recommendations for the frequency of future HIV testing, and provide intensified interventions to men at highest risk. Therefore, we constructed and validated a multivariable risk score predictive of HIV acquisition among MSM.

Methods

Design Overview

We used data from Public Health—Seattle and King County (PHSKC) STD Clinic electronic records to develop a model predictive of HIV acquisition among MSM. We validated these scores using data from control group participants of Project Explore.²⁴

Risk Score Development: STD Clinic Repeat Testers

We included all MSM who tested for HIV >1 time in the PHSKC STD clinic between October 2001 and May 2008 and who tested HIV-negative by second-generation enzyme immunoassay (EIA) at their initial HIV test during the study period. Starting in 2003, all MSM negative on HIV EIA were also tested using pooled HIV RNA.²⁵ To account for the window period of HIV EIA, we excluded men from the analysis if <30 days elapsed between their first and last HIV testing visits if they were not tested using pooled RNA.

Baseline Measurement

PHSKC STD clinic providers use a structured form to record client histories before HIV testing. In a face-to-face encounter, clinicians collect information on demographic variables and self-reported substance use, STD history, and sexual behavior. Clinicians ask all clients whether they have had sex with men, women, or both in the prior year. We classified men who reported sex with another man in the prior year as MSM. For MSM, sexual behavior data include the number of male sex partners (anal and oral combined) and how often clients used condoms (always, usually, sometimes, never) during insertive and receptive anal intercourse with HIV-positive, HIV-negative, or HIV-unknown partners. These data use a 12-month recall period. In contrast, for substance use history, clinicians ask MSM whether they have ever used methamphetamine, injection drugs, inhaled nitrites, or crack and, if so, to provide the most recent month and year they used these substances. Syphilis was diagnosed using rapid plasma reagin confirmed with Treponema pallidum particle agglutination assay. Urethral gonococcal and chlamydial infections were diagnosed using Aptima Combo 2 (Genprobe Inc., San Diego, CA) or culture, while rectal and pharyngeal gonococcal and chlamydial infections were diagnosed by culture. Gonococcal cultures used modified Thayer-Martin media and chalmydial cultures used McCoy cells.

Model Building

We selected 6 binary variables on the basis of simplicity and epidemiologic evidence for inclusion in the predictive model: age less than 40 years,²⁶ nonwhite and non-Asian/Pacific Islander^26,27 (API) race/ethnicity, current laboratory diagnosis of a bacterial STD (gonorrhea, Chlamydia, or early syphilis) or of having ever had a bacterial STD,^8,28–30 use of methamphetamine or inhaled nitrites in the prior 6 months,^4,7,29,31,32 ≥10 male sex partners in the prior year,^4,32 and unprotected anal intercourse with a partner of unknown or positive HIV status (nonconcordant unprotected anal intercourse, UAI) in the prior year.^4,33 This model is the full model.

We considered the inclusion of herpes simplex virus type-2 (HSV-2) infection given its high prevalence among MSM and its association with HIV acquisition³⁴; however, excluded it because only 10% of men were tested for HSV-2 at their first HIV testing visit and only 8% reported a history of HSV-2 infection. Despite the different physiological effects of methamphetamine and inhaled nitrites, we grouped methamphetamine and inhaled nitrite use into one variable for simplicity and to facilitate uptake of the prediction models by clinicians. Additionally, these substances are used in similar contexts^35–39 and the magnitude of associations between HIV acquisition and these substances are similar.^4,7,29,31,32 We used a 6-month recall period for methamphetamine and inhaled nitrite use to be consistent with previous reports and to improve the reliability of the measures over those using longer recall periods.^4,7,29,31,32 We performed analyses with separate variables for methamphetamine and inhaled nitrites use and analyses with a 12-month recall period for the composite variable; the results were very similar to those in this manuscript and are not presented.

Because one’s age and race/ethnicity are immutable characteristics, we tested a second, simple model that excluded the variables indicating age and race/ethnicity. We also used a stepwise selection procedure using the Akaike Information Criterion to create a more parsimonious model from the initial 6-variable model.^24,40,41 This procedure yielded a model that included all the predictors except for nonconcordant UAI in the prior year. Because nonconcordant UAI was the strongest predictor of incident HIV infection in a cross-sectional analysis,³³ we kept it in our prediction model and did not evaluate a model without this predictor.

We used Cox proportional hazards models to estimate hazard ratios (HR) for HIV acquisition. Because clinicians and disease intervention specialists concentrate on current reports of risk and diagnoses of STD in counseling clients, the values of the predictors in the models were those at baseline; predictors were not modeled as time-varying covariates. The initial time point for survival modeling was the date of first negative HIV EIA. For MSM who tested HIV-positive by reactive EIA, we defined time-of-acquisition as the midpoint between the date of their positive EIA and the date of their last negative EIA. For men who tested HIV-positive by pooled RNA screening but HIV-negative by EIA at the same visit, we defined time-of-acquisition as the date of the positive RNA test. MSM were censored at their last HIV testing visit if they had not tested HIV-positive in the STD Clinic by 12 May 2008. Additionally, we repeated model building and development using the predictor values recorded at clients’ second visits at which they tested HIV-negative; the results of those analyses were similar to those using clients’ first visits (data not given here).

For the full and simple models, all predictors met the proportional hazards assumption as assessed with log(−log) plots of the survival function, Schoenfeld residuals and associated test statistics,⁴² and including interaction terms with time for each predictor variable. Additionally, we found no evidence for interactions between individual risk predictors assessed by a global likelihood ratio test.

Based on the coefficients from the Cox proportional hazards regression models, we created integer weights for each variable. We calculated these weights by multiplying the model coefficients by 10 (the choice of multiplier is arbitrary) and rounding to the nearest whole integer. The weights rank the risk predictors in relative importance and dictate how one assigns integer point values for each risk predictor for a given individual. The assigned points are then summed to compute that individual’s risk score.

We also performed interval-censored analyses assuming a log-logistic distribution to model the odds of HIV acquisition associated with each risk predictor.^43,44 These methods yielded similar results to the Cox proportional hazards models presented here.

Stability of Simple Model Predictors

To assess the consistency of risk among men evaluated in our clinic on multiple occasions, we used data collected from men on their first visit to the clinic each year for the first 4 years of follow-up to calculate κ statistics for the variables in the simple model.⁴⁵

Risk Score Validation: Project Explore

The control arm of Project Explore comprised the population for model validation. Project Explore was a randomized behavioral intervention trial that enrolled HIV-negative MSM age >16 from 6 US cities, including Seattle, between January 1999 and February 2001. Men were eligible if they reported having anal sex with ≥1 men in the past year, but were excluded if they reported a mutually monogamous relationship lasting ≥2 years with an HIV-negative male partner. At enrollment and every 6 months thereafter, Explore participants completed an audio computer-assisted self-interview to assess sexual behavior, substance use, and recent STD diagnoses in the prior 6 months, and were tested for HIV by EIA. Project Explore followed participants for 48 months or until January 2003, whichever came first, and 87% of men in the control arm completed the study.⁴⁶ The variables in Project Explore that corresponded to the predictors selected in the STD clinic sample were: methamphetamine or inhaled nitrites use in the prior 6 months; nonconcordant UAI in the prior 6 months; bacterial STD in the prior 6 months; and, ≥10 sex partners in the prior 6 months. All of these variables are based on participant self-report.

Analysis

Using the full and simple models derived from the STD clinic sample, we calculated 2 risk scores for each humans in the control arm of Project Explore based on baseline assessments. We calculated the annual incidence of HIV infection for each quartile of risk score. We assessed model calibration by calculating the ratio of expected HIV infections to observed HIV infections (expected-observed ratio),¹² where calibration is defined as the extent to which the risk predicted by the model reflects the risk in the observed in population.⁴⁷ The 95% confidence intervals for HIV incidence and the expected-observed ratio assume that the observed HIV infections follow a Poisson distribution. We also assessed model calibration graphically by comparing plots of observed and predicted survival estimates for each risk group.²⁴

To assess the ability of the risk scores to discriminate between men who acquired HIV and those who did not, we calculated the area under time-dependent receiver operating characteristic curves (AUC) for “survival” at 1 year of follow-up and at 4 years of follow-up with nonparametric bootstrap 95% confidence intervals using the methods of Heagerty et al.^48,49

All analyses were conducted using STATA 10.0 (StataCorp, College Station, TX) and R 2.6.1 (The R Project for Statistical Computing). The University of Washington Human Subjects Division approved all study procedures.

Results

Study Populations and HIV Incidence

Between October 2001 and May 2008, PHSKC STD clinic staff evaluated 6026 MSM who did not acknowledge previously testing HIV-positive; 290 (5%) of these men tested HIV-positive and 5736 (95%) men tested HIV-negative. Of these 5736 HIV-negative men, 3827 (67%) did not return for HIV testing during the study period and 1909 (33%) returned at least once for HIV testing (median: 3 visits, range: 2–28), and 1903 men tested for HIV again ≥30 days after their initial EIA and comprise our sample for model development.

Compared with men who tested for HIV only once during the study period, at baseline repeat testers were more likely to be white (68% vs. 71%, P = 0.01) and <40 years of age (70% vs. 80%, P <0.001), but less likely to use crack or cocaine (5% vs. 3%, P <0.001) or injection drugs (4% vs. 2%, P <0.001) in the prior 6 months. Men who did not return to test were less likely than men who tested repeatedly to test positive for bacterial STD (12% vs. 20%, P <0.001) at their only or first visits, respectively, and to report having ever been diagnosed with a bacterial STD (28% vs. 33%, P <0.001). Men who did and did not return for testing reported similar rates of methamphetamine and inhaled nitrite use in the prior 6 months and of nonconcordant UAI in the prior year.

Repeat HIV testers in the STD clinic sought testing over a median of 1.6 years (range: 30 days–6.7 years) and experienced 101 HIV infections (annual incidence = 2.57%; 95% confidence interval [CI]: 2.09%, 3.12%). Of the 2081 Explore control group participants, 144 acquired HIV over a median of 3 years (range: 6months–4 years) for an annual incidence of 2.32% (95% CI: 1.96%, 2.73%).

Explore participants were older, more likely to be Hispanic, and reported greater rates of partner change than STD clinic clients (Table 1). Explore participants were also more likely to report using inhaled nitrites and methamphetamine in the preceding 6 months than STD clinic testers. MSM tested at the STD clinic were more likely to be API than Explore participants, and more likely to be diagnosed with a bacterial STD at their first HIV testing visit than Explore participants in the 6 months preceding their enrollment interview. Fewer STD clinic clients reported having nonconcordant UAI in the preceding 12 months than Explore participants did in the prior 6 months.

TABLE 1: Baseline Socio-Demographic Characteristics and Sexual and Drug Use Behavior of PHSKC Repeat HIV Testers (2001–2008) and Project Explore Control Arm Participants (1999–2003)

Agreement of Simple Model Risk Predictors Within an Individual Over Time

Certain variables may be poor predictors of HIV infection if they are intermittent as opposed to consistent. For MSM tested at the STD clinic, we evaluated the agreement between the predictors of the simple model each year over the first 4 years of follow-up. The κ statistics were 0.41 for use of methamphetamine or inhaled nitrites in the prior 6 months, 0.47 for ≥10 male sex partners in the prior year, 0.28 for nonconcordant UAI in the prior year, and 0.67 for history, or current diagnosis, of bacterial STD.

Risk Score Evaluation and Validation

We used the full and simple prediction models to generate scores for each participant in the control arm of Project Explore based on their baseline assessments (Table 2). The mean (standard deviation) scores for the full and simple models were 11.7 (6.9) and 5.6 (6.0), respectively; the median (range) scores were 11 (0–31) and 3 (0–19), respectively. The incidence of HIV and the HR for HIV infection increased with quartile of risk score (test of trend P <0.0001 for each index; Table 3). For groups of men who differ by one unit in the full risk score, men with the higher risk score experience a risk of HIV 1.08 times greater than that experienced by men with the lower risk score (95% CI: 1.06, 1.11, P <0.001). For groups of men who differ by one category (quartile) of the full risk score, men in the higher category experience a risk of HIV 1.61 times greater than that experience by men in the lower category (95% CI: 1.39, 1.86; P <0.001). Comparable HRs for the simple model were similar.

TABLE 2: Multivariable Prediction Models for HIV Acquisition Among MSM Developed in PHSKC Repeat HIV Testers, 2001 to 2008

TABLE 3: Application of the Full and Simple MSM HIV Acquisition Prediction Models to Project Explore Control Arm Participants (1999–2003)

The full and simple models were well calibrated in the validation sample (Table 4). The expected annual rate of HIV infection was 2.35% according to the full model (expected-observed ratio: 1.01; 95% CI: 0.97, 1.06) and 2.34% according to the simple model (expected-observed ratio: 1.01; 95% CI: 0.97, 1.05). Inspection of plots comparing Kaplan-Meier survival curves and Cox proportional hazards survival curves for each risk quartile (data not shown), also indicated acceptable model calibration.

TABLE 4: Overall Calibration and Discrimination of the Full and Simple MSM HIV Acquisition Prediction Models Applied to PHSKC Repeat STD Clinic Testers (2001–2008) and Project Explore Control Arm Participants (1999–2003)

Model discrimination measured by time-dependent AUC estimates was 0.68 (95% CI: 0.62, 0.75) at 1 year and 0.67 (95% CI: 0.62, 0.71) at 4 years for the full model in Project Explore (Table 4). The simple model exhibited similar discrimination, 0.67 (95% CI: 0.60, 0.75) at 1 year and 0.66 (95% CI: 0.61, 0.71) at 4 years. Because our results may be influenced by temporal trends in sexual risk, substance use, and STI diagnoses over time and by the potential inclusion of Seattle Explore participants in the development sample, we performed 2 sensitivity analyses. The first assessed the impact of restricting model development and validation to the years 2001 to 2003 during which the STD clinic and Explore study periods overlapped. The second excluded the 371 Seattle participants from the Project Explore control arm on model discrimination. Time-dependent AUC estimates at 1 and 4 years were similar in magnitude and not different statistically from our original estimates (data not shown).

Because the full and simple models performed similarly based on their calibration and discrimination, Table 5 presents the sensitivity, specificity, and cumulative HIV incidence for groups of men defined by the score cut-offs for the simple model over 1 year and 4 years of follow-up. Figure 1 displays the cumulative incidence of HIV infection over time for each quartile of the simple risk score in Project Explore. Cumulative incidence at 4 years was 3.9% (95% CI: 2.4%, 6.2%) in men with a score of 0, 5.3% (95% CI: 2.1%, 8.5%) in men with scores 1 to 3, 9.5% (95% CI: 4.6%, 14.4%) in men with scores 4 to 11, and 14.3% (95% CI: 7.5%, 21.1%) in men with scores ≥12.

TABLE 5: Sensitivity, Specificity, and Cumulative Incidence of HIV Acquisition for Risk Scores Defined by the Simple Model, PHSKC Repeat Testers (2001–2008), and Project Explore Control Arm (1999–2003)

Figure 1.: Cumulative incidence of HIV infection the control arm of Project Explore, 1999 to 2003, according to risk quartiles defined by the simple prediction model developed among MSM seen in the Public Health—Seattle and King County STD Clinic, 2001 to 2008.

Discussion

We used clinical and behavioral data collected from an STD clinic to develop 2 risk scores predictive of HIV infection, and validated those scores using data from a large multicenter behavioral intervention trial. Although the scores we developed would optimally undergo additional validation in other populations,^50,51 we believe our simple score can be useful to clinicians and others in counseling MSM about their risk of HIV infection, and might be used to identify persons who require intensified interventions or more frequent HIV testing. The resources necessary to conduct biomedical and behavioral HIV prevention trials depend on HIV incidence.^46,52,53 Thus, this score might also help researchers identify MSM at high risk of HIV infection for prevention trials with HIV acquisition as the primary outcome.

Despite the model’s simplicity and excellent calibration, its discriminatory accuracy was only modest (AUC: 0.66). However, this finding should be interpreted with the understanding that extremely high relative risks (>100) are required to generate risk predictor models with high AUC estimates,^12,54–56 and that such models are extremely uncommon in clinical practice. The AUC estimates that we report are similar to those of other risk models, such as the Framingham risk score for coronary heart disease (AUC: 0.63–0.83),⁵⁷ that are commonly used to guide clinical decisions.

Although the calculation of the risk scores may seem daunting, the scores we present are not more complicated than the Framingham risk score⁹ or the CHADS score,¹⁴ tools that clinicians commonly use to estimate patients’ risk of coronary heart disease and the risk of stroke among persons with atrial fibrillation, respectively. Our MSM risk score should not require the use of a computer for calculation, and may be suitable for use in nonclinical settings or on the Internet. We are currently developing a public health website that will allow MSM to calculate their score and explore their risk of HIV acquisition. An approach to calculating the simple risk score and estimating a humans’ 4-year risk of acquiring HIV is presented in Figure 2.

Figure 2.: Estimating a patient’s 4-year risk of HIV acquisition according to the simple model.

The ideal risk score cut-off may be one that leads to the follow-up of a number of patients that the resources of a clinic, provider, or community-based organization can accommodate. We found that using a score that identifies men with any risk predictor (i.e., risk score ≥1), our simple model would identify 83% of STD clinic patients and 86% of Explore participants who acquired HIV over 4 years, but would require follow-up of approximately 70% of the 2 populations. Using a higher risk score identifies a population at greater risk, but a smaller proportion of all men who acquire HIV. Currently, our STD clinic employs the simple model to identify men with ≥1 risk predictor for more frequent HIV testing, follow-up counseling by telephone, and reminder notices to return for repeat HIV testing.

The strongest predictors of HIV acquisition in our models were use of methamphetamine or inhaled nitrites and a history, or current diagnosis, of bacterial STD, findings that highlight the importance of routinely asking MSM about these drugs and of concentrating prevention efforts in persons with bacterial STD. Although nonconcordant UAI has typically been the strongest risk factor for testing HIV-positive in cross-sectional studies,³³ it was a weaker predictor of HIV acquisition in our models. This difference may reflect the distinction between behaviors or characteristics that operate at the individual-level and those that operate at a network-level. For example, substance use and STD may indicate that an individual circulates within a sexual network more conducive to HIV transmission, while the report of nonconcordant UAI does not always indicate this network-level risk. Additionally, we found that the agreement in reports of nonconcordant UAI over 4 years of follow-up was only fair. This fair agreement likely diminishes our predictive models’ discriminatory ability. We overestimate the risk of HIV acquisition for men who report nonconcordant UAI at baseline, but do not practice nonconcordant UAI during the follow-up period. Conversely, we may underestimate the risk of HIV acquisition for men who do not report nonconcordant UAI at baseline, but engage in nonconcordant UAI during the follow-up period.

Our study is subject to important limitations. First, our risk score was derived from a population of MSM who repeatedly tested in a single US STD clinic. Not all MSM in our clinic tested serially during the study period, and studies suggest that risk behavior may influence HIV testing behavior.^58,59 On the other hand, our models performed reasonably well among MSM from 6 US cities who tested at 6-month interval as part of Project Explore, suggesting that the prediction models can be applied to a more diverse population of MSM. However, as our development and validation samples were composed mostly of white MSM, we cannot be sure of our models’ performance in racial and ethnic minority MSM. Additionally, we cannot be sure of our models’ applicability to populations and geographic areas with lower rates of methamphetamine and inhaled nitrites use.

Second, data collected from the PHSKC STD Clinic and Project Explore were dissimilar in important ways. Project Explore employed audio computer-assisted self-interview and collected behavioral and STD diagnosis data for 6-month period, while clinicians at the PHSKC STD Clinic collected behavioral data through face-to-face interviews, and sexual behavior questions focused on the prior 12 months. Also, data concerning bacterial STD were based on self-report and diagnoses on the day of HIV testing in the STD clinic, but reflected only self-reported history of bacterial STD in the prior 6 months in Project Explore. Misclassification would diminish model discrimination. For example, men in the STD clinic who practiced nonconcordant UAI might not acknowledge this behavior because of social desirability bias associated with face-to-face interviews. Therefore, the predictive models would overestimate the risk of HIV acquisition in men who did not report nonconcordant UAI in both the development and validation samples.

Prediction models are improved through repeated validation, augmentation, and examination in research and practice.^9,15,17 We present our prediction models as a step in the process of developing consistent guidelines for HIV prevention practice. Optimally, future work should augment and validate these prediction models in diverse settings and measure the acceptability and utility of prevention efforts guided by prediction models.^50,51 However, even without such additional research, we believe our prediction model can be useful in counseling MSM and in prioritizing intensified prevention efforts to the MSM at highest risk for HIV.