Derivation and Validation of an HIV Risk Prediction Score Among Gay, Bisexual, and Other Men Who Have Sex With Men to Inform PrEP Initiation in an STD Clinic Setting : JAIDS Journal of Acquired Immune Deficiency Syndromes

Secondary Logo

Journal Logo

Epidemiology

Derivation and Validation of an HIV Risk Prediction Score Among Gay, Bisexual, and Other Men Who Have Sex With Men to Inform PrEP Initiation in an STD Clinic Setting

Tordoff, Diana M. MPHa; Barbee, Lindley A. MD, MPHb,c; Khosropour, Christine M. PhD, MPHa,c; Hughes, James P. PhD, MSd; Golden, Matthew R. MD, MPHa,b,c

Author Information
JAIDS Journal of Acquired Immune Deficiency Syndromes: November 01, 2020 - Volume 85 - Issue 3 - p 263-271
doi: 10.1097/QAI.0000000000002438

Abstract

INTRODUCTION

HIV pre-exposure prophylaxis (PrEP) is a highly effective biomedical HIV prevention tool. When taken correctly, PrEP reduces the risk of HIV acquisition by over 90% in gay, bisexual, and other men who have sex with men (MSM), the population at highest risk of acquiring HIV in the United States.1 Although Centers for Disease Control and Prevention (CDC) guidelines suggest criteria that clinicians can use to identify persons to whom they should offer PrEP, the 2019 U.S. Preventive Services Task Force (USPSTF) PrEP recommendation states that no validated tools exist for identifying populations at greatest risk for HIV.2–4 Ensuring that PrEP reaches the populations at greatest risk for infection is critical to maximizing the intervention's public health impact.

Risk prediction tools can help clinicians calculate a patient's risk for a medical condition. Three published articles have evaluated risk prediction tools designed to predict the risk of future HIV acquisition among MSM in the United States. Our group in Seattle used data collected in our sexually transmitted disease (STD) clinic from 2001 to 2008 to develop a risk score that we subsequently used to inform local HIV testing and PrEP implementation guidelines.5,6 However, the data used to develop that score are now over a decade old and may not accurately predict MSM's contemporary HIV risk. Smith et al7 used data from 2 cohort studies conducted from 1998 to 2001 to develop another risk score, the HIV Incidence Risk Index for MSM (HIRI-MSM), which was instrumental in the development of national PrEP guidelines. However, like the prior Seattle score, the HIRI-MSM may no longer be valid. Finally, investigators at the LGBTQ Clinic in Los Angeles developed a more contemporary risk score that may be more discriminative than other published scores. This score relies on a relatively large number of predictors not typically collected in clinical settings, but is supported through an online tool that is accessible to patients.8,9

We used data collected in our STD clinic in Seattle from 2001 to 2015 to develop and validate HIV risk prediction scores for MSM, including a more contemporary score, that clinicians can use to counsel patients about their HIV risk and identify men to prioritize for PrEP initiation and frequent HIV testing.

METHODS

Study Population and Data Collection

We combined 2 clinical and public health surveillance data sources to form a retrospective cohort of MSM in Seattle, WA. The cohort comprised MSM who attended the Public Health—Seattle & King County (PHSKC) STD clinic and for whom we observed at least one negative HIV test result between October 1, 2001, and December 31, 2015. Seroconversions were defined as a subsequent HIV-positive test at the STD clinic or a positive test reported in Washington State's Enhanced HIV/AIDS Reporting System as of December 31, 2015. Staff at the Washington State Department of Health used a probabilistic record linkage algorithm to create a deidentified data set that links patient data from the PHSKC STD clinic to the enhanced HIV/AIDS Reporting System database.10 This research was deemed to be exempt from human subjects research status by Washington State and University of Washington Institutional Review Boards.

Predictors of HIV acquisition were derived from data collected as part of routine clinical care in the PHSKC STD clinic. This includes information on patient demographics, sexual behavior, self-reported sexually transmitted infection (STI) history, and substance use, as well as HIV/STI test results from the date of clinic visit. Sexual behavior questions were collected by clinicians through face-to-face interviews until 2010, after which the clinic instituted a computer-assisted self-interview for the collection of behavioral information. We have previously described our clinic's data collection procedures.11 All HIV testing was conducted in the PHSKC laboratory. The clinic used second-generation HIV enzyme immunoassay (EIA) until 2010 (Vironostika HIV-1 Microelisa System; bioMerieux, Durham, NC or rLAV Genetic System; Bio-Rad Laboratories, Hercules, CA), a third-generation EIA from 2010 to 2011 (Genetic Systems HIV1/2 Plus O EIA, Bio-Rad Laboratories, Redmond, WA), and fourth-generation EIA thereafter (Bio-Rad GS HIV Combo Ag/Ab EIA, Hercules, CA). Also, from 2003 to 2011, the clinic performed pooled HIV RNA testing on all MSM tested for HIV (OraQuick, OraSure Technologies Inc., Bethlehem, PA until 2013; INSTI, bioLytical Laboratories, Richmond, BC, after 2013). Throughout the study period, clinicians tested for gonorrhea and chlamydia at all exposed anatomical sites. Before 2011, the clinic used culture to diagnose rectal and pharyngeal gonorrhea and chlamydia. Since 2011, the clinic has used a nucleic acid amplification test, Aptima Combo 2 (Hologic, San Diego, CA), to diagnose extragenital gonorrhea and chlamydia. Syphilis was diagnosed by a combination of darkfield microscopy and rapid-plasma reagin testing, with Treponema pallidum particle agglutination assay used for confirmatory testing.

Risk Score Development

We split the cohort randomly 2:1 to create derivation and validation data sets, respectively, using the derivation sample to develop the predictive models and risk scores, and the validation data set to test the score's calibration and discrimination. We used Cox proportional hazards models with time-varying covariates and robust standard errors to estimate the hazard of testing positive for HIV associated with behavioral and clinical predictor variables ascertained during each clinic visit during which patients tested for HIV. The time-to-event was defined as the number of days from an STD clinic visit with a negative HIV test to the date of the first positive test for HIV. Patients could appear in the data set multiple times, corresponding to each negative HIV test result. All persons who did not test HIV-positive were censored 5 years after their last HIV-negative test or on December 31, 2015, the final day of the study period, to minimize the impact of potential out-migration on study outcomes. We chose the 5-year censoring period as the shortest timeframe that captured >90% of seroconversion events. The use of time-varying covariates allowed us to change patient risks based on serial clinical assessments, as occurs in clinical practice. Our models initially included all a priori defined predictors of future HIV infection. We then used a stepwise procedure akin to the Akaike information criterion to determine which variables to retain in the model, removing one predictor at a time. We choose the model with the lowest Akaike information criterion as the final best-fit models.12

Based on the literature, we considered the following potential predictor variables: condomless receptive anal intercourse (CRAI); condomless insertive anal intercourse (CIAI); any HIV-positive sex partners in the prior 12 months; any unknown status sex partners in the prior 12 months; history of bacterial STIs (eg, gonorrhea and chlamydia) in the past year and diagnosed at the current visit by anatomical site (rectal, urethral, and pharyngeal); syphilis; genital herpes diagnosis based on culture/direct fluorescent-antibody of a genital lesion showing herpes simplex virus 2 (HSV-2), positive serology for HSV-2, or self-reported genital herpes (herpes diagnosis was defined as ongoing for all periods following the first diagnosis); methamphetamine or “popper” (inhaled amyl nitrates) use in the prior 12 months; age; race/ethnicity; and number of sex partners in the prior 12 months. All predictors met the proportional hazards assumptions.

To obtain score weights for each predictor included in the final model, we multiplied the Cox proportional hazard beta coefficients by 10 and rounded to the nearest integer. Integer weights were then summed to create a risk score for each individual. We developed 3 risk scores. The first score included all data from 2001 to 2015 (full model), and a second included only data collected from 2011 to 2015 (modern model), the period during which extragenital nucleic acid amplification test testing came into use and the early period of PrEP availability.13 The third (Seattle PrEP Score) simplified the modern model by combining related variables into a composite predictor to include only 4 items, and predictors were assigned equal score weights.

Risk Score Evaluation

We used a standard internal validation approach in which the validation sample was used to test the calibration and discrimination of each risk score.14,15 This approach was chosen because its simplicity allowed us to use the same validation data set (limited to the appropriate time period) for all 3 risk scores. Using the score weights developed on the derivation sample, we calculated a risk score for each individual in the validation data set. Model calibration and discrimination was graphically determined by plotting the receiver operating characteristic (ROC) curve for each model. Analytically, we evaluated the area under the ROC curve (AUC) and the sensitivity, specificity, 1-year cumulative incidence, and number needed to treat (NNT) for both derivation and validation samples using different cutoffs for each score. In this context, the NNT is the number of individuals who would need to initiate PrEP to avert one incident HIV infection in the subsequent year, and assumes 100% PrEP effectiveness.

We compared the performance of the Seattle PrEP Score to previously published HIV risk prediction scores for which we had comparable data using the entire cohort (derivation and validation samples) between 2011 and 2015: the Menza score, HIRI-MSM, San Diego Early Test (SDET), and current CDC PrEP guidelines.5,7,16,17 For each score, we used score cutoffs recommended in the original study. Variables were modified from the original model where noted, depending on availability of measures in the PHSKC STD clinic data. Although the SDET was developed by Hoenigl et al to predict early acute HIV infection among persons being tested for HIV, not future incident infection, we compare it with our model because prior studies have evaluated it as an HIV risk score.8,16,18 We were unable to compare our model to the score developed by Beymer et al8 because we did not have data on history of intimate partner violence and race/ethnicity of sex partners, which were predictors in that model. We chose not to compare our score to symptom-based risk models,19,20 models that were are not MSM-specific,21–24 or models developed for non-U.S. populations.25–32 All analyses were performed using Stata version 14.0 (Stata Corps, TX).

RESULTS

Between 2001 and 2015, 16,448 MSM tested HIV-negative at the PHSKC STD clinic during 56,722 visits. MSM participants were predominantly White (66%), and had a mean age of 32 years (interquartile range = 26–42). Randomly splitting the cohort 2:1 into a derivation sample (N = 37,814 visits) and a validation sample (N = 18,908 visits) ensured similar distributions of demographic, clinical, and behavioral characteristics between samples (Table 1). Compared to visits in 2001–2010, men with visits in 2011–2015 were significantly more likely to report using poppers, having an HIV-positive sex partner, condomless anal sex, or history of a bacterial STI. They were also more likely to be diagnosed with extragenital chlamydia or gonorrhea at their clinic visit (see Supplemental Digital Content, https://links.lww.com/QAI/B507). Participants contributed a total of 172,854 person-years of passive follow-up and 640 seroconversions events (3.9%) over the 14.3-year follow-up period. The annual cumulative incidence of HIV diagnosis was 1.17% [95% confidence interval (CI): 1.08% to 1.27%].

TABLE 1. - Baseline Characteristics of Gay, Bisexual, and Other Men Who Have Sex With Men, 2001–2015
Characteristic Derivation Sample Validation Sample P
N individuals 13,527 9234
N visits 37,814 18,908
Person-time (yr) 114,064 56,890
Days from baseline clinic visit to HIV diagnosis, next clinic visit, or censoring, median (IQR) 1302 (287–1825) 1289 (283–1825) 0.667
HIV diagnoses, no. (%) 440 (1.2) 200 (1.1) 0.261
HIV incidence, per 100 person-yr, (95% CI) 1.20 (1.09 to 1.32) 1.10 (0.96 to 1.27) 0.171
Age, median (IQR) 33 (26–42) 33 (26–42) 0.335
Race/ethnicity, no. (%)
 White 24,693 (65.3) 12,397 (65.6) 0.264
 Black 4168 (11.0) 1994 (10.6)
 Asian 2123 (5.6) 1128 (6.0)
 Hispanic 1907 (5.0) 919 (4.9)
 Native American/Alaskan Native 452 (1.2) 221 (1.2)
 Multiracial/Other/Unknown 4471 (11.8) 2249 (11.9)
STI diagnoses, no. (%)*
 Urethral gonorrhea 1354 (3.6) 676 (3.6) 0.902
 Rectal gonorrhea 1226 (3.2) 648 (3.4) 0.463
 Pharyngeal gonorrhea 1332 (3.5) 648 (3.4) 0.878
 Urethral chlamydia 1189 (3.1) 696 (3.2) 0.761
 Rectal chlamydia 1319 (3.5) 648 (3.4) 0.830
 Pharyngeal chlamydia 158 (0.4) 74 (0.4) 0.895
 Syphilis 891 (2.4) 426 (2.3) 0.642
 Herpes 5733 (15.2) 2781 (14.7) 0.154
Self-reported history of STI
 Gonorrhea 6389 (16.9) 3181 (16.8) 0.828
 Chlamydia 5174 (13.7) 2588 (13.7) 0.988
 Syphilis 1893 (5.0) 939 (5.0) 0.837
Sexual behavior
 No. of sex partners, median (IQR) 5 (2–10) 5 (2–10) 0.158
 Any condomless anal intercourse, no. (%) 19,381 (51.3) 9731 (51.5) 0.635
 Any CRAI, no. (%) 14,966 (39.6) 7460 (39.45) 0.897
 Any CIAI, no. (%) 16,324 (43.2) 8211 (43.4) 0.707
 Any HIV-infected sex partners, no. (%) 4836 (12.8) 2464 (13.0) 0.608
 Any anonymous partners, no. (%) 7700 (20.4) 3977 (21.0) 0.063
Substance use
 Methamphetamine 1968 (5.2) 984 (5.2) 0.999
 Inhaled nitrate (ie, “poppers”) 4182 (11.6) 2073 (11.0) 0.731
Randomized at the visit level. There are a total of 16,448 people.
*Bacterial STIs are those diagnosed at the baseline clinical visit; Herpes diagnosis includes herpes seropositivity, clinical diagnosis (FA, PCR, culture) or self-report.
Prior 12 months.
IQR, interquartile range.

Risk Score Derivation and Validation

The full prediction model included 13 predictors (Table 2). The modern prediction model, which restricted data to 2011–2015, included only 7 predictors. The 3 strongest predictors of incident HIV in the full model were retained in the modern model: methamphetamine use, syphilis diagnosis, and any CRAI in the past year. The Seattle PrEP Score collapsed the 4 predictors related to bacterial STIs in the modern model into a composite predictor variable (ie, current gonorrhea or syphilis diagnosis, or self-reported gonorrhea or syphilis in the past year), and also included methamphetamine use, CRAI, and 10 or more partners in the past year.

TABLE 2. - Multivariate Cox Proportional Hazards Regression Models of Predictors Associated With Incident HIV Infection Among MSM in the Derivation Sample
Predictor Variable Full Model, 2001–2015 Modern Model, 2011–2015 Seattle PrEP Score, 2011–2015
HR (95% CI) β P Point HR (95% CI) β P Point HR (95% CI) β P Point
Methamphetamine use* 2.16 (1.64 to 2.86) 0.77 <0.001 8 2.35 (1.48 to 3.74) 0.86 <0.001 9 2.51 (1.59 to 3.95) 0.92 <0.001 1
Syphilis diagnosis 2.00 (1.26 to 3.16) 0.69 0.003 7 1.32 (0.56 to 1.31) 0.28 0.524 3
CRAI* 1.68 (1.23 to 2.32) 0.52 0.001 5 1.77 (1.04 to 3.02) 0.57 0.035 6 2.72 (1.65 to 4.50) 0.99 <0.001 1
Self-reported history of chlamydia* 1.64 (1.28 to 2.09) 0.49 <0.001 5
Self-reported history of syphilis* 1.61 (1.16 to 2.23) 0.48 0.004 5 2.14 (1.36 to 3.35) 0.76 0.001 8
10 or more sex partners* 1.49 (1.21 to 1.85) 0.40 <0.001 4 1.78 (1.21 to 2.62) 0.58 0.003 6 1.78 (1.22 to 2.59) 0.57 0.003 1
Inhaled nitrate (ie, “poppers”)* 1.46 (1.13 to 1.88) 0.38 0.004 4
Age 32 yrs or younger 1.44 (1.18 to 1.76) 0.36 <0.001 4
Rectal chlamydia diagnosis 1.40 (0.97 to 2.03) 0.34 0.075 3
Rectal gonorrhea diagnosis 1.36 (0.92 to 1.99) 0.30 0.119 3 1.27 (0.72 to 2.24) 0.24 0.407 2
Self-reported history of gonorrhea* 1.26 (1.00 to 1.58) 0.23 0.047 2 1.61 (1.12 to 2.30) 0.47 0.010 5
CIAI* 1.26 (0.96 to 1.65) 0.23 0.099 2
Any HIV-positive sex partners* 1.25 (0.96 to 1.62) 0.22 0.100 2
Gonorrhea or syphilis diagnosis or self-reported history* 1.97 (1.36 to 2.84) 0.68 <0.001 1
Our models initially included all a priori defined predictors. We then used a stepwise procedure to determine which variables to retain in the model, removing one predictor at a time. We choose the model with the lowest Akaike information criterion as the final best-fit models. The Akaike information criterion was 7278.8 for the Full Model, 2120.8 for the Modern Model, and 2166.4 for the Seattle PrEP Score. Points are calculated as the p value multiplied by 10 and rounded to the nearest integer. In the Seattle PrEP score, we assigned a uniform point value of 1 to each risk factor for simplified implementation.
*Prior 12 months.
Diagnosed at the baseline STD clinic visit.
Composite variable that combines the 4 STI-related variables included in the Modern Model.
HR, hazard ratio.

Model discrimination as measured by the AUC was 0.73 (95% CI: 0.71 to 0.76) for the full model, 0.69 (95% CI: 0.65 to 0.74) for the modern model, and 0.69 (95% CI: 0.64 to 0.72) for the Seattle PrEP Score (Table 3). For the full model, using a score cutoff ≥11 was chosen to identify MSM at substantial risk of HIV acquisition. This corresponded to 76.1% sensitivity and 63.0% specificity. One-year cumulative incidence was 0.4% for scores 0–10 and 2.2% for scores ≥11. The full model was modestly calibrated, as indicated by the ROC curves in Figure 1, as well as the comparability of AUC, sensitivity, specificity, and incidence estimated in the derivation and validation samples (Table 3). These measures were all slightly lower in the validation data set. Sensitivity, specificity, and incidence for all possible score cutoffs are provided in the Supplemental Digital Content, https://links.lww.com/QAI/B507.

TABLE 3. - Overall Discrimination of Risk of Incident HIV Infection in the Full Model (2001–2015), Modern and Simple Models (2011–2015) in Both the Derivation and Validation Samples
Model and Sample AUC (95% CI) Sensitivity, % Specificity, % High Risk for HIV Acquisition* Low Risk for HIV Acquisition*
Person-Years Seroconversions, No. Incidence Rate (95% CI) NNT (95% CI) Person-Years Seroconversions, No. Incidence Rate (95% CI) NNT (95% CI)
Full model
 Derivation 0.73 (0.71 to 0.76) 76.1 63.0 14,853 321 2.16 (1.94 to 2.41) 46 (41 to 52) 27,309 100 0.37 (0.30 to 0.45) 273 (224 to 332)
 Validation 0.72 (0.68 to 0.75) 73.0 63.2 9495 142 1.50 (1.27 to 1.76) 67 (57 to 79) 18,988 52 0.27 (0.21 to 0.36) 365 (278 to 479)
Modern model
 Derivation 0.69 (0.65 to 0.74) 65.9 67.6 4233 87 2.06 (1.67 to 2.34) 49 (39 to 60) 11,234 47 0.42 (0.31 to 0.56) 239 (180 to 318)
 Validation 0.61 (0.54 to 0.67) 50.7 66.8 2569 32 1.25 (0.88 to 1.76) 80 (57 to 114) 6732 30 0.45 (0.31 to 0.64) 224 (157 to 321)
Seattle PrEP score
 Derivation 0.69 (0.64 to 0.73) 62.3 69.6 3911 82 2.09 (1.69 to 2.60) 35 (27 to 46) 11,557 52 0.45 (0.34 to 0.59) 169 (136 to 211)
 Validation 0.60 (0.54 to 0.66) 46.3 69.0 2401 33 1.21 (0.84 to 1.74) 83 (58 to 119) 6899 29 0.48 (0.34 to 0.67) 209 (149 to 294)
*High risk is defined as individuals with a risk score of 11 or higher in the Full Model, a risk score of 7 or higher in the Modern Model, and a risk score of 2 or higher in the Seattle PrEP Score. The chosen score cutoffs for the Modern Model and Seattle PrEP score correspond to an individual having 2 or more risk factors.
Per 100 person-years.
The NNT is the number of individuals in each group who would need initiate and adhere to PrEP for one year to avoid one incidence HIV infection in the subsequent 1-year period. The NNT is calculated as 1/(Risk Difference), where the risk difference is equal to 1-year incidence, because we aim to achieve an HIV incidence of zero.
AUC, area under the receiver operating curve.

F1
FIGURE 1.:
Receiver Operating Characteristic (ROC) curves for derivation and validation samples.

Under the modern model, a score cutoff ≥7 had 65.9% sensitivity and 67.6% specificity for identifying persons with an incident HIV diagnosis. One-year cumulative incidence was 0.4% for scores 0–6 and 2.1% for scores ≥7. Under the Seattle PrEP Score, a score cutoff ≥2 had 62.3% sensitivity and 69.6% specificity. One-year cumulative incidence was 0.5% for scores 0–1 and 2.1% for scores ≥2. The modern model and Seattle PrEP Score were not well calibrated: the ROC, AUC, and sensitivity were significantly lower in the validation sample as compared to the derivation sample.

Because the modern model and the Seattle PrEP Score performed similarly, we provide a clinical implementation tool based on the Seattle PrEP Score, shown in Figure 2. One-year incidence was <0.5% among MSM with no points, 0.5%–1% among MSM with one point, 1%–2% among MSM with 2 points, 2%–5% among MSM with 3 points, and ≥5% among MSM with all 4 points. The NNT represents the number of MSM needed to initiate and adhere to PrEP for one year to avoid one incident HIV infection in the subsequent year, assuming PrEP is 100% effective in preventing HIV. The NNT for the groups defined above was 351, 145, 62, 29, and 13, respectively.

F2
FIGURE 2.:
Clinical implementation of the Seattle PrEP score to guide PrEP initiation among gay, bisexual, and other men who have sex with men. Individuals with 2 or more risk factors are at higher risk for HIV acquisition, and are recommended to initiate PrEP.

Comparison With Other Scores

When comparing the performance of previously published HIV risk prediction scores as well as current CDC recommendations for PrEP, all scores had comparable AUC (Table 4). The Menza score and CDC recommendations had the highest sensitivities (91.7% and 90.7%, respectively) at the cost of having very low specificities (13.3% and 34.3%, respectively). The Seattle PrEP Score and SDET were substantially more specific and had the lowest NNT (40 and 53, respectively).

TABLE 4. - Comparison With Alternative HIV Risk Prediction Models and PrEP Recommendations, Using Combined Data From the Derivation and Validation Samples, 2011–2015
Model Variables (Score Weight) AUC (95% CI) MSM Who Satisfy Criteria, % Sensitivity, % Specificity, % 1-yr NNT
Seattle PrEP score Methamphetamine use (1)
Any CRAI (1)
≥10 male sex partners (1)
Current diagnosis or self-reported history of GC/syphilis (1)
0.66 (0.62 to 0.69) 30.9 57.1 69.4 40
Menza score Methamphetamine use or inhaled nitrates (11)*
Current diagnosis or self-reported history of bacterial STI (4)
≥10 male sex partners (3)
CAI with HIV-positive or unknown status sex partner (1)
0.66 (0.62 to 0.70) 86.7 91.7 13.3 91
Smith's HIRI-MSM Age 18–28 (8), 29–40 (5), 41–48 (2)
>10 male sex partners (7), 6–10 male sex partner (5)*
CRAI with HIV-positive sex partner (10)* Any HIV-positive sex partners (6)*
Combination of CIAI with HIV-positive sex partners, and ≥5 sex partners (6)*
Methamphetamine use (5)*
Inhaled nitrates use (3)*
0.61 (0.57 to 0.65) 62.7 76.6 37.4 77
Hoenigl's SDET CAI with HIV-positive sex partner (3)
Combination of any CRAI and ≥5 male sex partners (3)
≥10 male sex partners (2)
Self-reported history of bacterial STI (2)
0.63 (0.59 to 0.67) 33.1 56.6 67.1 53
CDC 2018§ Any CAI (1)
Any HIV-positive sex partners (1)
Self-reported history of bacterial STI (1)*
Injection drug use in past 6 mo (1)
0.62 (0.60 to 0.65) 66.0 90.7 34.3 73
Menza and the Seattle PrEP score recommend a cutoff ≥2, Smith's HIRI-MSM recommends a cutoff of ≥10, and Hoenigl's SDET recommends a cutoff ≥5.
An exposure period of 12 months was used unless otherwise noted. We were unable to compare our model to Beymer et al8 due to lack of comparable variables in our data on history of intimate partner violence and race/ethnicity of sexual partners. We chose not to compare our score to symptom-based risk models,19,20 models that were are not MSM-specific,21–23 or developed for non-U.S. populations.25–30,32
*Original model used 6 months; here, we used 12 months.
Modification of the original model because number of HIV-positive partners was unavailable, as per Hoenigl.16 Smith originally specified >1 HIV-positive sex partners with a score of 8, versus exactly 1 HIV-positive sex partner with a score of 4.
Modification of the original model because number of HIV-positive partners was unavailable, as per Hoenigl.16 Smith originally specified 5 or more CIAI sex acts within the prior 6 months.
§The CDC recommends PrEP for a “gay or bisexual man who has had anal sex without using a condom or been diagnosed with an STD in the past 6 months,” who has an HIV-positive sex partner, or has injected any drugs in the past 6 months.
GC, gonorrhea; CAI, condomless anal intercourse; CIAI, condomless insertive anal intercourse.

DISCUSSION

We used clinical and behavioral data collected from MSM attending an STD clinic in Seattle and linked it to HIV surveillance data to develop and validate 3 risk scores predictive of future HIV acquisition. We believe that the Seattle PrEP Score, based on contemporary data that correspond to the era of treatment as prevention and the early period of PrEP availability, could be useful in helping clinicians and public health agencies identify men who would benefit from PrEP and frequent HIV testing.

Our findings are consistent with prior MSM risk score analyses. The 5 scores we evaluated, as well as one that we were not able to assess,8 contain common elements: All scores include condomless anal intercourse (some scores integrate this with having an HIV-positive partner, or number of partners), 5 include measures of bacterial STI history, and 4 include a measure of substance use. These commonalities highlight that these risks factors are consistently associated with HIV acquisition.

The different risk scores differ in their complexity, sensitivity, specificity, and the NNT with PrEP to avert an incident HIV infection. The Seattle PrEP Score, which includes only 4 criteria each of which is given equal weight, is the simplest score and, at least in our clinic population, has the lowest NNT (ie, requires treating the fewest persons to prevent an HIV infection). This may lend itself to easier implementation in clinical settings compared to previously published scores with complex scoring schemes and as many as 13 predictors.5,7,8,16 Although the Seattle PrEP Score merits further validation in other settings and using more contemporary data, we believe that clinicians should universally recommend PrEP—not just discuss or offer PrEP—to men with 2 or more of the score's criteria. Clinicians should discuss PrEP with all MSM, and engage in a more nuanced discussion of risk and patient preference among men with one point based on the score, a group at intermediate risk for future infection.

Strengths of this study include the large sample size, longitudinal design, long follow-up period, and high ascertainment of seroconversions through Washington State HIV surveillance. An additional strength of our analysis is the evaluation of a risk score using relatively contemporary data collected from 2011 to 2015. We observed differences in the models we developed using our full data set compared to only more contemporary data. Several behaviors, including popper use, having an HIV-positive sex partner, and chlamydia, were not predictive of HIV acquisition in the 2011–2015 period. The differences in predictors of HIV acquisition may be reflective of observed temporal changes in sexual behavior among MSM, changes in STI testing technologies, and changes in the level of viral suppression among MSM in King County.33 Therefore, the contemporary scores presented in this study may be preferable to risk prediction scores developed in-whole or in-part using pre-2011 data.5,7,8,16

This study has several limitations. First and foremost, the generalizability of our risk score is uncertain. Our study population was predominantly White, and race/ethnicity were not significant predictors of HIV acquisition despite known racial disparities in HIV incidence in Seattle and elsewhere in the United States.34 Two studies have demonstrated that HIV risk prediction tools that emphasize sexual behavior or methamphetamine use perform poorly in majority Black MSM populations in the Southern United States and Chicago.18,35 Although sensitivity analyses did not show statistical differences in the Seattle PrEP Score's performance among racial and ethnic minority MSM (Supplemental Digital Content, https://links.lww.com/QAI/B507), we were underpowered to detect such differences, and our findings related to this should be interpreted with caution. Given the absence of data validating the Seattle PrEP Score or other risk prediction scores in diverse samples of MSM, we believe that the best use of these scores is in identifying persons at elevated risk for HIV acquisition, a conclusion aligned with USPSTF recommendations.4 Particularly among Black MSM, a low score cannot be used to define a patient as being at low risk for HIV, and our STD clinic seeks to preferentially provide PrEP to Black and Latino MSM as part of an effort to addressing the disparate impact of HIV in minority populations. There is also uncertainty in the generalizability of this score to clinical settings other than STD clinics.

In addition, the models we evaluated using only data collected from 2011 to 2015 were not well calibrated between the derivation and validation samples. Our choice of validation method, although a standard approach, is inefficient relative to other methods that use the full data set (eg, bootstrapping) or those that do not randomly split the data (eg, temporal and external validation).14,15 Thus, our calibration results may, in part, also be a result of having fewer seroconversions during the shorter, more contemporary period and lower precision. Because our model was developed using PHSKC STD clinic data, we anticipated that our model would perform better in our study population than other models, a limitation that highlights the need for validation in other settings, particularly in more diverse populations. Our study also relied on passive follow-up of STD clinic patients, and it is possible that we did not capture some seroconversions if individuals moved out of Washington State before censoring. Last, the AUC we report suggests that our scores are only moderately discriminating. Although better risk scores would certainly be preferable, the AUC we observed is similar to that of CHADSVasc2, which is widely used to assess stroke risk and determine when to prescribe anticoagulation in patients with atrial fibrillation (AUC of 0.66), but lower than the Framingham score for cardiovascular disease (AUC of 0.76 in men, and 0.79 in women).36,37

In conclusion, we believe that the Seattle PrEP Score, a simple 4-item tool, can help clinicians and public health workers identify MSM at high risk for HIV acquisition. Although the performance of the score among racial minority MSM and in areas outside of the West Coast of the United States requires additional study, existing data support the conclusion that clinicians should recommend PrEP initiation to MSM with 2 or more of the criteria included in the score, and that men with these risks should test for HIV and other STIs on a quarterly basis.

ACKNOWLEDGMENTS

The authors thank all the Public Health—Seattle & King County STD Clinic patients and clinicians; Amy Bennet, MPH, who performed probabilistic record linkage, and Tom Jaenicke, MPH, at the Washington Department of Health.

REFERENCES

1. Riddell J, Amico KR, Mayer KH. HIV preexposure prophylaxis: a review. JAMA. 2018;319:1261–1268.
2. Chen A, Dowdy DW. Clinical effectiveness and cost-effectiveness of HIV pre-exposure prophylaxis in men who have sex with men: risk calculators for real-world decision-making. PLoS One. 2014;9:e108742.
3. Chou R, Evans C, Hoverman A, et al. Preexposure prophylaxis for the prevention of HIV infection: evidence report and systematic review for the US preventive Services Task Force. JAMA. 2019;321:2214.
4. Owens DK, Davidson KW, Krist AH, et al. Preexposure prophylaxis for the prevention of HIV infection: US preventive Services Task Force recommendation statement. JAMA. 2019;321:2203.
5. Menza TW, Hughes JP, Celum CL, et al. Prediction of HIV acquisition among men who have sex with men. Sex Transm Dis. 2009;36:547–555.
6. Golden MR, Lindquist S, Dombrowski JC. Public health–seattle & king county and Washington state department of health preexposure prophylaxis implementation guidelines, 2015. Sex Transm Dis. 2016;43:264–265.
7. Smith DK, Pals SL, Herbst JH, et al. Development of a clinical screening Index predictive of incident HIV infection among men who have sex with men in the United States. J Acquir Immune Defic Syndr. 2012;60:421–427.
8. Beymer MR, Weiss RE, Sugar CA, et al. Are Centers for disease Control and prevention guidelines for preexposure prophylaxis specific enough? Formulation of a personalized HIV risk score for pre-exposure prophylaxis initiation. Sex Transm Dis. 2017;44:49–57.
9. PrEP'd AF. Los Angeles LGBT Center. Available at: https://prephere.org/. Accessed July 25, 2019.
10. Avoundjian T, Dombrowski J, Sadinle M. Record linkage for public health action: a comparison of matching algorithms. In: Joint Statistical Meetings. Denver, CO; 2019. Available at: https://ww2.amstat.org/meetings/jsm/2019/onlineprogram/AbstractDetails.cfm?abstractid=304597. Accessed July 25, 2019.
11. Khosropour CM, Dombrowski JC, Swanson F, et al. Trends in serosorting and the association with HIV/STI risk over time among men who have sex with men. J Acquir Immune Defic Syndr. 2016;72:189–197.
12. Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–387.
13. Barbee LA, Dombrowski JC, Kerani R, et al. Effect of nucleic acid amplification testing on detection of extragenital gonorrhea and chlamydial infections in men who have sex with men sexually transmitted disease clinic patients. Sex Transm Dis. 2014;41:168–172.
14. Cowley LE, Farewell DM, Maguire S, et al. Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature. Diagn Progn Res. 2019;3:1–23.
15. Altman DG, Vergouwe Y, Royston P, et al. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:1432–1435.
16. Hoenigl M, Weibel N, Mehta SR, et al. Development and validation of the San Diego Early Test Score to predict acute and early HIV infection risk in men who have sex with men. Clin Infect Dis. 2015;61:468–475.
17. HIV Basics: PrEP. Centers for Disease Control and Prevention. Available at: https://www.cdc.gov/hiv/basics/prep.html. Accessed February 4, 2019.
18. Jones J, Hoenigl M, Siegler AJ, et al. Assessing the performance of 3 human immunodeficiency virus incidence risk scores in a cohort of Black and white men who have sex with men in the South. Sex Transm Dis. 2017;44:297–302.
19. Lin TC, Gianella S, Tenenbaum T, et al. A simple symptom score for acute human immunodeficiency virus infection in a san Diego community-based screening program. Clin Infect Dis. 2018;67:105–111.
20. Dijkstra M, de Bree GJ, Stolte IG, et al. Development and validation of a risk score to assist screening for acute HIV-1 infection among men who have sex with men. BMC Infect Dis. 2017;17:425.
21. Haukoos JS, Lyons MS, Lindsell CJ, et al. Derivation and validation of the denver human immunodeficiency virus (HIV) risk score for targeted HIV screening. Am J Epidemiol. 2012;175:838–846.
22. Hsieh Y-H, Haukoos JS, Rothman RE. Validation of an abbreviated version of the Denver HIV Risk Score for prediction of HIV infection in an urban ED. Am J Emerg Med. 2014;32:775–779.
23. Boileau C, Bruneau J, Al-Nachawati H, et al. A prognostic model for HIV seroconversion among injection drug users as a tool for stratification in clinical trials. J Acquir Immune Defic Syndr. 2005;39:489–495.
24. Marcus JL, Hurley LB, Krakower DS, et al. Use of electronic health record data and machine learning to identify candidates for HIV pre-exposure prophylaxis: a modelling study. Lancet HIV. 2019;6:e688–e695.
25. Wahome E, Fegan G, Okuku HS, et al. Evaluation of an empiric risk screening score to identify acute and early HIV-1 infection among MSM in Coastal Kenya. AIDS. 2013;27:2163–2166.
26. Wahome E, Thiong’o AN, Mwashigadi G, et al. An empiric risk score to Guide PrEP targeting among MSM in coastal Kenya. AIDS Behav. 2018;22:35–44.
27. Kahle EM, Hughes JP, Lingappa JR, et al. An empiric risk scoring tool for identifying high-risk heterosexual HIV-1–Serodiscordant couples for targeted HIV-1 prevention. J Acquir Immune Defic Syndr. 2013;62:339–347.
28. Yin L, Zhao Y, Peratikos MB, et al. Risk prediction score for HIV infection: development and internal validation with cross-sectional data from men who have sex with men in China. AIDS Behav. 2018;22:2267–2276.
29. Balkus JE, Brown E, Palanee T, et al. An empiric HIV risk scoring tool to predict HIV-1 acquisition in african women. J Acquir Immune Defic Syndr. 2016;72:333–343.
30. Balkus JE, Brown ER, Palanee-Phillips T, et al. Performance of a validated risk score to predict HIV-1 acquisition among African women participating in a trial of the dapivirine vaginal ring. J Acquir Immune Defic Syndr. 2017;77:1.
31. Pintye J, Drake AL, Kinuthia J, et al. A risk assessment tool for identifying pregnant and postpartum women who may benefit from pre-exposure prophylaxis (PrEP). Clin Infect Dis. 2016;64:ciw850.
32. Wand H, Reddy T, Naidoo S, et al. A simple risk prediction algorithm for HIV transmission: results from HIV prevention trials in KwaZulu natal, South Africa (2002–2012). AIDS Behav. 2018;22:325–336.
33. HIV/AIDS Epidemiology Report 2018. Vol 87. Available at: https://www.kingcounty.gov/depts/health/communicable-diseases/hiv-std/patients/∼/media/depts/health/communicable-diseases/documents/hivstd/2018-hiv-aids-epidemiology-annual-report.ashx. Accessed May 9, 2019.
34. Golden MR, Bennett AB, Dombrowski JC, et al. Achieving the goals of the national HIV/AIDS strategy. Sex Transm Dis. 2016;43:269–276.
35. Lancki N, Almirol E, Alon L, et al. Preexposure prophylaxis guidelines have low sensitivity for identifying seroconverters in a sample of young Black MSM in Chicago. AIDS. 2018;32:383–392.
36. Friberg L, Rosenqvist M, Lip GYH. Evaluation of risk stratification schemes for ischaemic stroke and bleeding in 182 678 patients with atrial fibrillation: the Swedish Atrial Fibrillation cohort study. Eur Heart J. 2012;33:1500–1510.
37. D'Agostino RB, Vasan RS, Pencina MJ, et al. General cardiovascular risk profile for use in primary care: the Framingham heart study. Circulation. 2008;117:743–753.
Keywords:

HIV; MSM; risk prediction; PrEP; incidence

Supplemental Digital Content

Copyright © 2020 Wolters Kluwer Health, Inc. All rights reserved.