Development of a Clinical Screening Index Predictive of Incident HIV Infection Among Men Who Have Sex With Men in the United States : JAIDS Journal of Acquired Immune Deficiency Syndromes

Secondary Logo

Journal Logo

Epidemiology and Prevention

Development of a Clinical Screening Index Predictive of Incident HIV Infection Among Men Who Have Sex With Men in the United States

Smith, Dawn K. MD, MS, MPH; Pals, Sherri L. PhD; Herbst, Jeffrey H. PhD; Shinde, Sanjyot PhD; Carey, James W. PhD, MPH

Author Information
JAIDS Journal of Acquired Immune Deficiency Syndromes: August 1, 2012 - Volume 60 - Issue 4 - p 421-427
doi: 10.1097/QAI.0b013e318256b2f6
  • Free



To achieve the vision of the National HIV/AIDS Strategy that the United States becomes “a place where new HIV infections are rare,”1 substantial focus must be placed on engaging more men who have sex with men (MSM) in prevention activities anticipated to have high public health impact. In 2009, MSM (with or without injection drug use) accounted for 64% of estimated incident HIV infections. Among adolescent and young adult African American MSM aged 13–29 years, the number of new HIV diagnoses increased 12% annually between 2006 and 2009.2

Recent evidence demonstrates that daily oral preexposure prophylaxis (PrEP) with a pill containing tenofovir and emtricitabine substantially reduces the risk of HIV acquisition by MSM when medication adherence is high and when PrEP is provided in combination with risk-reduction counseling, condom provision, and regular HIV and sexually transmitted infection (STI) testing (with treatment when indicated).3 Models strongly suggest that in the United States, PrEP will have the highest impact and cost-effectiveness when it is targeted to MSM at significant risk for acquiring HIV infection.4,5

Unlike most behavioral and structural interventions with proven efficacy in reducing HIV risk behaviors among MSM,6–8 PrEP medications must be prescribed by a licensed clinician, and so specific implementation challenges for delivery of PrEP-related services in busy health care settings are called into play. Clinicians need tools to help them rapidly assess which subset of their MSM patients may be most appropriate to discuss PrEP. These tools can also help assess the need for other intensive HIV prevention activities when sexual histories indicate a significant risk of acquiring HIV infection.

A variety of brief tools are widely used in primary care settings to screen for alcohol dependence (CAGE,9 AUDIT10), cognitive mental status (MMSE11), depression,12 suicide risk,13 and other conditions. These tools identify patients in need of further evaluation for clinical diagnosis to guide treatment plans. To support the introduction and broader implementation of PrEP, a brief screening index is needed by clinicians to help them identify and screen their MSM patients to systematically determine whether they may be at significant risk of HIV acquisition.


We identified 2 existing large datasets from HIV prevention trials that enrolled HIV-uninfected MSM in the United States, collected both risk behavior data and HIV test results at standard intervals (every 6 months), and that had at least 100 HIV seroconversions over the course of the trial. With data from the VAXGEN 00414,15 and Project EXPLORE16 trials, in each dataset, we modeled which combination of reported risk behaviors at each visit best predicted HIV status at the subsequent study visit (ie, predicting short-term seroconversion risk). VAXGEN 004 data were used to develop the risk index, and data from the Project EXPLORE trial for a validation analysis.

The VAXGEN 004 trial enrolled 4643 HIV-uninfected MSM from 57 US sites during 1998–1999 into a randomized placebo-controlled trial of a recombinant HIV-1 envelope glycoprotein subunit (rgp120) vaccine that proved ineffective at 36 months of follow-up. We excluded 129 participants who reported injection drug use at any point before or during the trial and 146 participants whose only visit was the baseline visit. Self-reported risk behaviors, including sexual activity and alcohol and drug use, and occurrence of STIs were assessed by use of standard interviewer-administered questionnaires at baseline and every 6 months thereafter. HIV-1 status was determined at each study visit by a standard HIV-1 enzyme-linked immunosorbent assay; enzyme immunoassay–positive tests underwent confirmatory immunoblot assay; and for those confirmed antibody positive at any visit, a stored serum specimen from the prior study visit was tested by a highly sensitive and specific nucleic acid–based amplification test. There were 320 HIV infections in both arms combined; including only non-IDU US participants.

Project EXPLORE enrolled 4,295 HIV-uninfected MSM from 6 US cities during 1999–2001 into a randomized, controlled behavioral intervention trial in which the active arm did not significantly reduce HIV incidence compared with the control intervention arm. At baseline and follow-up visits every 6 months, trained interviewers collected self-reported history of STIs and use of antiretroviral postexposure prophylaxis. Audio-computer–assisted self-interviewing was used to collect sexual and substance-use behaviors. At each visit, blood samples were tested for HIV antibodies by enzyme-linked immunosorbent assay; if positive, retested in duplicate, and repeatedly reactive samples were confirmed by western-blot assay or immunofluorescence assay. There were 259 HIV infections in both arms combined; 171 infections remained after dropping from the analysis; 745 injection drug users; and 182 participants whose only visit was the baseline visit.

For both trial datasets, we included MSM in both the active and control arms of the trials. We feel this is appropriate for 2 reasons as follows: (1) our analysis was designed to determine the predictive capability of combinations of reported risk behaviors on HIV incidence regardless of which intervention was received, and (2) in both trials, statistically significant differences in HIV incidence were not found when comparing the intervention and control arms.

A review of the questionnaires used in VAXGEN 004 was conducted to select questions about sexual behaviors, noninjection drug use, and STI diagnosis; all factors that have been associated with HIV incidence in many prior studies. We limited analysis to variables drawn from questions that were answerable at a single clinical encounter (eg, self-report of STI diagnoses rather than laboratory test results). The Project EXPLORE questionnaire included comparable variables to the VAXGEN 004 questionnaire and so was used as a validation dataset.

Candidate variables were age at visit; sexual behavior in the 6 months before the visit (ie, total number of male sex partners, total number of HIV+ male sex partners, number of times had unprotected receptive anal sex, and number of times had unprotected insertive anal sex—each by reported HIV-negative, HIV-positive, or unknown status of partner); noninjection substance use in the 6 months before the visit (ie, poppers, amphetamines, hallucinogents, cocaine, or sildenafil); any self-reported STI; and whether the person resided in one of the top 10 metropolitan statistical areas in the United States ranked by HIV prevalence.

Generalized estimating equations were used to fit logistic regression models to adjust for over-time correlation in variables. Continuous variables were categorized anticipating a need to provide for ease of administration. We used a bootstrap and backward elimination procedure to select variables for the final model. For each bootstrap sample drawn (with replacement) from the original dataset, we fit a full model with all of the candidate variables, then removed variables one at a time, starting with the least significant variable. This procedure stopped when all variables remaining in the model were significant (P < 0.05). We repeated the bootstrap and backward elimination procedure 1000 times.17,18

To obtain point scores for the index from final model variables, we multiplied the regression coefficients for each by 10 and rounded to the nearest integer.19 We then summed the point values for all variables in the model to get a total high-risk MSM index score and assessed different score cutoffs by computing sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve. The same procedures were used to validate the index based on Project EXPLORE data.


Our study used data from 4386 eligible VAXGEN participants. In regard to racial/ethnic composition, 86.5% were non-Hispanic white, 5.9% Hispanic, 3.8% non-Hispanic black, 1.6% Asian/Pacific Islander, and 2.3% classified as “other.” The Project EXPLORE subsample included 3368 men. Of these, 74.6% were non-Hispanic white, 14.0% Hispanic, 5.6% non-Hispanic black, 2.6% Asian/Pacific Islander, and 3.1% “other.” MSM participants in VAXGEN were assessed at 24,391 visits over time, whereas those in Project EXPLORE were assessed at 15,582 visits. During each visit, men were tested for possible HIV seroconversion. They also were asked about their drug and sex behaviors during the 6 months before the visit; the values for these variables could change at any visit. Table 1 shows visit-level characteristics of the men in both studies for age, and 2 drug use, and 4 sex behavior variables.

Characteristics of MSM From VAXGEN 004 (1998–1999) and Project EXPLORE (1999–2001), at Visits Included in Data Analysis, United States*

We compared the area under the ROC curve for models containing all candidate variables to models containing variables found significant in at least 70%, 60%, 40%, and 30% of bootstrap samples. We chose a final model that included variables significant in ≥30% of the bootstrap samples because it increased the area under the ROC curve substantially over models with fewer variables but was still parsimonious compared with the model containing all of the candidate variables. Areas under the ROC curve for models with variables found significant in 70%, 60%, 40%, and 30%, and all of the bootstrap samples were 0.717, 0.730, 0.735, 0.743, and 0.749, respectively. Variables in the final model were age at visit; sexual behavior in the 6 months before the visit (ie, total number male sex partners, total number HIV+ male sex partners, number times had unprotected receptive anal sex with any HIV status partner, number of times had unprotective insertive anal sex with a reported HIV-positive partner); and noninjection drug-use behavior in the 6 months before the visit [ie, amphetamine and amyl nitrate (poppers) use] (Table 2). For ease of administration, we then combined 3 variables with similar scores (unprotected receptive anal sex with HIV-positive, HIV-negative, and unknown status) into a single variable indicating unprotected receptive anal sex with a male partner of any HIV serostatus. Reanalysis with the combined variable yielded point values and an area under the curve very similar to the original analysis (data not shown). The area under the curve for the final model was 0.738 for the VAXGEN 004 sample (Fig. 1A) and 0.721 for the Project EXPLORE sample (Fig. 1B).

Final Multivariate Model, VAXGEN 004 Sample, 1998–1999, United States
A, ROC curve for HIRI-MSM—VAXGEN 004. B, ROC curve for HIRI-MSM—Project EXPLORE.

Based on these results, HIV Incidence Risk Index for MSM (HIRI-MSM) scores were computed. We selected a cutoff of 10 to identify MSM at substantial risk of HIV infection. This cutoff was associated with a sensitivity of 84% (ie, of those who became HIV infected by the next study visit, 84% had an HIRI-MSM score of 10 or greater). Specificity at a cutoff of 10 was 45% (ie, of those who remained HIV negative at the next study visit, 45% had a HIRI-MSM score <10) (Table 3). Positive predictive value was 1.9% (i.e., of those with a HIRI-MSM score of 10 or more, 1.9% became HIV infected in the following interval) and negative predictive value was 99.5% (ie, of those with a HIRI-MSM score <10, 99.5% remained uninfected). When the final scoring system based on the VAXGEN 004 data was compared with results using the Project EXPLORE data, sensitivity and specificity were both slightly lower (81% and 38%, respectively) (Table 3). Positive predictive value and negative predictive value (NPV) were 1.2% and 99.5%

Sensitivity and Specificity of Cutoff Scores in VAXGEN 004 (1998–1999) and Project EXPLORE (1999–2001) datasets, United States

As a final step, we developed an administration format for clinicians, adapting the questions used in the VAXGEN 004 trial to elicit data for the final set of 6 variables for simplicity and adding scoring (Table 4).

HIRI-MSM—Administration and Scoring Format


We analyzed data from a large longitudinal study of initially uninfected MSM in the United States that contained results of periodic behavioral assessments and HIV test results to develop a brief index for clinical use to screen for risk of incident HIV infection. Weighted responses to 6 items yielded a scoring system that has good sensitivity although reducing the number of men who may need to have more extensive behavioral assessment regarding specific sexual and drug-use behaviors that can increase their risk for HIV acquisition. The results obtained in the development dataset were validated in a second longitudinal dataset.

To use HIRI-MSM, it will first be necessary for clinicians to identify the MSM in their male patient population. Several studies have shown that many MSM do not identify their sexual partner preference to their primary care providers,20,21 and that primary care providers infrequently ask sexual histories in a manner that facilitates disclosure of male–male sex.22,23 We recommend clinicians routinely ask a straightforward question of all adult and adolescent male (and female) patients as follows: “Do you have sex with men, women, or both?”. This question can be modified for specific situations. For example, with adolescents or other patients not known to be sexually active, it can be preceded by “…have you had sex with anyone in the past (time)…?”24

After determining that a male patient has sex with other men, the 7 HIRI-MSM questions can be administered and scored. Basic HIV/STI risk-reduction messages are appropriate for MSM with scores less than 10. MSM with scores of 10 or greater should have an in-depth assessment of their sexual behaviors and the context of their risk behavior (eg, does the patient have an HIV-infected regular sex partner or report substance/use abuse). This will allow clinicians to make decisions about provision of PrEP25or referral for other intensive HIV prevention interventions (eg, behavioral counseling or substance abuse treatment).

There are limitations to the analysis used to develop HIRI-MSM. The data were collected from a clinical vaccine trial population not the general population of MSM. Although the VAXGEN 004 trial participants were recruited from 57 sites across the United States, they are not representative of the population of MSM with incident HIV infections. For example, 86% of the participants were white, whereas African Americans accounted for 37% of estimated incident infections among MSM in the United States.2 In addition, prevalence of HIV infection in the local community or sexual network will increase (high prevalence) or decrease (low prevalence) the likelihood of encountering an HIV-infected sexual partner, given any specific risk behaviors. The VAXGEN 004 and EXPLORE trials are a decade old, and risk factors for HIV acquisition may have changed over time in different populations. However, more recent data appropriate to perform these analyses were not available. Due to concerns about limited prevalence data available for this analysis, and likely to be available to clinical care providers, HIRI-MSM could only use a crude prevalence variable (person resides in top 10 HIV prevalence metropolitan statistical area), which was not related to incidence in the Project EXPLORE dataset. However, when clinicians are able to estimate the community HIV prevalence based on local surveillance data, the prevalence in their clinic population or demographic and social network characteristics of an individual MSM patient that may be associated with higher or lower HIV risk of HIV acquisition (eg, race/ethnicity), they may be able to take them into account in contextualizing the HIRI-MSM score.

It is reassuring that the items identified in this analysis are similar to those identified in an analysis of a Seattle sexually transmitted disease clinic database used to develop a scoring system to predict the 4-year risk of HIV incidence.26 In that study, analyses were restricted to participants in the control arm although we included MSM in both the control and active arms.

The Centers for Disease Control and Prevention (CDC) has collaborated with Emory University to conduct user and provider acceptability testing of the HIRI-MSM delivered on a computer tablet.27 Both MSM and clinicians found it brief and easily completed either on a table or paper (Jeb Jone, MPH, oral communication, July 2011). The risk index could also be provided in multiple formats (paper, web application, smartphone application) for patient self-administration or clinician administration. The CDC is currently developing similar risk indices for use with injection drug users and heterosexual men and women.

We believe the HIRI-MSM can be useful to assist primary care, STI, and other providers to routinely ask questions that will help them identify unacknowledged MSM in their practice, and then quickly screen for factors that indicate a potential substantial risk of acquiring HIV infection. It is designed to take only 5–10 minutes to complete and can accommodate the documented time pressures under which providers deliver recommended-preventive screening and care.28 CDC estimates that 2.6% of US males aged 13 and older have engaged in male–male sex in any given year,29 but only a subset of these engage in sexual behaviors in contexts that place them at high risk of HIV acquisition. All MSM should be provided with regular HIV and STI testing and basic risk-reduction counseling and condom provision. The National HIV Behavioral Surveillance survey conducted in 2008 found that 14% of MSM reported receiving individual (and 7% group) behavioral preventive intervention care in the prior year.30 It is critical that the subset of MSM most at risk for becoming HIV infected be provided with high impact intensive behavioral (eg, specialized counseling) or biomedical (eg, PrEP) interventions if we are to lower the rate of new HIV infections occurring among them.


1. Office of National AIDS Policy. National HIV/AIDS Strategy for the United States: July 2010. 2010. Available at: Accessed October 20, 2010.
2. Prejean J, Song RG, Hernandez A, et al.. Estimated HIV Incidence in the United States, 2006–2009. Plos One. 2011;6(8):e17502.
3. Grant RM, Lama JR, Anderson PL, et al.. Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. N Engl J Med. 2010;363:2587–2599.
4. Paltiel AD, Freedberg KA, Scott CA, et al.. HIV preexposure prophylaxis in the United States: impact on lifetime infection risk, clinical outcomes, and cost-effectiveness. Clin Infect Dis. 2009;48:806–815.
5. Desai K, Sansom SL, Ackers ML, et al.. Modeling the impact of HIV chemoprophylaxis strategies among men who have sex with men in the United States: HIV infections prevented and cost-effectiveness. AIDS. 2008;22:1829–1839.
6. Centers for Disease Control and Prevention. Compendium of evidence-based HIV behavioral interventions. 2010; Available at: Accessed April 6, 2011.
7. Johnson WD, Diaz RM, Flanders WD, et al.. Behavioral interventions to reduce risk for sexual transmission of HIV among men who have sex with men. Cochrane Database Syst Rev. 2008;16(3):CD001230.
8. Herbst JH, Beeker C, Mathew A, et al.. The effectiveness of individual-, group-, and community-level HIV behavioral risk-reduction interventions for adult men who have sex with men: a systematic review. Am J Prev Med. 2007;32(4 suppl):S38–S67.
9. Chan AWK, Pristach EA, Welte JW. Detection by the CAGE of alcoholism or heavy drinking in primary care outpatients and the general population. J Subst Abuse. 1994;6:123–135.
10. Volk RJ, Steinbauer JR, Cantor SB, et al.. The alcohol use disorders identification test (AUDIT) as a screen for at-risk drinking in primary care patients of different racial/ethnic backgrounds. Addiction. 1997;92:197–206.
11. Crum RM, Anthony JC, Bassett SS, et al.. Population-based norms for the mini-mental state examination by age and educational level. JAMA. 1993;269:2386–2391.
12. Sharp LK, Lipsky MS. Screening for depression across the lifespan: a review of measures for use in primary care settings. Am Fam Physician. 2002;66:1001–1008.
13. Gaynes BN, West SL, Ford CA, et al.. Screening for suicide risk in adults: a summary of the evidence for the U.S. preventive services task force. Ann Intern Med. 2004;140:822–835.
14. Flynn NM, Forthal DN, Harro CD, et al.. Placebo-controlled phase 3 trial of a recombinant glycoprotein 120 vaccine to prevent HIV-1 infection. J Infect Dis. 2005;191:654–665.
15. Bartholow BN, Buchbinder S, Celum C, et al.. HIV sexual risk behavior over 36 months of follow-up in the world's first HIV vaccine efficacy trial. J Acquir Immune Defic Syndr. 2005;39:90–101.
16. Koblin B, Chesney M, Coates T, et al.. Effects of a behavioural intervention to reduce acquisition of HIV infection among men who have sex with men: the EXPLORE randomised controlled study. Lancet. 2004;364:41–50.
17. Steyerberg EW, Harrell FE Jr, Borsboom GJ, et al.. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774–781.
18. Makuch RW, Rosenberg PS, Mulshine J. Identifying prognostic factors in binary outcome data: an application using liver function tests and age to predict liver metastases. Stat Med. 1988;7:843–856.
19. Moons KG, Harrell FE, Steyerberg EW. Should scoring rules be based on odds ratios or regression coefficients? J Clin Epidemiol. 2002;55:1054–1055.
20. Bernstein KT, Liu KL, Begier EM, et al.. Same-sex attraction disclosure to health care providers among new York City men who have sex with men: implications for HIV testing approaches. Arch Intern Med. 2008;168:1458–1464.
21. Petroll AE MK. Physician awareness of sexual orientation and preventive health recommendations to men who have sex with men. Sex Transm Dis. 2011;38:63–67.
22. Hinchliff S, Gott M, Galena E. ‘I daresay I might find it embarrassing': general practitioners' perspectives on discussing sexual health issues with lesbian and gay patients. Health Soc Care Community. 2005;13:345–353.
23. Kurth AE, Holmes K, Hawkins R, et al.. National survey of clinic sexual histories for sexually transmitted infection and HIV screening. Sex Transm Dis. 2005;32:370–376.
24. Nusbaum MR, Hamilton CD. The proactive sexual health history. Am Fam Physician. 2002;66:1705–1712.
25. Smith DK, Grant RM, et al., Centers for Disease Control and Prevention. Interim guidance: preexposure prophylaxis for the prevention of HIV infection in men who have sex with men. MMWR Morb Mortal Wkly Rep. 2011;60:65–68.
26. Menza TW, Hughes JP, Celum CL, et al.. Prediction of HIV acquisition among men who have sex with men. Sex Transm Dis. 2009;36:547–555.
27. Jones J SP, Stephenson R, Taussig JA. An iPad-based risk assessment tool for PrEP administration: opinions of MSM. Presented at: National HIV Prevention Conference; 2011; Atlanta, GA.
28. Yarnall KSH, Pollak KI, Ostbye T, et al.. Primary care: is there enough time for prevention? Am J Public Health. 2003 93:635–641.
29. Hitt E. Medscape Medical News: men who have sex with men have a 40-fold higher risk for HIV, Syphilis. 2010; Available at: Accessed April 8, 2011.
30. Finlayson TJ, Le B, Smith A, et al., CDC. HIV risk, prevention, and testing behaviors among men who have sex with men–National HIV Behavioral Surveillance System, 21 U.S. cities, United States, 2008. MMWR Surveillance Summ. 2011;60:1–34.

HIV; risk behaviors; incidence; screening; MSM

Copyright © 2012 Wolters Kluwer Health, Inc. All rights reserved.