Objective: To develop and validate an easy-to-use prediction model for HIV acquisition among men who have sex with men (MSM).
Methods: We developed prediction models using medical records data from an STD clinic (2001–2008) and validated these models using data from the control arm of Project Explore, an HIV prevention trial (1999–2003).
Results: Of 1903 MSM who tested for HIV more than once in the development sample, 101 acquired HIV over 6.7 years of follow-up. Annual HIV incidence was 2.57% (95% confidence interval [CI]: 2.09%, 3.12%). During 4 years of follow-up of 2081 Project Explore control arm participants, 144 acquired HIV for an incidence of 2.32% (95% CI: 1.96%, 2.73%). A prediction model that included variables indicating use of methamphetamine or inhaled nitrites in the prior 6 months, unprotected anal intercourse with a partner of positive or unknown HIV status in the prior year, ≥10 male sex partners in the prior year, and current diagnosis or history of bacterial sexually transmitted infection was well calibrated overall (expected-observed ratio = 1.01; 95% CI: 0.97, 1.05) and had modest discriminatory accuracy at 1 year (area under the receiver-operator characteristic curve = 0.67; 95% CI: 0.60, 0.75) and at 4 years (area under the receiver-operator characteristic curve = 0.66; 95% CI: 0.61, 0.71). Over 4 years, cumulative incidence ranged from 3.9% to 14.3% for groups of men defined by the prediction model.
Conclusions: A new risk score was predictive of HIV acquisition and could assist providers in counseling MSM and in targeting intensified prevention to MSM at greatest risk for HIV infection. Its accuracy requires further evaluation.
A validated prediction model for human immunodeficiency virus acquisition among men who have sex with men may prove useful in counseling men who have sex with men and in prioritizing prevention efforts to men who have sex with men at highest risk.
From the *The Center for AIDS and STD, University of Washington, Seattle, Washington; †Department of Epidemiology, University of Washington, Seattle, Washington; ‡Department of Public Health—Seattle and King County, Seattle, Washington; and Departments of §Global Health, ∥Medicine, and ¶Biostatistics, University of Washington, Seattle, Washington
The authors thank Deborah Donnell and Marla Husnik from the Statistical Center for HIV/AIDS Research and Prevention and Roxanne Kerani and Fred Koch from Public Health—Seattle and King County for their technical assistance.
Project Explore was supported by the HIV Prevention Trials Network and sponsored by the National Institutes of Health and the US Office of AIDS Research and also supported by NIH T32 AI07140 (to T.W.M.).
Project Explore was not conducted independent of the funding sources. However, funding organizations had no involvement in the current study’s design; data collection, analysis, or interpretation; or in the decision to publish this manuscript.
Correspondence: Timothy W. Menza, BA, Harborview Medical Center, 325 Ninth Ave, Box 359931, Seattle, WA 98104. E-mail: firstname.lastname@example.org.
Received for publication January 10, 2009, and accepted April 1, 2009.
Men who have sex with men (MSM) continue to comprise the greatest number of HIV infections in the US and many other nations.1–3 However, HIV risk is not evenly distributed among all MSM,4–8 and efforts to prevent HIV might be more effectively employed if they stratified men based on validated risk criteria.
Clinical prediction models are an efficient means of identifying individuals at high risk for a medical problem who might benefit from treatment or intensified prevention efforts. Scientists have created models to predict the risk of developing coronary heart disease,9 diabetes,10 breast cancer,11,12 lung cancer,13 and stroke among persons with atrial fibrillation,14 and findings from these models have contributed to the development of treatment and prevention guidelines for these diseases.15–20
Several studies have identified characteristics, behaviors, and contexts that associated with HIV acquisition,4–8 but few studies have used these factors to predict the probability of future HIV acquisition.21–23 To our knowledge, no studies have created prediction models that are specific to the MSM population. Prediction models may help clinicians better counsel clients about their HIV risk, provide appropriate referrals to further services, make recommendations for the frequency of future HIV testing, and provide intensified interventions to men at highest risk. Therefore, we constructed and validated a multivariable risk score predictive of HIV acquisition among MSM.
We used data from Public Health—Seattle and King County (PHSKC) STD Clinic electronic records to develop a model predictive of HIV acquisition among MSM. We validated these scores using data from control group participants of Project Explore.24
Risk Score Development: STD Clinic Repeat Testers
We included all MSM who tested for HIV >1 time in the PHSKC STD clinic between October 2001 and May 2008 and who tested HIV-negative by second-generation enzyme immunoassay (EIA) at their initial HIV test during the study period. Starting in 2003, all MSM negative on HIV EIA were also tested using pooled HIV RNA.25 To account for the window period of HIV EIA, we excluded men from the analysis if <30 days elapsed between their first and last HIV testing visits if they were not tested using pooled RNA.
PHSKC STD clinic providers use a structured form to record client histories before HIV testing. In a face-to-face encounter, clinicians collect information on demographic variables and self-reported substance use, STD history, and sexual behavior. Clinicians ask all clients whether they have had sex with men, women, or both in the prior year. We classified men who reported sex with another man in the prior year as MSM. For MSM, sexual behavior data include the number of male sex partners (anal and oral combined) and how often clients used condoms (always, usually, sometimes, never) during insertive and receptive anal intercourse with HIV-positive, HIV-negative, or HIV-unknown partners. These data use a 12-month recall period. In contrast, for substance use history, clinicians ask MSM whether they have ever used methamphetamine, injection drugs, inhaled nitrites, or crack and, if so, to provide the most recent month and year they used these substances. Syphilis was diagnosed using rapid plasma reagin confirmed with Treponema pallidum particle agglutination assay. Urethral gonococcal and chlamydial infections were diagnosed using Aptima Combo 2 (Genprobe Inc., San Diego, CA) or culture, while rectal and pharyngeal gonococcal and chlamydial infections were diagnosed by culture. Gonococcal cultures used modified Thayer-Martin media and chalmydial cultures used McCoy cells.
We selected 6 binary variables on the basis of simplicity and epidemiologic evidence for inclusion in the predictive model: age less than 40 years,26 nonwhite and non-Asian/Pacific Islander26,27 (API) race/ethnicity, current laboratory diagnosis of a bacterial STD (gonorrhea, Chlamydia, or early syphilis) or of having ever had a bacterial STD,8,28–30 use of methamphetamine or inhaled nitrites in the prior 6 months,4,7,29,31,32 ≥10 male sex partners in the prior year,4,32 and unprotected anal intercourse with a partner of unknown or positive HIV status (nonconcordant unprotected anal intercourse, UAI) in the prior year.4,33 This model is the full model.
We considered the inclusion of herpes simplex virus type-2 (HSV-2) infection given its high prevalence among MSM and its association with HIV acquisition34; however, excluded it because only 10% of men were tested for HSV-2 at their first HIV testing visit and only 8% reported a history of HSV-2 infection. Despite the different physiological effects of methamphetamine and inhaled nitrites, we grouped methamphetamine and inhaled nitrite use into one variable for simplicity and to facilitate uptake of the prediction models by clinicians. Additionally, these substances are used in similar contexts35–39 and the magnitude of associations between HIV acquisition and these substances are similar.4,7,29,31,32 We used a 6-month recall period for methamphetamine and inhaled nitrite use to be consistent with previous reports and to improve the reliability of the measures over those using longer recall periods.4,7,29,31,32 We performed analyses with separate variables for methamphetamine and inhaled nitrites use and analyses with a 12-month recall period for the composite variable; the results were very similar to those in this manuscript and are not presented.
Because one’s age and race/ethnicity are immutable characteristics, we tested a second, simple model that excluded the variables indicating age and race/ethnicity. We also used a stepwise selection procedure using the Akaike Information Criterion to create a more parsimonious model from the initial 6-variable model.24,40,41 This procedure yielded a model that included all the predictors except for nonconcordant UAI in the prior year. Because nonconcordant UAI was the strongest predictor of incident HIV infection in a cross-sectional analysis,33 we kept it in our prediction model and did not evaluate a model without this predictor.
We used Cox proportional hazards models to estimate hazard ratios (HR) for HIV acquisition. Because clinicians and disease intervention specialists concentrate on current reports of risk and diagnoses of STD in counseling clients, the values of the predictors in the models were those at baseline; predictors were not modeled as time-varying covariates. The initial time point for survival modeling was the date of first negative HIV EIA. For MSM who tested HIV-positive by reactive EIA, we defined time-of-acquisition as the midpoint between the date of their positive EIA and the date of their last negative EIA. For men who tested HIV-positive by pooled RNA screening but HIV-negative by EIA at the same visit, we defined time-of-acquisition as the date of the positive RNA test. MSM were censored at their last HIV testing visit if they had not tested HIV-positive in the STD Clinic by 12 May 2008. Additionally, we repeated model building and development using the predictor values recorded at clients’ second visits at which they tested HIV-negative; the results of those analyses were similar to those using clients’ first visits (data not given here).
For the full and simple models, all predictors met the proportional hazards assumption as assessed with log(−log) plots of the survival function, Schoenfeld residuals and associated test statistics,42 and including interaction terms with time for each predictor variable. Additionally, we found no evidence for interactions between individual risk predictors assessed by a global likelihood ratio test.
Based on the coefficients from the Cox proportional hazards regression models, we created integer weights for each variable. We calculated these weights by multiplying the model coefficients by 10 (the choice of multiplier is arbitrary) and rounding to the nearest whole integer. The weights rank the risk predictors in relative importance and dictate how one assigns integer point values for each risk predictor for a given individual. The assigned points are then summed to compute that individual’s risk score.
We also performed interval-censored analyses assuming a log-logistic distribution to model the odds of HIV acquisition associated with each risk predictor.43,44 These methods yielded similar results to the Cox proportional hazards models presented here.
Stability of Simple Model Predictors
To assess the consistency of risk among men evaluated in our clinic on multiple occasions, we used data collected from men on their first visit to the clinic each year for the first 4 years of follow-up to calculate κ statistics for the variables in the simple model.45
Risk Score Validation: Project Explore
The control arm of Project Explore comprised the population for model validation. Project Explore was a randomized behavioral intervention trial that enrolled HIV-negative MSM age >16 from 6 US cities, including Seattle, between January 1999 and February 2001. Men were eligible if they reported having anal sex with ≥1 men in the past year, but were excluded if they reported a mutually monogamous relationship lasting ≥2 years with an HIV-negative male partner. At enrollment and every 6 months thereafter, Explore participants completed an audio computer-assisted self-interview to assess sexual behavior, substance use, and recent STD diagnoses in the prior 6 months, and were tested for HIV by EIA. Project Explore followed participants for 48 months or until January 2003, whichever came first, and 87% of men in the control arm completed the study.46 The variables in Project Explore that corresponded to the predictors selected in the STD clinic sample were: methamphetamine or inhaled nitrites use in the prior 6 months; nonconcordant UAI in the prior 6 months; bacterial STD in the prior 6 months; and, ≥10 sex partners in the prior 6 months. All of these variables are based on participant self-report.
Using the full and simple models derived from the STD clinic sample, we calculated 2 risk scores for each humans in the control arm of Project Explore based on baseline assessments. We calculated the annual incidence of HIV infection for each quartile of risk score. We assessed model calibration by calculating the ratio of expected HIV infections to observed HIV infections (expected-observed ratio),12 where calibration is defined as the extent to which the risk predicted by the model reflects the risk in the observed in population.47 The 95% confidence intervals for HIV incidence and the expected-observed ratio assume that the observed HIV infections follow a Poisson distribution. We also assessed model calibration graphically by comparing plots of observed and predicted survival estimates for each risk group.24
To assess the ability of the risk scores to discriminate between men who acquired HIV and those who did not, we calculated the area under time-dependent receiver operating characteristic curves (AUC) for “survival” at 1 year of follow-up and at 4 years of follow-up with nonparametric bootstrap 95% confidence intervals using the methods of Heagerty et al.48,49
All analyses were conducted using STATA 10.0 (StataCorp, College Station, TX) and R 2.6.1 (The R Project for Statistical Computing). The University of Washington Human Subjects Division approved all study procedures.
Study Populations and HIV Incidence
Between October 2001 and May 2008, PHSKC STD clinic staff evaluated 6026 MSM who did not acknowledge previously testing HIV-positive; 290 (5%) of these men tested HIV-positive and 5736 (95%) men tested HIV-negative. Of these 5736 HIV-negative men, 3827 (67%) did not return for HIV testing during the study period and 1909 (33%) returned at least once for HIV testing (median: 3 visits, range: 2–28), and 1903 men tested for HIV again ≥30 days after their initial EIA and comprise our sample for model development.
Compared with men who tested for HIV only once during the study period, at baseline repeat testers were more likely to be white (68% vs. 71%, P = 0.01) and <40 years of age (70% vs. 80%, P <0.001), but less likely to use crack or cocaine (5% vs. 3%, P <0.001) or injection drugs (4% vs. 2%, P <0.001) in the prior 6 months. Men who did not return to test were less likely than men who tested repeatedly to test positive for bacterial STD (12% vs. 20%, P <0.001) at their only or first visits, respectively, and to report having ever been diagnosed with a bacterial STD (28% vs. 33%, P <0.001). Men who did and did not return for testing reported similar rates of methamphetamine and inhaled nitrite use in the prior 6 months and of nonconcordant UAI in the prior year.
Repeat HIV testers in the STD clinic sought testing over a median of 1.6 years (range: 30 days–6.7 years) and experienced 101 HIV infections (annual incidence = 2.57%; 95% confidence interval [CI]: 2.09%, 3.12%). Of the 2081 Explore control group participants, 144 acquired HIV over a median of 3 years (range: 6months–4 years) for an annual incidence of 2.32% (95% CI: 1.96%, 2.73%).
Explore participants were older, more likely to be Hispanic, and reported greater rates of partner change than STD clinic clients (Table 1). Explore participants were also more likely to report using inhaled nitrites and methamphetamine in the preceding 6 months than STD clinic testers. MSM tested at the STD clinic were more likely to be API than Explore participants, and more likely to be diagnosed with a bacterial STD at their first HIV testing visit than Explore participants in the 6 months preceding their enrollment interview. Fewer STD clinic clients reported having nonconcordant UAI in the preceding 12 months than Explore participants did in the prior 6 months.
Agreement of Simple Model Risk Predictors Within an Individual Over Time
Certain variables may be poor predictors of HIV infection if they are intermittent as opposed to consistent. For MSM tested at the STD clinic, we evaluated the agreement between the predictors of the simple model each year over the first 4 years of follow-up. The κ statistics were 0.41 for use of methamphetamine or inhaled nitrites in the prior 6 months, 0.47 for ≥10 male sex partners in the prior year, 0.28 for nonconcordant UAI in the prior year, and 0.67 for history, or current diagnosis, of bacterial STD.
Risk Score Evaluation and Validation
We used the full and simple prediction models to generate scores for each participant in the control arm of Project Explore based on their baseline assessments (Table 2). The mean (standard deviation) scores for the full and simple models were 11.7 (6.9) and 5.6 (6.0), respectively; the median (range) scores were 11 (0–31) and 3 (0–19), respectively. The incidence of HIV and the HR for HIV infection increased with quartile of risk score (test of trend P <0.0001 for each index; Table 3). For groups of men who differ by one unit in the full risk score, men with the higher risk score experience a risk of HIV 1.08 times greater than that experienced by men with the lower risk score (95% CI: 1.06, 1.11, P <0.001). For groups of men who differ by one category (quartile) of the full risk score, men in the higher category experience a risk of HIV 1.61 times greater than that experience by men in the lower category (95% CI: 1.39, 1.86; P <0.001). Comparable HRs for the simple model were similar.
The full and simple models were well calibrated in the validation sample (Table 4). The expected annual rate of HIV infection was 2.35% according to the full model (expected-observed ratio: 1.01; 95% CI: 0.97, 1.06) and 2.34% according to the simple model (expected-observed ratio: 1.01; 95% CI: 0.97, 1.05). Inspection of plots comparing Kaplan-Meier survival curves and Cox proportional hazards survival curves for each risk quartile (data not shown), also indicated acceptable model calibration.
Model discrimination measured by time-dependent AUC estimates was 0.68 (95% CI: 0.62, 0.75) at 1 year and 0.67 (95% CI: 0.62, 0.71) at 4 years for the full model in Project Explore (Table 4). The simple model exhibited similar discrimination, 0.67 (95% CI: 0.60, 0.75) at 1 year and 0.66 (95% CI: 0.61, 0.71) at 4 years. Because our results may be influenced by temporal trends in sexual risk, substance use, and STI diagnoses over time and by the potential inclusion of Seattle Explore participants in the development sample, we performed 2 sensitivity analyses. The first assessed the impact of restricting model development and validation to the years 2001 to 2003 during which the STD clinic and Explore study periods overlapped. The second excluded the 371 Seattle participants from the Project Explore control arm on model discrimination. Time-dependent AUC estimates at 1 and 4 years were similar in magnitude and not different statistically from our original estimates (data not shown).
Because the full and simple models performed similarly based on their calibration and discrimination, Table 5 presents the sensitivity, specificity, and cumulative HIV incidence for groups of men defined by the score cut-offs for the simple model over 1 year and 4 years of follow-up. Figure 1 displays the cumulative incidence of HIV infection over time for each quartile of the simple risk score in Project Explore. Cumulative incidence at 4 years was 3.9% (95% CI: 2.4%, 6.2%) in men with a score of 0, 5.3% (95% CI: 2.1%, 8.5%) in men with scores 1 to 3, 9.5% (95% CI: 4.6%, 14.4%) in men with scores 4 to 11, and 14.3% (95% CI: 7.5%, 21.1%) in men with scores ≥12.
We used clinical and behavioral data collected from an STD clinic to develop 2 risk scores predictive of HIV infection, and validated those scores using data from a large multicenter behavioral intervention trial. Although the scores we developed would optimally undergo additional validation in other populations,50,51 we believe our simple score can be useful to clinicians and others in counseling MSM about their risk of HIV infection, and might be used to identify persons who require intensified interventions or more frequent HIV testing. The resources necessary to conduct biomedical and behavioral HIV prevention trials depend on HIV incidence.46,52,53 Thus, this score might also help researchers identify MSM at high risk of HIV infection for prevention trials with HIV acquisition as the primary outcome.
Despite the model’s simplicity and excellent calibration, its discriminatory accuracy was only modest (AUC: 0.66). However, this finding should be interpreted with the understanding that extremely high relative risks (>100) are required to generate risk predictor models with high AUC estimates,12,54–56 and that such models are extremely uncommon in clinical practice. The AUC estimates that we report are similar to those of other risk models, such as the Framingham risk score for coronary heart disease (AUC: 0.63–0.83),57 that are commonly used to guide clinical decisions.
Although the calculation of the risk scores may seem daunting, the scores we present are not more complicated than the Framingham risk score9 or the CHADS score,14 tools that clinicians commonly use to estimate patients’ risk of coronary heart disease and the risk of stroke among persons with atrial fibrillation, respectively. Our MSM risk score should not require the use of a computer for calculation, and may be suitable for use in nonclinical settings or on the Internet. We are currently developing a public health website that will allow MSM to calculate their score and explore their risk of HIV acquisition. An approach to calculating the simple risk score and estimating a humans’ 4-year risk of acquiring HIV is presented in Figure 2.
The ideal risk score cut-off may be one that leads to the follow-up of a number of patients that the resources of a clinic, provider, or community-based organization can accommodate. We found that using a score that identifies men with any risk predictor (i.e., risk score ≥1), our simple model would identify 83% of STD clinic patients and 86% of Explore participants who acquired HIV over 4 years, but would require follow-up of approximately 70% of the 2 populations. Using a higher risk score identifies a population at greater risk, but a smaller proportion of all men who acquire HIV. Currently, our STD clinic employs the simple model to identify men with ≥1 risk predictor for more frequent HIV testing, follow-up counseling by telephone, and reminder notices to return for repeat HIV testing.
The strongest predictors of HIV acquisition in our models were use of methamphetamine or inhaled nitrites and a history, or current diagnosis, of bacterial STD, findings that highlight the importance of routinely asking MSM about these drugs and of concentrating prevention efforts in persons with bacterial STD. Although nonconcordant UAI has typically been the strongest risk factor for testing HIV-positive in cross-sectional studies,33 it was a weaker predictor of HIV acquisition in our models. This difference may reflect the distinction between behaviors or characteristics that operate at the individual-level and those that operate at a network-level. For example, substance use and STD may indicate that an individual circulates within a sexual network more conducive to HIV transmission, while the report of nonconcordant UAI does not always indicate this network-level risk. Additionally, we found that the agreement in reports of nonconcordant UAI over 4 years of follow-up was only fair. This fair agreement likely diminishes our predictive models’ discriminatory ability. We overestimate the risk of HIV acquisition for men who report nonconcordant UAI at baseline, but do not practice nonconcordant UAI during the follow-up period. Conversely, we may underestimate the risk of HIV acquisition for men who do not report nonconcordant UAI at baseline, but engage in nonconcordant UAI during the follow-up period.
Our study is subject to important limitations. First, our risk score was derived from a population of MSM who repeatedly tested in a single US STD clinic. Not all MSM in our clinic tested serially during the study period, and studies suggest that risk behavior may influence HIV testing behavior.58,59 On the other hand, our models performed reasonably well among MSM from 6 US cities who tested at 6-month interval as part of Project Explore, suggesting that the prediction models can be applied to a more diverse population of MSM. However, as our development and validation samples were composed mostly of white MSM, we cannot be sure of our models’ performance in racial and ethnic minority MSM. Additionally, we cannot be sure of our models’ applicability to populations and geographic areas with lower rates of methamphetamine and inhaled nitrites use.
Second, data collected from the PHSKC STD Clinic and Project Explore were dissimilar in important ways. Project Explore employed audio computer-assisted self-interview and collected behavioral and STD diagnosis data for 6-month period, while clinicians at the PHSKC STD Clinic collected behavioral data through face-to-face interviews, and sexual behavior questions focused on the prior 12 months. Also, data concerning bacterial STD were based on self-report and diagnoses on the day of HIV testing in the STD clinic, but reflected only self-reported history of bacterial STD in the prior 6 months in Project Explore. Misclassification would diminish model discrimination. For example, men in the STD clinic who practiced nonconcordant UAI might not acknowledge this behavior because of social desirability bias associated with face-to-face interviews. Therefore, the predictive models would overestimate the risk of HIV acquisition in men who did not report nonconcordant UAI in both the development and validation samples.
Prediction models are improved through repeated validation, augmentation, and examination in research and practice.9,15,17 We present our prediction models as a step in the process of developing consistent guidelines for HIV prevention practice. Optimally, future work should augment and validate these prediction models in diverse settings and measure the acceptability and utility of prevention efforts guided by prediction models.50,51 However, even without such additional research, we believe our prediction model can be useful in counseling MSM and in prioritizing intensified prevention efforts to the MSM at highest risk for HIV.
1. Subpopulation estimates from the HIV incidence surveillance system–United States, 2006. MMWR Morb Mortal Wkly Rep 2008; 57:985–989.
2. Hall HI, Song R, Rhodes P, et al. Estimation of HIV incidence in the United States. JAMA 2008; 300:520–529.
3. Baral S, Sifakis F, Cleghorn F, et al. Elevated risk for HIV infection among men who have sex with men in low- and middle-income countries 2000–2006: A systematic review. PLoS Med 2007; 4:e339.
4. Buchbinder SP, Vittinghoff E, Heagerty PJ, et al. Sexual risk, nitrite inhalant use, and lack of circumcision associated with HIV seroconversion in men who have sex with men in the United States. J Acquir Immune Defic Syndr 2005; 39:82–89.
5. Koblin BA, Husnik MJ, Colfax G, et al. Risk factors for HIV infection among men who have sex with men. AIDS 2006; 20:731–739.
6. Hall HI, Byers RH, Ling Q, et al. Racial/ethnic and age disparities in HIV prevalence and disease progression among men who have sex with men in the United States. Am J Public Health 2007; 97:1060–1066.
7. Plankey MW, Ostrow DG, Stall R, et al. The relationship between methamphetamine and popper use and risk of HIV seroconversion in the multicenter AIDS cohort study. J Acquir Immune Defic Syndr 2007; 45:85–92.
8. Schwarcz SK, Kellogg TA, McFarland W, et al. Characterization of sexually transmitted disease clinic patients with recent human immunodeficiency virus infection. J Infect Dis 2002; 186:1019–1022.
9. Lloyd-Jones DM, Larson MG, Beiser A, et al. Lifetime risk of developing coronary heart disease. Lancet 1999; 353:89–92.
10. Griffin SJ, Little PS, Hales CN, et al. Diabetes risk score: Towards earlier detection of type 2 diabetes in general practice. Diabetes Metab Res Rev 2000; 16:164–171.
11. Gail MH, Brinton LA, Byar DP, et al. Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J Natl Cancer Inst 1989; 81:1879–1886.
12. Tice JA, Cummings SR, Smith-Bindman R, et al. Using clinical factors and mammographic breast density to estimate breast cancer risk: Development and validation of a new predictive model. Ann Intern Med 2008; 148:337–347.
13. Cassidy A, Duffy SW, Myles JP, et al. Lung cancer risk prediction: A tool for early detection. Int J Cancer 2007; 120:1–6.
14. Gage BF, Waterman AD, Shannon W, et al. Validation of clinical classification schemes for predicting stroke: Results from the National Registry of Atrial Fibrillation. JAMA 2001; 285:2864–2870.
15. Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report. Circulation 2002; 106:3143–3421.
16. Bach PB, Niewoehner DE, Black WC. Screening for lung cancer: The guidelines. Chest 2003; 123(suppl 1):83S–88S.
17. Chobanian AV, Bakris GL, Black HR, et al. Seventh report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure. Hypertension 2003; 42:1206–1252.
18. Qaseem A, Snow V, Sherif K, et al. Screening mammography for women 40 to 49 years of age: A clinical practice guideline from the American College of Physicians. Ann Intern Med 2007; 146:511–515.
19. Fuster V, Ryden LE, Cannom DS, et al. ACC/AHA/ESC 2006 Guidelines for the Management of Patients with Atrial Fibrillation: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines and the European Society of Cardiology Committee for Practice Guidelines. (Writing Committee to Revise the 2001 Guidelines for the Management of Patients With Atrial Fibrillation): Developed in collaboration with the European Heart Rhythm Association and the Heart Rhythm Society. Circulation 2006; 114:e257–e354.
20. Nutrition recommendations and interventions for diabetes: A position statement of the American Diabetes Association. Diabetes Care 2007; 30(suppl 1):S48–S65.
21. Boileau C, Bruneau J, Al-Nachawati H, et al. A prognostic model for HIV seroconversion among injection drug users as a tool for stratification in clinical trials. J Acquir Immune Defic Syndr 2005; 39:489–495.
22. Chen Z, Branson B, Ballenger A, et al. Risk assessment to improve targeting of HIV counseling and testing services for STD clinic patients. Sex Transm Dis 1998; 25:539–543.
23. Richert CA, Peterman TA, Zaidi AA, et al. A method for identifying persons at high risk for sexually transmitted infections: Opportunity for targeting intervention. Am J Public Health 1993; 83:520–524.
24. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15:361–387.
25. Stekler J, Swenson PD, Wood RW, et al. Targeted screening for primary HIV infection through pooled HIV-RNA testing in men who have sex with men. AIDS 2005; 19:1323–1325.
27. Choi KH, McFarland W, Neilands TB, et al. An opportunity for prevention: Prevalence, incidence, and sexual risk for HIV among young Asian and Pacific Islander men who have sex with men, San Francisco. Sex Transm Dis 2004; 31:475–480.
28. Wasserheit JN. Epidemiological synergy: Interrelationships between human immunodeficiency virus infection and other sexually transmitted diseases. Sex Transm Dis 1992; 19:61–77.
29. Page-Shafer K, Veugelers PJ, Moss AR, et al. Sexual risk behavior and risk factors for HIV-1 seroconversion in homosexual men participating in the Tricontinental Seroconverter Study, 1982–1994. Am J Epidemiol 1997; 146:531–542.
30. Truong HM, Grant RM, McFarland W, et al. Routine surveillance for the detection of acute and recent HIV infections and transmission of antiretroviral resistance. AIDS 20:2193–2197.
31. Chesney MA, Barrett DC, Stall R. Histories of substance use and risk behavior: Precursors to HIV seroconversion in homosexual men. Am J Public Health 1998; 88:113–116.
32. Burcham JL, Tindall B, Marmor M, et al. Incidence and risk factors for human immunodeficiency virus seroconversion in a cohort of Sydney homosexual men. Med J Aust 1989; 150:634–639.
33. Golden MR, Brewer DD, Kurth A, et al. Importance of sex partner HIV status in HIV risk assessment among men who have sex with men. J Acquir Immune Defic Syndr 2004; 36:734–742.
34. Brown EL, Wald A, Hughes JP, et al. High risk of human immunodeficiency virus in men who have sex with men with herpes simplex virus type 2 in the EXPLORE study. Am J Epidemiol 2006; 164:733–741.
35. Shoptaw S, Reback CJ, Freese TE. Patient characteristics, HIV serostatus, and risk behaviors among gay and bisexual males seeking treatment for methamphetamine abuse and dependence in Los Angeles. J Addict Dis 2002; 21:91–105.
36. Romanelli F, Smith KM, Thornton AC, et al. Poppers: Epidemiology and clinical management of inhaled nitrite abuse. Pharmacotherapy 2004; 24:69–78.
37. Colfax GN, Mansergh G, Guzman R, et al. Drug use and sexual risk behavior among gay and bisexual men who attend circuit parties: A venue-based comparison. J Acquir Immune Defic Syndr 2001; 28:373–379.
38. Reback CJ. The Social Construction of a Gay Drug: Methamphetamine Use Among Gay and Bisexual Males in Los Angeles. Los Angeles, CA: City of Los Angeles, 1997.
39. Schwarcz S, Scheer S, McFarland W, et al. Prevalence of HIV infection and predictors of high-transmission sexual risk behaviors among men who have sex with men. Am J Public Health 2007; 97:1067–1075.
40. Akaike H. Information theory and an extension of the maximum likelihood principle. In: Petrov B, Csaki F, eds. Second International Symposium on Inference Theory. Budapest, Hungary: Akademiai Kiado, 1973:267–281.
41. Hosmer DW, Lemeshow S. Applied Survival Analysis. New York, NY: Wiley, 1999.
42. Grambsch P, Therneau T. Proportional hazards tests and diagnostics based on weighted residuals. Biometrika 1994:515–526.
43. Griffin JT, Fraser C, Gras L, et al. The effect on treatment comparisons of different measurement frequencies in human immunodeficiency virus observational databases. Am J Epidemiol 2006; 163:676–683.
44. Lindsey JC, Ryan LM. Tutorial in biostatistics methods for interval-censored data. Stat Med 1998; 17:219–238.
45. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960; 20:37–46.
46. Koblin B, Chesney M, Coates T. Effects of a behavioral intervention to reduce acquisition of HIV infection among men who have sex with men: The EXPLORE randomized controlled study. Lancet 2004; 364:41–50.
47. Harrell FE Jr. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York, NY: Springer-Verlag, 2001.
48. Heagerty PJ, Lumley T, Pepe MS. Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics 2000; 56:337–344.
49. Heagerty PJ, Zheng Y. Survival model predictive accuracy and ROC curves. Biometrics 2005; 61:92–105.
50. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med 1999; 130:515–524.
51. Reilly BM, Evans AT. Translating clinical research into clinical practice: Impact of using prediction rules to make decisions. Ann Intern Med 2006; 144:201–209.
52. Kamb ML, Fishbein M, Douglas JM Jr, et al. Efficacy of risk-reduction counseling to prevent human immunodeficiency virus and sexually transmitted diseases: a randomized controlled trial. Project RESPECT Study Group. JAMA 1998; 280:1161–1167.
53. Imrie J, Stephenson JM, Cowan FM, et al. A cognitive behavioural intervention to reduce sexually transmitted infections among gay men: Randomized trial. BMJ 2001; 322:1451–1456.
54. Pepe MS, Janes H, Longton G, et al. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 2004; 159:882–890.
55. Wald NJ, Hackshaw AK, Frost CD. When can a risk factor be used as a worthwhile screening test? BMJ 1999; 319:1562–1565.
56. Ware JH. The limitations of risk factors as prognostic tools. N Engl J Med 2006; 355:2615–2617.
57. D’Agostino RB Sr, Grundy S, Sullivan LM, et al. Validation of the Framingham coronary heart disease prediction scores: Results of a multiple ethnic groups investigation. JAMA 2001; 286:180–187.
58. MacKellar DA, Valleroy LA, Secura GM, et al. Repeat HIV testing, risk behaviors, and HIV seroconversion among young men who have sex with men: A call to monitor and improve the practice of prevention. J Acquir Immune Defic Syndr 2002; 29:76–85.
© Copyright 2009 American Sexually Transmitted Diseases Association
59. Kellerman SE, Lehman JS, Lansky A, et al. HIV testing within at-risk populations in the United States and the reasons for seeking or avoiding HIV testing. J Acquir Immune Defic Syndr 2002; 31:202–210.