Brief Report

Validation of a Quantitative HIV Risk Prediction Tool Using a National HIV Testing Cohort

Abstract

Routine screening is recommended for HIV detection. HIV risk estimation remains important. Our goal was to validate the Denver HIV Risk Score using a national cohort from the Centers for Disease Control and Prevention. Patients of 13 years and older were included, 4,830,941 HIV tests were performed, and 0.6% newly diagnosed infections were identified. Of all visits, 9% were very low risk (HIV prevalence = 0.20%), 27% low risk (HIV prevalence = 0.17%), 41% moderate risk (HIV prevalence = 0.39%), 17% high risk (HIV prevalence = 1.19%), and 6% very high risk (HIV prevalence = 3.57%). The Denver HIV Risk Score accurately categorized patients into different HIV risk groups.

INTRODUCTION

In the United States, over 200,000 individuals are infected but remain undiagnosed with the HIV.¹ Screening efforts are at the forefront of HIV prevention in the United States. A principal goal of the National HIV/AIDS Strategy is to reduce the proportion of patients living with undiagnosed HIV infection to 10% by 2015.² To improve identification of HIV-infected persons, both the Centers for Disease Control and Prevention (CDC) and the US Preventive Services Task Force (USPSTF) recommend routine nontargeted HIV screening.^3,4

Numerous studies of nontargeted HIV screening have shown this approach to successfully identify patients with HIV infection, although the effectiveness of such large-scale screening has been judged as modest,^5,6 whereas others have raised concerns about its costs and inefficiencies.^5,7,8 Risk-based HIV screening still remains a viable alternative to nontargeted HIV screening.^9,10 Although the concept of targeted HIV screening has existed for decades,¹¹ and risk characteristics have been extensively studied,^9,12 specific targeted strategies remain largely undefined and have not been broadly evaluated in practice.^13–15

In 2012, the Denver HIV Risk Score (DHRS), an empirically derived clinical prediction instrument, was developed to help clinicians estimate patients' probabilities of being infected with HIV, thus informing HIV screening and prevention counseling.¹⁶ The DHRS consists of 6 demographic and risk behavior variables that, when applied to a patient, result in a score; individuals with scores ≥30 are considered at increased risk for HIV infection (see Appendix, Supplemental Digital Content, https://links.lww.com/QAI/A621). The external validity of the DHRS has not been assessed broadly, and its validity in areas where HIV prevalence is high or where demographic and risk behavior characteristics differ is unknown.¹⁷ The goal of this study was to validate the DHRS using data from a national HIV testing cohort from the CDC. We hypothesized that HIV prevalence would significantly increase with DHRS values ≥30 and this relationship would be consistent across all geographic regions of the United States.

METHODS

Study Design

We performed a secondary analysis of the CDC's National HIV Program Evaluation and Monitoring System (PEMS). PEMS is a national data reporting system of a standardized set of HIV prevention variables, secure web-based software, and a range of data collection support services. This study was approved by our institutional review board.

Population and Setting

We used HIV testing data that were collected from all CDC-funded HIV testing sites throughout the United States (except Massachusetts, North Dakota, Ohio, and Rhode Island) from January 1, 2008, to December 31, 2010. We included patients of 13 years and older, with no other exclusions. Testing data were not restricted according to testing venue and included emergency departments (EDs), hospitals, outpatient clinics, sexually transmitted diseases and HIV counseling and testing sites, community-based organizations, blood banks, plasma centers, and correctional facilities.

Data Collection and Study Variables

Collection and reporting of PEMS data are required by health departments funded through CDC HIV prevention cooperative agreements. A standardized data collection instrument is completed by individuals providing testing services and is submitted through the enhanced HIV/AIDS Reporting System (eHARS). The eHARS is a browser-based HIV/AIDS surveillance system deployed at all state health departments.

The data set provided for this study included the following variables: patient birth year, gender, race/ethnicity, testing history and result, and risk characteristics including sex with a female, sex with a male, sex with a partner with known HIV infection, sex with a partner who injects drugs, sex with a male who has sex with other males, injection drug use and sharing of drug injection equipment, and date of HIV testing, geographic location of testing, and test results.

Outcomes

The primary outcome was a newly diagnosed HIV infection, defined by a confirmed HIV-positive test result, and the patient self-reporting as having not previously tested positive for HIV infection. The secondary outcome was all confirmed HIV-positive test results, defined as a positive HIV test result (ie, with either conventional or rapid enzyme immunoassay) with confirmation by supplemental testing (eg, Western blot).

Statistical Analyses

Statistical analyses were performed using SAS 9.3 (SAS Institute, Inc., Cary, NC). Observations were assigned scores according to the DHRS and categorized into the following 5 DHRS groups: <20 (very low risk), 20–29 (low risk), 30–39 (moderate risk), 40–49 (high risk), and ≥50 (very high risk). Proportions are reported as percentages with 95% confidence intervals (CIs). Calibration is reported as predicted versus observed HIV prevalence, and discrimination is reported as the area under the receiver operating characteristics curve. Unit of analysis was testing events unless otherwise specified. Additional analyses were performed stratifying by geographic region as defined by the US Census Bureau (see Appendix, Supplemental Digital Content, https://links.lww.com/QAI/A621).¹⁸ Best-case and worst-case sensitivity analyses were also performed to estimate the effect of missing data on complete case results (see Appendix, Supplemental Digital Content, https://links.lww.com/QAI/A621).

RESULTS

During the 3-year period, 4,830,941 tests were reported with complete data, representing the principal cohort for our analyses. Of these, the median age was 28 years (interquartile range: 22–40 years), 50% were male, 46% were black, 33% were white or other, and 21% were Hispanic; additionally, 30,080 (0.6%) were newly diagnosed with HIV infection.

Table 1 shows the distribution of the DHRS variables by primary and secondary outcomes. The prevalence of newly diagnosed HIV infection within each of the 5 DHRS risk groups was 0.20% (95% CI: 0.19% to 0.20%) (n = 856/432,674), 0.17% (95% CI: 0.16% to 0.17%) (n = 2168/1,312,427), 0.39% (95% CI: 0.38% to 0.40%) (n = 7771/2,003,857), 1.19% (95% CI: 1.16% to 1.21%) (n = 9617/811,501), and 3.57% (95% CI: 3.50% to 3.65%) (n = 9668/270,482), respectively (Fig. 1). When all confirmed HIV infections were considered, the DHRS performed similarly.

TABLE 1: Risk Score Variables for the Complete Case Cohort (N = 4,830,941), Stratified by HIV Diagnosis and Test Result, CDC PEMS Data, 2008–2010

FIGURE 1: Prevalence of HIV infection within each risk score category for both newly identified HIV infection and all HIV infections, CDC PEMS data, 2008–2010. The Denver HIV Risk Score ranges from −4 to +73. Bars = 95% CI.

The top 3 risk groups (scores ≥30) represented only 63% (n = 3,085,840/4,830,941) of the cohort, yet 90% (n = 27,056/30,080) of newly diagnosed HIV infections, whereas the top 2 risk groups (scores ≥40) represented only 22% (n = 1,081,983/4,830,941) of the cohort, yet 64% (n = 19,285/30,080) of newly diagnosed HIV infections. The DHRS demonstrated excellent calibration (regression slope: 1.09) and good discrimination (area under receiver operating characteristics curve: 0.77) (see Appendix, Supplemental Digital Content, https://links.lww.com/QAI/A621).

DISCUSSION

In a large national HIV testing cohort, the DHRS accurately categorized individuals into different HIV risk groups, and a DHRS ≥30 identified individuals at significantly increased risk of being HIV-infected. It also demonstrated increasing HIV prevalence as the DHRS increased with the highest risk group having a DHRS ≥50. Results were similar across all geographic regions of the United States, strongly supporting the validity and generalizability of the DHRS for predicting HIV risk.

To the best of our knowledge, the DHRS is the only instrument to quantify a patient's probability of being infected with HIV and was developed to help clinicians identify patients with HIV infection.¹⁶ After its development, however, a clear broader external validation was needed. The results of this study mirror the performance of the DHRS when first tested in ED populations from Cincinnati, OH,¹⁶ and Baltimore, MD,¹⁹ thus confirming its fidelity as a valid prediction tool.

The CDC recommends nonrisk-based HIV screening for adolescents and adults based,³ in part, on the concern that risk-based screening is less effective.^3,20 In 2013, the US Preventive Services Task Force affirmed the CDC's recommendation, not because comparative effectiveness of different screening methods exists but because individuals diagnosed and linked into care benefit from treatment while simultaneously reducing viral transmission.⁴ We believe that the DHRS may be a valuable tool to help achieve goals established by the National HIV/AIDS Strategy,² especially when used in environments where HIV screening resources are limited or when repeat testing is warranted.

According to the DHRS, individuals, 26–54 years of age, male, and black or Hispanic, regardless of sexual orientation, risk behaviors, or previous HIV testing, should be considered at increased risk for HIV infection (DHRS ≥30). Our results support the notion that certain demographic groups, regardless of risk behaviors, should be routinely tested for HIV infection. This is consistent with recent expanded testing efforts by the CDC to target communities with high disease burdens.²¹ Conversely, the DHRS identifies groups at such low risk for HIV infection as to not necessitate routine testing. We identified a threshold of ≥30 as having the best compromise between sensitivity and specificity; however, using different DHRS thresholds in different settings to inform screening practices may be warranted depending on the population being served and the HIV testing resources available.

The DHRS may also contribute meaningfully to current or future HIV screening paradigms. Nontargeted screening has been most widely studied in EDs with results indicating that a large proportion (nearly 80%) of eligible patients do not complete HIV testing.⁶ The DHRS may serve as an important adjunct to nontargeted screening by helping to quantify the risk of HIV infection for both clinicians and patients, thus helping to establish joint decision-making regarding consent for testing. Furthermore, high-risk patients who know their risk may be more likely to accept testing, especially when involved in discussions with their clinician. Moreover, as the proportion of undiagnosed individuals in the United States becomes smaller (<10%), nontargeted screening will likely become relatively less effective and efficient when compared with more focused screening methods. Finally, it remains unknown whether certain clinical venues, especially nontraditional HIV testing settings such as EDs and urgent cares, will be able to routinely provide nontargeted HIV screening, or whether a targeted approach is better suited for these sites.^22–24

Risk-based testing has historically failed because it was rooted in subjective assessment and without standardization. The DHRS is a simple structured tool that if used as part of routine HIV screening may help identify most patients with undiagnosed HIV infection while conserving scarce public health resources. People at greatest risk for HIV infection, including those who are economically disadvantaged, men who have sex with men, blacks, and Latinos, are disproportionately underinsured and rely heavily on publically funded settings or EDs for HIV testing.²⁵ Without an insurance payer, the burden of paying for HIV testing falls to the institution offering the test or the cost shifts to other patients with a payer source, a liability many administrators are reticent to take on.^25,26 Although previous research supports the notion that “universal HIV testing” is cost-effective from a societal perspective,^27,28 this means relatively little to administrators, clinicians, or public health officials as they plan annual budgets and seek funding needed to pay for large numbers of tests. Until a clearer understanding of how funding will be provided for HIV testing, the DHRS may be used to prioritize how and when HIV testing is offered in public settings. Future research will require comparative and cost-effective evaluations of routine risk-based HIV screening using the DHRS to nonrisk-based HIV screening.

This study has limitations. The data were not specifically collected for purposes of validating the DHRS; as such, data may have been misclassified or missing. Given the size of the data set from a diverse range of HIV testing venues and results of sensitivity analyses (see Appendix, Supplemental Digital Content, https://links.lww.com/QAI/A621), we believe the effects of misclassification or missingness were small. The primary outcome included subjective reports by individuals of a new diagnosis, and it is possible that in some instances these reports were incorrect; we specifically included our secondary outcome to help assess the impact of potential misclassification of the primary outcome. Also, data are reported at the test level and not at the individual level; as such, it was not possible to link the results of repeated testing events for the same person. However, the definition of a confirmed HIV-positive testing event minimized this limitation for persons who were newly identified because records for which there was a current HIV-positive test result and a history of a previous HIV-positive test were excluded from the definition of this outcome. Finally, data included in this study represented HIV testing funded by the CDC. Besides the 4 states that specifically declined participation, other non–CDC-funded HIV testing occurred that was not captured in the data set and thus may have contributed to selection bias. Given the large number of observations, however, we believe our results are generalizable to all geographic regions and HIV testing venues in the United States.

In summary, the DHRS accurately categorized patients into groups with different HIV prevalence and serves as a simple tool for quantifying risk and identifying individuals for HIV screening. The DHRS may contribute substantially to future HIV screening efforts, especially as the number of those with undiagnosed infection declines.

ACKNOWLEDGMENTS

The authors are indebted to the following individuals from CDC: John Beltrami, Dale Stratford, Natasha Hollis and Guoshen Wang. None of these individuals received compensation as part of this study.