JAIDS Journal of Acquired Immune Deficiency Syndromes:
Predictive Accuracy of the Veterans Aging Cohort Study Index for Mortality With HIV Infection: A North American Cross Cohort Analysis
Justice, Amy C. MD, PhD*; P. Modur, Sharada PhD†; Tate, Janet P. ScD, MPH‡; Althoff, Keri N. PhD, MPH†; Jacobson, Lisa P. ScD, MS†; Gebo, Kelly A. MD, MPH§; Kitahata, Mari M. MD, MPH‖; Horberg, Michael A. MD¶; Brooks, John T. MD#; Buchacz, Kate PhD#; Rourke, Sean B. PhD**; Rachlis, Anita MD††; Napravnik, Sonia PhD‡‡; Eron, Joseph MD‡‡; Willig, James H. MD§§; Moore, Richard PhD‖‖; Kirk, Gregory D. MD, PhD, MPH¶¶; Bosch, Ronald PhD##; Rodriguez, Benigno MD***; Hogg, Robert S. PhD†††; Thorne, Jennifer MD, PhD‡‡‡; Goedert, James J. MD§§§; Klein, Marina MD, MSc‖‖‖; Gill, John MD¶¶¶; Deeks, Steven MD###; Sterling, Timothy R. MD****; Anastos, Kathryn MD††††; Gange, Stephen J. PhD†; for the NA-ACCORD and VACS Project Teams
*Department of Internal Medicine, Yale University and the Veterans Affairs Healthcare System, West Haven, CT
†Department of Epidemiology, Johns Hopkins University, Baltimore, MD
‡Department of Biostatics, Yale University and the Veterans Affairs Healthcare System, West Haven, CT
§Department of Infectious Diseases, Johns Hopkins University, Baltimore, MD
‖Department of Allergy and Infectious Diseases, University of Washington, Seattle, WA
¶Division of Research, Mid-Atlantic Permanente Research Institute, Rockville, MD
#Division of HIV AIDS Prevention, Centers for Disease Control and Prevention, Atlanta, GA
Departments of **Psychiatry and
††Medicine, Ontario Treatment Network, University of Toronto, Toronto, Canada
‡‡Department of Medicine, University of North Carolina Chapel Hill, Chapel Hill, NC
§§Department of Internal Medicine, University of Alabama at Birmingham, Birmingham, AL
Departments of ‖‖Medicine and
¶¶Epidemiolody, Johns Hopkins University, Baltimore, MD
##Department of Biostatiscics, Harvard Medical School, Boston, MA
***Department of Medicine, Case Western Reserve University, Cleveland, OH
†††Department of Health Sciences, British Columbia Centre for Excellence and HIV/AIDS and Simon Fraser University, Vancouver, Canada
‡‡‡Department of Ophthalmology and Epidemiology, Johns Hopkins University, Baltimore, MD
§§§Division of Cancer Epidemiology and Genetics, National Institutes of Health, Bethesda, MD
‖‖‖Department of Medicine, McGill University, Montreal, Canada
¶¶¶Department of Medicine, University of Calgary, Calgary, Canada
###Department of Medicine, University of California—San Francisco, San Francisco, CA
****Department of Medicine, Vanderbilt University, Nashville, TN
††††Department of Medicine, Einstein Medical School, New York, NY.
Correspondence to: Amy C. Justice, MD, PhD, Veterans Affairs Connecticut Healthcare System-11ACSLG, Building 35a, Room 2-212, 950 Campbell Avenue, West Haven, CT 06516 (e-mail: firstname.lastname@example.org).
Supported financially by National Institutes of Health: NIAAA (U10-AA13566), NHLBI (R01-HL095136; R01-HL090342; RCI-HL100347), and NIA (R01-AG029154; K23 AG024896). Ms. J.P. Tate was supported by the Training Program in Environmental Epidemiology funded under grant no.T32 ES07069. This work was supported by grants from the National Institutes of Health: U01-AI069918, U01-AA013566, UO1-AI-35042, 5-MO1-RR-00052 (GCRC), UO1-AI-35043, UO1-AI-35039, UO1-AI-35040, UO1-AI-35041, U01-AI38855: ALLRT; U01-AI38858; U01-AI68634; U01-AI68636; AI-69432; AI-69434, UO1-AI-35004, UO1-AI-31834, UO1-AI-34994, UO1-AI-34989, UO1-AI-34993, and UO1-AI-42590, UO1-HD-32632, UL1-RR024131, P30-AI27757; K23-AI-61-0320, P30-AI27767, P30-AI50410; RR025747, P30-AI54999, R01-DA04334; R01-DA12568, R01-MH54907, R24-AI067039, N02-CP55504; Z01-CP010176, AHQ290-01-0012, K01-AI071754, K24-00432; R01-DA11602, K01-AI071725, R01 AG026250 to K.A. Gebo. This work was also supported by the Centers for Disease Control (CDC200-2006–18797), the Canadian Institutes for Health Research (CIHR: TGF-96118; HCP-97105; CBR-86906; CBR-94036; KRS-86251; 169621), and the Canadian Trials Network (project number 242).
The authors have no conflicts of interest to disclose.
The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention or those of the Veterans Affairs.
Received June 26, 2012
Accepted November 13, 2012
Background: By supplementing an index composed of HIV biomarkers and age (restricted index) with measures of organ injury, the Veterans Aging Cohort Study (VACS) index more completely reflects risk of mortality. We compare the accuracy of the VACS and restricted indices (1) among subjects outside the Veterans Affairs Healthcare System, (2) more than 1–5 years of prior exposure to antiretroviral therapy (ART), and (3) within important patient subgroups.
Methods: We used data from 13 cohorts in the North American AIDS Cohort Collaboration (n = 10, 835) limiting analyses to HIV-infected subjects with at least 12 months exposure to ART. Variables included demographic, laboratory (CD4 count, HIV-1 RNA, hemoglobin, platelets, aspartate and alanine transaminase, creatinine, and hepatitis C status), and survival. We used C-statistics and net reclassification improvement (NRI) to test discrimination varying prior ART exposure from 1 to 5 years. We then combined Veterans Affairs Healthcare System (n = 5066) and North American AIDS Cohort Collaboration data, fit a parametric survival model, and compared predicted to observed mortality by cohort, gender, age, race, and HIV-1 RNA level.
Results: Mean follow-up was 3.3 years (655 deaths). Compared with the restricted index, the VACS index showed greater discrimination (C-statistics: 0.77 vs. 0.74; NRI: 12%; P < 0.0001). NRI was highest among those with HIV-1 RNA <500 copies per milliliter (25%) and age ≥50 years (20%). Predictions were similar to observed mortality among all subgroups.
Conclusions: VACS index scores discriminate risk and translate into accurate mortality estimates over 1–5 years of exposure to ART and for diverse patient subgroups from North American.
With the advent of effective antiretroviral therapy (ART), the spectrum of disease experienced by those with HIV infection has changed. Viral suppression is common1 and incident AIDS-defining events are rare.2 Yet, those with HIV infection continue to experience excess mortality3,4 which is incompletely described by age, CD4 count, and HIV-1 RNA alone.5
Despite ART, chronic HIV infection seems to exacerbate generic pathophysiologic processes associated with aging which increase physiologic vulnerability relative to demographically similar HIV-uninfected individuals.6–8 Consistent with current treatment guidelines,9 HIV providers routinely monitor general indicators of organ system injury including hemoglobin, platelets, aspartate transaminase (AST) and alanine transaminase (ALT), creatinine, and viral hepatitis C virus infection (HCV) but have no index with which to integrate these data into an overall estimate of disease burden or mortality risk. Such a comprehensive measure would be useful as a means of more effectively motivating behavior change in the clinical setting,10 improved risk stratification in the analysis of observational data,11 and more effective randomized trials.12 For example, indices such as the Framingham risk index have enhanced research and care in cardiovascular disease,13 and several geriatric risk indices are enhancing research and care for those aging without HIV infection.14
Although the cumulative evidence supporting the accuracy and generalizability of the Veterans Aging Cohort Study (VACS) index exceeds that for any prior HIV risk index, the VACS index builds upon important prior research.15–22 Most prior indices emphasized AIDS-defining conditions, CD4 cell count, and HIV-1 RNA. Some recognized the importance of age and anemia.16,20 However, much has changed since these indices were developed and validated. Specifically, the increasing role of multiorgan system injury (reflected by FIB-4, eGFR, and hemoglobin) and of HCV, and the decreasing role of AIDS-defining illnesses, CD4 count, and HIV-1 RNA. By including FIB-4, HCV, eGFR, hemoglobin, and age, and placing less weighting upon CD4 count and HIV-1 RNA, the VACS index better reflects more of the major common pathways of physiologic injury among those on ART. As a result, the VACS index discriminates risk of mortality more effectively than an index restricted to CD4 count, HIV-1 RNA, and age (restricted index).23,24
Importantly, the discrimination of the VACS index rivals that of indices in clinical use including the Framingham index13 and those recommended for use among geriatric patients.14 Nevertheless, prognostic indices developed in one sample [those within the Veterans Affairs Healthcare System (VA)] may not generalize to a new sample or important subgroups.25 Further, indices effective at one particular point in clinical care (ART initiation) may not generalize beyond treatment initation.25 We use data from the North American AIDS Cohort Collaboration (NA-ACCORD) to test the generalizability of the VACS index outside the VA and at differing intervals of exposure to ART. We then combine data from NA-ACCORD and VA to translate index scores to an estimated absolute risk of mortality and compare predicted to observed mortality by cohort and subgroups defined by sex, age, race, and HIV-1 RNA titer.
NA-ACCORD has been described in detail.26–28 It is a multisite collaboration of interval and clinical cohort studies in the United States and Canada and represents the North American region of the International epidemiologic Databases to Evaluate AIDS (IeDEA). VACS has also been described in detail.29 Although VACS is a participating cohort within NA-ACCORD, we separated VACS patients from the NA-ACCORD for this analysis to demonstrate the generalizability of our findings outside the Veterans Healthcare Administration.
We used data from 13 NA-ACCORD cohorts that routinely collect and contribute the laboratory data required for construction of the VACS index. All included cohorts monitor deaths at local sites and regionally using death registries. United States cohorts also check for deaths using national registries (Social Security Administration or National Death Index). Among these cohorts, eligible subjects were HIV-infected individuals on ART for at least 1 year from 2000 to 2007 (n = 15,938). Of these, 10,835 (68%) had complete data after 12 months of ART (from 90 days before to 180 days after) and constituted our sample with full data (complete cases). Using the same eligibility criterion, 5066 VACS subjects were available for analyses. Both VACS and NA-ACCORD studies are approved by affiliated institutional review boards.
VACS and Restricted Index Scores
The development and internal validity of the VACS index has been described (Table 1).30 It includes age and routinely monitored laboratory tests as follows: CD4 count, HIV-1 RNA, hemoglobin, platelets, AST, ALT, creatinine, and HCV status. Composite markers of liver and renal injury (FIB-4 and eGFR) are computed. FIB-4, composed of AST, ALT, platelets, and age, has been validated as an indicator of liver fibrosis [FIB-4 = (years of age × AST)/(platelets in 100/L × square root of ALT)].31 eGFR, composed of serum creatinine age, gender, and race, was included as a validated indicator of impaired renal function [eGFR = 186.3 × (creatinine)−1.154 × (age)−0.203 × (0.742 for women) × (1.21 if black)].32 HCV status was defined as positive if the patient ever had a positive antibody test or detectable virus before the anchoring point of our analysis (12 months of ART). Points are added to calculate score. The restricted index was developed solely for the purposes of comparing the accuracy and generalizability of an index restricted to CD4 count, HIV-1 RNA, and age with that of a more completely specified index. All predictors are categorized according to previously established cut points.5,30–33
Statistical analyses were conducted by S.M., J.P.T., and S.G.. Using the point system described above, VACS and restricted index scores were assigned to each subject at 1, 2, 3, 4, and 5 years of ART exposure. We evaluated the discrimination of the indices at these anchoring points. Observation time ended at death or was censored at the date of last follow-up, December 30, 2007 (administrative censoring), or 5 years from each anchoring point, whichever came first. For year 1, laboratories were obtained −90 to+180 days of the anchor date; and ±180 days for years 2–5. These 5 anchors were then used to assess and compare discrimination of the indices using Cox proportional hazards models and Harrell C-statistics among NA-ACCORD subjects only. We also measured C-statistics among NA-ACCORD subjects after stratifying by sex (men, women), age (<50 years, ≥50 years), race (white, black, and Hispanic), and HIV-1 RNA (<500 copies/mL, ≥500 copies/mL). Proportion of NA-ACCORD subjects reclassified by VACS index compared with restricted index was calculated using the method by Cook and Ridker34 and Cook35 (see Appendix).
Of 15,938 eligible NA-ACCORD subjects, 32% were missing at least 1 required laboratory value and were excluded from the complete case analyses. Those with complete data differed from those with missing data on gender, race, injection drug use, hemoglobin, platelets, AST, ALT, and FIB-4. We applied multiple imputation methods to the entire NA-ACCORD sample; results were similar compared with the complete case analyses (see Appendix).
To translate scores to predicted mortality with maximal precision, we combined NA-ACCORD and VA subjects and fit a parametric (gamma) regression model predicting all-cause mortality using VACS index score as the only predictor. This model provided an equation for calculating predicted mortality over 1–5 years for each value of the VACS index score (Fig. 1). Five-year mortality predictions were compared graphically with observed mortality between NA-ACCORD and VA subjects separately and among designated subgroups. For each 5-point interval of score (collapsed if necessary to maintain at least 5 deaths and 10 survivors in each interval), a Kaplan–Meier mortality estimate and 95% confidence interval were calculated.
Characteristics of the Population
The NA-ACCORD sample (n = 10,835) included 2982 women, 2407 people ≥50 years, and 3557 black individuals. After 1 year of ART, 77% of the entire sample had HIV-1 RNA <500 copies per milliliter, 34% had hemoglobin values between 12–13.9 g/dL, 10% had hemoglobin values between 10 and 11.9 g/dL, 25% had FIB-4 consistent with fibrosis (>3.25), 6% had stage III renal insufficiency (eGFR < 60 mL/min), and 24% had HCV coinfection (Table 1). The overall mortality was 1.6 per 100 person-years. Median scores were 16 with a 1%–99% range of 0–80 for the VACS index and 10 with a range of 0–71 for the restricted index.
Prognostic Accuracy in NA-ACCORD
Among NA-ACCORD subjects overall, the VACS index was more discriminating of all-cause mortality than the restricted index (Table 2, C-statistics: 0.77 vs. 0.74) and among men and women; whites, blacks, and Hispanics; those <50 years and ≥50 years of age; and those with HIV-1 RNA <500 copies per milliliter and ≥500 copies per milliliter (P < 0.0001 in all cases). Discrimination of the restricted index declined with increased prior ART exposure (C-statistics: 0.74 at 2-year anchor; 0.72 at 5-year anchor), whereas discrimination of the VACS index remained strong (C-statistics: 0.79 at 2 years; 0.81 at 5 years). When compared with the restricted index, the VACS index resulted in the reclassification of 53% of patients as follows: 22% to a higher risk group and 31% to a lower risk group. The net gains in reclassification proportions at 5 years were 9% for survivors and 3% for those who died for a net reclassification improvement (NRI) of 12% (P < 0.0001). NRI was 25% among those with undetectable HIV-1 RNA and 20% among those 50 years and older. When 5-year Kaplan-Meier–observed mortality estimates were graphed, the VACS index demonstrated a finer gradation of risk for more patients and a wider range of observed mortality compared with the restricted index (Figs. 1A, B).
When components of the VACS index were evaluated separately and in combination in NA-ACCORD and VACS cohorts (Table 3), we found that age alone offered modest risk discrimination (C-statistics: 0.56, 0.59, respectively). HIV biomarkers were more discriminating in NA-ACCORD than VACS data (C-statistics: 0.72, 0.67). Organ system biomarkers were less discriminating in NA-ACCORD than VACS data (C-statistics: 0.70, 0.75). When age, HIV biomarkers, and organ system biomarkers were combined, discrimination was improved in both cohorts (C-statistics: 0.78 for both).
Calibration of Model Predictions Using Combined NA-ACCORD and VA Data
When a parametric survival model was calculated (Fig. 2), predicted mortality was similar to observed mortality at 5 years for both NA-ACCORD and VA subjects (Figs. 3A, B) and when stratified by important subgroups (Figs. 3C–J).
The VACS index provided a more discriminating prediction of all-cause mortality among HIV-infected subjects from North America on ART than the restricted index. This was true overall, with increasing exposure to ART, and among important subgroups, most notably among persons with low HIV-1 RNA and those ≥50 years of age—2 rapidly growing populations in treatment. Based on established criteria,13,25 the VACS index has demonstrated excellent generalizability and is likely to accurately predict mortality among HIV-infected patients on ART in North America. Importantly, after demonstrating that this translation is accurate in demographically and clinically diverse subgroups, we provide a table and nomogram (Fig. 2; Table 1) and a website (http://vacs.med.yale.edu) to facilitate calculating VACS index scores and translating them to predicted mortality rates. Potential applications for the VACS index include patient management and clinical research.
C-statistics are a commonly employed metric for evaluating the discrimination of prognostic indices.36 In uncensored data, the C-statistic is the likelihood that, if any 2 subjects were drawn from the sample, the subject with the higher score would die before the subject with the lower score. Although these categories are somewhat arbitrary, C-statistics between 0.50 and 0.59 are considered poor; 0.60 and 0.69, fair; 0.70 and 0.79, good; 0.80 and 0.89 very good; and above 0.89, excellent.14 Although restricted index C-statistics ranged from 0.63 to 0.76 (“fair” to “good”), VACS index C-statistics ranged from 0.70 to 0.81 (“good” to “very good”). Discrimination was particularly better among those with undetectable HIV-1 RNA (C-statistics: 0.67 vs. 0.74) and those older than 50 years of age (C-statistics: 0.63 vs. 0.70). C-statistics for the VACS index for all-cause mortality meet or surpass those reported for prognostic indices used in clinical practice including the Framingham index for predicting cardiovascular disease and validated indices predicting all-cause mortality among aging HIV-uninfected individuals.13,37
A newer metric, developed and popularized by the methodologists working on the Framingham Risk score, is the NRI.34,35 This is calculated by separating those who died and those who lived and asking in each group whether the VACS index resulted in a change in risk classification compared with the restricted index. Among those who died, a higher risk classification is considered an improvement and a lower risk classification is considered an error. Among those who lived, a lower risk classification is considered an improvement and a higher classification is considered an error. The NRI is the sum of the differences. The net gain in reclassification proportions at 5 years was 9% for survivors and 3% for those who died for an overall statistic of 12% (P < 0.0001). Further, the NRI was even higher among those with undetectable HIV-1 RNA (25%) and those 50 years and older (20%). These NRIs suggest a highly clinically significant improvement in discrimination35,38 and are greater than improvements seen by the addition of D-dimer to the VACS index.24
For maximal clinical and research utility, providers and investigators need a means of translating VACS index scores to mortality risk. We combined NA-ACCORD and VACS data to provide as precise a translation as possible. We then considered the accuracy of this translation by cohort and among important subgroups. Because such translations depend upon the overall mortality rate in the cohort,13,25,37 we conducted this work among cohorts with uniform access to regional and/or national death registries. In these analyses, the predicted mortality based upon VACS index score was similar to observed mortality among veteran (VACS) and nonveteran (NA-ACCORD) subjects; and among: men and women; black and non-black patients; those <50 years and those ≥50 years old; and those with HIV-1 RNA <500 copies per milliliter and those with ≥500 copies per milliliter.
To understand how the VACS index reclassifies risk, consider an HIV-infected 45-year-old man who, after 12 months of ART, has a CD4 count of 500 cells per cubic millimeter and an undetectable HIV-1 RNA but is HCV coinfected with a FIB-4 >3.25. He, like 1 in 4 NA-ACCORD subjects, was assigned 0 points by the restricted index with a 2% predicted 5-year mortality. Using the VACS index, he was assigned 5 points for HCV coinfection and 25 points for his FIB-4 value for a score of 30 and a predicted 5-year mortality of 12%. Fifty-three percent of NA-ACCORD subjects assigned a score of 0 by the restricted index were assigned a higher score by the VACS index.
Having an accurate, generalizable, responsive, and feasible method for estimating individual risk can improve effectiveness and efficiency of chronic disease management in major ways.39–41. First, it can inform decision making when an intervention puts the patient at some immediate risk for long-term gain.42,43 This is true whenever patients are asked to undergo a risk of immediate harm from treatment in the hope of averting long-term disease incidence or progression—commonly the case in cancer screening and primary and secondary prophylaxis for cardiovascular disease and stroke. It is also true when considering aggressive treatment protocols (toxic chemotherapy, organ transplant, or major surgery) for cancer or heart disease. Second, it can motivate patients to modify health behaviors such as adherence to medication, smoking, diet, exercise, and alcohol use by quantifying the impact these changes have on risk and by charting progress after modifying risk.44–48 Third, it can identify patients in need of intensive management either with respect to site of care (outpatient, inpatient, intensive care unit, skilled nursing facility, nursing home) or care management (case management, frequency of follow-up).
Of note, none of these applications require that an index identify all modifiable sources of risk, only that it accurately, generalizably, responsively, and feasibly estimate risk of mortality—including risk associated with the modifiable factors of interest.25 Because many sources of modifiable risk have a similar common pathway to physiologic injury, it is not efficient or feasible for a single index to include all modifiable sources of risk. Instead, separate analyses can map changes in risk score associated with changes in modifiable factors of interest. We are currently undertaking a series of analyses demonstrating this for adherence to ART, alcohol use, smoking, and substance use, but these are beyond the scope of this article.
The VACS index also predicts cardiovascular mortality,49 hospitalization, and medical intensive care unit admission50 and is correlated with functional performance51 and fragility fractures.52 It offers an improved means of balancing patient enrollment by severity of illness in randomized trials and of controlling for disease severity in observational analyses, and it may eventually prove a useful intermediate outcome for interventional and observational research. Because the discrimination of the VACS index for mortality is maintained over extended prior exposure to ART, it also offers a means of charting a patient's progress over time.
Although the utility of the VACS index for clinical management can only be established through a randomized trial comparing management with and without the index, evidence to date suggests that it offers useful insight. We have previously shown that hemoglobin, FIB-4, eGFR, CD4 count, and HIV-1 RNA, change substantially in response to ART initiation, not always in the same direction,33 and that the VACS index is more responsive to ART initiation and differing levels of ART adherence than the restricted index.33 Third, we have shown that the VACS index is more correlated with biomarkers of inflammation (interleukin 6), microbial translocation (D-dimer), and hyper coagulability (sCD14) than the restricted index.5 Taken together, these data suggest that the VACS index provides a more comprehensive means of tracking disease burden, including the effects of chronic inflammation, over time.
Further, to facilitate use of the VACS index in the clinical setting, we have developed a web site calculator accessible via smart phone or computer with an automatic conversion of the VACS index score to a risk estimate (http://vacs.med.yale.edu). The site includes regularly updated links to supporting evidence for the index. As we develop information regarding how behavior change alters risk, we will include this information as an additional link for the calculator. We also provide SAS programming for any who wish to include the calculation as part of their electronic medical record system or for analyses of grouped clinical data (www.vacohort.org).
Our analysis has substantial strengths. We demonstrated the generalizability of the VACS index in a large independent sample on ART over differing periods of ART exposure and among important subgroups of patients.25 By combining NA-ACCORD and VA samples, we were powered to precisely translate VACS index scores to predicted mortality and to consider whether predicted mortality matched observed deaths overall and among important subgroups. Further, the VACS index is based on laboratory tests currently ordered in the course of routine management and therefore offers enhanced clinical insight requiring only that providers calculate and interpret the score. We have simplified this process by providing a web site and smart phone calculator (http://vacs.med.yale.edu). Eventually the index could be calculated by the clinical laboratory (or the electronic medical record) every time component tests are ordered, as is the current practice for eGFR.
A limitation of any large observational study is missing data. Although subjects with missing values in NA-ACCORD tend to have more liver disease and less anemia, the imputed analyses yielded results similar to complete cases (Appendix), suggesting that missing data did not compromise our findings. Further, the VACS index may be improved in the future. Our choice of risk factors was based on prior work,16,20,21,23,53 a desire to base the index on consistently measured metrics such as clinical laboratory tests, and the need to validate findings. As the population with HIV ages, higher age thresholds will likely become relevant. Of note, we have evaluated whether the following: body mass index, lipid profiles, smoking status, hypertension54; inflammatory biomarkers (D-dimer, interleukin 6, and soluble CD14)24; and functional status55 improve the discrimination of the VACS index. Although many of these predict mortality in unadjusted analyses, VACS index scores co vary with these factors. When added to the VACS index, only D-dimer and/or sCD14 resulted in risk reclassification that exceeded 1%. Additional factors (such as D-dimer) may be added to the VACS index in the future if they can be consistently measured and they improve discrimination sufficiently to justify added cost and complexity.13,35
In summary, we have demonstrated that a novel index composed of routine clinical data can predict mortality among HIV-infected individuals on ART with good to very good discrimination and consistent calibration across important subgroups. Measures of general organ system function included in the VACS index substantially enhance discrimination. Although it would be a reasonable precaution to verify the calibration of the VACS index among younger patients and subjects outside North America, predicted mortality from the VACS index is likely generalizable to HIV-infected individuals older than 30 years of age in care in North America. Among these individuals, the VACS index is ready for clinical and research application.
The authors would like to thank all individuals involved with the NA-ACCORD collaboration, including staff, investigators, and patients, for their valuable contributions to this work.
Risk reclassification is an approach that evaluates the improvement of prognostic models with the addition of new markers.56 It addresses limitations in discrimination metrics such as changes in the C-statistic by evaluating how individuals are reassigned into risk categories with the addition of new predictors.34 Risk reclassification tables are primarily used to compare models with and without single factors; we applied the technique to examine models with and without a set of covariates.35
We evaluated reclassification from a model that used the “restricted index” (calculated using age at anchoring time, CD4 count, and log10 HIV viral load) with a model that used the full VACS index (calculated using all the variables in the restricted index and hemoglobin, baseline HCV status, FIB4, and eGFR). All variables were categorized as shown in the main article (Table 1). Note that one participating cohort, with only 5 subjects and no events, was dropped because predicted probabilities could not be estimated.
From the risk reclassification analysis, our specific methods follow. First, we fit a Cox regression model for 5-year mortality using the restricted index score, stratified by cohort to account for cross-cohort heterogeneity. We identified intervals of continuous ART for all subjects occurring between January 1, 2001, and December 30, 2007, and selected the anchoring time to be 1 year after the start of the study interval. We used the restricted index score calculated at 1 year of ART exposure (−90 days to 180 days) as the only variable in the regression. Follow-up continued from anchoring time to death date or was censored at the date of last follow-up, December 30, 2007 (administrative censoring), or 5 years from the anchoring point, whichever came first.
Based on this model, we calculated the predicted probability of death at 5 years for each subject using the Gibbs Sampling algorithm assuming noninformative priors (the default in SAS V9). These steps were repeated replacing the restricted index score with the full VACS index score.
Risk strata were derived using quintiles of predicted probabilities from the restricted model: 0 to <3%, 3 to <5%, 5 to <7%, 7 to <11%, and ≥11%. The decision to use quintiles was based on the comments by Pencina et al56 who noted that reclassification measures are heavily dependent on the choice of risk strata. Individuals were categorized based on their predicted probability from each model. (Appendix Table 1). Rows represent risk categorization from the restricted index model and columns that form the VACS index model. For example, 1586 of the 10,830 participants were categorized in the 0 to <3% risk stratum based on their predicted probability of death at 5 years by both the restricted index and VACS index models. However, 363 participants classified into the lowest risk stratum by the restricted index were reclassified by the VACS index into the 3 to <5% risk stratum.
APPENDIX TABLE 1 Str...Image Tools
Using these methods, 53% of 10,830 subjects were reclassified into different risk strata. The overall percentage reclassified gives an indication of how many subjects would change risk categories under the VACS model. Twenty-two percent moved to a higher risk group. Thirty-one percent moved to a lower risk group.
We sought to supplement these descriptive results with 2 additional evaluations. First, we computed a version of the Hosmer–Lemeshow goodness-of-fit statistic known as the reclassification calibration statistic for both the restricted index model and VACS index model.35 This is a comparison of observed and predicted 5-year cumulative mortality for each model. Predicted mortality came from the mean predicted probability of death for each cell from each model. We can then compute the expected number of events by multiplying the mean probability by the number of subjects in that cell. Observed 5-year cumulative mortality was obtained from Kaplan–Meier estimates of 5-year all-cause mortality for each cell. For example, for the first cell, the Kaplan–Meier estimate for 5-year all-cause mortality is 1.3%. We can also compute the Kaplan–Meier estimate of the observed number of “deaths” in a similar manner. For each cell with at least 20 subjects, we can find the squared differences between the observed and expected number of events and the χ2 goodness-of-fit test for each model separately. The restricted index had a strong lack of fit with a χ2 statistic of 220 on 19 degrees of freedom. The VACS index had a χ2 test statistic of 58 on 19 degrees of freedom (P = 0.0008) also indicating lack of fit but to a much lesser extent.
Second, we explored whether the VACS index offered clinically meaningful improvement over the restricted index using net NRI.56 NRI is the difference between the proportions moving up and down risk strata among those who died versus those who survived at 5 years. This measure is similar to the percent reclassified but distinguishes movements in the correct direction. The second layer of numbers within each cell pertains to the 571 individuals who died by 5 years. Among those who died, the cells above the diagonal in Appendix Table 1 correspond to (3 + 4 + 2 + 2 + 17 + 10 + 9 + 19 + 9 + 55) = 130 of 571 (23%) participants who moved up to a higher risk category. This means that for 130 people who were dead at the 5-year mark, classification improved using the VACS model. For the cells below the diagonal, (10 + 19 + 6 + 33 + 7 + 42) = 117 of 571 (20%) participants moved down to a lower risk category using the VACS index. Among those who died at 5 years, the overall improvement was 3% (23%–20%; P = 0.41). The third layer of numbers within each cell of Table 1 are 10,259 individuals who were alive (or censored) at 5 years. Of these, (360 + 164 + 41 + 13 + 556 + 217 + 58 + 433 + 129 + 337) = 2308 of 10,259 (23%) moved up to a higher risk category using the VACS index. Another (908 + 66 + 491 + 3+ 229 + 785 + 1 + 123 + 628) = 3234 of 10259 (32%) moved down to a lower risk category The overall gain in reclassification proportions for survivors at 5 years was 9% (32%–23%) and was significantly greater than zero (P = <0.0001). The NRI, computed by summing the overall improvement for those who died and for those alive at 5 years (3 + 9) is 12% and is also significantly different from zero (0 = <0.0001). NRI suggests the VACS index model for 5-year all-cause mortality results in significant improvement in performance compared with the restricted index. Cited Here...
Missing Data Methods
Based on the initial criteria for the study, we found 15,938 eligible subjects. Of these 10,835 (68%) subjects had complete information on both the outcome and all necessary laboratory measurements required to construct the VACS index score. Therefore, 32% of eligible subjects were missing a required laboratory value but were complete on all other variables of interest, including the outcome. Consequently, we were unable to construct a VACS and/or a restricted index score for these subjects. Patterns of missingness were largely arbitrary.
In an effort to characterize patients with missing data, we explored differences between those with complete and incomplete laboratories. These groups differed on almost all variables of interest (Appendix Table 2), but there had similar 5-year all-cause mortality (Appendix Fig. 1). Based on these initial findings, we explored use of multiple imputation methods that allowed us to forgo any strict assumptions regarding the appropriateness of the complete case analysis.
Multiple imputation was performed using PROC MI procedure in SAS V9 assuming multivariate normality for the distribution of laboratory values, and nonmonotone missingness. We used the Markov Chain Monte Carlo procedure to obtain 25 completed datasets. This method imputes missing values by simulating draws from the joint complete distribution via data augmentation.57 Convergence of the Markov Chain Monte Carlo simulation was verified using techniques proposed by Gelman et al.58 We also explored graphical diagnostics (Q–Q plots) and numerical values (means and ranges) of the imputations to verify that our imputed values were sampled within a reasonable range. Summary statistics for the imputed laboratory data (Appendix Table 3) indicate close agreement between complete cases and imputed datasets. Although mean VACS index and restricted index scores for the complete cases were lower than those from the imputed data, Q–Q plots revealed close agreement between observed and imputed scores (Appendix Fig. 2). Cox models were refitted in the 25 imputed datasets using the VACS index score and restricted index scores. Standard errors used for calculating confidence intervals were computed by combining the estimates using Rubin rules. The resulting hazard ratios (95% confidence intervals) based on the imputed data were 1.041 (1.038-1.044) for the VACS Index and 1.040 (1.037-1.043). Harrell C-statistics59 were computed (overall and for relevant subgroups) using the means from 25 imputation sets (Appendix Table 4). We also plotted calibration curves using the imputed data for both the VACS and restricted index models and found results similar to the complete case analysis (Appendix Fig. 3). We conclude that the results based on multiple imputation are similar to those based on the complete case analysis. For simplicity, the main article presents results from the complete case analysis only.
1. McKinnell JA, Willig JH, Westfall AO, et al. Antiretroviral prescribing patterns in treatment-naive patients in the United States. AIDS Patient Care STDS. 2010;24:79–85.
2. D'Arminio MA, Sabin CA, Phillips A, et al. The changing incidence of AIDS events in patients receiving highly active antiretroviral therapy. Arch Intern Med. 2005;165:416–423.
3. Hogg R, Lima V, Sterne JA, et al. Life expectancy of individuals on combination antiretroviral therapy in high-income countries: a collaborative analysis of 14 cohort studies. Lancet. 2008;372:293–299.
4. Losina E, Schackman BR, Sadownik SN, et al. Racial and sex disparities in life expectancy losses among HIV-infected persons in the united states: impact of risk behavior, late initiation, and early discontinuation of antiretroviral therapy. Clin Infect Dis. 2009;49:1570–1578.
5. Justice AC, Freiberg MS, Tracy R, et al; and The VACS Project Team. Does an index composed of clinical data reflect effects of inflammation, coagulation, and monocyte activation on mortality among those aging with HIV? Clin Infect Dis. 2012;54:984–994.
6. Justice AC. HIV and aging: time for a new paradigm. Curr HIV/AIDS Rep. 2010;7:69–76.
7. Deeks SG. Immune dysfunction, inflammation, and accelerated aging in patients on antiretroviral therapy. Top HIV Med. 2009;17:118–123.
8. Phillips AN, Neaton J, Lundgren JD. The role of HIV in serious diseases other than AIDS. AIDS. 2008;22:2409–2418.
9. Panel on Antiretroviral Guidelines for Adults and Adolescents. Guidelines for the Use of Antiretroviral Agents in HIV-1-Infected Adults and Adolescents. Washington, DC: Department Health and Human Services; 2011:1–166.
10. DiMatteo MR, Haskard KB, Williams SL. Health beliefs, disease severity, and patient adherence: a meta-analysis. Med Care. 2007;45:521–528.
11. Feinstein AR. Clinical biostatistics, XIV: the purposes of prognostic stratification. Clin Parmacol Ther. 1972;13:285–297.
12. Feinstein AR, Landis RJ. The role of prognostic stratification in preventing the bias permitted by random allocation of treatment. J Chron Dis. 1979;29:277–284.
13. Vasan RS. Biomarkers of cardiovascular disease: molecular basis and practical considerations. Circulation. 2006;113:2335–2362.
14. Yourman LC, Lee SJ, Schonberg MA, et al. Prognostic indices for older adults: a systematic review. JAMA. 2012;307:182–192.
15. Turner BJ, Kelly JV, Ball JK. A severity classification system for AIDS hospitalizations. Med Care. 1989;27:423–437.
16. Justice AC, Feinstein AR, Wells CK. A new prognostic staging system for the acquired immunodeficiency syndrome. N Engl J Med. 1989;320:1388–1393.
17. Egger M, May M, Chene G, et al. Prognosis of HIV-1-infected patients starting highly active antiretroviral therapy: a collaborative analysis of prospective studies. Lancet. 2002;360:119–129.
18. May M, Royston P, Egger M, et al. Development and validation of a prognostic model for survival time data: application to prognosis of HIV positive patients treated with antiretroviral therapy. Stat Med. 2004;23:2375–2398.
19. Mocroft AJ, Johnson MA, Sabin CA, et al. Staging system for clinical AIDS patients. Lancet. 1995;346:12–17.
20. Mocroft A, Kirk O, Barton SE, et al. Anaemia is an independent predictive marker for clinical prognosis in HIV-infected patients from across Europe. EuroSIDA study group. AIDS. 1999;13:943–950.
21. Harris RJ, Sterne JA, Abgrall S, et al. Prognostic importance of anaemia in HIV type-1-infected patients starting antiretroviral therapy: collaborative analysis of prospective cohort studies. Antivir Ther. 2008;13:959–967.
22. Lundgren JD, Mocroft A, Gatell JM, et al. A clinically prognostic scoring system for patients receiving highly active antiretroviral therapy: results from the EuroSIDA study. J Infect Dis. 2002;185:178–187.
23. Justice AC, McGinnis KA, Skanderson M, et al. Towards a combined prognostic index for survival in HIV infection: the role of ‘non-HIV' biomarkers. HIV Med. 2009;11:143–151.
24. Justice AC, Freiberg MS, Tracy R, et al. Does an index composed of clinical data reflect effects of inflammation, coagulation, and monocyte activation on mortality among those aging with HIV? Clin Infect Dis. 2012;54:984–994.
25. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med. 1999;130:515–524.
26. Gange SJ, Kitahata MM, Saag MS, et al. Cohort profile: the North American AIDS Cohort Collaboration on Research and Design (NA-ACCORD). Int J Epidemiol. 2007;36:294–301.
27. Lau B, Gange SJ, Moore RD. Interval and clinical cohort studies: epidemiological issues. AIDS Res Hum Retroviruses. 2007;23:769–776.
28. Lau B, Gange SJ, Moore RD. Risk of non-AIDS-related mortality may exceed risk of AIDS-related mortality among individuals enrolling into care with CD4+ counts greater than 200 cells/mm3. J Acquir Immune Defic Syndr. 2007;44:179–187.
29. Fultz SL, Skanderson M, Mole LA, et al. Development and verification of a virtual cohort using the National VA Health Information System. Med Care. 2006;44:S25–S30.
30. Tate JP, Justice AC, Hughes MD, et al. An internationally generalizable risk index for mortality after one year of antiretroviral therapy. AIDS. 2012 Nov 6. [Epub ahead of print]
31. Sterling RK, Lissen E, Clumeck N, et al. Development of a simple noninvasive index to predict significant fibrosis in patients with HIV/HCV coinfection. Hepatology. 2006;43:1317–1325.
32. Stevens LA, Coresh J, Greene T, et al. Assessing kidney function—measured and estimated glomerular filtration rate. N Engl J Med. 2006;354:2473–2483.
33. Tate JP, Justice AC; for the VACS Project Team. Change in a prognostic index for survival in HIV infection after one year on cART by level of adherence. Presented at: 48th Annual Meeting of the Infectious Disease Society of America; 21–24 October 2010, Vancouver, British Columbia.
34. Cook NR, Ridker PM. Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann Intern Med. 2009;150:795–802.
35. Cook NR. Assessing the incremental role of novel and emerging risk factors. Curr Cardiovasc Risk Rep. 2010;4:112–119.
36. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115:928–935.
37. D'Agostino RB Sr, Grundy S, Sullivan LM, et al. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 2001;286:180–187.
38. Cook NR. Comments on ‘Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27:191–195.
39. Kent DM, Rothwell PM, Ioannidis JP, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials. 2010;11:85.
40. Hayward RA, Kent DM, Vijan S, et al. Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis. BMC Med Res Methodol. 2006;6:18.
41. Eddy DM, Adler J, Patterson B, et al. Individualized guidelines: the potential for increasing quality and reducing costs. Ann Intern Med. 2011;154:627–634.
42. Braithwaite RS. Can life expectancy and QALYs be improved by a framework for deciding whether to apply clinical guidelines to patients with severe comorbid disease? Med Decis Making. 2011;31:582–595.
43. Braithwaite RS, Concato J, Chang CC, et al. A framework for tailoring clinical guidelines to comorbidity at the point of care. Arch Intern Med. 2007;167:2361–2365.
44. Chan CW, Witherspoon JM. Health risk appraisal modifies cigarette smoking behavior among college students. J Gen Intern Med. 1988;3:555–559.
45. Dapp U, Anders JA, von Renteln-Kruse W, et al. A randomized trial of effects of health risk appraisal combined with group sessions or home visits on preventive behaviors in older adults. J Gerontol A Biol Sci Med Sci. 2011;66:591–598.
46. Kreuter MW, Strecher VJ. Do tailored behavior change messages enhance the effectiveness of health risk appraisal? Results from a randomized trial. Health Educ Res. 1996;11:97–105.
47. Maron DJ, Forbes BL, Groves JR, et al. Health-risk appraisal with or without disease management for worksite cardiovascular risk reduction. J Cardiovasc Nurs. 2008;23:513–518.
48. Stuck AE, Kharicha K, Dapp U, et al. The PRO-AGE study: an international randomised controlled study of health risk appraisal for older persons based in general practice. BMC Med Res Methodol. 2007;7:2.
49. Justice AC, Tate JP, Freiberg MS, et al. Reply to Chow et al. Clin Infect Dis. 2012 Jun 10. [Epub ahead of print]
50. Akgun KM, Pisani MA, Fried TR, et al. Risk factors for medical intensive care Unit admission in HIV infected veterans. AIDS. 2012 Oct 29. [Epub ahead of print]
51. Erlandson KM, Allshouse AA, Jankowski C. Prospective comparison of three functional assessments with the Veteran's Aging Cohort Study index in virologically suppressed HIV-infected adults. Presented at: 2nd International Workshop on HIV and Aging; Baltimore, Maryland, October 27–28, 2011. Abstract #8.
52. Womack JA, Goulet JL, Gibert C, et al. Increased risk of fragility fractures among HIV infected compared to uninfected male veterans. PLoS One. 2011;6:e17217.
53. May MT, Sterne JA, Costagliola D, et al. HIV treatment response and prognosis in Europe and North America in the first decade of highly active antiretroviral therapy: a collaborative analysis. Lancet. 2006;368:451–458.
54. Tate J, Freiberg M, Justice AC. Do risk factors for cardiovascular disease improve VACS index prediction of all cause mortality? Presented at: International Cohorts Meeting; Athens, Greece, March 27–29, 2012.
55. Oursler K., Tate J., Gill T., et al; and Veterans Aging Cohort Study Project Team. Comorbidity is predictive of Muscle Strength in HIV-infected veterans: results from the vacs index. Presented at: 19th Conference on Retroviruses and Opportunistic Infections; Seattle, Washington, March 8, 2012.
56. Pencina MJ, D’Agostino RB, Vasan SR. Evaluating the additive predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:207–212.
57. Schafer JL. Analysis of Incomplete Multivariate Data. London, United Kingdom: Chapman and Hall CRC Press; 1997.
58. Gelman A, Carlin JB, Stern HS, et al. Bayesian Data Analysis. London, United Kingdom: Chapman and Hall CRC Press; 2003.
59. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–387.
This article has been cited 3 time(s).
Journal of Clinical and Experimental NeuropsychologyThe aggregate effects of multiple comorbid risk factors on cognition among HIV-infected individualsJournal of Clinical and Experimental Neuropsychology
Drugs & AgingThe Next Therapeutic Challenge in HIV: PolypharmacyDrugs & Aging
Journal of Infectious DiseasesAssociation of Functional Impairment with Inflammation and Immune Activation in HIV Type 1-Infected Adults Receiving Effective Antiretroviral TherapyJournal of Infectious Diseases
HIV; aging; prognosis
© 2013 Lippincott Williams & Wilkins, Inc.
Highlight selected keywords in the article text.