The VACS index provided a more discriminating prediction of all-cause mortality among HIV-infected subjects from North America on ART than the restricted index. This was true overall, with increasing exposure to ART, and among important subgroups, most notably among persons with low HIV-1 RNA and those ≥50 years of age—2 rapidly growing populations in treatment. Based on established criteria,13,25 the VACS index has demonstrated excellent generalizability and is likely to accurately predict mortality among HIV-infected patients on ART in North America. Importantly, after demonstrating that this translation is accurate in demographically and clinically diverse subgroups, we provide a table and nomogram (Fig. 2; Table 1) and a website (http://vacs.med.yale.edu) to facilitate calculating VACS index scores and translating them to predicted mortality rates. Potential applications for the VACS index include patient management and clinical research.
C-statistics are a commonly employed metric for evaluating the discrimination of prognostic indices.36 In uncensored data, the C-statistic is the likelihood that, if any 2 subjects were drawn from the sample, the subject with the higher score would die before the subject with the lower score. Although these categories are somewhat arbitrary, C-statistics between 0.50 and 0.59 are considered poor; 0.60 and 0.69, fair; 0.70 and 0.79, good; 0.80 and 0.89 very good; and above 0.89, excellent.14 Although restricted index C-statistics ranged from 0.63 to 0.76 (“fair” to “good”), VACS index C-statistics ranged from 0.70 to 0.81 (“good” to “very good”). Discrimination was particularly better among those with undetectable HIV-1 RNA (C-statistics: 0.67 vs. 0.74) and those older than 50 years of age (C-statistics: 0.63 vs. 0.70). C-statistics for the VACS index for all-cause mortality meet or surpass those reported for prognostic indices used in clinical practice including the Framingham index for predicting cardiovascular disease and validated indices predicting all-cause mortality among aging HIV-uninfected individuals.13,37
A newer metric, developed and popularized by the methodologists working on the Framingham Risk score, is the NRI.34,35 This is calculated by separating those who died and those who lived and asking in each group whether the VACS index resulted in a change in risk classification compared with the restricted index. Among those who died, a higher risk classification is considered an improvement and a lower risk classification is considered an error. Among those who lived, a lower risk classification is considered an improvement and a higher classification is considered an error. The NRI is the sum of the differences. The net gain in reclassification proportions at 5 years was 9% for survivors and 3% for those who died for an overall statistic of 12% (P < 0.0001). Further, the NRI was even higher among those with undetectable HIV-1 RNA (25%) and those 50 years and older (20%). These NRIs suggest a highly clinically significant improvement in discrimination35,38 and are greater than improvements seen by the addition of D-dimer to the VACS index.24
For maximal clinical and research utility, providers and investigators need a means of translating VACS index scores to mortality risk. We combined NA-ACCORD and VACS data to provide as precise a translation as possible. We then considered the accuracy of this translation by cohort and among important subgroups. Because such translations depend upon the overall mortality rate in the cohort,13,25,37 we conducted this work among cohorts with uniform access to regional and/or national death registries. In these analyses, the predicted mortality based upon VACS index score was similar to observed mortality among veteran (VACS) and nonveteran (NA-ACCORD) subjects; and among: men and women; black and non-black patients; those <50 years and those ≥50 years old; and those with HIV-1 RNA <500 copies per milliliter and those with ≥500 copies per milliliter.
To understand how the VACS index reclassifies risk, consider an HIV-infected 45-year-old man who, after 12 months of ART, has a CD4 count of 500 cells per cubic millimeter and an undetectable HIV-1 RNA but is HCV coinfected with a FIB-4 >3.25. He, like 1 in 4 NA-ACCORD subjects, was assigned 0 points by the restricted index with a 2% predicted 5-year mortality. Using the VACS index, he was assigned 5 points for HCV coinfection and 25 points for his FIB-4 value for a score of 30 and a predicted 5-year mortality of 12%. Fifty-three percent of NA-ACCORD subjects assigned a score of 0 by the restricted index were assigned a higher score by the VACS index.
Having an accurate, generalizable, responsive, and feasible method for estimating individual risk can improve effectiveness and efficiency of chronic disease management in major ways.39–41. First, it can inform decision making when an intervention puts the patient at some immediate risk for long-term gain.42,43 This is true whenever patients are asked to undergo a risk of immediate harm from treatment in the hope of averting long-term disease incidence or progression—commonly the case in cancer screening and primary and secondary prophylaxis for cardiovascular disease and stroke. It is also true when considering aggressive treatment protocols (toxic chemotherapy, organ transplant, or major surgery) for cancer or heart disease. Second, it can motivate patients to modify health behaviors such as adherence to medication, smoking, diet, exercise, and alcohol use by quantifying the impact these changes have on risk and by charting progress after modifying risk.44–48 Third, it can identify patients in need of intensive management either with respect to site of care (outpatient, inpatient, intensive care unit, skilled nursing facility, nursing home) or care management (case management, frequency of follow-up).
Of note, none of these applications require that an index identify all modifiable sources of risk, only that it accurately, generalizably, responsively, and feasibly estimate risk of mortality—including risk associated with the modifiable factors of interest.25 Because many sources of modifiable risk have a similar common pathway to physiologic injury, it is not efficient or feasible for a single index to include all modifiable sources of risk. Instead, separate analyses can map changes in risk score associated with changes in modifiable factors of interest. We are currently undertaking a series of analyses demonstrating this for adherence to ART, alcohol use, smoking, and substance use, but these are beyond the scope of this article.
Although the utility of the VACS index for clinical management can only be established through a randomized trial comparing management with and without the index, evidence to date suggests that it offers useful insight. We have previously shown that hemoglobin, FIB-4, eGFR, CD4 count, and HIV-1 RNA, change substantially in response to ART initiation, not always in the same direction,33 and that the VACS index is more responsive to ART initiation and differing levels of ART adherence than the restricted index.33 Third, we have shown that the VACS index is more correlated with biomarkers of inflammation (interleukin 6), microbial translocation (D-dimer), and hyper coagulability (sCD14) than the restricted index.5 Taken together, these data suggest that the VACS index provides a more comprehensive means of tracking disease burden, including the effects of chronic inflammation, over time.
Further, to facilitate use of the VACS index in the clinical setting, we have developed a web site calculator accessible via smart phone or computer with an automatic conversion of the VACS index score to a risk estimate (http://vacs.med.yale.edu). The site includes regularly updated links to supporting evidence for the index. As we develop information regarding how behavior change alters risk, we will include this information as an additional link for the calculator. We also provide SAS programming for any who wish to include the calculation as part of their electronic medical record system or for analyses of grouped clinical data (www.vacohort.org).
Our analysis has substantial strengths. We demonstrated the generalizability of the VACS index in a large independent sample on ART over differing periods of ART exposure and among important subgroups of patients.25 By combining NA-ACCORD and VA samples, we were powered to precisely translate VACS index scores to predicted mortality and to consider whether predicted mortality matched observed deaths overall and among important subgroups. Further, the VACS index is based on laboratory tests currently ordered in the course of routine management and therefore offers enhanced clinical insight requiring only that providers calculate and interpret the score. We have simplified this process by providing a web site and smart phone calculator (http://vacs.med.yale.edu). Eventually the index could be calculated by the clinical laboratory (or the electronic medical record) every time component tests are ordered, as is the current practice for eGFR.
A limitation of any large observational study is missing data. Although subjects with missing values in NA-ACCORD tend to have more liver disease and less anemia, the imputed analyses yielded results similar to complete cases (Appendix), suggesting that missing data did not compromise our findings. Further, the VACS index may be improved in the future. Our choice of risk factors was based on prior work,16,20,21,23,53 a desire to base the index on consistently measured metrics such as clinical laboratory tests, and the need to validate findings. As the population with HIV ages, higher age thresholds will likely become relevant. Of note, we have evaluated whether the following: body mass index, lipid profiles, smoking status, hypertension54; inflammatory biomarkers (D-dimer, interleukin 6, and soluble CD14)24; and functional status55 improve the discrimination of the VACS index. Although many of these predict mortality in unadjusted analyses, VACS index scores co vary with these factors. When added to the VACS index, only D-dimer and/or sCD14 resulted in risk reclassification that exceeded 1%. Additional factors (such as D-dimer) may be added to the VACS index in the future if they can be consistently measured and they improve discrimination sufficiently to justify added cost and complexity.13,35
In summary, we have demonstrated that a novel index composed of routine clinical data can predict mortality among HIV-infected individuals on ART with good to very good discrimination and consistent calibration across important subgroups. Measures of general organ system function included in the VACS index substantially enhance discrimination. Although it would be a reasonable precaution to verify the calibration of the VACS index among younger patients and subjects outside North America, predicted mortality from the VACS index is likely generalizable to HIV-infected individuals older than 30 years of age in care in North America. Among these individuals, the VACS index is ready for clinical and research application.
The authors would like to thank all individuals involved with the NA-ACCORD collaboration, including staff, investigators, and patients, for their valuable contributions to this work.
Risk reclassification is an approach that evaluates the improvement of prognostic models with the addition of new markers.56 It addresses limitations in discrimination metrics such as changes in the C-statistic by evaluating how individuals are reassigned into risk categories with the addition of new predictors.34 Risk reclassification tables are primarily used to compare models with and without single factors; we applied the technique to examine models with and without a set of covariates.35
We evaluated reclassification from a model that used the “restricted index” (calculated using age at anchoring time, CD4 count, and log10 HIV viral load) with a model that used the full VACS index (calculated using all the variables in the restricted index and hemoglobin, baseline HCV status, FIB4, and eGFR). All variables were categorized as shown in the main article (Table 1). Note that one participating cohort, with only 5 subjects and no events, was dropped because predicted probabilities could not be estimated.
From the risk reclassification analysis, our specific methods follow. First, we fit a Cox regression model for 5-year mortality using the restricted index score, stratified by cohort to account for cross-cohort heterogeneity. We identified intervals of continuous ART for all subjects occurring between January 1, 2001, and December 30, 2007, and selected the anchoring time to be 1 year after the start of the study interval. We used the restricted index score calculated at 1 year of ART exposure (−90 days to 180 days) as the only variable in the regression. Follow-up continued from anchoring time to death date or was censored at the date of last follow-up, December 30, 2007 (administrative censoring), or 5 years from the anchoring point, whichever came first.
Based on this model, we calculated the predicted probability of death at 5 years for each subject using the Gibbs Sampling algorithm assuming noninformative priors (the default in SAS V9). These steps were repeated replacing the restricted index score with the full VACS index score.
Risk strata were derived using quintiles of predicted probabilities from the restricted model: 0 to <3%, 3 to <5%, 5 to <7%, 7 to <11%, and ≥11%. The decision to use quintiles was based on the comments by Pencina et al56 who noted that reclassification measures are heavily dependent on the choice of risk strata. Individuals were categorized based on their predicted probability from each model. (Appendix Table 1). Rows represent risk categorization from the restricted index model and columns that form the VACS index model. For example, 1586 of the 10,830 participants were categorized in the 0 to <3% risk stratum based on their predicted probability of death at 5 years by both the restricted index and VACS index models. However, 363 participants classified into the lowest risk stratum by the restricted index were reclassified by the VACS index into the 3 to <5% risk stratum.
Using these methods, 53% of 10,830 subjects were reclassified into different risk strata. The overall percentage reclassified gives an indication of how many subjects would change risk categories under the VACS model. Twenty-two percent moved to a higher risk group. Thirty-one percent moved to a lower risk group.
We sought to supplement these descriptive results with 2 additional evaluations. First, we computed a version of the Hosmer–Lemeshow goodness-of-fit statistic known as the reclassification calibration statistic for both the restricted index model and VACS index model.35 This is a comparison of observed and predicted 5-year cumulative mortality for each model. Predicted mortality came from the mean predicted probability of death for each cell from each model. We can then compute the expected number of events by multiplying the mean probability by the number of subjects in that cell. Observed 5-year cumulative mortality was obtained from Kaplan–Meier estimates of 5-year all-cause mortality for each cell. For example, for the first cell, the Kaplan–Meier estimate for 5-year all-cause mortality is 1.3%. We can also compute the Kaplan–Meier estimate of the observed number of “deaths” in a similar manner. For each cell with at least 20 subjects, we can find the squared differences between the observed and expected number of events and the χ2 goodness-of-fit test for each model separately. The restricted index had a strong lack of fit with a χ2 statistic of 220 on 19 degrees of freedom. The VACS index had a χ2 test statistic of 58 on 19 degrees of freedom (P = 0.0008) also indicating lack of fit but to a much lesser extent.
Second, we explored whether the VACS index offered clinically meaningful improvement over the restricted index using net NRI.56 NRI is the difference between the proportions moving up and down risk strata among those who died versus those who survived at 5 years. This measure is similar to the percent reclassified but distinguishes movements in the correct direction. The second layer of numbers within each cell pertains to the 571 individuals who died by 5 years. Among those who died, the cells above the diagonal in Appendix Table 1 correspond to (3 + 4 + 2 + 2 + 17 + 10 + 9 + 19 + 9 + 55) = 130 of 571 (23%) participants who moved up to a higher risk category. This means that for 130 people who were dead at the 5-year mark, classification improved using the VACS model. For the cells below the diagonal, (10 + 19 + 6 + 33 + 7 + 42) = 117 of 571 (20%) participants moved down to a lower risk category using the VACS index. Among those who died at 5 years, the overall improvement was 3% (23%–20%; P = 0.41). The third layer of numbers within each cell of Table 1 are 10,259 individuals who were alive (or censored) at 5 years. Of these, (360 + 164 + 41 + 13 + 556 + 217 + 58 + 433 + 129 + 337) = 2308 of 10,259 (23%) moved up to a higher risk category using the VACS index. Another (908 + 66 + 491 + 3+ 229 + 785 + 1 + 123 + 628) = 3234 of 10259 (32%) moved down to a lower risk category The overall gain in reclassification proportions for survivors at 5 years was 9% (32%–23%) and was significantly greater than zero (P = <0.0001). The NRI, computed by summing the overall improvement for those who died and for those alive at 5 years (3 + 9) is 12% and is also significantly different from zero (0 = <0.0001). NRI suggests the VACS index model for 5-year all-cause mortality results in significant improvement in performance compared with the restricted index.
Missing Data Methods
Based on the initial criteria for the study, we found 15,938 eligible subjects. Of these 10,835 (68%) subjects had complete information on both the outcome and all necessary laboratory measurements required to construct the VACS index score. Therefore, 32% of eligible subjects were missing a required laboratory value but were complete on all other variables of interest, including the outcome. Consequently, we were unable to construct a VACS and/or a restricted index score for these subjects. Patterns of missingness were largely arbitrary.
In an effort to characterize patients with missing data, we explored differences between those with complete and incomplete laboratories. These groups differed on almost all variables of interest (Appendix Table 2), but there had similar 5-year all-cause mortality (Appendix Fig. 1). Based on these initial findings, we explored use of multiple imputation methods that allowed us to forgo any strict assumptions regarding the appropriateness of the complete case analysis.
Multiple imputation was performed using PROC MI procedure in SAS V9 assuming multivariate normality for the distribution of laboratory values, and nonmonotone missingness. We used the Markov Chain Monte Carlo procedure to obtain 25 completed datasets. This method imputes missing values by simulating draws from the joint complete distribution via data augmentation.57 Convergence of the Markov Chain Monte Carlo simulation was verified using techniques proposed by Gelman et al.58 We also explored graphical diagnostics (Q–Q plots) and numerical values (means and ranges) of the imputations to verify that our imputed values were sampled within a reasonable range. Summary statistics for the imputed laboratory data (Appendix Table 3) indicate close agreement between complete cases and imputed datasets. Although mean VACS index and restricted index scores for the complete cases were lower than those from the imputed data, Q–Q plots revealed close agreement between observed and imputed scores (Appendix Fig. 2). Cox models were refitted in the 25 imputed datasets using the VACS index score and restricted index scores. Standard errors used for calculating confidence intervals were computed by combining the estimates using Rubin rules. The resulting hazard ratios (95% confidence intervals) based on the imputed data were 1.041 (1.038-1.044) for the VACS Index and 1.040 (1.037-1.043). Harrell C-statistics59 were computed (overall and for relevant subgroups) using the means from 25 imputation sets (Appendix Table 4). We also plotted calibration curves using the imputed data for both the VACS and restricted index models and found results similar to the complete case analysis (Appendix Fig. 3). We conclude that the results based on multiple imputation are similar to those based on the complete case analysis. For simplicity, the main article presents results from the complete case analysis only.
1. McKinnell JA, Willig JH, Westfall AO, et al. Antiretroviral prescribing patterns in treatment-naive patients in the United States. AIDS Patient Care STDS. 2010;24:79–85.
2. D'Arminio MA, Sabin CA, Phillips A, et al. The changing incidence of AIDS events in patients receiving highly active antiretroviral therapy. Arch Intern Med. 2005;165:416–423.
3. Hogg R, Lima V, Sterne JA, et al. Life expectancy of individuals on combination antiretroviral therapy in high-income countries: a collaborative analysis of 14 cohort studies. Lancet. 2008;372:293–299.
4. Losina E, Schackman BR, Sadownik SN, et al. Racial and sex disparities in life expectancy losses among HIV-infected persons in the united states: impact of risk behavior, late initiation, and early discontinuation of antiretroviral therapy. Clin Infect Dis. 2009;49:1570–1578.
5. Justice AC, Freiberg MS, Tracy R, et al; and The VACS Project Team. Does an index composed of clinical data reflect effects of inflammation, coagulation, and monocyte activation on mortality among those aging with HIV? Clin Infect Dis. 2012;54:984–994.
6. Justice AC. HIV and aging: time for a new paradigm. Curr HIV/AIDS Rep. 2010;7:69–76.
7. Deeks SG. Immune dysfunction, inflammation, and accelerated aging in patients on antiretroviral therapy. Top HIV Med. 2009;17:118–123.
8. Phillips AN, Neaton J, Lundgren JD. The role of HIV in serious diseases other than AIDS. AIDS. 2008;22:2409–2418.
9. Panel on Antiretroviral Guidelines for Adults and Adolescents. Guidelines for the Use of Antiretroviral Agents in HIV-1-Infected Adults and Adolescents. Washington, DC: Department Health and Human Services; 2011:1–166.
10. DiMatteo MR, Haskard KB, Williams SL. Health beliefs, disease severity, and patient adherence: a meta-analysis. Med Care. 2007;45:521–528.
11. Feinstein AR. Clinical biostatistics, XIV: the purposes of prognostic stratification. Clin Parmacol Ther. 1972;13:285–297.
12. Feinstein AR, Landis RJ. The role of prognostic stratification in preventing the bias permitted by random allocation of treatment. J Chron Dis. 1979;29:277–284.
13. Vasan RS. Biomarkers of cardiovascular disease: molecular basis and practical considerations. Circulation. 2006;113:2335–2362.
14. Yourman LC, Lee SJ, Schonberg MA, et al. Prognostic indices for older adults: a systematic review. JAMA. 2012;307:182–192.
15. Turner BJ, Kelly JV, Ball JK. A severity classification system for AIDS hospitalizations. Med Care. 1989;27:423–437.
16. Justice AC, Feinstein AR, Wells CK. A new prognostic staging system for the acquired immunodeficiency syndrome. N Engl J Med. 1989;320:1388–1393.
17. Egger M, May M, Chene G, et al. Prognosis of HIV-1-infected patients starting highly active antiretroviral therapy: a collaborative analysis of prospective studies. Lancet. 2002;360:119–129.
18. May M, Royston P, Egger M, et al. Development and validation of a prognostic model for survival time data: application to prognosis of HIV positive patients treated with antiretroviral therapy. Stat Med. 2004;23:2375–2398.
19. Mocroft AJ, Johnson MA, Sabin CA, et al. Staging system for clinical AIDS patients. Lancet. 1995;346:12–17.
20. Mocroft A, Kirk O, Barton SE, et al. Anaemia is an independent predictive marker for clinical prognosis in HIV-infected patients from across Europe. EuroSIDA study group. AIDS. 1999;13:943–950.
21. Harris RJ, Sterne JA, Abgrall S, et al. Prognostic importance of anaemia in HIV type-1-infected patients starting antiretroviral therapy: collaborative analysis of prospective cohort studies. Antivir Ther. 2008;13:959–967.
22. Lundgren JD, Mocroft A, Gatell JM, et al. A clinically prognostic scoring system for patients receiving highly active antiretroviral therapy: results from the EuroSIDA study. J Infect Dis. 2002;185:178–187.
23. Justice AC, McGinnis KA, Skanderson M, et al. Towards a combined prognostic index for survival in HIV infection: the role of ‘non-HIV' biomarkers. HIV Med. 2009;11:143–151.
24. Justice AC, Freiberg MS, Tracy R, et al. Does an index composed of clinical data reflect effects of inflammation, coagulation, and monocyte activation on mortality among those aging with HIV? Clin Infect Dis. 2012;54:984–994.
25. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med. 1999;130:515–524.
26. Gange SJ, Kitahata MM, Saag MS, et al. Cohort profile: the North American AIDS Cohort Collaboration on Research and Design (NA-ACCORD). Int J Epidemiol. 2007;36:294–301.
27. Lau B, Gange SJ, Moore RD. Interval and clinical cohort studies: epidemiological issues. AIDS Res Hum Retroviruses. 2007;23:769–776.
28. Lau B, Gange SJ, Moore RD. Risk of non-AIDS-related mortality may exceed risk of AIDS-related mortality among individuals enrolling into care with CD4+ counts greater than 200 cells/mm3. J Acquir Immune Defic Syndr. 2007;44:179–187.
29. Fultz SL, Skanderson M, Mole LA, et al. Development and verification of a "virtual" cohort using the National VA Health Information System. Med Care. 2006;44:S25–S30.
30. Tate JP, Justice AC, Hughes MD, et al. An internationally generalizable risk index for mortality after one year of antiretroviral therapy. AIDS. 2012 Nov 6. [Epub ahead of print]
31. Sterling RK, Lissen E, Clumeck N, et al. Development of a simple noninvasive index to predict significant fibrosis in patients with HIV/HCV coinfection. Hepatology. 2006;43:1317–1325.
32. Stevens LA, Coresh J, Greene T, et al. Assessing kidney function—measured and estimated glomerular filtration rate. N Engl J Med. 2006;354:2473–2483.
33. Tate JP, Justice AC; for the VACS Project Team. Change in a prognostic index for survival in HIV infection after one year on cART by level of adherence. Presented at: 48th Annual Meeting of the Infectious Disease Society of America; 21–24 October 2010, Vancouver, British Columbia.
34. Cook NR, Ridker PM. Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann Intern Med. 2009;150:795–802.
35. Cook NR. Assessing the incremental role of novel and emerging risk factors. Curr Cardiovasc Risk Rep. 2010;4:112–119.
36. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115:928–935.
37. D'Agostino RB Sr, Grundy S, Sullivan LM, et al. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 2001;286:180–187.
38. Cook NR. Comments on ‘Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27:191–195.
39. Kent DM, Rothwell PM, Ioannidis JP, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials. 2010;11:85.
40. Hayward RA, Kent DM, Vijan S, et al. Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis. BMC Med Res Methodol. 2006;6:18.
41. Eddy DM, Adler J, Patterson B, et al. Individualized guidelines: the potential for increasing quality and reducing costs. Ann Intern Med. 2011;154:627–634.
42. Braithwaite RS. Can life expectancy and QALYs be improved by a framework for deciding whether to apply clinical guidelines to patients with severe comorbid disease? Med Decis Making. 2011;31:582–595.
43. Braithwaite RS, Concato J, Chang CC, et al. A framework for tailoring clinical guidelines to comorbidity at the point of care. Arch Intern Med. 2007;167:2361–2365.
44. Chan CW, Witherspoon JM. Health risk appraisal modifies cigarette smoking behavior among college students. J Gen Intern Med. 1988;3:555–559.
45. Dapp U, Anders JA, von Renteln-Kruse W, et al. A randomized trial of effects of health risk appraisal combined with group sessions or home visits on preventive behaviors in older adults. J Gerontol A Biol Sci Med Sci. 2011;66:591–598.
46. Kreuter MW, Strecher VJ. Do tailored behavior change messages enhance the effectiveness of health risk appraisal? Results from a randomized trial. Health Educ Res. 1996;11:97–105.
47. Maron DJ, Forbes BL, Groves JR, et al. Health-risk appraisal with or without disease management for worksite cardiovascular risk reduction. J Cardiovasc Nurs. 2008;23:513–518.
48. Stuck AE, Kharicha K, Dapp U, et al. The PRO-AGE study: an international randomised controlled study of health risk appraisal for older persons based in general practice. BMC Med Res Methodol. 2007;7:2.
49. Justice AC, Tate JP, Freiberg MS, et al. Reply to Chow et al. Clin Infect Dis. 2012 Jun 10. [Epub ahead of print]
50. Akgun KM, Pisani MA, Fried TR, et al. Risk factors for medical intensive care Unit admission in HIV infected veterans. AIDS. 2012 Oct 29. [Epub ahead of print]
51. Erlandson KM, Allshouse AA, Jankowski C. Prospective comparison of three functional assessments with the Veteran's Aging Cohort Study index in virologically suppressed HIV-infected adults. Presented at: 2nd International Workshop on HIV and Aging; Baltimore, Maryland, October 27–28, 2011. Abstract #8.
52. Womack JA, Goulet JL, Gibert C, et al. Increased risk of fragility fractures among HIV infected compared to uninfected male veterans. PLoS One. 2011;6:e17217.
53. May MT, Sterne JA, Costagliola D, et al. HIV treatment response and prognosis in Europe and North America in the first decade of highly active antiretroviral therapy: a collaborative analysis. Lancet. 2006;368:451–458.
54. Tate J, Freiberg M, Justice AC. Do risk factors for cardiovascular disease improve VACS index prediction of all cause mortality? Presented at: International Cohorts Meeting; Athens, Greece, March 27–29, 2012.
55. Oursler K., Tate J., Gill T., et al; and Veterans Aging Cohort Study Project Team. Comorbidity is predictive of Muscle Strength in HIV-infected veterans: results from the vacs index. Presented at: 19th Conference on Retroviruses and Opportunistic Infections; Seattle, Washington, March 8, 2012.
56. Pencina MJ, D’Agostino RB, Vasan SR. Evaluating the additive predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:207–212.
57. Schafer JL. Analysis of Incomplete Multivariate Data. London, United Kingdom: Chapman and Hall CRC Press; 1997.
58. Gelman A, Carlin JB, Stern HS, et al. Bayesian Data Analysis. London, United Kingdom: Chapman and Hall CRC Press; 2003.
59. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–387.