Notably, among the dichotomous retention measures the 6-month gap and HRSA HAB measure demonstrated sensitivity for VL suppression of 82% and 91%, respectively, with lower specificity (39% and 27%), whereas the dichotomous missed visit measure displayed specificity of 82%, with lower sensitivity (42%). Thresholds to define retention for other measures were selected to provide a balance of sensitivity and specificity (Tables 4 and 5), although other cut points may be applied with an associated trade-off of sensitivity for specificity and vice versa. Additional analyses using a missing equals failure approach for patients lacking 12-month VL measures (18%) yielded consistent findings with primary analyses, albeit with larger parameter estimates and c-statistic values for retention measures, particularly, for kept visit–based measures (Table 5).
Our study is the first to evaluate the prognostic value of 6 commonly used measures in predicting VL suppression and the correlation among HIV retention in care measures in the same patient sample. A recent study from the HIV Research Network identified strong correlations (concordance correlation coefficients range = 0.67–0.88) among retention measures calculated based on kept visits only, which was corroborated by our analyses.20 We extend this work by further examining an additional 3 measures that incorporate no-show visits and additionally comparing the prognostic value of the 6 measures for HIV VL suppression. Overall, considerable variability was observed among these measures in categorizing patients as being retained, ranging from one-third of patients with no missed visits (perfect visit adherence) to over three-quarters of patients meeting the HRSA HAB retention measure. Although these varying definitions translated to a broad range of correlations across retention measures (Spearman coefficients range = 0.16–0.85), each measure demonstrated a strong and statistically significant (P < 0.001) relationship with VL suppression. Accordingly, our data suggest that there is no clear gold standard to measure retention in care and that any of the evaluated measures may have a role depending on visit data availability, the questions being addressed, and the principal rationale and goals of measuring “retention.” Moreover, there may be merit to using multiple retention measures, particularly in research settings, and using one measure that incorporates missed visits and another based solely on kept visits.
As anticipated, improved prognostic value for VL suppression was observed for multilevel retention measures compared with dichotomous measures. By allowing more granular categorization of patient retention, the missed visit count measure, visit adherence, and 4-month constancy measures allowed for better discriminatory capacity in predicting VL suppression, as indicated by higher c-statistic values. For example, among the two-thirds of patients categorized as “not retained” by the dichotomous no-show visit measure, a broad range of counts of missed visits and of visit adherence was observed. The enhanced variability captured by these latter 2 measures translates into improved prognostic capacity for VL suppression and is perhaps best visualized by the multiple points incorporated into the ROC curves, in contrast to the single point used for dichotomous measures (Fig. 1).
However, dichotomous measures clearly have value, offering advantages including face validity and less complex programming and computational and analytic demands.15 Moreover, the dichotomous missed visit, 6-month gap and HRSA-HAB measures were all strongly associated with VL suppression, albeit with reduced discriminatory capacity. An interesting study finding was the variability in defining patients as retained across dichotomous retention measures, which translated into robust differences in the sensitivity and specificity of these measures in predicting VL suppression. The dichotomous no-show measure categorized only one-third of the sample as retained, resulting in high specificity (82%) of this measure in relation to VL suppression. In other words, 82% of persons without viral suppression had at least one no-show visit during the year. In contrast, the 6-month gap and HRSA HAB measures categorized 68% and 77% of patients as retained, respectively. This translated into high sensitivity of these measures (82% and 91%, respectively) in predicting VL suppression. In other words, 8 or 9 of 10 persons with viral suppression met these standards for retention, respectively. No measure, however, had both high sensitivity and high specificity. These fascinating relationships highlight the potential to use multiple retention measures; one including missed visits and the other based on kept visits only, as they seem to provide complementary information regarding measurement of retention yet are each significantly associated with VL suppression with large effect sizes. Future studies should evaluate the prognostic value of composite measures of retention integrating 2 or more of the measures examined here.
Our findings are germane to contemporary clinical and public health issues related to HIV treatment and prevention. In recent years, the importance of retention as a key step on the HIV treatment continuum has received heightened attention.2,4 Our findings indicate that the operational definition chosen to measure retention can have far-reaching implications in assessing this component, with subsequent downstream implications for estimates of persons on ART and achieving VL suppression. The US National HIV/AIDS Strategy set 80% retention among HRSA Ryan White CARE Act clients as a goal by 2015, with retention measured using the HRSA HAB measure.12 Among our sample, 77% of patients achieved retention according to this measure, which would have been widely variable, ranging from 33% to 68%, if other measures had been used. This observation is of particular importance when comparing retention in care across settings and studies, as the measure used, and the duration of the observation period can have a dramatic impact on interpretation of findings and the inference that may be drawn. It is imperative to ensure consistency of measures when evaluating similarities or differences in retention across settings.
Prior studies have established significant associations between the retention measures under study and HIV biomarker and clinical outcomes.2,16–19 The present study extends this work by evaluating VL suppression across retention measures among the same study sample, showing strong and statistically significant associations for each measure. The large parameter estimates observed are in accordance with prior studies and underscore the critical role of retention in care as a key step along the treatment continuum that ultimately leads to VL suppression. However, the retention measures studied demonstrated only modest discriminatory capacity (c-statistic = 0.59–0.69) for VL suppression. It is anticipated that including other steps in the treatment continuum, notably ART adherence, to retention measures as an additional independent variable in statistical models would improve overall prognostic value for VL suppression. Notably, a significant association between retention and ART adherence was observed in a previous study that compared visit constancy with pharmacy refills.16 Future research should examine the discriminatory capacity of models, including both measures of retention and ART adherence, in relation to HIV biomarker and other clinical outcomes.
Beyond individual health outcomes, considerable improvement in retention in care at the population level is essential to achieving the potential success of ART treatment as prevention initiatives.4,10,13 There is great need for substantial improvement in retention in care, along with other steps across the treatment continuum, if we are to meaningfully increase the proportion of HIV-infected Americans with suppressed VL levels from current estimates of 19%–28%.4,8,9 The impact of treatment as prevention approaches is predicated on the collective success of public health, medical and supportive service providers, and affected communities in dramatically increasing these estimates.
Our study has limitations. By focusing on established clinic patients to allow for a sample with a comparable observation period, we cannot comment on retention measures in persons newly establishing HIV medical care, which has been evaluated in other studies.6,17,24,25 Established patients may be more likely to be adherent to ART regardless of their retention status, and this may place a limit on the discriminatory ability of these measures. We are unable to systematically capture and account for patients who may have transferred their medical care during the 1-year observation period, which could impact calculation of retention measures. Similarly, deaths during the 1-year observation period were not systematically captured, which may have resulted in underreporting of retention, although it is unclear whether this would introduce systematic bias when making comparisons across retention measures. Moreover, patients, who died, likely had missing 12-month VL values and were excluded from primary analyses. Our examination of retention measures was for a relatively short period of time. Additional research should evaluate these measures over longer time intervals. Although our 6 sites serve diverse populations across the United States, our findings might not translate to other domestic and international treatment settings or to nonacademically affiliated clinics. We also note study strengths, including the clinic-wide capture of high-quality patient-level visit utilization and sociodemographic and clinical data. In addition, the evaluation of measures incorporating missed clinic visits in addition to those based solely on kept visits is novel and extends recent work comparing only this latter group of retention measures.20
In summary, 6 commonly used measures of retention in care demonstrated considerable variability in categorizing retention, translating to a wide range of correlations among these measures. In general, stronger associations were observed among measures incorporating missed visits and among those based solely on kept visits, with potentially complementary information provided when using measures from these 2 groups. Despite the observed heterogeneity across retention measures, each demonstrated strong and statistically significant relations with VL suppression, albeit with variable discriminatory capacity. Taken together, our findings suggest that as for ART adherence, there is no clear gold standard to measure retention in HIV care, and that each measure studied may have value and utility according to setting and circumstance.
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the US Centers for Disease Control and Prevention (CDC) or Health Resources and Services Administration (HRSA). The authors thank the patients, providers, and clinical and research personnel at the 6 study sites and the CDC and HRSA administrative and data management teams. The authors also thank the Retention in Care Study Group: Boston University Medical Center: M. -L. Drainoni [Principal Investigator (PI)], C. Ferreira, L. Koppelman, R. Lewis, M. McDoom, M. Naisteter, K. Osella, G. Ruiz, P. Skolnik, M. Sullivan (PI); SUNY Downstate Medical Center: S. Gibbs-Cohen, E. Desrivieres, M. Frederick, K. Gravesande, S. Holman, H. Johnson, T. Taylor, T. Wilson (PI); University of Alabama-Birmingham: S. Batey, S. Gaskin, M. Mugavero (PI), J. Murphree, J. Raper, M. Saag (PI), S. Thogaripally, J. Willig, A. Zinski; Baylor College of Medicine: M. Arya, D. Bartholomew, T. Biggs, H. Budhwani, J. Davila, T. Giordano (PI), N. Miertschin, S. Payne, W. Slaughter; Johns Hopkins University: M. Jenckes, J. Keruly (PI), A. McCray, M. McGann, R. Moore (PI), M. Otterbein, L. Zhou; University of Miami: C. Garzon, J. Jean-Simon, K. Mercogliano, L. Metsch (PI), A. Rodriguez (PI), G. Saint-Jean, M. Shika; Federal: L. Cheever, HRSA; F. Malitz, HRSA; R. Mills, HRSA; J. Craw, CDC/ICF-Macro; L. Gardner, CDC; S. Girde, CDC/ICF-Macro; G. Marks, CDC; Mountain Plains AETC; L. Bradley-Springer; M. Corwin.
1. Aberg JA, Kaplan JE, Libman H, et al.. Primary care guidelines for the management of persons infected with human immunodeficiency virus: 2009 update by the HIV Medicine Association of the Infectious Diseases Society of America. Clin Infect Dis. 2009;49:651–681.
2. Horstmann E, Brown J, Islam F, et al.. Retaining HIV-infected patients in care: where are we? Where do we go from here? Clin Infect Dis. 2010;50:752–761.
3. Thompson MA, Mugavero MJ, Amico KR, et al.. Guidelines for improving entry into and retention in care
and antiretroviral adherence
for persons with HIV: evidence-based recommendations from an International Association of Physicians in AIDS Care Panel. Ann Intern Med. 2012;156:817–833.
4. Gardner EM, McLees MP, Steiner JF, et al.. The spectrum of engagement in HIV care and its relevance to test-and-treat strategies for prevention of HIV infection. Clin Infect Dis. 2011;52:793–800.
5. Giordano TP, Suarez-Almazor ME, Grimes RM. The population effectiveness of highly active antiretroviral therapy: are good drugs good enough? Curr HIV/AIDS Rep. 2005;2:177–183.
6. Mugavero MJ, Amico KR, Westfall AO, et al.. Early retention in HIV care and viral load
suppression: implications for a test and treat approach to HIV prevention. J Acquir Immune Defic Syndr. 2012;59:86–93.
7. Marks G, Gardner LI, Craw J, et al.. Entry and retention in medical care among HIV-diagnosed persons: a meta-analysis. AIDS. 2010;24:2665–2678.
8. Marks G, Gardner LI, Craw J, et al.. The spectrum of engagement in HIV care: do more than 19% of HIV-infected persons in the US have undetectable viral load
? Clin Infect Dis. 2011;53:1168–1169; author's reply 1169–1170.
9. Centers for Disease Control and Prevention (CDC). Vital signs: HIV prevention through care and treatment—United States. MMWR Morb Mortal Wkly Rep. 2011;60:1618.
10. Das M, Chu PL, Santos GM, et al.. Decreases in community viral load
are accompanied by reductions in new HIV infections in San Francisco. PLoS One. 2010;5:e11068.
11. Mugavero MJ, Napravnik S, Cole SR, et al.. Viremia copy-years predicts mortality among treatment-naive HIV-infected patients initiating antiretroviral therapy. Clin Infect Dis. 2011;53:927–935.
13. Dieffenbach CW, Fauci AS. Universal voluntary testing and treatment for prevention of HIV transmission. JAMA. 2009;301:2380–2382.
14. Cohen MS, Chen YQ, McCauley M, et al.. Prevention of HIV-1 infection with early antiretroviral therapy. N Engl J Med. 2011;365:493–505.
15. Mugavero MJ, Davila JA, Nevin CR, et al.. From access to engagement: measuring retention in outpatient HIV clinical care. AIDS Patient Care STDS. 2010;24:607–613.
16. Giordano TP, Gifford AL, Clinton White A, et al.. Retention in care
: a challenge to survival with HIV infection. Clin Infect Dis. 2007;44:1493–1499.
17. Mugavero MJ, Lin HY, Willig JH, et al.. Missed visits and mortality among patients establishing initial outpatient HIV treatment. Clin Infect Dis. 2009;48:248–256.
18. Park WB, Choe PG, Kim SH, et al.. One-year adherence
to clinic visits after highly active antiretroviral therapy: a predictor of clinical progress in HIV patients. J Intern Med. 2007;261:268–275.
19. Cabral HJ, Tobias C, Rajabiun S, et al.. Outreach program contacts: do they increase the likelihood of engagement and retention in HIV primary care for hard-to-reach patients? AIDS Patient Care STDS. 2007;21(suppl 1):S59–S67.
20. Yehia BR, Fleishman JA, Metlay JP, et al.. Comparing different measures of retention in outpatient HIV care. AIDS. 2012;26:1131–1139.
21. Gardner LI, Marks G, Craw J, et al.. Brief clinic-wide intervention promotes retention in care
among HIV patients (Oral, Abstract ID #2018). Presented at: 2011 National HIV Prevention Conference; August 2011; Atlanta, GA.
22. Gardner L, Marks G, Craw J, et al.. A low-effort clinic-wide intervention improves attendance for HIV primary care. Clin Infect Dis 2012;55:1124–1134.
23. Melnikow J, Kiefe C. Patient compliance and medical research: issues in methodology. J Gen Intern Med. 1994;9:96–105.
24. Giordano TP, Visnegarwala F, White AC Jr, et al.. Patients referred to an urban HIV clinic frequently fail to establish care: factors predicting failure. AIDS Care. 2005;17:773–783.
25. Metsch LR, Pereyra M, Messinger S, et al.. HIV transmission risk behaviors among HIV-infected persons who are successfully linked to care. Clin Infect Dis. 2008;47:577–584.
Keywords:© 2012 Lippincott Williams & Wilkins, Inc.
retention in care; adherence; engagement in care; viral load