Secondary Logo

Journal Logo

Clinical Science

HIV Laboratory Monitoring Reliably Identifies Persons Engaged in Care

Dean, Bonnie B. PhD, MPH*; Hart, Rachel L.D. MS*; Buchacz, Kate PhD; Bozzette, Samuel A. MD, PhD*,‡; Wood, Kathy BSN*; Brooks, John T. MD the HOPS Investigators

Author Information
JAIDS Journal of Acquired Immune Deficiency Syndromes: February 1, 2015 - Volume 68 - Issue 2 - p 133-139
doi: 10.1097/QAI.0000000000000406



Engagement in ongoing HIV care is recognized as an important clinical performance indicator,1 and attendance at biannual medical encounters has been proposed as a minimum US standard for engagement in care.2–6 Measuring attendance poses a major challenge for both medical providers and public health authorities,1 HIV-related medical visit attendance is not nationally reportable,7 and methods for monitoring attendance at the clinic level are not standardized. Under current standards of care,8–10 a patient's HIV RNA viral load (VL) and CD4 cell count (CD4) are typically measured coincident with clinical evaluation every 3–6 months after establishing care. These laboratory data (ie, measured values and dates of testing) are now reportable as part of routine HIV surveillance activities to the CDC's HIV/AIDS Reporting System to track national trends. Various medical and public health entities have proposed using these laboratory reports as proxy indicators for assessing engagement in care.1 Using data from the HIV Outpatient Study (HOPS), we examined the relationship between patient encounters and VL and CD4 laboratory reports to assess the extent to which the latter can be used as a surrogate for actual engagement in care. Because HOPS is a multisite longitudinal cohort, which collects both encounter and laboratory information for its participants, this data set is particularly apt to answer the question regarding whether laboratory data can adequately be used to assess engagement in care.


The HIV Outpatient Study

The HOPS is an open, prospective, observational cohort study of HIV-infected patients enrolled at 9 public universities and private HIV specialty clinics located in 6 US cities. Patients must be at least 18 years of age and receiving HIV care at one of the clinics to be enrolled. Research coordinators located in each clinic abstract sociodemographic characteristics, diagnoses, treatments, encounters, and laboratory values from patients' medical charts and enter these data into an electronic database for central processing and analysis. Since its inception, the HOPS protocol has been reviewed and approved annually by the CDC or each local site's institutional review board. The study protocol conforms to the guidelines of the US Department of Health and Human Services for the protection of human subjects in research; all participants have provided written informed consent.

Study Population and Inclusion Criteria

In our analyses, we included data from HOPS patients with at least 1 HIV laboratory test, either a CD4 or VL measurement, performed from January 2010 through December 2011. We did not require a minimum number of medical encounters. We classified the following as medical encounters: routine visits, initial visits, provider visits, consultations, pharmacist encounters, outpatient surgical procedures, and any other or unknown visit types. Long-term rehabilitation, home care visits, telephone calls, research study visits, hospital stays, post-hospital follow-up visits, emergency room visits, and visits that were scheduled but not attended by the patient (ie, no-shows) were not included. CD4 and VL tests were excluded if the laboratory reported the sample volume was inadequate for testing (comprising 0.6% of specimens reported in the database)—failed attempts to test are not nationally reportable, and this circumstance should have warranted a repeat specimen collection event and test. We also excluded data from 1 HOPS site that did not provide HIV primary care (ie, provided solely HIV specialty care).

Definitions of Variables

The index date (ie, initiation of observation) for all patients was the date on which a specimen was collected for the first VL or CD4 measurement during the study period. We defined “engaged in care by encounters” as having at least 2 medical encounters within 12 months that were at least 90 days apart. We defined the proxy measure, “engaged in care by laboratory monitoring,” as having at least 2 HIV laboratory test results reported within 12 months and at least 90 days apart. We classified an HIV laboratory testing event as linked with a medical encounter if the specimen was collected within 3 weeks before or after an HIV-related medical encounter. Suppression of VL was defined as <400 copies per milliliter.

Analysis Methods

We calculated the sensitivity, specificity, positive predictive value, and negative predictive value of being engaged by laboratory monitoring as a measure for being engaged in care by medical encounters as the gold standard. We compared the characteristics of patients for whom all laboratory testing events were linked with a medical encounter and patients for whom at least 1 laboratory testing event was not linked with a medical encounter through χ2 test.

For patients whose engagement in care was misclassified, we examined their characteristics by using the proxy measure of engagement in care by laboratory monitoring as compared with medical encounters. Using χ2 tests and Fisher exact tests, as appropriate, we compared demographic characteristics of patients classified as false positive (ie, incorrectly classified as engaged) or false negative (ie, incorrectly classified as not engaged) with those of patients whose engagement was correctly classified through the proxy measure.

For laboratory testing events not linked with medical encounters, we used a generalized estimating equations model7 to explore associated patient characteristics present at the time the specimen was collected while controlling for correlations arising from multiple observations per patient. We used bivariate and multivariate logistic regression to assess characteristics associated with a laboratory testing event without an associated medical encounter. Missing values were multiply imputed for race/ethnicity, education, insurance, HIV transmission risk group, antiretroviral therapy (ART) exposure, VL suppression, previous VL suppression, and CD4.11 Multiple imputation fills in plausible responses for missing values based on existing values within the data, resulting in multiple imputed data sets; the outcomes resulting from analysis on these multiple data sets are then combined to estimate the outcome from the original data set. We included variables with univariate associations at P value <0.05 in multivariable models.11 All analyses were performed using SAS version 9.2 (SAS Institute, Cary, NC). We used the HOPS data set available as of March 31, 2012, for this analysis.


Laboratory Testing Events and Medical Encounters

We included in our analysis 10,321 VL and CD4 laboratory testing events for 2909 unique HOPS patients seen during 2010–2011. During the same period, a total of 20,928 medical encounters were recorded. The median duration between patients' first and last medical encounter was 14.5 months (interquartile range: 8.4–18.8, minimum: 0.0, maximum: 23.7). Of the 10,321 laboratory testing events, 519 were for CD4 measures alone, 1194 were for VL measures alone, and 8608 were events when both CD4 and VL were measured. The median time between a laboratory testing event and the nearest clinical visit was 0 days (interquartile range: 0–1.5); the median time varied by HOPS site from 0 to 6 days.

A total of 73.6% of laboratory testing events occurred on the same day as a medical encounter (see Figure S1, Supplemental Digital Content, and 89.3% of laboratory testing events occurred within the 3 weeks before or after a medical encounter. For 2148 (73.8%) patients, all laboratory testing events were classified as associated with a medical encounter (ie, occurred within 3 weeks before or after the medical encounter). The remaining 26.2% of patients with at least 1 HIV laboratory testing event performed outside the 3-week window (ie, classified as not associated with a medical encounter) were similar to patients who had each HIV laboratory testing event associated with a medical encounter in terms of age at index date, gender, education, disease stage, and HIV transmission risk group but were more likely to have been white non-Hispanic, to have been privately insured, to have been ART naive at index date, and to have been cared for at a private facility (Table 1).

Characteristics of Patients With At Least 1 HIV Laboratory Test Not Associated With a Medical Encounter Versus Patients With All Tests Associated With Medical Encounters, HOPS Data, 2010–2011 (N = 2909 Participants)*

Laboratory Testing Events as a Proxy for Engagement in Care

In our sample, 88.2% of patients met criteria for being engaged in care using medical encounter data. Engagement in care by encounters ranged from 76.2% to 97.8% across participating HOPS sites. Using the proxy measure of care by laboratory monitoring, 77.3% of patients were engaged in care, ranging from 47.9% to 96.1% across participating HOPS sites. Using the proxy of engagement in care by laboratory monitoring to measure engagement in care by encounters had a sensitivity of 85.7%, a specificity of 86.0%, and positive and negative predictive values of 97.9% and 44.5%, respectively (Table 2). Overall, 14.2% of patients were misclassified. Three hundred sixty-six (12.6% overall, 88.4% of all misclassifications) patients were falsely classified as not engaged in care (falsely negative); that is, those patients were engaged in care by medical encounters but did not meet the definition of engagement in care by laboratory monitoring (see Figure S2, Supplemental Digital Content, Among the patients misclassified as not engaged in care by laboratory measures (22.7% overall), 55.5% (366/660) were actually engaged by encounters (Table 2). Only 48 (1.7%) patients were misclassified as falsely engaged in care; that is, they appeared engaged in care by laboratory monitoring but were not actually engaged according to medical encounters.

Cross-Tabulation of Health Resources and Services Administration Encounter-Based Definition of Engagement in Care by Laboratory-Based Definition, HOPS Data, 2010–2011 (N = 2909 Participants)*

Patients who were misclassified as not engaged through the proxy laboratory measure but who were actually engaged through medical encounters (false negatives) were more likely to be younger, female, Hispanic, or non-Hispanic black race/ethnicity and to belong to the high-risk heterosexual transmission risk group compared with those who were correctly classified (Table 3). In contrast, participants who were misclassified as engaged in care through the proxy laboratory measure but who were actually not engaged through encounters (false positives) were more likely to be of non-Hispanic black or non-Hispanic white race/ethnicity compared with those correctly classified, although there were small numbers of false positives (N = 48) from which to ascertain patterns.

Characteristics of Patients With Engagement Correctly Classified Versus Incorrectly Classified by the Proxy Laboratory Measure, HOPS Data, 2010–2011 (N = 2909 Participants)*

Odds of an HIV Laboratory Testing Event Being Unassociated With a Medical Encounter

In bivariate analysis, younger age, non-Hispanic white race/ethnicity, receiving care in a private HOPS clinic, private insurance, and being ART naive at index date were associated with greater odds of having a laboratory test without a medical encounter (Table 4). When evaluated in a multivariate logistic generalized estimating equations model, receiving care in a private HOPS clinic and being ART naive at index date remained independently associated with having a laboratory testing event without an associated medical encounter (Table 4).

Odds of HIV Laboratory Testing Event Not Being Associated With a Medical Encounter, HOPS Data, 2010–2011 (N = 10,321 Laboratory Events)*


Using data from a multisite cohort study, we found that monitoring HIV laboratory testing events as a proxy for engagement in care reliably mirrored actual engagement as recorded by medical encounters. The proxy measure had a very high positive predictive value (97.9%) for actual engagement, and fewer than 15% of patients' engagement statuses were misclassified. Most misclassifications were falsely negative (ie, laboratory testing events were absent when the patient was actually engaged in care by encounters) with only very few false positives (ie, a person not engaged in care by encounters appeared to be engaged by laboratory testing events). The low negative predictive value (44.5%) indicated that a substantial fraction of persons who did not seem to be engaged in care by laboratory measures (ie, falsely negative) were indeed engaged. Thus, the proxy measure functioned well at predicting engagement in care but was biased toward misclassifying a modest number of patients as not engaged in care rather than engaged; this circumstance resulted in a slight underestimation of care engagement based on laboratory measures.

Our estimate of engagement by laboratory monitoring exceeds the estimates of both Gardner et al and Hall et al (67% and 56%, respectively, among persons linked to care) but was similar to estimates reported by Sabharwal et al.12–14 Each of these estimates relied on laboratory data to infer attendance at clinical encounters among persons in care; Gardner et al synthesized published data, Hall et al analyzed data from 2 CDC national surveillance systems, the National HIV/AIDS Surveillance system and the Medical Monitoring Project, and Sabharwal et al analyzed New York City Registry data. Our higher estimate likely reflects the demography of our cohort, which in comparison with national data has a lower percentage of patients at risk of poor retention in care (ie, our population has a greater proportion of participants who are of older age, male gender, and white non-Hispanic race/ethnicity)13,15 and potentially reflects practice patterns and experience levels of HOPS physicians at our specialty clinics. Another explanation for why our estimate exceeds the estimates of engagement in care by laboratory monitoring published by others is that both laboratory and encounter information were gathered from the same source, reducing the probability of ascertainment bias and improving the linkage between laboratory tests and clinical visits when both actually occur. Similar to Sabharwal et al, when used to define establishment in care, laboratory data are likely to underestimate the true frequency.

Poor retention in care is associated with increased morbidity and mortality.1,16–18 Monitoring engagement can help identify and prioritize patients at risk. Our findings suggest that surveillance and other administrative databases that capture CD4 and VL testing could be used effectively by public health departments to assist HIV care providers in their jurisdiction in identifying patients who may have disengaged from care. Public health departments could also ensure that patients who, from the provider's perspective, seem to have disengaged are receiving at least a minimum standard of care by checking whether an HIV laboratory result has been reported from another source within their jurisdiction.

In practice, physicians do not necessarily order laboratory tests on the same day as a medical encounter. In some settings, depending on provider and patient preferences, patients may have blood drawn for laboratory tests before or after the medical encounter. We found in the HOPS that 89.3% of laboratory testing events occurred within the 3 weeks before or after a medical encounter. Increasing the window to 1 month did not appreciably improve the sensitivity and specificity of the performance measure (data not shown). However, we cannot determine whether this same window would be adequate for assessing correspondence of laboratory testing events and medical encounters in other models of care, such as health maintenance organizations, other unified health systems with multiple service points, or federally qualified health centers. We found that patients who were ART naive and received care in a private practice were more likely to have had a laboratory testing event not associated with a medical encounter. We have not conducted qualitative interviews with providers or patients to better understand this observation. We hypothesize that this demographic may comprise patients who had no overt illness requiring physician contact but required close laboratory monitoring to determine when ART should be started (ie, ART naive) or for whom there was a lower barrier, such as incentive from the provider's perspective, to ensure patients attended a reimbursable medical encounter (ie, private practice).

Our analyses had a number of strengths and some limitations. HOPS is a robust prospective HIV cohort database composed of 20,928 encounters and 10,321 laboratory measurements recorded during 2010–2011 and representing care of 2909 patients. The HOPS database consists of a dynamic cohort of patients who have engaged in HIV care and consented to participate in a research study; thus, our findings cannot be necessarily extrapolated to other populations not engaged in specialty HIV care. Patients were categorized for analyses based on whether they had laboratory tests associated with medical encounters or not regardless of the number of laboratory tests or medical encounters recorded during the study period. Thus, different patients could have had varying numbers of medical encounters contributing to their classification. It is possible that some patients had laboratory testing conducted outside the purview of the HOPS, so that they might have appeared to have no laboratory data when they were actually engaged. This would have led to an overestimation of false-negative classification, although it was small to begin with in this study.

In conclusion, monitoring dates of CD4 cell count and plasma HIV RNA VL testing events have been proposed as a means to measure engagement in clinical care in circumstances where attendance records are unavailable or difficult to analyze. We found that in a well-characterized cohort study where both sets of information (ie, testing data and attendance records) were available from a data collection system designed to maximize completeness of their capture, HIV laboratory testing data reliably estimated attendance, and that such estimates were biased substantially more toward underestimation rather than overestimation. The generalizability of our findings may be limited. We recommend repeating our analyses with data sets derived from patients in clinical environments that differ from the HOPS, including at the national level (eg, the Medical Monitoring Project), to further evaluate the performance characteristics of this engagement metric.


1. Mugavero MJ, Lin HY, Willig JH, et al.. Missed visits and mortality among patients establishing initial outpatient HIV treatment. Clin Infect Dis. 2009;48:248–256.
2. US Department of Health and Human Services. Health resources and services administration. HIV/AIDS Programs. HAB HIV performance measures. Available at: July, 2008. Accessed January 4, 2013.
3. Horberg MA, Aberg JA, Cheever LW, et al.. Development of national and multiagency HIV care quality measures. Clin Infect Dis. 2010;51:732–738.
4. Mugavero MJ, Westfall AO, Zinski A, et al.. Measuring retention in HIV care: the elusive gold standard. J Acquir Immune Defic Syndr. 2012;61:574–580.
5. Thompson MA, Mugavero MJ, Amico KR, et al.. Guidelines for improving entry into and retention in care and antiretroviral adherence for persons with HIV: evidence-based recommendations from an International Association of Physicians in AIDS Care panel. Ann Intern Med. 2012;156:817–833.
6. Institute of Medicine. Monitoring HIV Care in the United States: Indicators and Data Systems. Washington, DC: The National Academies Press; 2012.
7. Mahle GK, Tang T, Shouse L, et al.. Using the HIV surveillance system to monitor the National HIV/AIDS Strategy. Am J Public Health. 2013;103:141–147.
8. Panel on Antiretroviral Guidelines for Adults and Adolescents. Guidelines for the use of antiretroviral agents in HIV-1-infected adults and adolescents. Department of health and human services. Available at: Accessed February 26, 2013.
9. Aberg JA, Kaplan JE, Libman H, et al.. Primary care guidelines for the management of persons infected with human immunodeficiency virus: 2009 update by the HIV medicine Association of the Infectious Diseases Society of America. Clin Infect Dis. 2009;49:651–681.
10. Thompson MA, Aberg JA, Hoy JF, et al.. Antiretroviral treatment of adult HIV infection: 2012 recommendations of the International Antiviral Society-USA panel. JAMA. 2012;308:387–402.
11. Little RJA, Rubin DJ. Statistical Analysis With Missing Data. Vol 4. New York, NY: Wiley; 1987.
12. Gardner EM, McLees MP, Steiner JF, et al.. The spectrum of engagement in HIV care and its relevance to test-and-treat strategies for prevention of HIV infection. Clin Infect Dis. 2011;52:793–800.
13. Hall HI, Frazier EL, Rhodes P, et al.. Differences in human immunodeficiency virus care and treatment among subpopulations in the United States. JAMA Intern Med. 2013;173:1337–1344.
14. Sabharwal CJ, Braunstein SL, Robbins R, et al.. Optimizing the use of surveillance data for monitoring the care status or persons recently diagnosed with HIV in NYC. J Acquir Immune Defic Syndr. 2014;65:571–578.
15. Higa DH, Marks G, Crepaz N, et al.. Interventions to improve retention in HIV primary care: a systematic review of U.S. studies. Curr HIV/AIDS Rep. 2012;9:313–325.
16. Giordano TP, Gifford AL, White AC Jr, et al.. Retention in care: a challenge to survival with HIV infection. Clin Infect Dis. 2007;44:1493–1499.
17. Park WB, Choe PG, Kim SH, et al.. One-year adherence to clinic visits after highly active antiretroviral therapy: a predictor of clinical progress in HIV patients. J Intern Med. 2007;261:268–275.
18. Berg MB, Safren SA, Mimiaga MJ, et al.. Nonadherence to medical appointments is associated with increased plasma HIV RNA and decreased CD4 cell counts in a community-based HIV primary care clinic. AIDS Care. 2005;17:902–907.


The HIV Outpatient Study Investigators include the following persons and sites: J.T.B., K.B., Marcus D. Durham, Division of HIV/AIDS Prevention, National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention (NCHHSTP), Centers for Disease Control and Prevention (CDC), Atlanta, GA; Harlen Hays, Kathleen C. Wood, Darlene Hankerson, R.D., Thilakavathy Subramanian, Carl Armon, and Bonnie Dean, Cerner Corporation, Vienna, VA; Sam Bozzette, Cerner Corporation, Vienna, VA at the time of analysis and manuscript preparation; Frank J. Palella, Joan S. Chmiel, Carolyn Studney, and Saira Jahangir, Feinberg School of Medicine, Northwestern University, Chicago, IL; Kenneth A. Lichtenstein and Cheryl Stewart, National Jewish Medical and Research Center Denver, CO; John Hammer, Kenneth S. Greenberg, Barbara Widick, and Rosa Franklin, Rose Medical Center, Denver, CO; Bienvenido G. Yangco and Kalliope Halkias, Infectious Disease Research Institute, Tampa, FL; Doug Ward and Troy Thomas, Dupont Circle Physicians Group, Washington, DC; Jack Fuhrer, Linda Ording-Bauer, Rita Kelly, and Jane Esteves, State University of New York (SUNY), Stony Brook, NY; Ellen M. Tedaldi, Ramona A. Christian, Faye Ruley, Dania Beadle, and Princess Graham, Temple University School of Medicine, Philadelphia, PA; Richard M. Novak, Andrea Wendrow, and Renata Smith, University of Illinois at Chicago, Chicago, IL; and Benjamin Young and Barb Widick, APEX Family Medicine, Denver, CO.


HIV; retention in care; CD4 cell count; plasma HIV RNA viral load

Supplemental Digital Content

Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.