Increased reengagement of out-of-care HIV patients using Lost & Found, a clinic-based intervention : AIDS

Secondary Logo

Journal Logo


Increased reengagement of out-of-care HIV patients using Lost & Found, a clinic-based intervention

Linthwaite, Blakea; Kronfli, Nadinea,b; Marbaniang, Ivanc; Ruppenthal, Lucianab; Lessard, Davida; Engler, Kima; Lebouché, Bertranda,b,d; Cox, Josepha,b,c

Author Information
doi: 10.1097/QAD.0000000000003147



The UNAIDS Fast-Track targets to end the HIV epidemic by 2030 prioritized reengagement of people with HIV (PWH) who are out of care (OOC) [1]. Specific definitions for ‘OOC’ or ‘loss-to-follow-up’ vary but all include some criteria related to gaps in HIV care [2–6]. OOC patients are at risk for individual-level negative health outcomes leading to broader population-level health and economic impacts. These include suboptimal combination antiretroviral treatment (cART) adherence [7,8], viremia [8–10], development of HIV resistance [11,12], opportunistic infections [13], hospitalizations [14], mortality [15–17], and secondary HIV transmission [9,18].

Data-to-Care (D2C) is ‘a public health strategy that uses HIV surveillance data and other data sources’ to identify people with gaps in HIV testing or care, subsequently directing them for reengagement by public-health officials [5,6,19–27]. In combination with services to locate, contact, and facilitate access to HIV medical care, D2C has been shown to be an effective strategy [24] and is recommended by the US Centers for Disease Control and Prevention (CDC) for improving the HIV continuum of care [19,20]

Surveillance-based D2C strategies, however, are limited in their capacity to accurately identify OOC patients [3,28–31] and, as such, often require supplemental input from clinicians [25,30]. The large, multijurisdictional datasets or data-sharing agreements used in surveillance-based D2C [20,24,26,32–34] may also be infeasible in some jurisdictions because of legal barriers or poor public-health infrastructure [35,36], further limiting the utility of these strategies in some contexts.

Two studies utilizing clinic-based approaches for identification and reengagement of OOC patients successfully reengaged approximately half of OOC patients [4,37]. Although these clinic-based interventions show promise, evaluations of effectiveness were limited. Most studies of reengagement interventions have reported descriptive results only and none have been guided by implementation or sustainability frameworks [3–6,21–27,29–33,37–41]. This lack of theoretical framing may limit their long-term uptake and sustainability [42].

Lost & Found is a clinic-based intervention developed and delivered using implementation science, informed by previous efforts to identify and reengage OOC patients [4–6,37,41,43]. In contrast to traditional D2C, Lost & Found leverages available clinical datasets and related clinical knowledge to improve identification and reengagement of OOC patients.

This study aims to determine effectiveness of Lost & Found as a reengagement intervention. Specifically, we conducted a descriptive analysis of reengagement outcomes and a pre-post evaluation of total reengagement and time to reengagement among OOC patients.


Results are drawn from a type II implementation-effectiveness hybrid pilot study of Lost & Found [44]. A full study protocol is published elsewhere [43] and selected details are discussed here. This study was approved by the McGill University Health Centre (MUHC) research ethics board (2018-4369).


The MUHC is a large public quaternary care hospital in Montréal, Canada. Multidisciplinary care for adult patients with chronic viral illnesses, such as HIV, hepatitis B, and hepatitis C, is provided at the Chronic Viral Illness Service (CVIS-MUHC) clinic. Over 90% of patients who received care at the CVIS-MUHC in 2018 were PWH (n = 1777). Before Lost & Found, approximately 10% of patients did not return for care annually, and no formal system to identify or reengage patients existed [43]. Ad hoc and periodic efforts by clinical nurses to identify and reengage OOC patients, using patient lists, were limited by several barriers including competing work priorities, staffing shortages, technology limitations, and lack of decision support to accurately identify OOC patients [43].

Lost & found intervention

Lost & Found is an intervention to identify and reengage OOC patients into HIV care. It was implemented at the CVIS-MUHC between April 2018 and 2019 to circumvent barriers to OOC patient reengagement and reduce HIV care attrition. It consists of two core, evidence-based elements adapted to the clinic: identification and documentation of OOC patients; and systematic contact of OOC patients.

For the first core element – identification and documentation of OOC patients – a list of OOC patients and an OOC risk prediction tool (OOC-RPT; Fig. 1) were used. Both were integrated into the clinic's electronic medical record (EMR) database. An automated portion of the OOC-RPT (Fig. 1, step 1: triage) used available clinical information to categorize patients as high, intermediate, or low risk of HIV disease progression [43]. On the basis of time since last appointment and risk category, patients were classified as engaged in care or possibly OOC. Possible OOC patients were placed on a real-time OOC list, visible upon launching the EMR application (Supplemental Digital Content 1, Using this information, nurses then accessed patients’ clinical files to validate OOC status, either retaining or removing them from the list (Fig. 1, step 2: nurse validation). Step 1 of the OOC-RPT was designed to capture any patient potentially requiring reengagement by CVIS-MUHC nurses. Consequently, it is highly inclusive, reflecting nurses’ concerns about potentially excluding at-risk patients. These possible misclassifications of OOC are corrected by step 2 of the OOC-RPT, when nurses validate OOC status and retain only those patients in need of reengagement.

Fig. 1:
Out of care risk prediction tool.

In the second core element, confirmed OOC patients were contacted. This consisted of prioritization of higher risk patients, nurse-delivered phone calls, multiple contact attempts (if necessary), and motivational communication. The OOC list was arranged to hierarchically organize patients, with high-risk patients preceding intermediate-risk and low-risk patients, as a means of facilitating prioritization of phone calls. The OOC list was managed by nurses who attempted to contact patients at regular intervals until they were reengaged or it was learned that care was being received elsewhere. Recommended timelines for contacting and reengaging patients were co-developed with nurses but they were encouraged to use clinical judgement in determining the urgency with which individual patients should be reengaged [43].

Participants and analyses

Objective I: descriptive analysis

All CVIS-MUHC patients in the EMR database during the 12-month implementation phase were included. The primary outcomes of interest were identification by the OOC-RPT, confirmation of OOC status by nurses, contact attempts, and successful reengagement. We also provide other outcomes from nurses’ OOC status validations, such as the number of patients followed elsewhere and reasons for non-reengagement. For patients marked possibly OOC and reengaged multiple times throughout the implementation phase, only reengagement attempts related to the first OOC event were analysed. We report both overall and risk-category-stratified (high, intermediate, low) intervention metrics, including number of contact attempts needed to reengage patients and time to reengagement, as well as routinely collected clinical (e.g. CD4+ cell counts, HIV viral load) and sociodemographic information.

Objective II: pre-post analyses

For the pre-post analyses, we retroactively applied the OOC-RPT to patients in the pre-implementation (15 April 2017 to 15 April 2018) and implementation (15 April 2018 to 15 April 2019) phases excluding information from the nurse validation step of the implementation phase. This modified version of the OOC-RPT, a ‘pseudo-RPT’, was used to minimize differential misclassification bias of OOC status with respect to reengagement as changes to OOC statuses through nursing input were not possible in the pre-implementation phase (i.e. the comparator) as they were in the implementation phase. Excluding this information nondifferentially increases the total number of people included in each phase, all of whom would have been marked ‘not OOC’ by nurses. Consequently, we render the groups more comparable, but with the trade-off of diluting any effect that may exist. More detailed explanations of these considerations are described in Supplemental Digital Content 2, All patients marked OOC by the pseudo-RPT in the implementation and pre-implementation phases were included in this analysis. For each patient, only the first OOC event in each phase was considered. OOC patients are right censored at the end of each phase, meaning their time since being marked OOC is terminated at reengagement or the end of the given study phase.

To determine the risk of reengagement for patients deemed OOC by the pseudo-RPT, we used a Poisson regression model with robust variance estimation and a log link function [43,45]. Robust Poisson regression for binary outcomes is a more reliable alternative to logistic regression and provides more easily interpreted results [46,47]. We modelled reengagement as a function of study phase, controlled for sex, age, and being born in Canada [43]. These variables were selected as they may be associated both with receiving the intervention and reengagement [43,48]. Other variables were thought unlikely to differ between the two phases, or were not available in the EMR system. The risk difference in reengagement between the two phases was determined using the same model but with an identity link function [45]. In addition to these preplanned analyses, we conducted a survival analysis using a Cox proportional hazards model to provide median time to reengagement. The results of this Cox model, as well as a supplemental propensity score-adjusted robust Poisson model, were also used to validate results from the preplanned analysis. Full results and analysis plans for these models are presented in Supplemental Digital Content 3,


Descriptive analyses

Intervention results over the 12-month implementation phase are summarized in Fig. 2. Of the 2354 patients with HIV in the clinical EMR database, 56% (n = 1312) were marked as possibly OOC using the OOC-RPT. Of this group, nurses determined 44% (n = 578) were followed elsewhere, 32% (n = 423) were OOC, 19% (n = 249) were still engaged in care, 3% (n = 33) were deceased, and 2% (n = 29) were of unknown status. Only six patients had their status changed from engaged in care to possible OOC. Notably, of the 249 patients marked as possibly OOC but still engaged, the large majority (75%) had medical follow-up visits booked just beyond the window period of their assigned risk category, while the remainder had shared care with another clinic or other arrangements. Among patients confirmed OOC, nurses attempted to contact 85% (n = 359); the remaining 15% (n = 64) had already scheduled appointments at the time of nurse review or were unreachable until they reengaged on their own.

Fig. 2:
Summary of Lost & Found outcomes: (a) among all patients in the clinical electronic medical record, (b) among out of care patients only, and (c) among out of care patients contacted but not yet reengaged.

Among the OOC patients for whom contact was attempted, 70% (n = 250) were successfully reengaged as of the end of the intervention/implementation phase. Of those not reengaged (n = 109), 46% (n = 50) were left a voice message, 30% (n = 33) had upcoming appointments beyond study end and the remaining 24% (n = 26) were unreachable, declined a reengagement visit, or had missed their reengagement visit. Two-thirds (64%; 21/33) of patients with upcoming appointments after the end of the implementation phase missed their first scheduled reengagement visit.

Baseline characteristics of contacted OOC patients (n = 359), stratified by reengagement, are presented in Table 1. The OOC-RPT, including nurse validation, classified 11% (n = 40) of patients as high risk, 73% (n = 263) intermediate risk, and 16% (n = 57) low risk.

Table 1 - Baseline characteristics of contacted out of care patients, stratified by reengagement.
OOC, contacted Contacted and reengaged Contacted, not yet reengaged
N 359 250 109
Sex [n (%)]
 Female 124 (34.5%) 87 (34.8%) 37 (33.9%)
 Male 235 (65.5%) 163 (65.2%) 72 (66.1%)
Age, median [IQR] 51 [42–57] 52 [42–58] 50 [40–56]
Born in Canada [n (%)] 155 (43.2%) 108 (43.2%) 47 (43.1%)
Years since HIV diagnosis, median [IQR] 16 [11–23] 17 [11–23] 16 [11–22]
Risk categorya
 High 40 (11.1%) 26 (10.4%) 14 (12.8%)
  CD4+ cell count <100 14 (3.9%) 12 (4.8%) 2 (1.8%)
  CD4+ cell count 100–200 + VL ≥40 12 (3.3%) 8 (3.2%) 4 (3.7%)
  New patient 3 (0.8%) 1 (0.4%) 2 (1.8%)
  Other/clinical judgement 11 (3.0%) 5 (2.0%) 6 (5.5%)
 Intermediate 262 (73.0%) 197 (78.8%) 65 (59.6%)
  CD4+ cell count 100–300 + VL < 40 32 (8.9%) 27 (10.8%) 5 (4.6%)
  CD4+ cell count ≥200 + VL ≥40 50 (13.9%) 29 (11.6%) 21 (19.3%)
  Non-ART polypharmacy (>5 non-antiretrovirals) 107 (29.8%) 83 (33.2%) 24 (22.0%)
  History of chronic HCV infectionb 30 (8.4%) 21 (8.4%) 9 (8.3%)
  Youth (<25 years old) 13 (3.6%) 11 (4.4%) 2 (1.8%)
  CD4+ cell count nadir <200 155 (43.2%) 126 (50.4%) 29 (26.6%)
  Other/clinical judgement 3 (0.9%) 2 (0.8%) 1 (0.9%)
 Low 57 (15.9%) 27 (10.8%) 30 (27.5%)
  CD4+ cell count ≥300 + VL <40 53 (14.8%) 25 (10.0%) 28 (25.7%)
  Other/clinical judgement 4 (1.1%) 2 (0.8%) 2 (1.8%)
History of chronic HCV infectionb [n (%)] 42 (11.7%) 30 (12.0%) 12 (11.0%)
CD4+ cell count nadirc (cells/μl), median [IQR] 184 [78–335] 164 [71–310] 274 [134–474]
Among contacted OOC patients at the CVIS-MUHC clinic from 15 April 2018 to 15 April 2019 (n = 359). ART, antiretroviral therapy; IQR, interquartile range; OOC, out of HIV care; VL, viral load.
aRisk category criteria, by risk category: High: CD4+ cell count less than 100 cells/μl (irrespective of viral load) or CD4+ cell count 100–200 cells/μl with viral load greater than 40 copies/ml or New patient. Intermediate: CD4+ cell count 100–300 cells/μl with viral load less than 40 copies/ml or CD4+ cell count greater than 200 cells/μl + viral load greater than 40 copies/ml or non-ART polypharmacy (>5 non-antiretrovirals) or Hx of chronic HCV infection (HCV RNA+) or Youth (<25 years old) or CD4+ cell count nadir less than 200 cells/μl. Low: CD4+ cell count greater than 300 cells/μl + viral load less than 40 copies/ml. Risk categories are mutually exclusive. For the criteria within each risk category, only the CD4+ cell count and viral load criteria are mutually exclusive from each other; otherwise multiple criteria can apply to the same patient. ‘Other’ refers to a nurse defined reason for classification into the given risk category.
bEver HCV RNA+.
cLowest CD4+ cell count on record before reengagement.

Clinical characteristics and intervention metrics for reengaged OOC patients are presented in Table 2. CD4+ cell counts and viral loads reflected the criteria of the OOC-RTP. Both at reengagement and last visit, patients in the low-risk category had the highest CD4+ cell counts and lowest viral loads, while patients in the high-risk category had the lowest CD4+ cell counts and highest viral loads. The occurrence of viremia (200+ copies/ml) increased from 12% (n = 30) at last visit to 15% (n = 38) at reengagement, despite 79% (n = 197) having undetectable viral loads (<40 copies/ml) at each time point.

Table 2 - Characteristics of patients contacted and reengaged.
Risk category Overall High riska Intermediate riska Low riska
N (%) 250 26 (10%) 197 (79%) 27 (11%)
CD4+ cell count
 CD4+ cell count; median [IQR] Reengagement 528 [351–744] 151 [72–314] 544 [379–770] 704 [574–850]
Previous visit 538 [350–731] 139 [84–257] 551 [378–730] 751 [612–873]
 CD4%, median [IQR] Reengagement 30 [21–38] 12 [5–17] 31 [23–38] 35 [30–41]
Previous visit 28 [21–37] 12 [7–14] 29 [23–37] 34 [30–38]
 CD4+/CD8+ ratio, median [IQR] Reengagement 0.70 [0.40–1.10] 0.20 [0.10–0.30] 0.70 [0.50–1.10] 1.10 [0.75–1.35]
Previous visit 0.70 [0.40–1.08] 0.20 [0.10–0.30] 0.70 [0.50–1.07] 1.00 [0.75–1.20]
 VL, median [IQR] Reengagement <40 [<40 to <40] 116 [<40 to 37 646] <40 [<40 to <40] <40 [<40 to <40]
Previous visit <40 [<40 to <40] 203 [<40 to 21 376] <40 [<40 to <40] <40 [<40 to <40]
 Undetectable VL [n (%)] Reengagement 197 (79%) 11 (42%) 162 (82%) 24 (89%)
Previous visit 197 (79%) 7 (27%) 163 (83%) 27 (100%)
 Viremia [n (%)] Reengagement 38 (15.2%) 12 (46.2%) 24 (12.2%) 2 (7.4%)
Previous visit 30 (12.0%) 13 (50.0%) 17 (8.6%) 0 (0.0%)
 Days from… (median [IQR]) …Previous visit to reengagement 315 [246–448] 229 [168–428] 304 [247–422] 444 [427–514]
…First contact attempt to reengagement 72 [34–136] 86 [37–107] 78 [34–145] 57 [34–78]
Any missed visits reengagement visits [n (%)] 92 (37%) 14 (54%) 73 (37%) 5 (19%)
Number of contact attempts, median [IQR] 2.0 [1.0–4.0] 4.0 [1.5–5.0] 2.0 [1.0–4.0] 2.0 [1.0–2.5]
Among OOC patients contacted and reengaged into care at the CVIS-MUHC clinic from 15 April 2018 to 15 April 2019 (n = 250). IQR, interquartile range; OOC, out of HIV care; VL, viral load.
aRisk category criteria: High: CD4+ cell count less than 100 cells/μl (irrespective of viral load) or CD4+ cell count 100–200 cells/μl with viral load greater than 40 copies/ml or new patient. Intermediate: CD4+ cell count 100–300 cells/μl + viral load less than 40 copies/ml or CD4+ cell count greater than 200 cells/μl with viral load greater than 40 copies/ml or non-ART polypharmacy (>5 non-antiretrovirals) or Hx of chronic HCV infection (HCV RNA+) or youth (<25 years old) or CD4+ cell count nadir less than 200 cells/μl. Low: CD4+ cell count greater than 300 cells/μl with viral load less than 40 copies/ml
bUndetectable = viral load under 40 copies/ml; viremia = viral load over 200 copies/ml.

The median time from previous visit to reengagement was 315 days (IQR 246–448), with 72 days (IQR 34–136) between first contact attempt and reengagement. Nurses made a median of two contact attempts (IQR 1.0–4.0) for each patient, and over a third of patients (37%; n = 92) missed at least one scheduled reengagement visit before actually being reengaged. High-risk patients took the longest to reengage and received the most contact attempts, requiring 86 days (IQR 37–107) and four contact attempts (IQR 1.5–5.0) compared with 78 days (IQR 34–145) and two attempts (IQR 1.0–4.0) for intermediate-risk patients, or 57 days (IQR 34–78) and two attempts (IQR 1.0–2.5) for low-risk patients. The proportion of patients who missed a scheduled reengagement visit also increased by risk category.

Pre-post analyses

Results are presented in Table 3. A total of 740 and 804 patients were marked OOC by the pseudo-RPT in the pre-implementation and implementation phases, respectively.

Table 3 - Results from the Poisson regression model with robust variance estimation, overall and by risk category.
Variableb Estimatea (95% CI) P value
All risk categories (implementation phase: n = 804; pre-implementation phase: n = 740)
 (Intercept) 0.576 (0.500–0.664) <0.001
 Imp 1.179 (1.022–1.359) 0.023
 Sex 0.918 (0.783–1.076) 0.290
 Age 0.999 (0.993–1.005) 0.795
 Canada 0.799 (0.686–0.931) 0.004
High-risk category (Implementation phase: n = 199; pre-implementation phase: n = 128)
 (Intercept) 0.665 (0.487–0.909) 0.011
 Imp 1.201 (0.879–1.640) 0.250
 Sex 0.835 (0.613–1.139) 0.256
 Age 1.005 (0.994–1.016) 0.391
 Canada 0.762 (0.533–1.090) 0.137
Intermediate-risk category (implementation phase: n = 481; pre-implementation phase: n = 493)
 (Intercept) 0.601 (0.505–0.716) <0.001
 Imp 1.116 (0.938–1.329) 0.216
 Sex 0.905 (0.742–1.104) 0.327
 Age 0.999 (0.992–1.007) 0.872
 Canada 0.889 (0.740–1.067) 0.207
Low-risk category (implementation phase: n = 124; pre-implementation phase: n = 119)
 (Intercept) 0.345 (0.222–0.536) <0.001
 Imp 1.658 (1.029–2.671) 0.038
 Sex 1.018 (0.606–1.710) 0.947
 Age 0.986 (0.965–1.007) 0.184
 Canada 0.536 (0.326–0.881) 0.014
CI, confidence interval.
aRisk for ’(Intercept)’, risk ratio otherwise.
b‘Imp’ is a binary variable for being marked out of HIV care (OOC) in the implementation phase, compared with the pre-implementation phase.‘Sex’ is a binary variable for the patient's sex, as documented in their electronic medical record, where females are assigned a value of 1.‘Age’ is the patients age, centered at age 50 years.‘Canada’ is a binary variable where patients born in Canada are assigned a value of 1.

After 1 year of the Lost & Found intervention, patients identified as OOC were 1.18 (95% CI 1.02–1.36) times as likely to be reengaged compared with those in the pre-implementation phase after controlling for sex, age, and being born in Canada. This is equivalent to reengaging 72 (95% CI 8.97–134.45) OOC patients who would not otherwise have been reengaged. On the basis of results from the Cox proportional hazards model adjusted for the same covariates, median time to reengagement for OOC patients receiving Lost & Found in the implementation phase was 162 days (95% CI 147–183) compared with 217 days (95% CI 197–245) in the pre-implementation phase.

When stratified by risk category, Lost & Found primarily impacted reengagement in the low-risk category, where OOC patients in the implementation phase were 1.66 (95% CI 1.03–2.67) times more likely to be reengaged compared with those in the pre-implementation phase. High-risk and intermediate-risk patients who received Lost & Found were no more likely to reengage than those in the pre-implementation phase.

The results from the propensity score-adjusted robust Poisson model and Cox proportional hazards model mirror these findings. Full results of propensity score and survival analyses, including outputs from the Cox model, adjusted cumulative incidence curves, as well as tests of the model's assumptions, are presented in Supplemental Digital Content 3,


Our study demonstrates that Lost & Found may be an effective alternative to centralized, D2C approaches in the identification and reengagement of OOC patients. Through the OOC-RPT, nurses were highly successful in validating the HIV care status of most patients identified as possibly OOC. Only 2% (n = 29) were of unknown care status, compared with 14–35% in other interventions [5,6,21,25–27]. Moreover, over a quarter of our original patient pool (n = 641; 27%) were identified as no longer active patients, and the majority were followed elsewhere (n = 578). This reduced the active care list during the first few months of the study, effectively focusing nurses’ reengagement efforts. Integration of the nurse validation step of the intervention appears to be important in the identification of OOC patients for whom reengagement efforts are needed. Compared with surveillance-based D2C approaches, where at least 43% are falsely marked OOC, Lost & Found appears more accurate [3,26,29–31,49]. This initial clean up and accuracy in identifying OOC patients may address some of the high effort and low yield concerns often attributed to reengagement efforts [50].

A large majority (70%) of OOC patients contacted during the implementation phase were successfully reengaged before study end. Although variability in OOC definitions and reporting with respect to patient disposition may limit the capacity for between-study comparisons, this proportion of reengaged patients is similar to others, ranging from 44 to 94% [4–6,22,26,27,37]. Many patients required several contact attempts and among the 33 patients with upcoming appointments at the end of the implementation phase, 61% missed their first scheduled visit. Other studies of interventions to identify and reengage OOC patients have not evaluated phone calls separately from a full intervention package, and as such provide little guidance about the number of contact attempts needed to reengage OOC patients [4–6,21,22,24–27]. Our results, along with a higher level of viremia at reengagement, highlight the importance of ongoing and timely reengagement in this population.

The effort required to reengage patients appears to increase by risk category. Despite being targeted for more rapid reengagement, high-risk patients required more time to reengage and more contact attempts [43]. Over half of high-risk patients (n = 14; 54%) missed at least one scheduled reengagement visit. Social determinants and related barriers (e.g. income, transportation, work or child-care conflicts) are likely major contributors to reengagement difficulties [51]. Information on sociodemographics and reasons for becoming OOC was collected as part of this study; these data will be analyzed and summarized in a forthcoming publication.

Results from our pre-post analyses indicate that OOC patients are more likely to reengage and to do so earlier in the context of Lost & Found. Although positive, these results represent a very conservative evaluation of the intervention's effectiveness. By using a pseudo-RPT in these analyses (i.e. an artificial version of the OOC-RPT where nurse validation is not included), we drastically increased inclusivity in how OOC patients were identified. This added patients in each phase who would normally have been removed from the OOC list by nurses, and likely biased our results to the null. This increase is demonstrated when comparing the number of patients captured by the pseudo-RPT in the implementation phase of the pre-post analysis (n = 804) to the number confirmed OOC of the descriptive analysis (n = 423), where the true OOC-RPT was used including nurse validation. A large difference existed for all risk categories but the largest proportional difference occurred in the high-risk category where only 26% (52/199) of those marked by the pseudo-RPT were truly OOC, indicating a high level of misclassification. The null bias is most evident in the high-risk and intermediate-risk groups of our stratified analyses. Despite nondifferential bias across all categories, we still observe an effect overall as well as in the low-risk category. The intervention is likely effective for other risk categories but requires further evaluation with more appropriate study designs.

This study has several limitations. First, while all pre-post designs are vulnerable to uncontrolled time-related confounding, to our knowledge, there were no major changes in clinical practice guidelines or how patients were followed over the period of analysis. Second, these analyses are limited to only the first OOC event in each period, primarily because of censoring imposed by the pre-post design and to ensure comparability between phases. As a result, we provide no information about patient-level effectiveness of Lost & Found after first successful reengagement or its impact on OOC rates over the long-term. There were 40 patients marked OOC more than once in the implementation phase, all but one of whom were reengaged again before the end of the study period. Third, our robust Poisson model does not account for censoring or provide information about timeliness of reengagement. These are resolved by our secondary analysis using a Cox proportional hazards model. Despite accounting for censoring, the Cox proportional model provided similar results to the robust Poisson model, with the exception that a significant effect of the intervention was observed in the intermediate-risk category of the stratified analysis (Supplemental Digital Content 3, Nonetheless, we only reported details of the robust Poisson analysis for consistency with our protocol and to simplify understanding of our results, given the challenges inherent in hazard ratio interpretation [52,53]. Finally, this study provides little information about effectiveness of the first core element, identifying OOC patients. The intermediate-risk category of the automated portion of our OOC-RPT is likely overly sensitive, largely because of the CD4+ cell counts nadir criterion. However, at the behest of nurses, the primary stakeholders in Lost & Found's implementation, OOC-RPT criteria were left unchanged. Their prioritization of patients with lower CD4+ cell counts nadirs is likely reflected in the higher median pre-reengagement CD4+ cell counts nadirs of contacted OOC patients not yet reengaged (median: 274 cells/μl; IQR: 134–474) compared with OOC patients successfully reengaged (164 cells/μl; IQR: 71–310). We depended on nurses’ clinical judgement to improve accuracy of the OOC-RPT but additional strategies could be used. For example, our OOC-RPT does not consider longitudinal information. Patient trajectories with regard to OOC could be important in determining, which patients will become OOC again [54].

Lost & Found may be a viable, decentralized alternative to D2C approaches to identify and reengage PWH who are OOC. This clinic-based intervention may be especially pertinent in settings with limited capacity to undertake public-health or surveillance-based D2C strategies. Most reengagement interventions depend upon ongoing human resources to supplement existing clinical care, including work by public-health employees or creating new roles in the clinical team [21–25,27,37]. Lost & Found was integrated into existing clinical practice and infrastructure; this meant early investment in EMR improvements and support from an internal facilitator to aid in implementation [43]. Costing is beyond the scope of this study but the limited human resource requirements for this approach may offset early technology-related costs for changes to the clinical EMR. As this intervention requires minimal ongoing financial support and has buy-in from primary implementation stakeholders, sustainability over the long-term is more likely. Consistent with the adaptation inherent to an implementation science approach, clinics with limited technological capacity might consider scaling back automated aspects of the OOC-RPT and creating simpler OOC definitions. Such definitions might prioritize missed visits, which are easily monitored and appear to be strong predictors of future OOC, HIV disease-related health outcomes as well as mortality [17,55–58]. However, this might come at the expense of targeted reengagement for higher risk groups.

Although promising, more robust evaluations of Lost & Found are needed, particularly given the many adjustments and assumptions to accommodate the study design and inadequate control group. A forthcoming mixed-methods analysis of implementation data will provide additional insights, specifically regarding the utility of implementation strategies as well as other factors in the delivery and sustainability of the intervention. Knowledge of both effectiveness and implementation outcomes could inform a future type-I stepped wedge cluster randomized control trial assessing the intervention's true potential.


The authors wish to acknowledge the patient and nurse participants who were essential to this study and the resulting improvements to patient care at the MUHC. These include nurse Emilie Bourbonnière, Melodie Brown, Mariko Duknic, Nancy Obas, and Luciana Ruppenthal. They also wish to acknowledge other clinicians and allied health professionals at the MUHC for their support of the project; Julie Bellingham and Chantal Burrell for their support in translating study documents; Costas Pexos and Jessica Lumia for supporting changes to the RISQ database; Gino Curadeau, Christian Machucha, Ellen Seguin, Vicky Lavazos, Jonathan Roger, Julia Slater, and Sharon Sandy for their ongoing support; and the I-Score patients committee for their assistance in developing study tools and procedures.

Funding: This study is funded by ViiV Healthcare.

Authors’ contributions: J.C. and N.K. are the co-principal investigators. J.C. is the study sponsor. B.Linthwaite, is the research coordinator and internal facilitator. L.R. is a nurse at the CVIS-MUHC. B.Linthwaite, J.C., N.K., and I.M. co-developed the analysis plan. B.Linthwaite, J.C., and N.K. were responsible for obtaining ethics approval. B.Linthwaite wrote the first draft of this article and the protocol for ethics review. B.Linthwaite, J.C., and N.K. collaborated to complete subsequent drafts. All authors commented and critically reviewed drafts of the article. All authors read and approved the final manuscript.

Conflicts of interest

J.C. has received consulting fees from ViiV Healthcare, Merck, and Gilead; research funding from ViiV Healthcare, Merck and Gilead; and payment for lectures from Gilead. N.K. has received consulting fees from ViiV Healthcare, Merck and Gilead; research funding from ViiV Healthcare and Gilead; and payment for lectures from Gilead. B. Lebouché has received consulting fees from ViiV Healthcare Merck and Gilead; research funding from Merck, Gilead, ViiV Healthcare; and payment for lectures from Merck, Gilead and ViiV Healthcare. All other authors declare that they have no competing interests.


1. UNAIDS. Understanding Fast-Track: Accelerating Action to End the AIDS Epidemic by 2030. 2015.
2. Chi BH, Yiannoutsos CT, Westfall AO, Newman JE, Zhou J, Cesar C, et al. International Epidemiologic Databases to Evaluate AIDS Collaboration. Universal definition of loss to follow-up in HIV treatment programs: a statistical analysis of 111 facilities in Africa, Asia, and Latin America. PLoS Med 2011; 8:e1001111.
3. Dombrowski JC, Bove J, Roscoe JC, Harvill J, Firth CL, Khormooji S, et al. ‘Out of Care’ HIV case investigations: a collaborative analysis across six states in the Northwest US. J Acquir Immune Defic Syndr 2017; 74: (Suppl 2): S81.
4. Keller J, Heine A, LeViere AF, Donovan J, Wilkin A, Sullivan K, et al. HIV patient retention: the implementation of a North Carolina clinic-based protocol. AIDS Care 2017; 29:627–631.
5. Udeagu C-CN, Webster TR, Bocour A, Michel P, Shepard CW. Lost or just not following up: public health effort to re-engage HIV-infected persons lost to follow-up into HIV medical care. AIDS 2013; 27:2271–2279.
6. Wohl AR, Dierst-Davies R, Victoroff A, James S, Bendetson J, Bailey J, et al. Implementation and operational research: the navigation program: an intervention to reengage lost patients at 7 HIV clinics in Los Angeles County, 2012–2014. J Acquir Immune Defic Syndr 2016; 71:e44–e50.
7. Crystal S, Sambamoorthi U, Moynihan PJ, McSpiritt E. Initiation and continuation of newer antiretroviral treatments among medicaid recipients with AIDS. J Gen Intern Med 2001; 16:850–859.
8. Robertson MM, Penrose K, Irvine MK, Robbins RS, Kulkarni S, Braunstein SL, et al. Impact of an HIV care coordination program on durable viral suppression. J Acquir Immune Defic Syndr 2019; 80:46–55.
9. Krentz HB, Vu Q, Gill MJ. The impact of ‘Churn’ on plasma HIV burden within a population under care. Open forum infectious diseases. 2019; USA: Oxford University Press, ofz203.
10. Mugavero MJ, Lin H-Y, Allison JJ, Giordano TP, Willig JH, Raper JL, et al. Racial disparities in HIV virologic failure: do missed visits matter?. J Acquir Immune Defic Syndr 2009; 50:100–108.
11. Bastard M, Pinoges L, Balkan S, Szumilin E, Ferreyra C, Pujades-Rodriguez M. Timeliness of clinic attendance is a good predictor of virological response and resistance to antiretroviral drugs in HIV-infected patients. PLoS One 2012; 7:e49091.
12. Rocheleau G, Brumme C, Shoveller J, Lima V, Harrigan P. Longitudinal trends of HIV drug resistance in a large Canadian cohort, 1996–2016. Clin Microbiol Infect 2018; 24:185–191.
13. Buchacz K, Lau B, Jing Y, Bosch R, Abraham AG, Gill MJ, et al. North American AIDS Cohort Collaboration on Research and Design (NA-ACCORD) of IeDEA. Incidence of AIDS-defining opportunistic infections in a multicohort analysis of HIV-infected persons in the United States and Canada, 2000–2010. J Infect Dis 2016; 214:862–872.
14. Gardner EM, Maravi ME, Rietmeijer C, Davidson AJ, Burman WJ. The association of adherence to antiretroviral therapy with healthcare utilization and costs for medical care. Appl Health Econ Health Policy 2008; 6:145–155.
15. Giordano TP, Gifford AL, White AC Jr, Almazor MES, Rabeneck L, Hartman C, et al. Retention in care: a challenge to survival with HIV infection. Clin Infect Dis 2007; 44:1493–1499.
16. Kendall C, Raboud J, Donelle J, Loutfy M, Rourke S, Kroch A, et al. Ontario HIV Treatment Network (OHTN) Cohort Study Team. Lost but not forgotten: a population-based study of mortality and care trajectories among people living with HIV who are lost to follow-up in Ontario, Canada. HIV Medicine 2019; 20:88–98.
17. Mugavero MJ, Lin H-Y, Willig JH, Westfall AO, Ulett KB, Routman JS, et al. Missed visits and mortality among patients establishing initial outpatient HIV treatment. Clinical Infectious Diseases 2009; 48:248–256.
18. Skarbinski J, Rosenberg E, Paz-Bailey G, Hall HI, Rose CE, Viall AH, et al. Human immunodeficiency virus transmission at each step of the care continuum in the United States. JAMA Intern Med 2015; 175:588–596.
19. Centers for Disease Control and Prevention. Effective interventions: data to care.
20. Sweeney P, DiNenno EA, Flores SA, Dooley S, Shouse RL, Muckleroy S, et al. HIV data to care—using public health data to improve HIV care and prevention. J Acquir Immune Defic Syndr 2019; 82:S1–S5.
21. Buchacz K, Chen M-J, Parisi MK, Yoshida-Cervantes M, Antunez E, Delgado V, et al. Using HIV surveillance registry data to re-link persons to care: the RSVP Project in San Francisco. PLoS One 2015; 10:e0118923.
22. Bove J, Golden MR, Dhanireddy S, Harrington RD, Dombrowski JC. Outcomes of a clinic-based, surveillance-informed intervention to relink patients to HIV care. J Acquir Immune Defic Syndr 2015; 70:262.
23. Dombrowski JC, Hughes JP, Buskin SE, Bennett A, Katz D, Fleming M, et al. A cluster randomized evaluation of a health department data to care intervention designed to increase engagement in HIV care and antiretroviral use. Sex Transm Dis 2018; 45:361–367.
24. Fanfair RN, Khalil G, Camp N, Brady K, DeMaria A, Villanueva M, et al.Health Department Randomized Trial to Re-engage Out-of-Care HIV Infected Persons - The Cooperative Re-Engagement Controlled Trial. epo Conference on Retroviruses and Opportunistic Infections (CROI). Seattle, Washington, USA; 2019 [abstract 1031].
25. Hart-Malloy R, Brown S, Bogucki K, Tesoriero J. Implementing data-to-care initiatives for HIV in New York state: assessing the value of community health centers identifying persons out of care for health department follow-up. AIDS Care 2018; 30:391–396.
26. Sweeney P, Hoyte T, Mulatu MS, Bickham J, Brantley AD, Hicks C, et al. Implementing a data to care strategy to improve health outcomes for people with HIV: a report from the care and prevention in the United States demonstration project. Public Health Rep 2018; 133: (2 Suppl): 60S–74S.
27. Tesoriero JM, Johnson BL, Hart-Malloy R, Cukrovany JL, Moncur BL, Bogucki KM, et al. Improving retention in HIV care through New York's expanded partner services data-to-care pilot. J Public Health Manag Pract 2017; 23:255–263.
28. Sachdev DD, Mara E, Hughes AJ, Antunez E, Kohn R, Cohen S, et al. ‘Is a Bird in the Hand Worth 5 in the Bush?’: a comparison of 3 data-to-care referral strategies on HIV care continuum outcomes in San Francisco. Open Forum Infectious Diseases. 2020; US: Oxford University Press, ofaa369.
29. Buskin SE, Kent JB, Dombrowski JC, Golden MR. Migration distorts surveillance estimates of engagement in care: results of public health investigations of persons who appear to be out of HIV care. Sex Transm Dis 2014; 41:35.
30. Hague JC, John B, Goldman L, Nagavedu K, Lewis S, Hawrusik R, et al. Using HIV surveillance laboratory data to identify out-of-care patients. AIDS Behav 2019; 23:78–82.
31. Krentz HB, McPhee P, Arbess G, Jackson L, Stewart AM, Bois D, et al. Reporting on patients living with HIV ‘disengaging from care’. Who is actually ‘lost to follow-up’?. AIDS Care 2020; 1–7.
32. Macomber KE, Viall A, Ramakrishnan V, Wilson J, Brandt M-G, Kinsinger L, et al. Operationalizing a data to care strategy in Michigan through cross-agency collaborations. J Acquir Immune Defic Syndr 2019; 82: (Suppl 1): S69–S73.
33. Ocampo JMF, Smart J, Allston A, Bhattacharjee R, Boggavarapu S, Carter S, et al. Improving HIV surveillance data for public health action in Washington, DC: a novel multiorganizational data-sharing method. JMIR Public Health Surveill 2016; 2:e3.
34. Hart-Malloy R, Rajulu DT, Johnson MC, Shrestha T, Spencer EC, Anderson BJ, et al. Cross-jurisdictional data to care: lessons learned in New York State and Florida. J Acquir Immune Defic Syndr 2019; 82 Suppl 1:S42–S46.
35. Gasner MR, Fuld J, Drobnik A, Varma JK. Legal and policy barriers to sharing data between public health programs in New York City: a case study. Am J Public Health 2014; 104:993–997.
36. Sweeney P, Gardner LI, Buchacz K, Garland PM, Mugavero MJ, Bosshart JT, et al. Shifting the paradigm: using HIV surveillance data as a foundation for improving HIV care and preventing HIV infection. Milbank Q 2013; 91:558–603.
37. Bean MC, Scott L, Kilby JM, Richey LE. Use of an outreach coordinator to reengage and retain patients with HIV in care. AIDS Patient Care STDs 2017; 31:222–226.
38. Mokotoff ED, Ruth KG, Benbow N, Sweeney P, Sapiano TN, McNaghten A. Data to care: lessons learned from delivering technical assistance to 20 health departments. J Acquir Immune Defic Syndr 2019; 82 Suppl 1:S74–S79.
39. National Alliance of State & Territorial AIDS Directors. Data to Care Hub.
40. National Alliance of State & Territorial AIDS Directors. Data points: a health department roadmap for enhancing data to care programs. Available at
41. Kiragga AN, Nalintya E, Morawski BM, Kigozi J, Park BJ, Kaplan JE, et al. Impact of nurse-targeted care on HIV outcomes among immunocompromised persons: a before-after study in Uganda. J Acquir Immune Defic Syndr 2016; 72:e32.
42. Walugembe DR, Sibbald S, Le Ber MJ, Kothari A. Sustainability of public health interventions: where are the gaps?. Health Research Policy and Systems 2019; 17:8.
43. Cox J, Linthwaite B, Engler K, Lessard D, Lebouché B, Kronfli N. A type II implementation-effectiveness hybrid quasi-experimental pilot study of a clinical intervention to re-engage people living with HIV into care,‘Lost & Found’: an implementation science protocol. Pilot Feasibility Stud 2020; 6:1–11.
44. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care 2012; 50:217–226.
45. Tsou T-S. Robust Poisson regression. J Stat Planning Inference 2006; 136:3173–3186.
46. Knol MJ, Le Cessie S, Algra A, Vandenbroucke JP, Groenwold RH. Overestimation of risk ratios by odds ratios in trials and cohort studies: alternatives to logistic regression. CMAJ 2012; 184:895–899.
47. Zou G. A modified poisson regression approach to prospective studies with binary data. Am J Epidemiol 2004; 159:702–706.
48. Bulsara SM, Wainberg ML, Newton-John TRO. Predictors of adult retention in HIV care: a systematic review. AIDS Behav 2018; 22:752–764.
49. Kunzweiler C, Kishore N, John B, Roosevelt K, Lewis S, Klevens RM, et al. Using HIV surveillance and clinic data to optimize data to care efforts in community health centers in Massachusetts: the Massachusetts Partnerships for Care Project. J Acquir Immune Defic Syndr 2019; 82 Suppl 1:S33–S41.
50. Dombrowski JC. Systems of care for vulnerable populations in the United States. In: Conference on Retroviruses and Opportunistic Infections (CROI). Seattle, Washington, USA. 2018.
51. Gill VC, Krentz HB. Patient perspectives on leaving, disengaging, and returning to HIV care. AIDS Patient Care STDs 2015; 29:400–407.
52. Hernán MA. The hazards of hazard ratios. Epidemiology 2010; 21:13–15.
53. Sutradhar R, Austin PC. Relative rates not relative risks: addressing a widespread misinterpretation of hazard ratios. Ann Epidemiol 2018; 28:54–57.
54. Lee H, Wu XK, Genberg BL, Mugavero MJ, Cole SR, Lau B, Hogan JW. Centers for AIDS Research Network of Integrated Clinical Systems (CNICS) Investigators. Beyond binary retention in HIV care: predictors of the dynamic processes of patient engagement, disengagement, and re-entry into care in a US clinical cohort. AIDS 2018; 32:2217–2225.
55. Zinski A, Westfall AO, Gardner LI, Giordano TP, Wilson TE, Drainoni M-L, et al. The contribution of missed clinic visits to disparities in HIV viral load outcomes. Am J Public Health 2015; 105:2068–2075.
56. Pence BW, Bengtson AM, Boswell S, Christopoulos KA, Crane HM, Geng E, et al. Who will show? Predicting missed visits among patients in routine HIV primary care in the United States. AIDS Behav 2019; 23:418–426.
57. Walburn A, Swindells S, Fisher C, High R, Islam K. Missed visits and decline in CD4 cell count among HIV-infected patients: a mixed method study. Int J Infect Dis 2012; 16:e779–e785.
58. Nabaggala MS, Parkes-Ratanshi R, Kasirye R, Kiragga A, Castlenuovo B, Ochaka I, et al. Re-engagement in HIV care following a missed visit in rural Uganda. BMC Res Notes 2018; 11:1–5.

data to care; HIV; implementation science; lost to follow-up; not in care; out of care; reengagement

Supplemental Digital Content

Copyright © 2021 The Author(s). Published by Wolters Kluwer Health, Inc.