Secondary Logo

Journal Logo

Infectious Disease

Far from MCAR

Obtaining Population-level Estimates of HIV Viral Suppression

Balzer, Laura B.a; Ayieko, Jamesb; Kwarisiima, Dalsonec; Chamie, Gabrield; Charlebois, Edwin D.e; Schwab, Joshuaf; van der Laan, Mark J.f; Kamya, Moses R.g; Havlir, Diane V.d; Petersen, Maya L.f

Author Information
doi: 10.1097/EDE.0000000000001215


Accurate population-level estimates of disease prevalence and treatment coverage are needed to quantify disease burden and evaluate the success of programs for epidemic control. The data available to inform such estimates, however, are often susceptible to differential missingness. In other words, the missing-completely-at-random (MCAR) assumption rarely, if ever, holds.1–4 The field of HIV prevention and treatment provides an illustrative example. Consider the Joint United Nations Program on HIV/AIDS (UNAIDS) 90-90-90 target for the year 2020: 90% of all HIV-positive persons should know their status; 90% of those who know their status should be receiving antiretroviral therapy (ART); and 90% of those receiving ART should have suppressed HIV viral replication.5 Multiplying these proportions together yields an overall target, referred to here as “population-level suppression”—73% of all HIV-positive persons should have suppressed HIV viral replication (Appendix). This target reflects the HIV care “cascade” from diagnosis, through treatment initiation and retention, to viral suppression.

While population-level suppression is widely used in assessing HIV care strategies, two recent systematic reviews noted the variability in both data quality and statistical approaches used for assessment.6,7 In particular, Granich et al.6 remarked on the challenges posed by incomplete data and inconsistent methodology, while Sabapathy et al.7 proposed a template to standardize data collection and evaluation. In this article, we provide an in-depth demonstration of the methods used to estimate population-level suppression in the Sustainable East Africa Research in Community Health (SEARCH) Study, a cluster-randomized trial in rural Kenya and Uganda (NCT01864603).8,9 We approach the missing data problem with a causal framework to define target parameters with counterfactuals, state identifiability assumptions sufficient to translate these targets into statistical quantities, and estimate the resulting statistical parameters.1–4,10–14 We refer the reader to companion articles for details on trial.8,9


In general, the total number of HIV-positive persons in a population is unknown, and individuals with known HIV status are not necessarily representative of the general population. If, for example, persons with higher health-seeking behavior are more likely to test and less likely to be infected, then an unadjusted estimate of HIV prevalence (i.e., the proportion with HIV among those tested) is likely to underestimate the true prevalence of HIV in the population, even in the context of community-wide testing, as was implemented in recent Universal-Test-and-Treat trials.9,15–18

Likewise, measurement of plasma HIV RNA levels (viral loads) among HIV-positive individuals is generally incomplete and often depends on factors associated with viral suppression. For example, if viral loads are only measured at HIV clinic visits, then viral suppression among measured individuals will overestimate suppression among all HIV-positive persons, including newly diagnosed individuals who are not yet in-care and previously diagnosed individuals who are out-of-care. These familiar missing data challenges can be illustrated with a directed acyclic graph or another causal modeling approach (Figure 1).14,19–24

Simplified directed acyclic graph to represent the challenges posed by incomplete HIV testing. Demographics and prior testing are common causes of current testing (the hypothetical intervention node) and underlying HIV status (possibly unobserved), both of which impact observed HIV status. Analogous challenges arise due to incomplete measurement of suppression among HIV-positive persons.

Overcoming these challenges requires knowledge of the data-generating process. Consider the 16 communities in the intervention arm of the SEARCH Study. After a door-to-door census, we conducted community-wide testing annually through multidisease health fairs, followed by out-of-facility testing for residents who did not attend the fair.25,26 We linked participants over successive years with a fingerprint biometric. Prior diagnosis of HIV and ART use was ascertained through linkage to clinic records.8,27 A recensus was conducted 3 years after follow-up to determine interim deaths, out-migrations, and in-migrations.9

With this measurement scheme in mind, we describe the methods used in Petersen et al.8 and Havlir et al.9 to characterize viral suppression in the intervention arm at study baseline t = 0, and annually thereafter t = {1, 2, 3}. These cross-sectional analyses provide snapshots of population-level suppression among an open cohort of adult (≥15 years) residents (allowing for entry due to age and in-migration, and exit due to death or outmigration). We note estimating viral suppression among a closed cohort of known baseline HIV-positive residents is a distinct goal, resulting in a different causal parameter, identifiability assumptions, and estimation approach.8,9

Causal Parameters

Let be an indicator that an individual is HIV positive at time t, irrespective of whether serostatus is measured. Likewise, let be a possibly unmeasured indicator of HIV viral suppression (<500 cps/mL) at time t. Population-level suppression is the conditional probability of viral suppression given HIV-positive status: , or equivalently, the joint probability of being HIV-positive with suppression, divided by the probability of being HIV-positive (i.e., HIV prevalence):

Ideally, anyone not already known to be HIV-positive (i.e., previously HIV-negative or HIV-unknown) would be tested at time t. Of course, this is never the case; further, missingness inherently depends on underlying HIV status—the status of an HIV-negative individual who does not test at time t is unknown, whereas the status of an HIV-positive individual not seen at time t might be known from prior testing. The problem is intensified after multiple rounds of community-wide testing, which provide multiple opportunities for prevalent HIV-positive persons to be diagnosed. To avoid this inherent dependence, we define as an indicator that an individual was seen at community-wide testing and had “known” HIV status at time t—due to a negative test result at time t, or a positive result at or before time t. We define observed HIV status as .

As with HIV testing, viral load measurement is incomplete; HIV-positive persons on whom viral load is measured may differ systematically from HIV-positive persons who are missing this measure. Define as an indicator of viral load measurement at time t, and define observed viral suppression as .

Three Sets of Identifiability Assumptions

In the above, population-level suppression was expressed in terms of underlying indicators of HIV seropositivity and viral suppression: . We now present three sets of identifiability assumptions to write the numerator and denominator in this expression as parameters of the observed data distribution (see Appendix).


Suppose we are willing to assume that HIV prevalence among those seen at time t is representative of HIV prevalence among those not seen, and that viral suppression among HIV-positive persons with viral loads measured at time t is representative of suppression among HIV-positive persons without measured viral loads. More formally, suppose we make the following randomization assumptions as applied to missing data:1–4,10–14 and . If these assumptions hold, the numerator of population-level suppression is identified as

and denominator as .27 Taking the ratio of these yields the unadjusted statistical parameter:

Baseline Adjustment

We can weaken the above assumptions on the missingness process by conditioning on baseline covariates. Specifically, let B denote mutually exclusive and exhaustive strata defined by age group, sex, and community of residence. Now suppose within each strata b, HIV prevalence among those seen at time t is representative of prevalence among those not seen, and within each strata b, suppression among HIV-positive persons with viral loads measured at time t is representative of suppression among HIV-positive persons without measured viral loads. More formally, we assume and .

Under these assumptions on missingness, we obtain the following identifiability result, corresponding to a hypothetical, dynamic intervention to first ensure knowledge of HIV status and then to ensure measurement of viral loads among HIV-positive persons:28–31

In words, this is the strata-specific probability of viral suppression, given measurement and HIV-positive status; multiplied by the strata-specific probability of being HIV-positive, given measurement; and then standardized with respect to the distribution of strata. Identification of the denominator, population-level prevalence, follows from above:

By taking the ratio of the numerator (Eq. 2) to the denominator (Eq. 3), we obtain a baseline-adjusted statistical parameter corresponding to population-level suppression under the above assumptions.

For the conditioning sets to be well defined, we also require the positivity assumption.11,32 Irrespective of age, sex, and community, there must be a positive probability of being seen with known HIV status , and for every strata in which some proportion of HIV-positive persons are seen, there must be a positive probability of viral load measurement: .

Time-varying Adjustment

While stratifying on certain baseline characteristics weakens our assumptions on missingness, there may be many other variables potentially impacting testing, underlying HIV status, and viral suppression among HIV-positive persons. In particular, ART use is a key determinant of viral suppression and may also be predictive of viral load measurement.

Define as an indicator of ART initiation before time t, and let denote the remaining observed variables that are potentially predictive of both viral suppression and its measurement: the full set of baseline demographics (e.g., age, sex, marital status, education, occupation, alcohol use, mobility, wealth index, and community) together with prior HIV testing and suppression. While viral suppression without ART is possible, the UNAIDS target is focused on ART-induced suppression (Appendix).5 Therefore, we set to zero for persons not on ART. We further assume complete measurement of ART use, acknowledging that if ART use is incompletely captured, this assumption will lead to underestimation of suppression. Finally, for HIV-positive persons who have initiated ART, we assume that conditional on baseline and time-updated covariates , suppression among those with a measured viral load at time t is representative of suppression among those with a missing viral load. More formally, we assume .

We also require the positivity assumption; all HIV-positive individuals who have initiated ART have a positive probability of having their viral load measured, regardless of their baseline and time-updated covariates: almost everywhere. Under these assumptions, we have the following identifiability result corresponding to a hypothetical intervention to ensure viral load measurement among ART initiators:28

where the summation generalizes to an integral for continuous covariates. In words, this is the proportion of individuals who have started ART (and are, by implication, HIV-positive) in the total population (including both HIV-positive and HIV-negative persons) multiplied by the adjusted probability of being suppressed, given prior ART initiation.

For the denominator of HIV prevalence, we also consider an expanded adjustment set , consisting of all baseline demographics and prior HIV testing. For the subgroup without a prior HIV diagnosis, we assume that conditional on , HIV prevalence among those tested at time t is representative of HIV prevalence among those not tested, or more formally, . We further assume positivity; previously undiagnosed persons have some chance of being tested regardless of their values: almost everywhere. Under these assumptions, we have the following identifiability result corresponding to a hypothetical intervention to ensure HIV status is known:28

where the summation generalizes to an integral for continuous covariates. In words, this is the proportion of the population previously known to be HIV-positive plus the adjusted proportion of the population newly known to be HIV-positive.

Taking the ratio of the numerator (Eq. 4) to the denominator (Eq. 5) yields a fully adjusted statistical parameter for population-level suppression under the above assumptions.

Estimation Approaches

The unadjusted parameter (Eq. 1) can be estimated with the empirical proportion of the population with measured viral suppression. The baseline-adjusted parameter (Eq. 2 ÷ Eq. 3) can also be estimated with empirical proportions. Specifically, we would generate covariate strata-specific estimates by taking empirical means, and then combine by standardizing across strata. A similar approach was used in the PopART (HPTN 071) Universal-Test-and-Treat trial with stratification factors including sex, age group, and community.17,33,34 This approach corresponds to G-computation when fully saturated regressions are used to estimate the conditional probability of the outcome, given measurement and the adjustment set (i.e., the “outcome regression”).28,35,36 It is further equivalent to inverse-weighting when fully saturated regressions are used to estimate the conditional probability of measurement, given the adjustment set (i.e., the “propensity score”).37–39

When the adjustment set is higher dimensional, such as for our fully adjusted parameter (Eq. 4 ÷ Eq. 5), alternative approaches are needed to smooth over values of the covariates with weak support.32 We could, for example, use logistic regression with two-way interactions to estimate the propensity scores for inverse-weighting. This approach was used in a sensitivity analysis in the Ya Tsie Universal-Test-and-Treat trial (also called the Botswana Combination Prevention Project).18,40

Another approach is targeted maximum likelihood estimation (TMLE), which offers efficiency gains over inverse-weighting and allows for flexible adjustment for a large set of covariates through machine learning.24,41 We refer the reader to42,43 for an introduction to TMLE and to44 for a detailed comparison with other methods, such as parametric G-computation and augmented inverse-weighting. Briefly, TMLE combines estimates of outcome regression with estimates of the propensity score to achieve a number of desirable properties. TMLE is double robust—it is consistent if either the outcome regression is consistently estimated or the propensity score is consistently estimated. TMLE is also a substitution estimator, potentially improving robustness under strong confounding or rare outcomes.44–47

In eAppendix 1;, we provide step-by-step descriptions of the unadjusted estimator, parametric G-computation, inverse-weighting estimator, and TMLE to adjust for missing outcomes.


In the SEARCH Study, the primary approach used TMLE to estimate the fully adjusted parameter (Eq. 4 ÷ Eq. 5). Within TMLE, Super Learner was implemented to estimate the outcome regressions and propensity scores (i.e., the conditional probability of measurement, given the adjustment set).48 Super Learner is an ensemble, machine learning method that uses cross-validation to build the optimal combination of predictions from a library of candidate algorithms. For ease of interpretation, reduced computational burden, and to avoid overfitting, our Super Learner library was limited to logistic regressions (with various screening algorithms), generalized additive models (with various screening algorithms), and the mean (corresponding to the unadjusted estimator). We implemented TMLE fully stratified on community, allowing the outcome regressions and propensity scores to vary by community. Full implementation details, including computing code, are available in eAppendix 2;

For comparison, we also present the use of empirical proportions to estimate the unadjusted parameter (Eq. 1), and the baseline-adjusted parameters (Eq. 2 ÷ Eq. 3) controlling for sex, age group (15–19, 20–29, 30–39, 40–49, 50–59, and 60+ years), and community.

We obtained statistical inference with influence curve standard errors, treating the community as the unit of independence. We conducted analyses in R v.3.6.1 (Vienna, Austria) with the ltmle_v1.1-0 and SuperLearner_v2.0-25 packages.49–51

Ethical Considerations

The SEARCH Study was approved by the ethics committees at the University of California, San Francisco; the Kenya Medical Research Institute; and Makerere University School of Medicine in Uganda. As previously described,9 oral informed consent was provided participation in the census and health fairs; written informed consent was provided to receive ART outside of country guidelines.


The baseline characteristics of the study participants have been described elsewhere.8,9,25 In brief, approximately one-third of the 79,818 residents enumerated in the baseline census were from each study region, and nearly half of participants were age 15–30 years; men constituted 45% (eTable 1; HIV status was determined on 89% (71,402) of residents at baseline (Table 1). After baseline, knowledge of HIV serostatus remained high with 77% of residents (69,175/90,047) seen at year 1, 75% (71,577/95,599) at year 2, and 81% (80,390/99,186) at year 3. There were no obvious demographic differences between the enumerated population and those with known HIV serostatus (eTable 1;

Number and Coverage of Residents Contributing to Unadjusted Estimates of Population-level HIV Viral Suppression at the Time of Annual Testing

Viral loads were measured for 76% of baseline HIV-positive residents (Table 1). Missing viral loads were more common at baseline due to early assay failures.26 Despite ~95% coverage of viral load measurement for the remaining years, baseline and time-varying characteristics differed for HIV-positive persons with measured versus missed HIV RNA levels (Table 2). In particular, HIV-positive women were more likely to have their viral load measured than HIV-positive men. After baseline, adolescents (15–24 years) were more likely to be missed than older adults (25+ years). Viral load measurement also differed notably by the time-varying characteristics; HIV-positive persons who were previously aware of their status, had evidence of starting ART, or had a history of suppressing viral replication were more likely to have their viral load measured than their counterparts.

Select Baseline and Time-varying Characteristics of HIV-positive Residents by Year and by Viral Load Measurement

Estimates of population-level suppression varied meaningfully depending on the identifiability assumptions employed (Figure 2). At baseline, the unadjusted approach suggested that half of all HIV-positive residents had suppressed viral replication (50%; 95% CI = 46%, 54%). Stratifying on age group, sex, and community slightly reduced the estimate to 49% (95% CI = 45%, 54%). The lowest estimate of 42% (95% CI = 37%, 46%) was obtained after adjusting for the full set of baseline and time-varying characteristics.

Estimates of population-level HIV viral suppression at the time of annual testing in the intervention arm of the SEARCH Study. Estimates were obtained with the empirical mean among those measured (“unadjusted”), stratifying on sex, age group, and community (“baseline adjustment”), and using targeted maximum likelihood estimation (TMLE) with Super Learner to adjust for both baseline and time-varying characteristics (“time-varying adjustment”). Horizontal dashed line at 73% suppression indicates UNAIDS target; vertical lines indicate 95% confidence intervals.

Deviations were pronounced in subsequent years (Figure 2). The unadjusted approach suggested that 80% (95% CI = 78%, 82%) of all HIV-positive residents were suppressed at year 1, 85% (95% CI = 83%, 86%) at year 2, and 85% (95% CI = 83%, 87%) at year 3. Estimates adjusted for baseline covariate-strata were similar: 79% (95% CI = 77%, 81%) at year 1, 84% (95% CI = 82%, 86%) at year 2, and 84% (95% CI = 83%, 86%) at year 3. Fully adjusted estimates were meaningfully lower: 71% (95% C = 69%, 73%) at year 1, 76% (95% CI = 74%, 78%) at year 2, and 79% (95% CI = 77%, 81%) at year 3.


In an open cohort of nearly 100,000 residents in rural Kenya and Uganda, we compared three approaches for missing outcomes: (1) an unadjusted approach, the empirical proportion among those measured; (2) stratification on age group, sex, and community; and (3) TMLE with Super Learner to adjust for the full set of baseline and time-varying covariates. Despite low levels of missingness, estimates diverged by identifiability assumptions. The unadjusted approach consistently yielded the highest estimates; the fully adjusted approach consistently yielded the lowest estimates.

In the SEARCH Study, HIV serostatus and HIV RNA viral levels were obtained through multidisease testing at health fairs with follow-up for nonparticipants.25 Unlike clinic-based ascertainment, this approach reaches HIV-positive persons who are in-care or who are out-of-care.9,16–18 As a result, the MCAR assumption may seem reasonable.1–4 However, deviations between the unadjusted estimates and adjusted ones suggest there were meaningful differences in the population measured and population missed with respect to, among other factors, prior diagnosis, ART use, and viral suppression.

Both adjusted approaches were built on causal assumptions that (distinct subsets of) observed participant characteristics were sufficient to control for the common causes of measurement and health outcomes.1–4 When assuming baseline covariates were sufficient, we fully stratified on sex, age group, and community; as a result, this was equivalent to a fully nonparametric approach for the outcome regression in G-computation and to a fully nonparametric approach for the propensity score in inverse-weighting. Beyond age, sex, and community, there were, however, additional differences between those with measured versus missing viral loads, including differences in postbaseline variables; specifically, persons without prior diagnosis, ART initiation, or viral suppression were less likely to have their viral load measured. Therefore, the primary approach in the SEARCH Study was to weaken the assumptions on the missingness process by conditioning on a larger set of baseline and time-updated covariates.8,9,27 We estimated the fully adjusted statistical parameter using TMLE with Super Learner.

It is important to note, however, that the primary approach still relied on nontestable assumptions. If there is an unmeasured cause of measurement and health outcomes, none of the identifiability assumptions will hold and the wished-for causal parameters will diverge from the statistical estimands. In the SEARCH Study, we aimed to minimize this divergence by conditioning on a large set of baseline and time-varying covariates and assessing the plausibility of assumptions with experts in HIV prevention. During estimation, we further aimed to avoid unsubstantiated assumptions and minimize bias due to model misspecification by using TMLE with Super Learner. This approach will provide a consistent point estimate if either the outcome regression or the propensity score is consistently estimated and will provide consistent inference (i.e., valid confidence intervals) when both are consistently estimated at sufficiently fast rates and satisfy the relevant empirical process conditions.52,53 Recent extensions to obtain double robust inference, while of interest, were not implemented in the current analyses.

While it is well recognized that estimates can only be interpreted causally if the identifiability assumptions hold, similar challenges apply when interpreting estimates adjusting for missing data.14 Specifically, our estimates are only truly representative of population-level metrics if the missingness assumptions hold13 and there is sufficient data support. These challenges, however, should not inspire inaction; instead, we often need to use available data to obtain timely estimates of disease burden and control under minimal assumptions, which are transparently stated and carefully evaluated.

In the running example, the identification choice had implications for policymaking and targeting resources. Both the unadjusted and baseline-adjusted approaches suggested the UNAIDS 90-90-90 target (73% suppression) was surpassed within 1 year of the intervention and the UNAIDS 95-95-95 target (86% suppression) was nearly achieved by the trial’s close.5 In contrast, the estimates controlling for time-updated covariates indicated the 90-90-90 target was achieved after 2 years, but there still was a substantial gap to the 95-95-95 target.

In summary, estimates of population-level HIV viral suppression continue to be the benchmark in assessing programmatic success in epidemic control. In four cross-sectional analyses of nearly 100,000 participants in the intervention arm of the SEARCH Study, we demonstrated the impact of missing data assumptions that can occur even in large studies with high measurement. While we focused on a specific example—estimating population-level HIV viral suppression—our results have broad applicability to missing data problems, commonly encountered in public health and medicine.3,4,10–13 We recommend adjustment for a large set of baseline and time-varying covariates that potentially influence both measurement and underlying status; TMLE with Super Learner is one approach to performing such adjustments.


We thank the Ministry of Health of Uganda and of Kenya; our research teams and administrative teams in San Francisco, Uganda, and Kenya; collaborators and advisory boards; and especially all the communities and participants involved in the study.


1. Rubin DB. Inference and missing data. Biometrika. 1976;63:581–592.
2. Little RJA, Rubin DB. Statistical Analysis with Missing Data. 2002.2nd ed. Hoboken, NJ: Wiley.
3. Perkins NJ, Cole SR, Harel O, et al. Principled approaches to missing data in epidemiologic studies. Am J Epidemiol. 2018;187:568–575.
4. Hughes RA, Heron J, Sterne JAC, Tilling K. Accounting for missing data in statistical analyses: multiple imputation is not always the answer. Int J Epidemiol. 2019;48:1294–1304.
5. Joint United Nations Programme on HIV/AIDS (UNAIDS). 90-90-90 An Ambitious Treatment Target to Help End the AIDS Epidemic. 2014. Available at: Accessed 14 October 2019.
6. Granich R, Gupta S, Hall I, Aberle-Grasse J, Hader S, Mermin J. Status and methodology of publicly available national HIV care continua and 90-90-90 targets: a systematic review. PLoS Med. 2017;14:e1002253.
7. Sabapathy K, Hensen B, Varsaneux O, Floyd S, Fidler S, Hayes R. The cascade of care following community-based detection of HIV in sub-Saharan Africa - a systematic review with 90-90-90 targets in sight. PLoS One. 2018;13:e0200737.
8. Petersen M, Balzer L, Kwarsiima D, et al. Association of implementation of a universal testing and treatment intervention with HIV diagnosis, receipt of antiretroviral therapy, and viral suppression in East Africa. JAMA. 2017;317:2196–2206.
9. Havlir DV, Balzer LB, Charlebois ED, et al. HIV testing and treatment with the use of a community health approach in rural Africa. N Engl J Med. 2019;381:219–229.
10. Robins JM. Robust estimation in sequentially ignorable missing data and causal inference models. In: 1999 Proceedings of the American Statistical Association. 2000:American Statistical Association; 6–10.
11. Robins JM, Rotnitzky A, Zhao LP. Estimation of regression coefficients when some regressors are not always observed. J Am Stat Assoc. 1994;89:846–866.
12. Bang H, Robins JM. Doubly robust estimation in missing data and causal inference models. Biometrics. 2005;61:962–973.
13. Mohan K, Pearl J, Tian J. Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ. Graphical models for inference with missing data. In: Advances in Neural Information Processing Systems 26. 2013Red Hook, NY: Curran Associates; 1277–1285.
14. Petersen ML, van der Laan MJ. Causal models and learning from data: integrating causal modeling and statistical estimation. Epidemiology. 2014;25:418–426.
15. Perriat D, Balzer LB, Hayes R, Lockman S, Walsh F, et al. Comparative assessment of five large-scale studies of universal HIV testing and treatment in Sub-Saharan Africa. J Int AIDS Soc. 2018;21:e25048.
16. Iwuji CC, Orne-Gliemann J, Larmarange J, et al.; ANRS 12249 TasP Study Group. Universal test and treat and the HIV epidemic in rural South Africa: a phase 4, open-label, community cluster randomised trial. Lancet HIV. 2018;5:e116–e125.
17. Hayes RJ, Donnell D, Floyd S, et al.; HPTN 071 (PopART) Study Team. Effect of universal testing and treatment on HIV incidence - HPTN 071 (PopART). N Engl J Med. 2019;381:207–218.
18. Makhema J, Wirth KE, Pretorius Holme M, et al. Universal testing, expanded treatment, and incidence of HIV infection in Botswana. N Engl J Med. 2019;381:230–242.
19. Neyman J. Sur les applications de la theorie des probabilites aux experiences agricoles: Essai des principes (In Polish). English translation by D.M. Dabrowska and T.P. Speed (1990). Stat Sci. 1923;5:465–480.
20. Rubin DB. Estimating causal effects of treatments in randomized and nonrandomized studies. J Educ Psychol. 1974;66:688–701.
21. Holland PW. Statistics and causal inference. J Am Stat Assoc. 1986;81:945–960.
22. Rubin DB. Comment: Neyman (1923) and causal inference in experiments and observational Studies. Stat Sci. 1990;5:472–480.
23. Pearl J. Causality: Models, Reasoning and Inference. 2009.2nd ed. New York, NY: Cambridge University Press.
24. van der Laan M, Rose S. Targeted Learning: Causal Inference for Observational and Experimental Data. 2011.New York: Springer.
25. Chamie G, Clark TD, Kabami J, et al. A hybrid mobile approach for population-wide HIV testing in rural east Africa: an observational study. Lancet HIV. 2016;3:e111–e119.
26. Jain V, Liegler T, Kabami J, et al.; SEARCH Collaboration. Assessment of population-based HIV RNA levels in a rural east African setting using a fingerprick-based blood collection method. Clin Infect Dis. 2013;56:598–605.
27. Balzer LB, Schwab J, Laan MJ, van der, Petersen ML. Evaluation of Progress Towards the UNAIDS 90-90-90 HIV Care Cascade: A Description of Statistical Methods Used in an Interim Analysis of the Intervention Communities in the SEARCH Study. 2017. University of California at Berkeley; Available at: Accessed 14 October 2019.
28. Robins JM. A new approach to causal inference in mortality studies with sustained exposure periods–application to control of the healthy worker survivor effect. Math Model. 1986;7:1393–1512.
29. Hernán MA, Lanoy E, Costagliola D, Robins JM. Comparison of dynamic treatment regimes via inverse probability weighting. Basic Clin Pharmacol Toxicol. 2006;98:237–242.
30. van der Laan MJ, Petersen ML. Causal effect models for realistic individualized treatment and intention to treat rules. Int J Biostat. 2007;3:Article 3.
31. Robins J, Orellana L, Rotnitzky A. Estimation and extrapolation of optimal treatment and testing strategies. Stat Med. 2008;27:4678–4721.
32. Petersen ML, Porter KE, Gruber S, Wang Y, van der Laan MJ. Diagnosing and responding to violations in the positivity assumption. Stat Methods Med Res. 2012;21:31–54.
33. Hayes R, Floyd S, Schaap A, et al.; HPTN 071 (PopART) Study Team. A universal testing and treatment intervention to improve HIV control: one-year results from intervention communities in Zambia in the HPTN 071 (PopART) cluster-randomised trial. PLoS Med. 2017;14:e1002292.
34. Floyd S, Ayles H, Schaap A, et al.; HPTN 071 (PopART) Study Team. Towards 90-90: Findings after two years of the HPTN 071 (PopART) cluster-randomized trial of a universal testing-and-treatment intervention in Zambia. PLoS One. 2018;13:e0197904.
35. Taubman SL, Robins JM, Mittleman MA, Hernán MA. Intervening on risk factors for coronary heart disease: an application of the parametric g-formula. Int J Epidemiol. 2009;38:1599–1611.
36. Snowden JM, Rose S, Mortimer KM. Implementation of G-computation on a simulated data set: demonstration of a causal inference technique. Am J Epidemiol. 2011;173:731–738.
37. Horvitz DG, Thompson DJ. A generalization of sampling without replacement from a finite universe. J Am Stat Assoc. 1952;47:663–685.
38. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70:41–55.
39. Hernán MA, Brumback B, Robins JM. Marginal structural models to estimate the causal effect of zidovudine on the survival of HIV-positive men. Epidemiology. 2000;11:561–570.
40. Gaolathe T, Wirth KE, Holme MP, et al. Botswana’s progress toward achieving the 2020 UNAIDS 90-90-90 antiretroviral therapy and virological suppression goals: a population-based survey. Lancet HIV. 2016;3:e221–230.
41. van der Laan MJ, Rubin DB. Targeted maximum likelihood learning. Int J Biostat. 2006;2:Article 11.
42. Schuler MS, Rose S. Targeted maximum likelihood estimation for causal inference in observational studies. Am J Epidemiol. 2017;185:65–73.
43. Luque-Fernandez MA, Schomaker M, Rachet B, Schnitzer ME. Targeted maximum likelihood estimation for a binary treatment: a tutorial. Stat Med. 2018;37:2530–2546.
44. Rose S, van der Laan MJ. van der Laan MJ, Rose S. Why TMLE? In: Targeted Learning: Causal Inference for Observational and Experimental Data. 2011.New York: Springer.
45. Sekhon JS, Gruber S, Porter KE, van der Laan MJ. van der Laan MJ, Rose S. Propensity-score-based estimators and C-TMLE. In: Targeted Learning: Causal Inference for Observational and Experimental Data. 2011:New York: Springer; 343–364.
46. Gruber S, van der Laan MJ. Targeted minimum loss based estimation of a causal effect on an outcome with known conditional bounds. Int J Biostat. 2012;8:21.
47. Balzer L, Ahern J, Galea S, van der Laan M. Estimating effects with rare outcomes and high dimensional covariates: knowledge is power. Epidemiol Methods. 2016;5:1–18.
48. van der Laan MJ, Polley EC, Hubbard AE. Super learner. Stat Appl Genet Mol Biol. 2007;6:Article25.
49. R Core Team. R: A Language and Environment for Statistical Computing. 2018. R Foundation for Statistical Computing; Available at: Accessed 14 October 2019.
50. Schwab J, Lendle S, Petersen M, van der Laan M. Ltmle: Longitudinal Targeted Maximum Likelihood Estimation. 2017. Available at:
51. Polley E, LeDell E, Kennedy C, van der Laan M. SuperLearner: Super Learner Prediction. 2018. Available at: Accessed July 15, 2019.
52. van der Laan MJ. Targeted estimation of nuisance parameters to obtain valid statistical inference. Int J Biostat. 2014;10:29–57.
53. Benkeser D, Carone M, Laan MJV, Gilbert PB. Doubly robust nonparametric inference on the average treatment effect. Biometrika. 2017;104:863–880.

Causal inference; HIV viral suppression; Machine learning; Missing data; Super Learner; TMLE

Supplemental Digital Content

Copyright © 2020 Wolters Kluwer Health, Inc. All rights reserved.