Using Mechanistic Models to Simulate Comparative Effectiveness Trials of Therapy and to Estimate Long-term Outcomes in HIV Care

Roberts, Mark S. MD, MPP*†‡; Nucifora, Kimberly A. MS§; Braithwaite, R. Scott MD, MS§

Medical Care:
doi: 10.1097/MLR.0b013e3181e2b744
Comparative Effectiveness

Background: In HIV care, it is difficult to decide when to initiate therapy, which drugs to use for initial treatment, and which drugs to use if drug resistance develops. With hundreds of possible drug regimens available and variable patterns of drug resistance, randomized controlled trials cannot answer all HIV treatment decisions. Mechanistic models of HIV infection can be used to conduct virtual therapeutic trials with the goal of predicting outcomes, some of which are long-term and may not fall within the time frame of a typical therapeutic trial.

Methods: We used a previously developed and validated model of HIV infection to replicate 2 arms of an HIV initial treatment trial (ACTG A5142) and predict long-term outcomes. The model incorporated data about biologic processes involved in the development of drug resistance.

Results: The model reproduced the proportion that developed AIDS (0.04 and 0.05 for the efavirenz arm and lopinavir arms, respectively, vs. 0.04 and 0.06 for the trial), the development of virologic failure (0.27 and 0.33 for the Efavirenz arm and lopinavir arms, respectively, vs. 0.24 and 0.37 for the trial), and drug resistance. The hazard ratio for the time to treatment failure, a combination of resistance and other causes (0.96 for the model vs. 0.75 for the trial; 95% confidence interval, 0.57–0.98), and changes in CD4 cell count, were less accurate. The model estimated longer-term life expectancy, quality-adjusted life expectancy, and HIV-related deaths.

Conclusions: Mechanistic models of HIV infections have the potential to be useful in comparative effectiveness research.

Author Information

From the *Department of Health Policy and Management, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania; †Section of Decision Sciences and Clinical Systems Modeling, Division of General Internal Medicine, Department of Medicine, School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania; ‡Department of Industrial Engineering, School of Engineering, University of Pittsburgh, Pittsburgh, Pennsylvania; and §Section of Value and Comparative Effectiveness, Division of General Internal Medicine, Department of Medicine, School of Medicine, New York University, New York, NY.

This work was partially supported by U01 AA013566-06 and R01 AA017385-01 from the National Institutes of Health.

Reprints: Mark S. Roberts, MD, MPP, Department of Health Policy and Management, Graduate School of Public Health, University of Pittsburgh, 130 De Soto Street, Room A620, Pittsburgh, PA 15261. E-mail:

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's Web site (

Article Outline

HIV infection is a serious international public health problem that affects over 30 million people worldwide and is associated with high rates of morbidity and mortality.1 Antiretroviral therapy (ART) is the current standard of care for HIV/AIDS. The therapy is designed to drive the replication rate of the virus as low as possible, and this involves the use of a combination of 3 or more antiretroviral drugs that affect different components of HIV replication and assembly. Although studies have shown that ART can substantially improve both the quality and length of life of HIV-infected patients,2–4 many questions remain to be answered about its use. For example, what is the optimal time to initiate therapy? This remains a topic of debate, with some arguing that the benefits of early treatment are outweighed by the risks of developing drug resistance.2,3,5–9 Once a patient is receiving ART, what criteria should be used to define treatment failure? And once treatment failure is declared, what new combination of drugs should the patient receive?

Given the hundreds of possible regimens available and given the variations in timing and patterns of drug resistance, it is not feasible to use randomized controlled trials (RCTs) to explore these complex issues related to HIV care. To determine the best time to initiate therapy, a controlled trial would require that patients be characterized in terms of their CD4 count and viral load and then be randomly assigned to different timing strategies. Even with simplistic stratification of these and other patient characteristics, such a trial would require over 20 arms. Questions about drug choices after treatment failure are even more complex. There are currently 6 classes of antiretroviral drugs, with multiple drugs in each class. Because the development of resistance to 1 drug does not confer complete resistance to other drugs in the same class, there are many possible combinations available for use if the initial treatment regimen fails.

Recognizing that RCTs cannot answer every question concerning treatment,10,11 some have suggested that modeling can help provide answers.12 The use of mathematical models as a tool for medical decision making is not yet common, but several important examples provide support for their use in appropriate circumstances. Investigators working with the Scientific Registry of Transplant Recipients developed a series of simulation models that are used to predict the effects of changes in organ allocation policy before the changes are implemented.13,14 The Archimedes model, a complex mathematical model that represents diabetes, lipid metabolism, metabolic pathways, cardiovascular disorders, and many other interrelated aspects of human physiology and disease in a single model,15–17 has been able to reproduce the results of many clinical trials that have taken place and to accurately predict the results of some trials before the trials were completed.18,19 In the case of HIV treatment, many mathematical models have been designed,20,21 but not all of them fully represent the stochastic nature of disease progression.22 It was this shortfall in available HIV models that prompted us to build more mechanistically realistic representations of disease in our own mathematical models.

Our goal in this article is not to provide a specific answer to a specific question but, rather, to demonstrate that mechanistic models of HIV infection have the potential to be useful in comparative effectiveness research. To demonstrate this, we used our previously developed and validated model of HIV infection to replicate 2 arms of an HIV initial treatment trial (ACTG A5142), and predict long-term outcomes.

Back to Top | Article Outline


Description of the Model

We previously constructed a probabilistic, second-order Monte Carlo model of the progression and treatment of HIV infection. Details of the model have been published elsewhere,23–25 and are described in more detail in an online appendix, Supplemental Digital Content 1, available at: Figure 1 depicts the model's basic structure and shows the ways in which the various components of the model influence one another. Patients with HIV infection are primarily characterized by a CD4 cell count and a viral load. The CD4 count is the chief predictor of HIV-related death, and the viral load is a major predictor of the CD4 count trajectory. Viral load is determined not only by internal viral load “set points” unique to the individual patient but also by the presence or absence of effective ART.

At the onset of simulation, the model provides each simulated patient with a series of characteristics, including a CD4 count, a viral load, and other clinical characteristics that are important in the prediction of survival, such as comorbid diseases. The patients proceed through the model 1 at a time, with events occurring at daily intervals. Each day, depending on the patient's CD4 count and multiple other factors, the patient may or may not develop an AIDS-related complication, may or may not start receiving an ART regimen that is based on treatment algorithms, and may or may not experience the progression of disease as depicted by a decline in the CD4 count. If the patient is receiving ART, he or she may develop a resistance mutation, may develop toxic reactions to ART, and may or may not continue ART use, with compliance related to sociodemographic characteristics and the presence or absence of side effects. Each patient proceeds through the model until death occurs either from HIV-related complications or from non–HIV-related causes. All events are captured and recorded, providing a complete lifetime history of the patient's CD4 count, viral load, ART use, complications, and mortality burden.

A unique attribute of the simulation is the mechanism by which viral resistance to a drug is determined. The model simulates viral replication and genotype-specific mutation rates, parameters that are not typically represented in models of HIV.20 The combination of a replication rate and a mutation rate produces mutations to various classes of ART. In a simulated patient, only through the presence of selection pressure does a mutation emerge. This means that a simulated patient can develop resistance only if he or she is receiving a drug to which the randomly occurring mutation confers resistance. Modeled this way, the development of resistance is an “emergent property” of the interrelated components of the model and is not simply a statistical estimate from an existing data source. The model is therefore a hybrid that represents some relationships mechanistically (the development of antiretroviral resistance) and other relationships statistically (the rate of CD4 decline or mortally given CD4 count) as delineated in Figure 1.

The patient remains on the initial trial-based regimen until they either develop intolerance or the regimen fails. Failure is defined as a viral load above 2.7 log units for the initial regimen and 3.7 for subsequent regimens. Failure would most likely be caused by 1 or more resistance mutations or by ART nonadherence, which reduce the effective viral load decrement for that regimen. A new 3 drug regimen is then chosen. Regimens consist of either a boosted PI, Efavirenz or Nevirapine and 2 NRTIs. After the initial randomized treatment, the choice of and sequencing of regimens is not a fixed, prespecified, input; rather, when a patient fails a regimen, the simulation selects a new regimen with the lowest likelihood of resistance and intolerance.

Back to Top | Article Outline
Validation of the Model

The model has been previously validated in 3 ways that are briefly described in this article: details can be found in the referenced manuscripts.

The first validation was made by demonstrating that the model could closely reproduce the important clinical events in a dataset on which it was calibrated. The model primarily used data from the Collaborations in HIV Outcomes–US (CHORUS) cohort to calibrate the natural history of disease and the effect of treatment. In validation studies, the model was able to accurately reproduce the time to treatment failure by cycle of therapy and to reproduce the overall survival rate in the CHORUS cohort.23

The second validation was made by comparing the model's mortality estimates with mortality data that had not been used to calibrate the model. These mortality data were from the Antiretroviral Treatment Cohort Collaboration, which included 12,574 patients from 13 different cohorts.26 The model's predictions of 3-year mortality closely matched the Antiretroviral Treatment Cohort Collaboration data in virtually all subgroups of patient age, CD4 count, and viral load.23

Like the second validation, the third validation used data that had not been used to calibrate the model. In the third validation, the data were derived from 2 clinical trials that were specifically designed to investigate the relationship between adherence to treatment and resistance to drugs. One study, the Highly Active Antiretroviral Therapy Observational Medical Evaluation and Research study, had a cohort of 1191 individuals who were being placed on an ART regimen for the first time.27 The other study, the Research on Access to Care in the Homeless study, had a cohort of 148 individuals, most of whom had taken ART before the study began.28 Figure 2, adapted from one of our earlier studies,25 demonstrates the ability of the model to predict the accumulation of antiviral resistance mutations in both cohorts, based only on the initial CD4 counts, viral loads, and medication adherence data from patients in the 2 studies.

The third validation demonstrated that the model could replicate clinically observable effects from trials and could replicate intermediate biomarkers of disease progression. Because the effect of taking ART directly reduces viral replication in the model, the model explicitly represents the dangers of poor compliance with ART. If a simulated patient does not take his or her medication, the effect of ART on viral replication is reduced, and this in turns increases the replication rate and therefore the mutation rate. If a mutation to 1 drug in a particular ART regimen develops, that mutation will take hold in a particular viral population. The mutation will be based on selection pressure that favors a specific genotype and will be conferred by the presence of the drug that preferentially represses replication of a nonimmune viral strain. In this case, the model offers an appropriate representation of the relationship between adherence and resistance accumulation—ie, it shows that a decrease in adherence leads to a cascade of increases in replication, mutations, and resistance acquisition. This validation is highlighted here because the ability of the model to predict the relationship between adherence, resistance development, and medication failure is crucial to its use in the evaluation of various strategies of when to switch therapy and what to switch to.

Back to Top | Article Outline
Virtual Replication of a Trial

To illustrate the usefulness of a mechanistic model of HIV infection in comparative effectiveness research, we chose to replicate 2 arms of the ACTG A5142 trial. The trial was designed to address the use of ART regimens with and without nucleoside reverse transcription inhibitors (NRTIs) in patients who had not been treated with ART previously. There were 3 arms of treatment: the efavirenz arm, which received a regimen of efavirenz plus 2 NRTIs; the lopinavir-ritonavir arm, which received a regimen of lopinavir-ritonavir plus 2 NRTIs; and the NRTI-sparing arm, which received a regimen of lopinavir-ritonavir plus efavirenz.29 Our simulations were designed to replicate the efavirenz arm and lopinavir-ritonavir arm (ie, the 2 NRTI-containing arms) for the average trial patient, who was 38 years old, had a CD4 count of 191 cells per microliter, and had a viral load of 104.8 copies per milliliter. The simulations were also designed to predict long-term outcomes of overall life expectancy, quality-adjusted life expectancy, and the percentage of patients who died of HIV-related causes. In our simulations, we adjusted the model's mechanistic machinery to reflect the particularly favorable adherence rates, and immunologic and virologic responses to ART that were observed in the trial.

For the current report, we incorporated adherence levels observed in ACTG 5142 (62% of patients with 100% adherence). For the remainder of patients without perfect adherence (38%), we assumed a level of adherence typical in large observational cohort studies (62% of doses taken as directed).23 Sequencing of regimens is not a fixed, prespecified, input; rather, when a patient fails a regimen, the simulation selects a new regimen with the lowest likelihood of resistance and intolerance. This flexibility is an important strength of the simulation that allows it to better approximate clinical care decisions than a simulation that hardwires particular regimens and sequences after initial failure.

While a strength of the simulation is that it has the capability to consider different scenarios regarding rate of development of new drugs and mechanistic classes, for the current analyses, we made the simplifying assumption that no new drugs or classes would be developed.

Back to Top | Article Outline


In the trial and in the model, patients in the efavirenz arm were less likely than patients in the lopinavir-ritonavir arm to experience treatment failure. As shown in Table 1, the model closely reproduced the trial results in terms of the mean elevation in CD4 count, the proportion of patients who were suppressed (as indicated by viral load), the proportion who developed AIDS during the trial, the proportion who developed virologic failure, and the proportion who developed virologic failure with the presence of at least 1 relevant resistance mutation. The hazard ratio for time to treatment failure, a complex combination of virologic failure and other causes, was 0.75 in the trial, with a 95% confidence interval of 0.57 to 0.98, while the hazard ratio in the model was 0.96, within the confidence limits of the trial, but less accurate than the more biologically based outcomes. In addition to reproducing the trial results, the model was able to estimate long-term outcomes. As shown in Table 2, the model predicted that the efavirenz arm would have a slightly longer life expectancy, a slightly higher quality-adjusted life expectancy, and fewer HIV-related deaths than would the lopinavir-ritonavir arm.

Back to Top | Article Outline


Our results demonstrate that a model that mechanistically represents the acquisition of resistance to antiretroviral agents by taking into account the viral replication rate, the mutation rate, and the presence of a specific selection pressure can closely reproduce the biologic results of an RCT of HIV treatment. This is important to demonstrate because a variety of factors make it impossible to turn to RCTs as the sole method for completing comparative effectiveness studies concerning HIV treatment. For example, there is a limit to the number of arms that an RCT can include and a limit to the length of time that an RCT can continue. The number of arms that would be required by RCTs to truly inform the decisions made by clinicians faced with individual patients who have specific resistance patterns, CD4 counts, and viral loads is prohibitive.

As noted in Figure 1, the model contains both mechanistic and statistical relationships. The development of HIV resistance (which is measured in clinical practice) is an emergent property of the model based on the interaction of a replication rate and a mutation rate (which is not measured in clinical practice) interacting with the presence of specific selection pressure from the medications that the virtual patient is talking. However, many other components of the model are estimated statistical relationships. For example, the rate of CD4 decline is estimated as a function of viral load (VL), the presence or absence of therapy, and the presence of 1 or more resistance mutations. The risk of death is estimated from large observational cohorts given the CD4 count, age, gender, and many other clinical characteristics of the patient. Most of the data for the calibration of the statistical components of the model were derived from observational cohorts, such as the Veteran Aging Cohort Study (VACS) and CHORUS studies. The previous validations of this model were conducted in other observational cohorts: this is the first attempt of this model to replicate the results of an RCT. It is interesting that the parameters that rely most on the mechanistic component of the model (the development of resistance and the occurrence of virologic failure) are very close to the values found in the trial, but other outcomes, such as the actual changes in CD4 counts, which are derived from statistical relationships estimated from observational cohorts, are less well reproduced. This supports, but does not prove, the notion that modeling the processes that create the data we observe in the world, provides a more flexible system than modeling the relationships in the observed data themselves.

We believe that mechanistic models of HIV infection have the potential to be useful in comparative effectiveness research. While RCTs may be the best choice to answer some types of questions (Is drug A superior to drug B? Is regimen A superior to regimen B?), models may be the best choice to answer questions about optimization of treatment (What is the optimal CD4 count and viral load at which to initiate therapy? What constellation of patient characteristics can mitigate the effects of treatment?). RCTs can answer the superiority question for the “average” characteristics of the intervention and control cohorts, but RCTs are not always able to answer the superiority question for clinically encountered subgroups with different characteristic. Models have the advantage of being able to deal with the large number of variations in patients, the large and growing number of potential drug combinations used in ART regimens, and the rapidly changing landscape of resistance patterns. Moreover, whereas RCTs can answer questions about short-term outcomes, models can be used to predict long-term outcomes if the models are able to simulate important clinical events that govern transitions between states of health. In the model described here, antiretroviral resistance and ART failure are treated as emergent properties of the interactions of different components of the model. This allows the model to make estimates of outcomes under conditions and treatment choice combinations that have not been used in the model's development, and it allows for the virtual conduct of clinical trials that have far too many arms to conduct in actuality.

An example of a benefit of the model's mechanistic representation of resistance development is its ability to accurately reflect the differential effect of various patterns of lack of adherence. For example, a patient who stops taking all his medications raises viral replication and mutation, but because there is no selection pressure, no resistant strain emerges, the wild type virus simply replicates more quickly. However, in a person who discontinues 1 or 2 medications in a multidrug regimen, there is both an increase in the replication and mutation rates and the presence of selection pressure from the medication that the patients is still taking.

We believe that mechanistic models will eventually be used in comparative effectiveness research. Recognition of the limits and advantages of specific types of models is the first step in elucidating the types of roles that models will play.

Back to Top | Article Outline


1. WHO. Global Summary of the AIDS epidemic, December 2007. 2009. Available at: Accessed July 28, 2009.
2. Ho DD. Time to hit HIV, early and hard. N Engl J Med. 1995;333:450–451.
3. Sterling TR, Chaisson RE, Keruly J, et al. Improved outcomes with earlier initiation of highly active antiretroviral therapy among human immunodeficiency virus-infected patients who achieve durable virologic suppression: longer follow-up of an observational cohort study. J Infect Dis. 2003;188:1659–1665.
4. Sterling TR, Chaisson RE, Moore RD. Initiation of highly active antiretroviral therapy at CD4+ T lymphocyte counts of >350 cells/mm3: disease progression, treatment durability, and drug toxicity. Clin Infect Dis. 2003;36:812–815.
5. Cohen CJ, Boyle BA. Antiretroviral therapy: the “when to start” debates. Clin Infect Dis. 2004;39:1705–1708.
6. Phillips AN, Lepri AC, Lampe F, et al. When should antiretroviral therapy be started for HIV infection? Interpreting the evidence from observational studies. AIDS. 2003;17:1863–1869.
7. Lane HC, Neaton JD. When to start therapy for HIV infection: a swinging pendulum in search of data. Ann Intern Med. 2003;138:680–681.
8. Ledergerber B, Lundgren JD, Walker AS, et al. Predictors of trend in CD4-positive T-cell count and mortality among HIV-1-infected individuals with virological failure to all three antiretroviral-drug classes. Lancet. 2004;364:51–62.
9. Hertogs K, Bloor S, Kemp SD, et al. Phenotypic and genotypic analysis of clinical HIV-1 isolates reveals extensive protease inhibitor cross-resistance: a survey of over 6000 samples. AIDS. 2000;14:1203–1210.
10. Volpp KG, Das A. Comparative effectiveness—thinking beyond medication A versus medication B. N Engl J Med. 2009;361:331–333.
11. Luce BR, Kramer JM, Goodman SN, et al. Rethinking randomized clinical trials for comparative effectiveness research: the need for transformational change. Ann Intern Med. 2009;151:206–209.
12. Basu A, Meltzer D. Modeling comparative effectiveness and the value of research. Ann Intern Med. 2009;151:210–211.
13. Schaubel DE, Dykstra DM, Murray S, et al. Analytical approaches for transplant research, 2004. Am J Transplant. 2005;5:950–957.
14. Wolfe RA, Schaubel DE, Webb RL, et al. Analytical approaches for transplant research. Am J Transplant. 2004;4(suppl 9):106–113.
15. Eddy DM, Schlessinger L. Validation of the Archimedes diabetes model. Diabetes Care. 2003;26:3102–3110.
16. Eddy DM, Schlessinger L. Archimedes: a trial-validated model of diabetes. Diabetes Care. 2003;26:3093–3101.
17. Schlessinger L, Eddy DM. Archimedes: a new model for simulating health care systems—the mathematical formulation. J Biomed Inform. 2002;35:37–50.
18. Stern M, Williams K, Eddy D, et al. Validation of prediction of diabetes by the Archimedes model and comparison with other predicting models. Diabetes Care. 2008;31:1670–1671.
19. Eddy DM, Schlessinger L, Kahn R. Clinical outcomes and cost-effectiveness of strategies for managing people at high risk for diabetes. Ann Intern Med. 2005;143:251–264.
20. Freedberg KA, Losina E, Weinstein MC, et al. The cost effectiveness of combination antiretroviral therapy for HIV disease. N Engl J Med. 2001;344:824–831.
21. Richter A, Hauber B, Simpson K, et al. A Monte Carlo simulation for modelling outcomes of AIDS treatment regimens. Pharmacoeconomics. 2002;20:215–224.
22. Shechter SM, Bailey MD, Schaefer AJ, et al. The optimal time to initiate HIV therapy under ordered health states. Oper Res. 2008;56:20–33.
23. Braithwaite RS, Justice AC, Chang CC, et al. Estimating the proportion of patients infected with HIV who will die of comorbid diseases. Am J Med. 2005;118:890–898.
24. Braithwaite RS, Kozal MJ, Chang CC, et al. Adherence, virological and immunological outcomes for HIV-infected veterans starting combination antiretroviral therapies. AIDS. 2007;21:1579–1589.
25. Braithwaite RS, Shechter S, Roberts MS, et al. Explaining variability in the relationship between antiretroviral adherence and HIV mutation accumulation. J Antimicrob Chemother. 2006;58:1036–1043.
26. Egger M, May M, Chene G, et al. Prognosis of HIV-1-infected patients starting highly active antiretroviral therapy: a collaborative analysis of prospective studies. Lancet. 2002;360:119–129.
27. Bangsberg DR, Charlebois ED, Grant RM, et al. High levels of adherence do not prevent accumulation of HIV drug resistance mutations. AIDS. 2003;17:1925–1932.
28. Harrigan PR, Hogg RS, Dong WW, et al. Predictors of HIV drug-resistance mutations in a large antiretroviral-naive cohort initiating triple antiretroviral therapy. J Infect Dis. 2005;191:339–347.
29. Riddler SA, Haubrich R, DiRienzo AG, et al. Class-sparing regimens for initial treatment of HIV-1 infection. N Engl J Med. 2008;358:2095–2106.

Cited By:

This article has been cited 1 time(s).

Medical Care
Comparative Effectiveness Research Methods: Symposium Overview and Summary
Lohr, KN
Medical Care, 48(6): S3-S6.
PDF (160) | CrossRef
Back to Top | Article Outline

mechanistic model; HIV infection; virtual trials

Supplemental Digital Content

Back to Top | Article Outline
© 2010 Lippincott Williams & Wilkins, Inc.