International guidelines for interventions and medical care promote health by enabling populations to benefit from the best scientific evidence and accumulated experience of the global community. However, setting guidelines is difficult, especially when the best clinical practice has to be balanced with practical constraints and concern for overall population health outcomes. The 2013 consolidated guidelines for the use of antiretrovirals to treat HIV, promulgated by the World Health Organization (WHO), replace several distinct guideline documents about the provision of antiretrovirals in different circumstances . In providing such guidance, the consequences of decisions must simultaneously be considered in many dimensions (morbidity, mortality, new infections, resistance, resource needs) and over a range of timescales, whilst also weighing the strength of various forms of evidence and accounting for the attendant uncertainties. These questions lend themselves to mathematical modeling and economic evaluation as a means to synthesize data in a transparent and precise way and to anticipate the implications of competing approaches for population health .
For the last several years, the HIV Modelling Consortium (www.hivmodelling.org) has worked with agencies including the WHO, Joint United Nations Programme on HIV/AIDS (UNAIDS), National Institutes of Health, the World Bank and the Bill & Melinda Gates Foundation, and with almost 100 different mathematical modelers, to provide robust analysis informing contemporary policy questions. In advance of the antiretroviral guidelines revision, the WHO invited the HIV Modelling Consortium to prepare analyses for the panels tasked with developing the 2013 consolidated antiretroviral guidelines, so that the insights about the health and cost consequences gained from the modelled analyses could inform the guideline development process. Crucially, the modeling was not done post hoc to justify decisions that had already been made, but rather to provide useful analysis for the questions that remained open at that time.
The first step of developing modeling analyses for the guidelines panels involved collaborative discussions between the WHO and HIV Modelling Consortium to define specific policy questions which modeling could usefully contribute. We focused on two broad issues: who should be initiated on antiretroviral therapy (ART), and how should patients on ART be monitored. We decided to approach these questions within the framework of cost-effectiveness analysis (CEA), as this explicitly compares the health benefits produced by a particular policy with the resources required to achieve those benefits and allows for a set of policies to be chosen that will generate the greatest health gains from available resources. The final step was to agree on how to model these processes. Given that our previous work has demonstrated a large influence of model choice on results [3,4], it was decided to base the analysis not on a single model but instead on a diverse set of existing epidemiological models. We believed that consensus findings across the models would increase confidence in this evidence to support policy, and that the discrepancies between the model analyses would signal important uncertainties that might have gone undetected in a single model.
Summary results have been presented elsewhere by Eaton and Menzies et al.  and Keebler and Revill et al. . In brief, we estimated that, in programmes with high ART coverage, expanding treatment eligibility to HIV-positive adults with CD4+ cell counts up to 500 cells/μl or even higher appears to be cost-effective, but this will need to be accompanied by large expansions in HIV testing and enrolment in care in order to maximize the potential reductions in HIV incidence . Moreover, although more sophisticated forms of patient monitoring—such as frequent viral load measurement—are expected to confer benefits, expanding ART coverage would be expected to generate greater population health benefits from limited resources .
However, these summary findings obscure the many additional considerations raised during our investigations. Therefore, we have developed this special supplement of AIDS to document our further analyses, explain the whole process we have undertaken, and describe its limitations. This introductory editorial also reflects on the role of modeling in guiding decision making, providing a critique that should inform how models are used for similar applications in the future.
First, Gopalappa et al. consider the impact and cost of a switch to ‘Option B+’ for the prevention of mother-to-child transmission, whereby all HIV-positive pregnant women are offered lifelong ART irrespective of their CD4+ cell count . This is amongst the most eye-catching recommendations in the new guidelines, and this analysis suggests that, in many settings, Option B+ would be a cost-effective use of resources, as compared with other options for prevention of mother-to-child transmission. However, when there exist treatment gaps for individuals in need of antiretrovirals for their own health, other analyses suggest that greater health gains can be realized by concentrating ART on those in greatest need.
Cambiano et al. address another consideration for the sustainable long-term provision of antiretrovirals: drug resistance . Their model includes the development and accumulation of resistance within HIV-infected persons taking ART as well as transmission of (potentially drug-resistant) HIV over a network of sexual partnerships, uniquely allowing the model to examine the population-level implications of within-host processes. According to that analysis, the number of HIV infections that are resistant to first-line treatment will increase with earlier ART initiation. But, since the overall number of HIV infections would be reduced, the number of people with nonsuppressed viral load carrying resistant virus is not expected to be much higher. This is a more useful indicator of the need for expensive second-line therapies and clinical complications than the prevalence of resistance mutations alone. Therefore, the large potential for growth in resistance does not appear to represent a strong argument against a policy of expanding treatment eligibility.
Sometimes health interventions are considered for a specific disease in isolation, ignoring the interactions between diseases and interventions against them. HIV and ART have much interaction with tuberculosis (TB). Pretorius and Menzies et al. investigate the impact of expansions of ART on the burden of TB. Similar to our main analysis, they synthesize the results of three independent mathematical models of HIV and TB transmission in South Africa and compare the results . Due to the immune reconstitution of patients on ART, which ameliorates the excess risk of TB disease, the authors find that expanding treatment eligibility and coverage could decrease TB incidence and mortality. All three models estimate that one TB case could be averted for each 10–13 additional person-years of ART provided in the population.
The natural history of hepatitis B and hepatitis C is also intertwined with both HIV and ART. Martin et al. consider the benefits of providing ART to coinfected persons earlier than to persons with HIV only . The authors find that, because tenofovir-based antiretroviral regimens impede hepatitis B associated liver disease progression, additional benefits may accrue from earlier treatment initiation for hepatitis B and HIV coinfected patients. However, doing so on a large scale would require screening for hepatitis B, the cost of which is likely to be prohibitive. In contrast, early initiation of hepatitis C coinfected patients was not expected to generate benefits beyond those for other HIV-infected patients.
Klein, Bershteyn and Eckhoff delve more deeply into the impact of some key modeling assumptions . They find that improving the retention of patients in care can dramatically affect the cost-effectiveness ratios for expanded ART programmes. These results highlight that the most important determinant for achieving the full benefits of ART might not be a policy change but rather the supporting strategies undertaken by programmes to implement these policies. The study also highlights the model parameters that are simultaneously among the most influential and the most uncertain, and which should therefore be of particular priority to better measure.
An important question for our overall analysis was whether optimal ART policies differ across epidemiological settings or in settings with different current levels of ART scale-up. This motivated our focus on four country case studies of generalized (South Africa, Zambia) and concentrated (India, Vietnam) epidemics, with independent mathematical models in each setting, and our decision to consider a ‘status quo’ baseline scenario whereby patterns of ART uptake, as well as sexual behaviours and the coverage of other interventions, would remain at their current levels. However, a more nuanced approach can be revealing, and in the setting of southern India, a primarily sex work driven epidemic, Mishra et al. examine how underlying patterns of risk, and changes in those patterns of risk in the future, are influential determinants of the impact and cost (and cost-effectiveness) of the various ART-based interventions considered . Reassuringly, however, the authors conclude that the central finding – that high ART coverage among commercial sex workers should be prioritized – is robust to these alternative assumptions.
Although we tackled the two broad policy questions – who to treat, and how to monitor those receiving treatment – separately, to the extent that expenditures for the corresponding interventions compete for the same resources and the impact of treatment interventions depend on the monitoring patients receive, they should be considered together. Braithwaite et al. do just that  and find that as resources permit, the priority should be to expand ART access and eligibility before transitioning to monitoring strategies that are currently more expensive. In settings with higher ART coverage, viral load monitoring is perhaps more likely to be favoured, as there are fewer opportunity costs associated with large numbers in need of treatment but not currently receiving antiretrovirals. Also, the benefits of viral monitoring are likely to be greater, as HIV transmission will be more strongly linked to levels of viral suppression and perhaps also the predictive value of tracking CD4+ cell counts would diminish as patients start ART at high CD4 levels. This finding is dynamic, and over time the cost of viral load testing will likely fall in the coming years while ART coverage should increase: this analysis implies that countries may pass a threshold when viral load monitoring becomes a greater priority.
A final paper, from the WHO staff , considers the use of modeling from the point of view of the guideline setting body and describes where models could help the process, as well as how modeling could more constructively contribute to policy formulation in the future. A key point raised is that factors beyond the cost-effectiveness recommendations from models will be considered when taking policy decisions. However, in our view, it should be incumbent upon any decision-maker to explicitly justify any decision counter to that which objective modelling and economic evaluation studies indicate would maximize population health.
The experience of undertaking these modeled analyses provides lessons relevant to future efforts of this nature. Modeling analyses, such as those that have been undertaken in this project, are being increasingly embraced in decision-making . Although this is a trend we support, we would emphasize the need to continue to critically evaluate modeling analyses. As modellers, we accept that analyses should be viewed with a healthy scepticism: capturing how a complex system works is a challenge, especially when empirical data on modelled processes (such as the current performance of the HIV care system, in this instance) are not always available. Policy evaluation requires that the implications of competing policies be projected many years into the future, yet this requires assumptions to be made about behaviour, health system performance, and resource availability, which will only be truly known after the fact, and as such, the level of uncertainty about modeled findings increases as the time horizon is lengthened. However, although this uncertainty cannot be ignored, it is properly understood as an inherent feature of the policy-making context rather than a consequence of undertaking modelled analyses per se. In fact, it is an important feature of these analyses that they can expresses the limits of current knowledge and can therefore motivate new research to reduce those uncertainties with greatest influence over the policy decision . Similarly, we should be clear that models will always be imperfect approximations of the processes they attempt to replicate, even the best models will never be ‘finished’ and all models should be subject to revision and improvement in the future (or be discarded) as new data emerge.
Relying on multiple models to produce analyses is novel in this field. It allows multiple methods and approaches to contribute to the results, but can also mean that some analyses are conducted in preexisting models that were ‘re-purposed’, which somewhat contradicts our general insistence that models be designed specifically for a particular question [16,17]. As this approach becomes more common, it might be implemented differently, through commissioning ‘purpose-built’ models in advance of guideline development.
We used the framework of a CEA in this work and used country data to construct ‘case studies’ for how a particular country health sector could allocate its HIV budget. In reality, however, an analysis in a country would be constructed in a somewhat different way. First, a wide range of feasible policy options may be considered, rather than, as we did, only considering a small set of conventional scenarios designed to stretch across a ‘universe of possibilities’. Second, countries should be in a much better position than the authors to know what can be afforded and what maximum cost would be acceptable for a certain increase in health. To some extent, we stepped away from this question by illustrating ‘best interventions’ within some constraint, but at times, we have resorted to comparisons with international benchmarks  that may have a weak relationship with the ability of individual countries to pay for healthcare interventions . Third, there should also be a careful examination of how costs fall on patients, the implications of the policy for equity and practical issues such as procurement. Our modeling effort here has focused on ART, but issues such as the impact of HIV prevention programmes and how their influences modulate the impact of ART have not received enough attention . Finally, countries may want to consider reallocation of funding across disease areas and broader budget categories, which these studies did not address. What we have provided can be considered to scope the questions raised and to provide an indication of how the overall balance of costs and benefits might be realized, but it is only a starting point for country policy-makers and international partners to view these questions through their own lens, considering their own constraints.
If guidelines in HIV and other areas increasingly aim to provide pragmatic advice to programmes, in addition to defining optimal clinical practice, then modelling and economic analyses will increasingly be used in the guideline-setting process. This would be an important and productive shift, but it will also lead to models being confronted with difficult questions that they may struggle to answer, whilst decision-makers will be confronted with new models that they may struggle to trust. Dialogue between those parties will be crucial, however. In an era in which so many efficacious and low-cost interventions are not used fully, and when more effective treatments are continually developed, programmes need guidance on how to allocate their precious resources more than ever.
We thank Katherine Bilsborrow and Britta Jewell for their assistance in coordinating this project, especially in its early stages. This work was funded by the Bill & Melinda Gates Foundation through the HIV Modelling Consortium and by the WHO. Members of the Bill and Melinda Gates Foundation had no input to the analysis or description of results. Staff from the WHO participated in the framing of the research questions but played no role in the selection or development of epidemiological models, conduct of the analyses or interpretation of the results.
Conflicts of interest
T.B.H. is a member of the WHO Programmatic Guidelines committee and a technical consultant to the Bill & Melinda Gates Foundation.
2. Garnett GP, Cousens S, Hallett TB, Steketee R, Walker N. Mathematical models in the evaluation of health programmes
3. Eaton JW, Johnson LF, Salomon JA, Barnighausen T, Bendavid E, Bershteyn A, et al. HIV treatment as prevention: systematic comparison of mathematical models of the potential impact of antiretroviral therapy on HIV incidence in South Africa
. PLoS Med
4. The HIV Modelling Consortium Treatment as Prevention Editorial Writing GroupHIV treatment as prevention: models, data, and questions: towards evidence-based decision-making
. PLoS Med
5. Eaton JW, Menzies NA, Stover J, Cambiano V, Chindelevitch L, Cori A, et al. Health benefits, costs, and cost-effectiveness of earlier eligibility for adult antiretroviral therapy and expanded treatment coverage: a combined analysis of 12 mathematical models
. Lancet Glob Health
6. Keebler D, Revill P, Braithwaite S, Phillips A, Blaser N, Bórquez A, et al. Cost-effectiveness of different strategies to monitor adults on antiretroviral treatment: a combined analysis of three mathematical models
. Lancet Glob Health
7. Gopalappa C, Stover J, Shaffer N, Mahy M. The costs and benefits of Option B+ for the prevention of mother-to-child transmission of HIV
8. Cambiano V, Bertagnolio S, Jordan MR, Pillay D, Perriëns JH, Venter F, et al. Predicted levels of HIV drug resistance in South Africa: potential impact of expanding diagnosis, retention, and eligibility criteria for antiretroviral therapy initiation
9. Pretorius C, Menzies NA, Chindelevitch L, Cohen T, Cori A, Eaton JW, et al. The potential effects of changing HIV treatment policy on tuberculosis outcomes in South Africa: results from three tuberculosis-HIV transmission models
10. Martin NK, Devine A, Eaton JW, Miners A, Hallett TB, Foster GR, et al. Modeling the impact of early antiretroviral therapy for adults coinfected with HIV and hepatitis B or C in South Africa
11. Klein DJ, Bershteyn A, Eckhoff PA. Dropout and re-enrollment: implications for epidemiological projections of treatment programs
12. Mishra S, Mountain E, Pickles M, Vickerman P, Shastri S, Gilks C, et al. Exploring the population-level impact of antiretroviral treatment: the influence of baseline intervention context
13. Braithwaite RS, Nucifora KA, Toohey C, Kessler J, Uhler LM, Mentor SM, et al. How do different eligibility guidelines for antiretroviral therapy affect the cost–effectiveness of routine viral load testing in sub-Saharan Africa?
14. Easterbrook PH, Doherty MC, Perriens JH, Barcarolo JL, Hirnschall GO. The role of mathematical modelling in the development of recommendations in the 2013 WHO consolidated antiretroviral therapy guidelines
15. Hallett TB, Eaton JW. A side-door into care cascade for HIV-infected patients?
. J Acquir Immune Defic Syndr
2013; 63 (Suppl 2):S228–S232.
16. Keeling MJ, Rohani P. Modeling infectious diseases in humans and animals
. Princeton, NJ:Princeton University Press; 2007.
17. Delva W, Wilson DP, Abu-Raddad L, Gorgens M, Wilson D, Hallett TB, et al. HIV treatment as prevention: principles of good HIV epidemiology modelling for public health decision-making in all modes of prevention and evaluation
. PLoS Med
18. Edejer TT-T, Baltussen R, Adam T, Hutubessy R, Acharya A, Evans D, Murray CJL. Making Choices in Health: WHO guide to costeffectiveness analysis (p. 318)
. 2003; Geneva:World Health Organisation, 1–329.
19. Revill P, Sculpher M. Cost effectiveness of interventions to tackle noncommunicable diseases