#### Introduction

From the beginning of the HIV epidemic, mathematical models were developed to better understand the progression of the disease ^{[1,2]}. Models were used for a variety of purposes: from biological models to assess the interaction between host and virus to epidemiological models to compare the costs and effects of prevention and treatment interventions, or to predict the dynamics of the epidemic. Mathematical models have proven to be a useful tool for informing decisions on alternative programs or interventions. However, the overwhelming amount of research published using mathematical models makes it difficult to assess the overarching take-home message of that research. One of the major challenges to distill the contributions of modeling to the HIV/AIDS treatment and prevention fields is related to the fact that modeling exercises are quite heterogeneous in terms of reliability. Unlike clinical trials, which have a clear set of characteristics and parameters that a credible study must meet, modeling publications are stunningly heterogeneous making it a challenging task to create a robust and widely accepted classification or ranking for quality.

The aim of our review is to present in a systematic way the enormous heterogeneity of HIV modeling exercises in terms of methods used and topics addressed and to analyze key aspects of the articles presented in terms of quality and robustness. The specific aims are first to determine whether there is any relationship between modeling method and the question addressed by the study, and second to review key methodological aspects of each study.

Several reviews of HIV/AIDS models have been conducted and focused on specific themes; however, none of them attempt to rank methodologies. For instance, in a review of drug resistance models among patients with HIV, Blower *et al.*^{[3]} provide an overview of the utility and applications of the models and a description of their main findings. In a similar review, Blower *et al.*^{[4]} discuss how models predict the evolution of drug resistance and use these models to predict the impact of antiretroviral therapy on drug resistance in Africa. In a different review, Blower *et al.*^{[5]} show that different models that assess the impact of antiretroviral and HIV vaccines have led to similar results but emphasize the need to contrast findings with empirical data. A review by Hellinger ^{[6]} provides a critical assessment on the methods used to estimate healthcare costs in models of HIV progression. The author finds that in some of the models key assumptions may be leading to imprecise cost estimates. In 1988, Foulkes ^{[7]} reviewed the methodological contributions made from the field of statistics to HIV/AIDS research. In his work, Foulkes present an overview of the most important advances of statistics and their application to specific topics within the HIV/AIDS epidemic. Some of the key contributions of statistical modeling include techniques to estimate transmission probabilities, time for AIDS to develop in patients with HIV, and how some studies have determined and analyzed the possible biases of their results. Foulkes concludes that communication among experts and statisticians is important to improve HIV research. Similarly, Wu's review compares four methods to estimate viral load decay and concludes that inappropriate model-fitting methods may conduce to different results ^{[8]}.

A recent issue of the *Current Opinion in HIV/AIDS* includes seven reviews articles of new developments in HIV mathematical models and address future research paths for epidemic modeling ^{[9–16]}. Different topics are discussed in this issue: methods for estimating incidence of HIV ^{[10]}, the importance of including data and results about sexual behavior from social research on mathematical transmission models ^{[11]}; transmission models of HIV in specific population as men who have sex with men and injectable drug users ^{[12]}; models of drug resistance transmission ^{[13]}; and models related to the potential impact of prevention interventions, specially biomedical interventions ^{[14–16]}. Findings refer the use of mathematical modeling to guide research ^{[15]}, achieve better knowledge of the epidemic and to provide informed healthcare policies ^{[10]}. The reviews describe that models designed to answer different research questions, usually require more complex structures than the models built to achieve only one specific aim ^{[16]}, they also remark the need to include current and better input data to feed the models ^{[11–13]}, and stress the importance of calibrating and validating the models ^{[15,16]}.

To determine whether there is an association between research question and modeling methods, we first identified the mathematical structure of each study and then grouped the articles by topic addressed. To evaluate key methodological aspects of the studies, we created a checklist of what we called ‘desirable’ features of a model and classified all the works reviewed accordingly. The checklist includes a justification of the modeling method used, a description of the model's structure, a description of the main parameters with a link to the source of information or assumptions stated, and whether any validation or sensitivity analysis was presented. To facilitate comparisons between the methods used, we conducted this systematic review of HIV progression models restricting our search to models of HIV progression in humans. To our knowledge, no systematic reviews of HIV progression models with similar objectives have been published in the literature.

#### Methods

##### Search strategy

We searched for articles published before February 2009 that described models of HIV progression in humans. Our study focuses on published, peer-reviewed literature. It is not meant to be a comprehensive review of HIV modeling projects, but rather a contribution to the discussion around the standards of practice in HIV modeling publications in scientific journals. Therefore, we have not included research published in the gray literature. HIV progression models were defined as studies in which a hypothetical cohort of patients with HIV was followed over the patients’ life span using biomarkers of HIV progression, such as CD4, viral load, or other indicators of progression for patients undergoing treatment, such as treatment failure or viral suppression. We included models describing the natural history of the disease, models evaluating prevention or treatment interventions, and models used to conduct cost-effectiveness analysis. We excluded models applied to HIV progression in animals, microbiological models or models of the dynamics of the epidemic that did not include disease progression.

The search was conducted in PubMed using the following terms: ‘disease progression’ [MeSH Terms] AND ‘models, Theoretical’ [MeSH Terms] AND ‘hiv infections’ [MeSH Terms] OR ‘Mathematics’ [MeSH Terms] AND ‘hiv infections’ [MeSH Terms] AND ‘disease progression’ [MeSH Terms] OR ‘models, Theoretical’ [MeSH Terms] AND ‘Antiviral Agents/therapeutic use’ [MeSH Terms] AND ‘hiv infections’ [MeSH Terms] OR ‘models, Theoretical’ [MeSH Terms] AND ‘Cost-benefit Analysis’ [MeSH Terms] AND ‘hiv infections’ [MeSH Terms]. We conducted an additional search to identify articles that would not be found under MeSH terms. (‘MeSH is the National Library of Medicine's controlled vocabulary thesaurus. It consists of sets of terms naming descriptors in a hierarchical structure that permits searching at various levels of specificity’ cited on: http://www.nlm.nih.gov/pubs/factsheets/mesh.html). The expression used is as follows: ‘disease progression AND models, theoretical AND HIV infections OR mathematics and HIV infections AND disease progression OR models, theoretical antiviral agents/therapeutic use and HIV infections OR models, theoretical and cost-benefit analysis AND hiv infections’.

We selected the articles to be included in the review in three phases. First, we excluded any article that obviously did not meet the inclusion criteria as based solely on the title. Second, we reviewed all the abstracts of the remaining articles and then excluded a second group of articles that did not meet our inclusion criteria. Finally, we read the remaining articles from the second phase to exclude any that did not meet the inclusion criteria. Table 1 presents the results of the two search strategies. Using MeSH terms, 1387 articles were retrieved, including 157 reviews. By reviewing the titles, we selected 257 articles, and 120 remained after we read their abstracts. We finally excluded 43 more during the third phase. In the second search (without MeSH terms), 2104 articles were retrieved from which 250 were reviews. We selected 235 titles, 89 abstracts, and kept 30 articles in the third phase. From the two search strategies, we selected and reviewed a total of 99 articles from which six were reviews.

##### Classification and analysis

We present our analysis in two ways. First, we classified the articles by modeling method and topic addressed. Secondly, we present a classification of the articles reviewed by a set of characteristics related to key methodological aspects of the modeling exercise. The objective of the first classification is to show the great heterogeneity of the literature included, which can also be useful to search for articles, and the objective of the second classification is to venture into the dimension of quality and robustness of the articles reviewed.

For the first part of the analysis, we identified and classified the articles in terms of the modeling method used to represent HIV progression and the main research question. The modeling method includes the mathematical structure of the model and the method used to handle time intervals across different stages of the disease. Models were classified according to three structure categories: state transition models, differential equations, and ‘other’ in which we grouped probabilistic models, quasiempirical models, and discrete simulation models.

State transition models, also called Markov models, predict how a patient moves from one disease stage to another during a defined time horizon ^{[17]}. Disease stages are defined as stages that are relevant in terms of HIV progression and the related costs. For instance, an acute stage is defined as a stage in which patients have not developed any opportunistic infection, which contrasts from the chronic stage. This distinction is also relevant from the economic perspective as different average costs are usually assumed for each stage. Transition probabilities determine how patients move through different disease stages. Some Markov models allow transition probabilities that are time-dependent ^{[18]}.

Models based on differential equations simulate how the size of a compartment changes with time ^{[19]}. Compartment in these models represent either specific groups of the population or stages of a disease; however, compared to state transition models, the transitions are continuous and defined by the structure of the system and the parameters used. Often models using differential equations represent the interaction between CD4 cells and HIV RNA at the cellular and molecular levels, and they are solved using microsimulation processes that simulate changes among patients with probabilistic or deterministic transitions.

The remaining models were classified as ‘other’. In this group, we found probabilistic models, quasiempirical, and static deterministic models. The first group estimates characteristics of the disease using equations with probabilistic distributions but do not define specific stages and transition between them as a Markov model. The quasiempirical models apply deterministic associations between parameters to simulate transitions between the compartments, which represent different kinds of populations (e.g., uninfected, infected). Static deterministic models use deterministic relationships to simulate disease progression.

In terms of the methods applied to handle time between intervals, we distinguished discrete and continuous models. In discrete models, time often relates to months or years between disease stages defined in terms of occurrence of relevant events or costs. Models classified as continuous are those in which movements between disease stages are deterministic and changes in parameters are often mathematically expressed as derivatives.

We classify the articles in eight categories according to their research question: studies that model the effectiveness of different antiretroviral drugs, models that simulate the HIV epidemic, articles modeling the impact of antiretroviral treatment, articles aimed at looking at different alternatives to optimize therapy, interventions to improve adherence and other healthcare policies, evaluation of prevention programs, timing of antiretroviral initiation, and effectiveness of prophylaxis alternatives.

For the second part of the analysis, we created a checklist of what we called ‘desirable’ features of a model. Most of the features pretend to evaluate how comprehensive the models are. Table 2 presents the items included in the checklist along with a description. Some of the items in the checklist are desired features of any modeling exercise, such as methods to handle uncertainty (sensitivity analysis) and comparability with other models (external validation) contributing to enhance both the quality and credibility of the models and credibility of the results. These two criteria are often evaluated in the cost-effectiveness literature ^{[20,21]}. Along with these items, the other pieces in the checklist were developed to set minimum elements that an HIV model should have for clarity and robustness. Each article was reviewed and classified by two authors and their conclusions contrasted. When the conclusions were different, the article was reviewed by another author, and results were discussed until a consensus was reached.

#### Results

Table 3 presents the distribution of articles by modeling method. Results show that discrete state transition models were the most common method used to model HIV progression. We found that 59 out of the 93 articles used state transition models. The second most common method used to model HIV progression was differential equations (26/93 articles). The remaining models were classified as ‘other’. Within this category we found three probabilistic models, four quasiempirical, and one static deterministic.

Table 4^{[22–110]} presents the classification by research question. The most frequent themes addressed in these articles include the impact of prevention interventions (19/93), models that compare the effectiveness of antiretroviral drugs (13/93), and those modeling the simulation of the epidemic in specific contexts, as particular countries or periods of time (12/93). A second group of articles evaluate the impact of antiretroviral treatment on different outcomes (11/93), such as disease progression, disease transmission, and economic outcomes; different alternatives to optimize antiretroviral (11/93); and the impact of other healthcare policies, such as interventions to improve treatment adherence (11/93). A third group of articles study the adequate time of antiretroviral initiation and the effectiveness of initiating and ending prophylaxis. Although the low sample size in each category crossing research question and modeling method (columns in Table 4) did not allow us to test for statistical differences ^{[111]}, in general, it does not seem to be an association between research topic and method used. (More than 50% of expected values are lower than 5 in the table crossing research question and modeling method. According to Conover ^{[111]}, such approximation is very poor and would not allow us to derive any statistical test.) However, we found that for three of the research questions (evaluation of antiretroviral drug effectiveness, timing of antiretroviral initiation, effectiveness or initiation or termination of prophylaxis) in all cases or for the vast majority Markov models are used and most often these are cost-effectiveness models. Other mathematical structures are also used for cost-effectiveness models but these are less frequent.

Figure 1^{[65,68,72,88]} presents the results of the features that we considered to be the ones that should be presented in HIV models of disease progression. We found that only 20% of the articles justify the type of modeling used; 67% clearly describe and explain the model; 62% use a diagram or other tool to describe the model; and 34% provide an appendix to explain the details of the model that were not included in the text. Most of the articles (78%) presented the characteristics of patients and disease state at baseline, and 77% listed the parameters required to feed the model with the source of information or assumptions stated.

Methods to conduct an external validation of the results of the models are presented in 13% of the articles reviewed. In these articles, validation methods include a comparison of the outcomes with results from cohorts or clinical trials or epidemiological data, such as mortality. Some articles validated their results by looking at specific outcomes, such as the percentage of patients reaching virological failure in a short period. Only two articles estimate the percentage error or variation of the model with respect to other clinical trials and cohort studies ^{[22,23]}.

We found that 85% of the articles complemented their results with some type of sensitivity analysis. Among those that developed a sensitivity analysis, 89% presented univariate, bivariate, and multivariate analysis. More complex methods were found in only 12% of the articles, which described the method of sensitivity analysis used: three articles used Latin hypercube ^{[24,25,50]}, six applied a probabilistic sensitivity analysis ^{[26–32]}, and one presented a threshold analysis ^{[33]}. Among those with sensitivity analysis, 78% show a table or a graph to support the results, and 57% describe the method used, although some of them present parameters and ranges of variation. Some models, specifically models using differential equations, conducted a stability analysis, which is equivalent to a sensitivity analysis ^{[34–37]}. None of the studies complied with the complete set of items in the checklist, but 6.5% cover at least 90% of them and 18% satisfy 80% of them.

#### Discussion

Mathematical modeling addresses a broad range of research questions that are challenging or at times impossible to address using empirical data. We reviewed 93 articles on HIV/AIDS disease progression whose findings and conclusions are based on mathematical modeling. The methods used to develop and parameterize these models are very heterogeneous. Among the selected articles, the majority use state transitions models followed by differential equation models. Models of HIV progression address a wide range of questions; however, it does not seem to be an association type of question explored and the modeling method used.

Moreover, our review shows that only 20% of the articles justify the type of model used; hence, how authors select a specific mathematical structure is unclear. Some models are not externally validated, and the robustness of the results is pondered only through sensitivity analyses. Articles did not always include a detailed description of the model, which made it difficult to evaluate the validity of the most important assumptions on which a model was based.

Understanding the structure of HIV/AIDS progression and the research questions addressed facilitate the comparison among studies and the assessments of their limitations. Modeling exercises can be extremely useful, but it is important to establish minimum criteria to determine the validity of the conclusions resulting from such exercises. Complete information about the parameters, including sources of information or model assumptions for each parameter, is necessary to assess the validity of the predictions. In addition, the lack of validation methods could result in poor estimations and, therefore, guide decisions based on incorrect information. In this review, we found that very few models are validated. A similar finding was reported in a review of Cambiano ^{[16]}, which describes than only one article out of 39 reviewed assessed the validity and internal consistency of the model.

Our review has some limitations. Although it was not within the scope of this article, a revision and assessment of the validity of the assumptions and parameters used to model disease progression would be useful. Given the large amount of HIV progression models in the literature, futures studies should compare the main assumptions behind modeling exercises that address the same research question. A discussion of this type, but only focused on prevention intervention models, is available in the review presented by Stover ^{[14]}. Similarly, future studies should analyze whether the models used in the articles are the most suitable to answer the research question addressed. Ideally, as new models are added to the vast HIV literature, a consensus on how to model key aspects of HIV progression should be reached among experts around the world**.** The checklist presented in this article is an example of the type of tools that could be used when reporting modeling exercises. This would be a helpful step toward standardizing the literature on HIV modeling and, improving their accessibility, comparability, and applicability for readers. The checklist could be seen as minimum requirements an HIV progression model should report description and justification of the mathematical structure used, patients characteristics at baseline, references or assumptions for all parameters included in the model, description and presentation for external validations, and sensitivity analyses. These minimum requirements can enhance understanding and comparability between models and more importantly this would force researchers to make explicit the most important assumptions and decisions made during the modeling process. Some of the criteria that are not HIV-specific could eventually be used in other non-HIV models.

The systematic review presented in this article is a more comprehensive assessment of HIV/AIDS progression models compared to other reviews published, and to our knowledge, it is the first to summarize the most common mathematical structures and research questions addressed and the first to provide a list of features that models should include. We hope that this attempt can inform deeper discussions in the future.

#### Acknowledgements

Contributors: Y.C.V., M.A.C., and S.B.A. were responsible for the design of the study, as well as the writing of the article. Y.C.V. did the search and data collection. Y.C.V., M.A.C., G.S. did the data analysis. All the authors contributed to the interpretation of the findings and approved the final manuscript.

Role of the funding source;

The funder CONACYT had no role in study design, data collection, analysis, interpretation or writing of the manuscript.

This work was supported by CONACYT, Consejo Nacional de Ciencia y Tecnología [Project key: SALUD-2005-C02-14520].

##### Conflicts of interest

Y.C.V., M.A.C, G.S., and S.B.A. have no conflict of interest.