Secondary Logo

Journal Logo

Methods

Visualizing Results From Infection Transmission Models: A Case Against “Confidence Intervals”

Lofgren, Eric T.

Author Information
doi: 10.1097/EDE.0b013e318258369b

Increasing computational power has made it possible to use stochastic models to study the transmission of infectious diseases. These models, which include stochastic compartmental models, network models, and agent-based models, have appealing properties when compared with their deterministic analogs. A stochastic approach is advisable for small populations, as it builds results from the aggregation of individual-level data rather than population averages (which treat human beings as continuous rather than discrete units). Additionally, the use of stochastic models allows the examination of model outcomes that are unlikely but not impossible, outcomes that are ignored by their deterministic cousins.

Most stochastic models lack an approachable analytic solution, and rely instead on simulation of the model thousands of times. This leaves the researcher with the challenge of presenting tremendous amounts of data. One approach adapts the familiar technique of presenting the mean or median and 95% interval of the model results. Although this presentation may be comfortingly familiar to most epidemiologists, it is problematic.

The confidence intervals produced by most statistical models have an agreed upon meaning (albeit one occasionally subject to misinterpretation) and well-established methods for their construction. In contrast, transmission models rarely have analytic methods for producing confidence intervals. Most intervals produced by transmission models are not true confidence intervals but rather prediction intervals. Instead of being a summarization of the value of a true estimate based on repeated sampling, the intervals are a summarization of a large number of samples and the variability of those results. There is no clear intuitive ordering of model results into percentiles, and different approaches may produce different interpretations of the same model. Furthermore, the results of many transmission models are multimodal, a property that is impossible to communicate with methods designed for unimodal distributions. Finally, outlying results are often of interest in a transmission model and are hidden when using an interval-based approach.

Fortunately, common data-analysis software packages have graphical capabilities that ameliorate these problems, allowing the visualization of both “common” outcomes and outlying data by showing all results, rather than reductionist summaries. The limitations of summarizing results with an interval approach are explored here using an example network model. Two alternative methods are presented.

METHODS

As an example, an outbreak was modeled across a social contact network, Goodreau's Faux Mesa High School. This network was simulated using an exponential random graph model fit to a real high school in the rural western United States using data from the National Longitudinal Study of Adolescent Health.1,2 This analysis uses the largest connected component of that network, which contains 120 students linked by 188 mutual connections (Fig. 1). The data for this network are available as part of the “statnet” R package.3

Figure 1
Figure 1:
Largest connected component of Goodreau's Faux Mesa High School social network. Each node (gray circle) represents a single individual, and edges (black lines) represent mutual connections.1–2

Disease transmission was simulated 1000 times with a percolation model, using a single starting infected individual. The probability of transmission was set at 0.5053 to yield a basic reproductive number (R0) of ∼1.80. This value was chosen to distinguish the stochastic and deterministic forms of the same process. In a deterministic setting, this disease is guaranteed to cause an epidemic. However, in a stochastic setting, the disease may become extinct before an epidemic begins.

The results of this model are presented using 4 approaches. In approach 1, the median, 2.5th and 97.5th percentile of the total epidemic size were determined. The multiple realizations that resulted in those values were plotted over time. In approach 2, the median, 2.5th and 97.5th percentile of currently infected persons were plotted against time. In contrast to approach 1, these intervals do not represent a single realization of the model, but rather a point-wise summarization of the entire simulation at each time point. In approach 3, all realizations of the model are simultaneously plotted as semi-transparent lines (to prevent overlapping trajectories from obscuring each other). Finally, in approach 4, all realizations are plotted as points. The resulting scatterplot was smoothed using a kernel density estimator to highlight areas of dense results.

Simulations were performed with EpiFire 2.33 (T Hladish, unpublished paper, 2011), and results were analyzed in R 2.13.4 The results from our implementation, as well as the source code to produce the figures that follow, are available as electronic supplements (eAppendix 1 and eAppendix 2, respectively [https://github.com/elofgren/Visualizing-Results-from-Infection-Transmission-Models]).

RESULTS

The epidemic simulation produced bimodal results, with 21% of realizations becoming extinct after the first infection, while the largest epidemic infected 87 people. The median epidemic size was 16 people, with any common measure of central tendency providing a poor summary of the results (Fig. 2).

Figure 2
Figure 2:
Final epidemic size of 1000 simulations of disease spread over a social network of 120 individuals with an R0 of 1.80. The most common epidemic size was 1 case (21%) with a median epidemic size of 16 cases.

For approach 1, the median epidemic size was 16 (with 2.5th and 97.5th percentiles of 0 and 66, respectively). Multiple realizations produced these values. Rather than make an arbitrary selection, 4 such realizations of the model for the median and both intervals are shown in Figure 3A. The graphical results show considerable heterogeneity for the median and upper interval. As their ordering is based on the total epidemic size, several median results appear above the upper interval results toward the beginning of the simulation.

For approach 2, the median and 2.5th and 97.5th percentiles of infected persons are shown at each time point in Figure 3B. The median- and upper-interval values do not represent a single particular realization. In many cases, the median and upper intervals take a noninteger value, which cannot represent an actual model outcome. In contrast, the lower interval represents a preponderance of model realizations. In approach 2, intervals are extremely sensitive to whether epidemics that have died out are regarded as being zero infected people or as missing at time points past their extinction (not shown).

Approach 3 depicts all 1000 model realizations (Fig. 3C). The multimodal nature of the results is shown in the areas of dense dark lines near n = 0 infections, as well as the modest epidemics ranging from 5 to 10 infections occurring from t = 2 to 10. Approach 4 similarly shows all 1000 model realizations as a smoothed density (Fig. 3D). Compared with approach 3, the extreme results—although still apparent—are visually deemphasized.

Figure 3
Figure 3:
Visual results of the simulated epidemic using approach 1 (A), approach 2 (B), approach 3 (C), and approach 4 (D). A, depicts the median and 95% confidence intervals based on the final size of the epidemic, with 4 choices for each as multiple realizations have the same final epidemic size. B, depicts the median and 95% confidence intervals based on the number of infected individuals at each time point. Epidemics that have an experienced stochastic extinction are treated as missing, the default assumption in many statistics and graphing packages. C, shows the trajectories of all model realizations, with dark areas indicating more than one overlapping trajectory being simultaneously plotted. D, plots all model realizations as individual points and the resulting scatterplot smoothed using kernel density estimation. Darker areas represent correspondingly denser concentrations of simulation results.

DISCUSSION

The importance of stochastic epidemic models will increase as computational resources become more available and simulations are used to address increasingly complex questions of infectious-disease transmission. Simultaneously, the ways that epidemiologists visualize these results will become more important, as these models elude compact numerical summarization.

Reductionist methods involving a median or mean and an upper and lower bound can be deceptive. Although simple to digest, these plots hide potentially important information. In addition to discarding data, the composition of these plots can be somewhat arbitrary. The “median and 95% confidence interval” may refer to the final epidemic size, to each time point in the simulation, or to the midpoint of the simulation (among others). Additionally, each realization may be treated as a distinct trajectory or as a set of points to be averaged. These choices can affect the conclusions drawn from a plot.

Visualizations that capture the full details of the simulation accept increased complexity in exchange for retaining more information. Such visualizations do not require ambiguous summarization and have correspondingly less opportunity for misinterpretation. Approaches 3 and 4 are capable of displaying complex (eg, multimodal) results, although still identifying the most common results through visual density. These figures can be overlaid with numerical results or particular realizations of interest (eFigure [https://github.com/elofgren/Visualizing-Results-from-Infection-Transmission-Models]), allowing for faster interpretation without loss of information.

Whole-data approaches also preserve outlying results, which are often of great interest in transmission models. Although these graphs require more initial explanation, the overall communication of results is better served with a plot appropriate to the nature of the data. Such graphs may be extended to other forms of analysis that generate large numbers of similar but distinct analyses, such as sensitivity analysis. Other approaches for representing the results of simulation models exist, including 3-dimensional plots and animation. This paper illustrates 2 common but inappropriate techniques and 2 informative alternatives that are tractable to code.

ACKNOWLEDGMENTS

I gratefully acknowledge the help and advice of Stephen R. Cole in the preparation of this manuscript.

REFERENCES

1. Resnick MD, Bearman PS, Blum RW, et al.. Protecting adolescents from harm. Findings from the National Longitudinal Study on Adolescent Health. J Am Med Assoc. 1997;278:823–832.
2. Hunter DR, Goodreau SM, Handcock MS. Goodness of Fit of Social Network Models. J Am Stat Assoc. 2008;103:248–258.
3. Handcock MS, Hunter DR, Butts CT, et al.. statnet: Software tools for the Statistical Modeling of Network Data. 2003. Available at: http://statnetproject.org. Accessed October 15, 2011.
4. R Development Core Team (2008). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0. Available at: http://www.R-project.org. 2008.
© 2012 Lippincott Williams & Wilkins, Inc.