Skip Navigation LinksHome > April 14, 2000 - Volume 14 - Issue 6 > Distinguishing efficacy, individual effectiveness and popula...

Distinguishing efficacy, individual effectiveness and population effectiveness of therapies

Muñoz, Alvaro; Gange, Stephen J.; Jacobson, Lisa P.

Free Access
Article Outline
Collapse Box

Author Information

Department of Epidemiology, Johns Hopkins University School of Public Health, 615 N. Wolfe Street, Baltimore, MD 21205, USA.

Sponsorship: Coordinating Centers of the Multicenter AIDS Cohort Study and the Women's Interagency HIV Study funded by grants U01-AI-35043 and U01-AI-42590 from the National Institutes of Health, Bethesda, MD, USA.

Received: 30 November 1999; accepted: 23 December 1999.

We read with great interest the article by Phillips et al. [1] and the editorial by Sabin [2] regarding the effectiveness of therapies in observational studies. Although both articles laudably detail issues regarding the use of observational studies for assessing the effect of treatments, the articles highlight the need to distinguish between three epidemiological concepts: efficacy, individual effectiveness and population effectiveness of therapies (Table 1). Clinical trials measure the efficacy of treatments, in which the responses of treated individuals are compared with untreated individuals, and randomization is expected to remove all confounding factors. Clinical trials demonstrate what works under controlled conditions with the limitation that the results are not necessarily applicable to real-world conditions; effects may differ in populations not represented in such trials or among those who do not precisely adhere to the prescribed regimens.

Table 1
Table 1
Image Tools

In observational studies, two types of effects can be measured. The first can be termed `individual effectiveness', which mimics clinical trials by using treatment data at the individual level. These analyses must, via stratification and regression, overcome the lack of randomization and confounding by indication, whereby those individuals at more advanced disease stages are the ones more likely to receive the therapies [3]. Measures of individual effectiveness supplement (and typically confirm) the results of clinical trials, but are usually subject to residual confounding [1]. The second can be termed `population effectiveness', which compares the occurrence of disease in the population when the most ill are treated to the occurrence of disease in the population when none or only a few are treated with a given therapy. As the introduction and use of therapies are closely linked to the calendar, the primary comparison may be characterized by time periods. To control for survival bias and overall disease progression, this approach requires the comparison of groups reaching a similar time at risk (e.g. duration of infection) in different eras defined by calendar periods [4–6]; or the comparison of groups of individuals with similar markers of disease progression at the beginning of treatment eras [7]. Measures of population effectiveness complement the efficacy measured in clinical trials and provide a key public health index: the amount by which disease burden is reduced when only some (typically the most ill) have been exposed to the therapy of interest. Comprehensive data collected by cohort studies are essential to eliminate possible ecological fallaciousness (i.e. effectiveness caused by changes over time different from the therapies of interest) [2,5]. This includes not only prospectively collected data on therapy use, but also information on access, healthcare utilization and practices, and adherence. These data are important in characterizing calendar periods with heterogeneous disease incidences and trajectories of markers of disease progression.

Phillips and colleagues [1] mimicked three clinical trials in each of three major European cohorts to estimate individual effectiveness. In addition to examining the extent by which the cohort studies confirmed the efficacy shown by clinical trials, the data could also have been analysed to provide measures of population effectiveness. Characterization of the introduction of different therapies at different calendar times into the cohorts and the corresponding changes of disease incidence is of central public health interest.

The agreement between inferences of individual effectiveness from observational studies and efficacy from clinical trials shown by Phillips et al. [1] is, in general, comforting. However, the largest (by roughly a factor of 10) cohort of the three analysed studies was the French cohort that provided findings that were inconsistent with the results of the clinical trials. Despite the cautionary note of a `trade-off between quantity and quality of data', one would have expected the largest cohort to have the maximum ability of controlling (adjusting away) the confounding by indication. This is particularly relevant to the individual effectiveness of the highly efficacious therapies containing a protease inhibitor. This inability to confirm the results of the trials illustrates the difficulty of removing strong confounding by indication unless we entertain the unlikely possibility that potent antiretroviral therapy was not effective in the French cohort.

It is possible that the French cohort resembles more of a surveillance registry [2] than a cohort study with active and close follow-up of subjects. Such cohort studies include observing participants at regular study visits to collect data and specimens for laboratory tests under standardized procedures, and the continuous monitoring and confirmation of outcomes. Often the lack of data on important confounders seriously imperils the ability of surveillance databases and registries to assess the effectiveness of interventions [2,8].

Alternatively, the largest cohort may appear to be discrepant from the clinical trials because the analytical approach used by the authors was rather simple compared with the complexities of the selection factors involved in who received treatment. Instead of the standard extension of including updates of the markers of disease progression (i.e. time-dependent covariates) in proportional hazards models, one needs to use more complex models incorporating the mediating role of markers in capturing the effect of therapies. In addition, there are other factors besides markers that influence the use of therapy (e.g. injecting drug users may be less likely to be prescribed complex regimens, and individuals with histories of drug interactions or toxicity may also be prescribed differently). It will be of great interest to know whether a more elaborate analysis renders better agreement between cohort studies and clinical trials regarding the individual effectiveness of therapies.

Although observational studies can provide important data on the effectiveness of therapies at the individual level, they face great challenges in fully mimicking what randomization accomplishes in a clinical trial setting. On the other hand, cohort studies offer the opportunity of providing important measures of the effectiveness of therapies at the population level that clinical trials cannot provide. From the public health perspective, population effectiveness quantifies the reduction of disease achieved by treating the subset of the population that needs therapies the most. In doing so, cohort studies are at the cornerstone of public health and policy.

Alvaro Muñoz

Stephen J. Gange

Lisa P. Jacobson

Back to Top | Article Outline


1. Phillips AN, Grabar S, Tassie JM, Costagliola D, Lundgren JD, Egger M. Use of observational databases to evaluate the effectiveness of antiretroviral therapy for HIV infection: comparison of cohort studies with randomized trials. AIDS 1999, 13: 2075–2082.

2. Sabin CA. Assessing the impact of highly active antiretroviral therapy on AIDS and death. AIDS 1999, 13: 2165–2166.

3. Ahdieh L, Gange S, Greenblatt R, et al. Selection by indication of potent antiretroviral therapy usage in a large cohort of HIV-infected women. Am J Epidemiol 2000, in press.

4. Muñoz A, Hoover D. Use of cohort studies for evaluating AIDS therapies. In:AIDS clinical trials. Finkelstein DM, Schoenfeld DA (editors). New York, NY: Wiley; 1995. pp. 423–446.

5. Detels R, Muñoz A, McFarlane G. et al. Effectiveness of potent antiretroviral therapy on time to AIDS and death in men with known HIV infection duration. JAMA 1998, 280: 1497–1503.

6. Veugelers PJ, Cornelisse PG, Craib KJ. et al. Models of survival in HIV infection and their use in the quantification of treatment benefits. Am J Epidemiol 1998, 148: 487–496.

7. Enger C, Graham NHM, Peng Y. Survival from early, intermediate, and late stages of HIV infection. JAMA 1996, 275: 1329–1334.

8. Gail M. Use of observational data for evaluating AIDS therapies. In:AIDS clinical trials. Finkelstein DM, Schoenfeld DA (editors). New York, NY: Wiley; 1995. pp. 403–422.

Cited By:

This article has been cited 40 time(s).

Journal of Epidemiology and Community Health
Using 'amenable mortality' as indicator of healthcare effectiveness in international comparisons: results of a validation study
Mackenbach, JP; Hoffmann, R; Khoshaba, B; Plug, I; Rey, G; Westerling, R; Parna, K; Jougla, E; Alfonso, J; Looman, C; McKee, M
Journal of Epidemiology and Community Health, 67(2): 139-146.
Journal of Maternal-Fetal & Neonatal Medicine
Does cesarean section prevent mortality and cerebral ultrasound abnormalities in preterm newborns?
Arpino, C; Brescianini, S; Ticconi, C; Di Paolo, A; D'Argenzio, L; Piccione, E; Curatolo, P
Journal of Maternal-Fetal & Neonatal Medicine, 20(2): 151-159.
Journal of Acquired Immune Deficiency Syndromes
Prognostic factors of combined viral load and CD4(+) cell count responses under triple antiretroviral therapy, aquitaine cohort, 1996-1998
Marimoutou, C; Chene, G; Mercie, P; Neau, D; Farbos, S; Morlat, P; Ceccaldi, J; Dabis, F
Journal of Acquired Immune Deficiency Syndromes, 27(2): 161-167.

American Journal of Respiratory and Critical Care Medicine
Measuring the effectiveness of inhaled corticosteroids for COPD is not easy!
Samet, JM
American Journal of Respiratory and Critical Care Medicine, 168(1): 1-2.
Clinical Infectious Diseases
Extensive implementation of highly active antiretroviral therapy shows great effect on survival and surrogate markers in vertically HIV-infected children
Resino, S; Bellon, JM; Resino, R; Navarro, ML; Ramos, JT; de Jose, MI; Mellado, MJ; Munoz-Frenandez, MA
Clinical Infectious Diseases, 38(): 1605-1612.

Clinical Infectious Diseases
Impact of discontinuation of initial protease inhibitor therapy on further virological response in a cohort of human immunodeficiency virus-infected patients
Le Moing, V; Chene, G; Leport, C; Lewden, C; Duran, S; Garre, M; Masquelier, B; Dupon, M; Raffi, F
Clinical Infectious Diseases, 34(2): 239-247.

Journal of Epidemiology and Community Health
Effectiveness of highly active antiretroviral therapy among HIV-1 infected women
Gange, SJ; Barron, Y; Greenblatt, RM; Anastos, K; Minkoff, H; Young, M; Kovacs, A; Cohen, M; Meyer, WA
Journal of Epidemiology and Community Health, 56(2): 153-159.

Journal of Clinical Epidemiology
Unmeasured confounding caused slightly better response to HAART within than outside a randomized controlled trial
Hansen, ABE; Gerstoft, J; Kirk, O; Mathiesen, L; Pedersen, C; Nielsen, H; Jensen-Fangel, S; Sorensen, HT; Obel, N
Journal of Clinical Epidemiology, 61(1): 87-94.
Journal of Acquired Immune Deficiency Syndromes
Time to AIDS from 1992 to 1999 in HIV-1-infected subjects with known date of infection
Tassie, JM; Grabar, S; Lancar, R; Deloumeaux, J; Bentata, M; Costagliola, D
Journal of Acquired Immune Deficiency Syndromes, 30(1): 81-87.

Archives of Pediatrics & Adolescent Medicine
Determinants of mother-to-infant human immunodeficiency virus 1 transmission before and after the introduction of zidovudine prophylaxis
de Martino, M; Galli, L; Tovo, PA; Gabiano, C; Pezzotti, P; Wagner, TM; Rezza, G; Osimani, P; De Mattia, D; Di Bari, C; Ruggeri, M; Baldi, F; Ciccia, M; Lanari, M; Masi, M; Venturi, V; Battisti, L; Duse, M; Chiriaco, PG; Cavallini, R; Dessi, C; Pintor, C; Anastasio, E; Sabatino, G; Sticca, M; Pomero, G; Bezzi, T; Chiappini, E; De Luca, M; Gervaso, P; Cecchi, MT; Bassetti, D; Gotta, C; Rosso, R; Timitilli, A; Tondo, U; Mussini, P; Bricalli, D; Bucceri, A; Ferraris, G; Giovannini, M; Mosca, F; Lipreri, R; Guarino, A; Plebani, A; Riva, E; Riva, S; Vigano, A; Zuccotti, GV; Cellini, M; Buffolano, W; Guarino, A; Tarallo, L; D'Elia, R; Giaquinto, C; Rampon, O; Dalle Nogare, ER; Romano, A; Caselli, D; Maccabruni, A; Consolini, R; Benaglia, G; Magnani, C; Anzidei, G; Pistilli, AMC; Gattinara, GC; Catania, S; Facente, C; Falconieri, P; Fundaro, C; Genovese, O; Rendeli, C; Bionda, S; Cristiano, L; Garetto, S; Riva, C; Palomba, E; Portelli, V; Mazza, A; Salvatore, C; Pellegatta, A; Molesini, M
Archives of Pediatrics & Adolescent Medicine, 156(9): 915-921.

Annals of Epidemiology
The logic for a conception-to-death cohort study
Eaton, WW
Annals of Epidemiology, 12(7): 445-451.
PII S1047-2797(01)00314-3
Clinical Infectious Diseases
Survival of HIV-infected injection drug users (IDUs) in the highly active antiretroviral therapy era, relative to sex- and age-specific survival of HIV-uninfected IDUs
Muga, R; Langohr, K; Tor, J; Sanvisens, A; Serra, I; Rey-Joly, C; Munoz, A
Clinical Infectious Diseases, 45(3): 370-376.
Medicina Clinica
Effect of highly active antiretroviral treatment in HIV infection: role of epidemiological studies
Perez-Hoyos, S
Medicina Clinica, 127(5): 175-176.

International Journal of Epidemiology
Changes over calendar time in the risk of specific first AIDS-defining events following HIV seroconversion, adjusting for competing risks
Babiker, A; Darbyshire, J; Pezzotti, P; Porter, K; Rezza, G; Walker, AS
International Journal of Epidemiology, 31(5): 951-958.

Bmc Infectious Diseases
Immunological predictors of CD4(+) T cell decline in antiretroviral treatment interruptions
Seoane, E; Resino, S; Moreno, S; de Quiros, JCLB; Moreno, A; Rubio, R; Gonzalez-Garcia, J; Arribas, J; Pulido, F; Munoz-Fernandez, M
Bmc Infectious Diseases, 8(): -.
Medicina Clinica
Health services utilization according to gander in a cohort of intravenous drug users infected with HIV
de la Hera, MG; Ruiz, I; Perez-Hoyos, S; Aguado, IH
Medicina Clinica, 120(): 489-493.

American Journal of Epidemiology
Effect of highly active antiretroviral therapy on time to acquired immunodeficiency syndrome or death using marginal structural models
Cole, SR; Hernan, MA; Robins, JM; Anastos, K; Chmiel, J; Detels, R; Ervin, C; Feldman, J; Greenblatt, R; Kingsley, L; Lai, SH; Young, M; Cohen, M; Munoz, A
American Journal of Epidemiology, 158(7): 687-694.
Translating efficacy into effectiveness in antiretroviral therapy
Fletcher, CV
Drugs, 67(): 1969-1979.

Journal of the National Cancer Institute
Highly active antiretroviral therapy and cervical squamous intraepithelial lesions in human immunodeficiency virus-positive women
Ahdieh-Grant, L; Li, R; Levine, AM; Massad, LS; Strickler, HD; Minkoff, H; Moxley, M; Palefsky, J; Sacks, H; Burk, RD; Gange, SJ
Journal of the National Cancer Institute, 96(): 1070-1076.
Journal of Antimicrobial Chemotherapy
Treatment comparisons in HIV infection: the benefits and limitations of observational cohort studies
Sabin, CA; Phillips, AN
Journal of Antimicrobial Chemotherapy, 47(4): 371-375.

Public Health Reports
Effect of race and/or ethnicity in use of antiretrovirals and prophylaxis for opportunistic infection: A review of the literature
Palacio, H; Kahn, JG; Richards, TA; Morin, SE
Public Health Reports, 117(3): 233-251.

Enfermedades Infecciosas Y Microbiologia Clinica
Analytical methods in cohort studies of patients with HIV infection
Jarrin, I; Geskus, R; Perez-Hoyos, S; del Amo, J
Enfermedades Infecciosas Y Microbiologia Clinica, 28(5): 298-303.
Clinical Infectious Diseases
Prevalence of clinical symptoms associated with highly active antiretroviral therapy in the women's interagency HIV study
Silverberg, MJ; Gore, ME; French, AL; Gandhi, M; Glesby, MJ; Kovacs, A; Wilson, TE; Young, MA; Gange, SJ
Clinical Infectious Diseases, 39(5): 717-724.

American Journal of Epidemiology
Selection by indication of potent antiretroviral therapy use in a large cohort of women infected with human immunodeficiency virus
Ahdieh, L; Gange, SJ; Greenblatt, R; Minkoff, H; Anastos, K; Young, M; Nowicki, M; Kovacs, A; Cohen, M; Munoz, A
American Journal of Epidemiology, 152(): 923-933.

Journal of Acquired Immune Deficiency Syndromes
Prevalence and correlates of highly active antiretroviral therapy switching in the Women's Interagency HIV Study
Kirstein, LM; Greenblatt, RM; Anastos, K; Levine, A; French, AL; Minkoff, H; Silver, S; Gange, SJ
Journal of Acquired Immune Deficiency Syndromes, 29(5): 495-503.

Epidemiology and Infection
Initiation, changes in use and effectiveness of highly active anti-retroviral therapy in a cohort of injecting drug users
Egea, JM; Muga, R; Sirera, G; Clotet, B; Tor, J; Ardevol, M; Rey-Joly, C; Munoz, A
Epidemiology and Infection, 129(2): 325-333.
Journal of Epidemiology and Community Health
Gender differences in progression to AIDS and death from HIV seroconversion in a cohort of injecting dug users from 1986 to 2001
de la Hera, MG; Ferreros, I; del Amo, J; de Olalla, PG; Hoyos, SP; Muga, R; del Romero, J; Guerrero, R; Hernandez-Aguado, I
Journal of Epidemiology and Community Health, 58(): 944-950.
American Journal of Epidemiology
Methods to assess population effectiveness of therapies in human immunodeficiency virus incident and prevalent cohorts
Tarwater, PM; Mellors, J; Gore, ME; Margolick, JB; Phair, J; Detels, R; Munoz, A
American Journal of Epidemiology, 154(7): 675-681.

British Journal of Clinical Pharmacology
Clinically informative measures of the effect of drugs or other interventions
Salmi, LR; Suissa, S; Chene, G; Salamon, R
British Journal of Clinical Pharmacology, 65(6): 935-941.
American Journal of Epidemiology
When to initiate highly active antiretroviral therapy: A cohort approach
Ahdieh-Grant, L; Yamashita, TE; Phair, JP; Detels, R; Wolinsky, SM; Margollck, JB; Rinaldo, CR; Jacobson, LP
American Journal of Epidemiology, 157(8): 738-746.
HIV Medicine
Triple nucleoside reverse transcriptase inhibitor- vs. nonnucleoside reverse transcriptase inhibitor-containing regimens as first-line therapy: efficacy and durability in a prospective cohort of French HIV-infected patients
Cuzin, L; Pugliese, P; Bugnon, F; Delpierre, C; Cua, E; Billaud, E; Massip, P; Raffi, F; Dellamonica, P
HIV Medicine, 6(6): 388-395.

Clinical Infectious Diseases
Clinical outcomes improve with highly active antiretroviral therapy in vertically HIV type-1-infected children
Resino, S; Resino, R; Bellon, JM; Micheloud, D; Gutierrez, MDG; de Jose, MI; Ramos, JT; Fontelos, PM; Ciria, L; Munoz-Fernandez, A
Clinical Infectious Diseases, 43(2): 243-252.

Effectiveness of highly-active antiretroviral therapy by race/ethnicity
Silverberg, MJ; Wegner, SA; Milazzo, MJ; McKaig, RG; Williams, CF; Agan, BK; Armstrong, AW; Gange, SJ; Hawkes, C; O'Connell, RJ; Ahuja, SK; Dolan, MJ
AIDS, 20(): 1531-1538.

AIDS Research and Human Retroviruses
Impact of highly active antiretroviral therapy on CD4(+) T cells and viral load of children with AIDS: A population-based study
Resino, S; Bellon, JM; Ramos, JT; Resino, R; Gurbindo, MD; Mellado, MJ; De Jose, MI; Munoz-Fernandez, M
AIDS Research and Human Retroviruses, 20(9): 927-931.

Intensive Care Medicine
Studying outcomes of intensive care unit survivors: the role of the cohort study
Dowdy, DW; Needham, DM; Mendez-Tellez, PA; Herridge, MS; Pronovost, PJ
Intensive Care Medicine, 31(7): 914-921.
Sexually Transmitted Infections
Factors influencing HIV progression in a seroconverter cohort in Madrid from 1985 to 1999
del Amo, J; del Romero, J; Barrasa, A; Perez-Hoyos, S; Rodriguez, C; Diez, M; Soriano, SGV; Castilla, J
Sexually Transmitted Infections, 78(4): 255-260.

Effectiveness of highly active antiretroviral therapy in Spanish cohorts of HIV seroconverters: differences by transmission category
Hernàndez-Aguado, I; and GEMES, ; Pérez-Hoyos, S; del Amo, J; Muga, R; del Romero, J; de Olalla, PG; Guerrero, R
AIDS, 17(3): 353-359.

PDF (264)
Healthcare use by varied highly active antiretroviral therapy (HAART) strata: HAART use, discontinuation, and naivety
Palacio, H; Li, X; Wilson, TE; Sacks, H; Cohen, MH; Richardson, J; Young, M; Muñoz, A; for the Women's Interagency HIV Study (WIHS),
AIDS, 18(4): 621-630.

PDF (129)
Immunologic and virologic response to highly active antiretroviral therapy in the Multicenter AIDS Cohort Study
Jacobson, LP; Yamashita, TE; Phair, JP; Muñoz, A; Margolick, JB; Detels, R; O'Brien, SJ; Mellors, JW; Wolinsky, SM
AIDS, 15(6): 735-746.

PDF (201)
Comparison of the effectiveness of non-nucleoside reverse transcriptase inhibitor-containing and protease inhibitor-containing regimens using observational databases
Ghani, AC; Henley, WE; Donnelly, CA; Mayer, S; Anderson, RM
AIDS, 15(9): 1133-1142.

PDF (151)
Back to Top | Article Outline

© 2000 Lippincott Williams & Wilkins, Inc.


Article Level Metrics