Approaches to Comparative Effectiveness Research in Multimorbid Populations
Maciejewski, Matthew L. PhD*,†; Bayliss, Elizabeth A. MD, MSPH‡,§
*Health Services Research and Development, Durham VA Medical Center
†Department of Medicine, Division of General Internal Medicine, Duke University Medical Center, Durham, NC
‡Institute for Health Research, Kaiser Permanente, Denver
§Department of Family Medicine, University of Colorado Denver, Aurora, CO
Supported by the Office of Research and Development, Health Services Research and Development Service, Department of Veterans Affairs. M.L.M. and E.A.B. received support from The Agency for Healthcare Research and Quality (AHRQ) Multiple Chronic Conditions Research Network R21HS019445 and R21HS019520, respectively. M.L.M. was also supported by a Research Career Scientist award from the Department of Veterans Affairs (RCS 10-391).
The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veteran Affairs, Duke University, Kaiser Permanente, or the University of Colorado.
M.L.M. has received consultation funds from Daichi Sankyo, Takeda Pharmaceuticals, the Surgical Review Corporation, ResDAC at the University of Minnesota, and owns stock in Amgen due to his spouse’s employment.
Reprints: Matthew L. Maciejewski, PhD, Department of Medicine, Division of General Internal Medicine, Duke University Medical Center, Durham, NC 27705. E-mail: email@example.com.
There is an urgent need for an evidence base to guide care for patients with multiple chronic medical conditions (MCC). Comparative effectiveness research (CER) has been touted as 1 solution to generating such evidence. However, the majority of CER topics and methods are designed to generate evidence applicable to single diseases. Generating evidence to guide the care of MCC populations requires thoughtful, and often alternative, approaches to using the existing armamentarium of CER methods. To initiate a dialog about appropriate methods for CER in MCC populations, we discuss advantages and disadvantages of experimental and quasi-experimental study designs for CER in MCC populations, estimating heterogeneity of treatment effects, developing meaningful outcome measures, and aligning morbidity measurement with relevant outcomes. Through an engaged dialog with clinicians, methodologists, and patients, evidence about strengths and limitations of alternative approaches, recommendations about preferred methods for CER in MCC can be developed to ensure that knowledge gaps are filled by valid evidence.
Current clinical guidelines are based on disease-specific goals and metrics. However, care based on such guidelines may or may not be relevant for persons with multiple chronic conditions (MCC). There is little evidence on which to base guidelines for this population, and applying disease-specific recommendations may result in unintended consequences, such as treatment-treatment and treatment-condition interactions.1–3 Further, applying disease-specific guidelines to persons with MCC may lead to markedly inefficient care delivery and impede shared decision making. Given the paucity of evidence to inform the care of persons with MCC, there is a need for epidemiologic, health services, and intervention studies to support the development of evidence-based guidelines that address MCC.4 Care delivery organizations, payers, quality assessment organizations, funders, patient advocates, professional societies, and others have recognized these knowledge gaps and are encouraging research studies directed at this population.
Although knowledge gaps are widely acknowledged,5–7 there is little guidance on research approaches to best address them. Comparative effectiveness research (CER) has been touted as 1 solution for generating an evidence base to inform MCC care.6 CER has been defined as “the conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat, and monitor health conditions in ‘real-world’ settings.”8 The single most challenging feature of conducting MCC CER is the heterogeneity of the study population: chronic conditions occur in multiple combinations, and these conditions and combinations have varying effects on different individuals. Because of heterogeneity, standard CER may not fill critical research gaps. Persons with MCC are routinely excluded from clinical trials due to concerns about generalizability,9 few interventions target the MCC population,10 relevant outcomes for this population have not been well defined, and studies require rigorous methods to account for population and treatment heterogeneity.
To initiate a dialog about approaches to CER in MCC populations, we discuss experimental and quasi-experimental study designs, methods for estimating heterogeneity of treatment effects (HTEs), development of meaningful outcome measures, and alignment of morbidity measurement with relevant outcomes (Fig. 1). We do not discuss specifics of covariate/risk adjustment in detail because these issues are addressed elsewhere.11,12 The discussion below focuses on measurement and adjustment for MCC itself as a covariate, because the need for covariate/risk adjustment depends on choice of study design and covariate selection is informed by the research question. Given the range of validated risk adjusters now available, measure selection should consider the outcome it was designed to predict and the population on which it was validated.
The primary audience of this overview is experienced researchers who may be applying standard CER methods to MCC populations. The secondary audience is clinicians and clinician researchers who are relatively new to research and want to design and conduct studies to improve the evidence base for MCC care. In our own work, we have found that application of standard methods to MCC populations is not straightforward. This overview is not meant to be a comprehensive guide or systematic review of study design, measurement, and statistical methods used in existing MCC literature. The discussion that follows was developed through a series of conversations between the authors facilitated by a series of meetings of AHRQ Multiple Chronic Conditions Research Network. The methodological challenges of conducting rigorous CER will only be overcome through continued dialog with clinicians, methodologists, and patients, to develop evidence (and recommendations for CER generally13–16) about strengths and limitations of alternative approaches to MCC evaluation.
APPROACHES TO STUDY DESIGN
Randomized Controlled Trials (RCTs)
Randomized trials are the gold standard for identifying the causal effect of an intervention on outcomes. That is, did this treatment work on the highly selected patients enrolled in this trial? Although RCTs have long been considered the gold standard for generating disease-specific evidence, traditional RCTs do not accommodate population heterogeneity. Methodologically RCTs minimize such heterogeneity through strict inclusion criteria. To be inclusive of the most prevalent MCC combinations, a classical RCT or practical clinical trial (PCT)17 designed to study the effect of a hypothetical treatment on persons with MCC would require multiple strata to account for different combinations of chronic conditions of varying severity in populations with different social, cultural, and care delivery contexts. Conservatively, such a multisite trial might be impractical in most cases due to the need to include several thousand or tens of thousands individuals and cost hundreds of millions of dollars.
At the other extreme is the N of 1 clinical trial—designed to address areas in which standard RCT-generated evidence may not generalize well to an individual patient and there are substantial questions about efficacy.18 The N of 1 trial can answer a clinically relevant question: did this treatment work on this patient? Such trials have the advantage of a priori specification of outcomes that are meaningful to an individual patient. However, having traded external for internal validity, N of 1 trials are unlikely to generate generalizable evidence for the MCC population at large. Further they can only be used for outcomes that are fully reversible after exposure—as N of 1 trials require alternating periods of exposure and placebo during which to measure outcomes. Although well-designed RCTs or PCTs driven by specific hypotheses relevant to common combinations of conditions will clearly have a role in evidence development, exclusively relying on RCTs for the MCC population is likely impractical given the urgency for timely and valid information.
Because of the inherent challenges of trials, quasi-experiments with large sample sizes are likely to be the workhorse study design for CER to inform care for persons with MCCs. Quasi-experiments answer a slightly different research question than RCTs. That is, is a given treatment associated with specific outcomes among the broadly selected patients enrolled in this study? With careful design, measurement, and analysis, this association may approach the causal effect that can be identified in a randomized trial, but that supposition is the subject of heated ongoing debate. Rigorous quasi-experiments that leverage administrative datasets representing large, geographically diverse populations (eg, Medicare and Medicaid claims data; electronic medical record data through the VA, HMO Research Network; large representative surveys such as the Medical Expenditure Panel Survey or the National Health Interview Survey) have the potential to generate evidence about the prevalence of specific MCC combinations,19 the health and economic outcomes associated with specific MCC combinations, and processes of care that may be effective for improving the health and healthcare for diverse MCC patients. Quasi-experimental CER can also identify variations in MCC prevalence and incidence, clinical and economic outcomes associated with MCC, and modifiable risk factors for specific populations that can inform designs of targeted randomized trials.
An important challenge to quasi-experimental studies that is particularly relevant to MCC research is ensuring internal validity (inferring causality and ruling out alternative explanations for observed associations). The internal validity of quasi-experimental results from single-condition samples has been questioned,20 and these concerns may be magnified in studies of MCC populations because the choice of control groups and potential causes of unobserved confounding may be more complicated (Table 1). Design, measurement, and analytic strategies to improve internal validity can be greatly informed by developing detailed conceptual models21,22 that illustrate potentially causative relationships between complexity and patient outcomes. Such models also clarify potential relationships between exposures and outcomes across different combinations of conditions (ie, addressing possible heterogeneity in the association between different conditions and an outcome) and the likely extent of measured and unmeasured confounding.
Internal validity of quasi-experimental studies can be further strengthened by including comparable control groups by using identical inclusion and exclusion criteria23; adjusting for risk differences between treatment and control groups; assessing nonequivalent outcomes (outcomes that would not be expected to change in response to the intervention)24,25; and conducting sensitivity analyses.23 Sensitivity analyses, in particular, are becoming a standard element of CER, but their application to MCC research requires consideration of stratification variables and methods that are likely to identify relevant subpopulations. These include accurate measurement of morbidity burden, and analytic methods to assess and manage population heterogeneity and are discussed below.
Depending on the research question of interest, morbidity burden can be treated as a variable to define the population of interest (inclusion criterion), as an independent (predictive) variable, and/or as a covariate to adjust the relationship between dependent and independent variables. Morbidity measurement is most commonly based on counts or weighted counts of specific conditions—usually drawn from administrative data such as diagnosis codes. Such data are easy to access electronically and measures can be efficiently calculated and applied in analyses.
Other morbidity measures include those based on pharmacy data, chart review, and self-report.26–33 The method of measurement matters—as morbidity measures are validated in different settings and against different outcomes (Table 1). For example, a morbidity measure valid in predicting hospitalization and mortality may or may not fully capture the association between morbidity burden and quality of life outcomes.34 Self-reported morbidity, although more labor intensive to collect, is more strongly associated with subjective outcomes and biopsychosocial constructs such as function and quality of life; whereas data-based measures are more strongly associated with objective outcomes such as mortality and cost of care.30,35 However, data-based and self-report measures are independently associated with multiple outcomes. For example, hospitalization and outpatient utilization are independently associated with morbidity quantified by self-report and diagnosis codes35 because these 2 morbidity measures are not perfectly correlated with one another: Self-reported morbidity captures a range of biopsychosocial constructs,30 whereas morbidity quantified by diagnosis code captures clinician assessment of morbidity to reflect service use. Depending on the research question and outcome of interest, complete assessment of morbidity burden may require both subjective (eg, self-report) and objective (eg, data based) measures.
OUTCOMES RELEVANT TO MULTIMORBIDITY
A second major challenge in conducting rigorous CER that can inform patient care is the choice of outcomes that are salient to patients with MCC (Table 1). Disease-specific outcomes are the standard for evaluating care of single chronic diseases. Such outcomes are necessary but insufficient for the MCC population as they reflect only a fragment of the multiphased care received by MCC patients and do not address the spectrum of issues faced by this population.2 For example, a woman with good prognosis breast cancer as well as hypertension, diabetes, and a family history of heart disease will likely benefit from goal-oriented management of cardiovascular risk throughout her cancer treatment. However, for the individual with late-stage lung cancer, aggressive lipid management even in the face of high cardiac risk is unlikely to change life expectancy.36
Outcomes that are not disease specific are relevant for persons with and without MCC. However, nondisease-specific outcomes may apply differently to the MCC population and it is unclear how to best apply more generic outcomes to this population. For example, rates of 30-day readmission vary by morbidity level—What are acceptable rates for the MCC population?37,38 What are appropriate utilization norms for persons with MCC who by definition need to use services? Would outcomes such as underuse of necessary services (eg, preventive care) and overuse of unnecessary services (suggested by the Choosing Wisely guidelines) reflect the competing demands and high overall utilization (respectively) of the MCC population?
Patient-reported outcomes (PROs) such as quality of life, subjective disease and symptom burden, and individualized goals of care are highly relevant for persons with MCC.39 However, routine use of these outcomes is limited as outcome availability currently lags behind patient need due to inefficiencies systematically collecting, storing, and retrieving such information. (This limitation also applies to collecting self-reported morbidity measures.) In addition to improving the systematic collection of PROs to maximize the ability to assess individualized outcomes in the future, it will be beneficial to develop and validate other relevant outcomes that can be accessed electronically and that reflect the care needs of persons with MCC.
With these challenges in mind, leaders at Medicare, National Quality Forum, National Committee for Quality Assurance, the Centers for Medicare and Medicaid Services, and others are encouraging the development of quality measures relevant to MCC patients.7 These quality measures and associated standards of care must be based on outcomes that are valid for heterogenous patient populations and sensitive to intervention and change over time. Development and validation of additional outcomes relevant to MCC patients is a research priority.
ASSESSING HETEROGENEITY IN EFFECT OF MCC ON OUTCOMES
A major strength of quasi-experimental designs for studies of large MCC populations is the ability to examine whether the average treatment effect generalizes to the entire sample or whether there is HTE. HTE reflects the potential for treatments or interventions to differentially affect specific MCC subpopulations—a particular risk given the diverse nature of the MCC patient population. Assessment of HTE can clarify whether the average treatment effect measured in an investigation generalizes to the entire study sample or alternatively represents a weighted average of modest treatment effects in some subgroups and larger treatment effects in other subgroups. Thus accurate assessment of HTE can minimize the risk of ecological fallacy in which incorrect conclusions about individuals may be inadvertently drawn from group-level results.
Studies of MCC populations are at particular risk of ecological fallacy due to HTEs40 because responsiveness to treatments may vary much more widely within MCC populations than within a population with a single condition. For example, the average improvement in adherence to ACE inhibitors 2 years after a copayment reduction was 4.6%, which represents a weighted average of 4.4% among patients adherent at baseline, 5.4% among patients who were somewhat adherent at baseline, and 9.7% among patients who were nonadherent at baseline.41 Analyses that aggregate subpopulations with different combinations of conditions have several advantages over condition-specific analyses (eg, greater statistical power, ability to assess common outcomes of interest such as all-cause healthcare utilization or mortality), but an aggregate sample is likely to be comprised of inherently heterogenous subgroups of MCC patients. Thus aggregation runs the risk of ecological fallacy when population averages do not accurately reflect averages of these distinct subgroups.40,42 HTE analytic methods are designed to minimize this risk.
The primary goals in addressing HTEs are to identify clinically coherent subgroups and to accurately estimate the relationship between population subgroups and outcomes of interest. Six approaches have been used in prior clinical studies to assess HTE, which may be applicable to MCC populations (Table 1). The first and most common approach is to conduct stratified analyses of the treatment group based on single covariates (eg, age, sex). This has several limitations40,43,44 including lack of power and increased risk of false positives if multiple binary comparisons are made.45,46 The second approach is to stratify based on a multidimensional construct, such as a self-reported or calculated morbidity measure.47,48 This approach has advantages over bivariate stratification as it incorporates a more comprehensive set of risk factors than a one-variable-at-a-time approach and reduces the likelihood of false positives because multiple comparisons are reduced. However, it has a significant disadvantage of potentially excluding important predictors of heterogeneity.
The third approach aggregates several self-report measures into a single index on which patients are then stratified.49 This approach has advantages over bivariate stratification because multiple self-report domains are combined into a more comprehensive predictor. This approach also has potential benefits compared with morbidity-based stratification because it can incorporate patient-centric psychosocial and contextual predictors that are unavailable in administrative claims. However, the clinical interpretation of study results in which subpopulations are identified with a score that aggregates multiple self-report measures is less clear and less feasible in actual clinical practice.
The fourth approach involves predictive risk modeling on the basis of clinical variables that are related to the benefits and risks of treatment, which has several strengths compared with prior approaches.50–53 For example, Kent et al51 developed a risk prediction model on the basis of clinical variables from medical record data to predict which patients at risk for myocardial infarction would benefit most from tissue plasminogen activator or streptokinase. Stratification on the basis of a predictive risk model reduces the number of false positives because the number of subgroup analyses is more limited, and it reduces the number of false negatives through the use of interaction terms to assess heterogeneity.46 This approach does have limitations, including the possibility that complex predictive risk models that include administrative data-based and survey-based covariates may be difficult to implement in clinical practice and different predictive risk models may classify the same patients into different groups.54 Thus predictive models should be designed for specific outcomes, should be based on relevant and obtainable variables, and should be based on a priori hypotheses about target subpopulations.
The fifth approach is an empirically driven iterative process known as recursive partitioning (RP). This method identifies subgroups that are homogenous in both outcome and predictors, as compared with predictive risk models based on logistic regression results that identify subgroups that are homogenous only in outcome.55 RP methods can identify several subgroups that may be more clinically intuitive (eg, older female patients with social support and high levels of resilience). In addition, RP directly accounts for interactions when identifying subgroups rather than requiring a priori selection of potential interactions as is the case in regression models,55 but may overfit the data.56 The sixth approach is latent class modeling, which can define distinct subpopulations based on differential exposure levels while modeling continuous and discrete outcomes.57 Demographic and clinical characteristics (including morbidity burden) can then be used to predict the likelihood of belonging to a particular treatment class, as has been done in several latent class analyses to examine HTEs.58–60
HTE methods provide an important set of analytic tools to address a significant challenge to conducting MCC investigations: defining relevant subpopulations for specific exposures and outcomes. In MCC CER, identification of these groups is more likely to include post hoc exploration of the relationship between subpopulations and outcomes than would be expected in investigations of single diseases. Certain HTE methods (such as stratification) can guide exploration through a priori identification of independent characteristics of interest such as coprevalent conditions, morbidity burden, cost of care, or personal or community contextual factors. Other HTE methods (such as latent class analyses or RP) derive potentially relevant subpopulations based on the data itself. Research is needed to compare results from these alternative methods. When choosing subcategories within a broad MCC population, investigators must be mindful that the goal of MCC research is to inform intervention development for the conduct of MCC trials, care management, and clinical guidelines.
We need both an evidence base to guide care for patients with MCC and an armamentarium of methods to generate evidence with internal and external validity.6 Descriptive studies that examine factors contributing to patient complexity and examine the health and economic consequences of multimorbidity are likely to be most effectively and efficiently conducted using combinations of conditions in nonrandomized studies. Evidence gleaned from such population-based quasi-experiments can inform more detailed observational or interventional studies comparing effective care strategies for specific MCC combinations.
These population-based MCC evaluations will be well served by using methods that address the potential for HTE; identifying relevant characteristics of MCC subpopulations; accurately measuring and stratifying by morbidity if needed; and using clear conceptual models to clarify relevant exposures, relevant outcomes, and the relationships between them. Outcomes should be chosen intentionally to either reflect the clinical needs of specific population subgroups or to be generalizable to the MCC population at large. Developing and validating relevant outcomes for this population is a priority.
Through an engaged dialog with clinicians, methodologists, and patients, evidence about strengths and limitations of alternative approaches, recommendations about preferred methods for CER in MCC can be developed to ensure that knowledge gaps are filled by valid evidence. This dialog has been initiated in part by expert panels assembled to elucidate a research agenda and make recommendations on the use of PROs.4,61 The publication of this Special Issue in Medical Care of research articles specific to MCC populations continues the discussion. Next steps should include convening MCC content experts from a range of disciplines, Federal agencies, foundations, advocacy organizations (eg, National Council on Aging, the Patient Centered Outcomes Research Institute), and patients to develop an agenda for developing methodologic standards and developing relevant outcomes specific to MCC populations.
Given the complex contextual factors surrounding MCC research, it is particularly important to expand the discussion beyond traditional research “silos.” For example, public and private payers around the country are experimenting with various approaches to manage the care of various MCC subpopulations, which may enable identification of best practices. Such scientific debate and discussion on methods and evidence generation will require meaningful input and collaboration from experimentalists and observationalists to meet the urgent need to understand and improve the health and healthcare of complex patients.
The authors thank John F. Steiner, MD, MSPH, Jennifer L. Ellis, MSPH, MBA, and 3 reviewers for helpful comments.
1. Boyd CM, Darer J, Boult C, et al..Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance.JAMA.2005;294:716–724.
2. Tinetti ME, Fried TR, Boyd CM.Designing health care for the most common chronic condition—multimorbidity.JAMA.2012;307:2493–2494.
3. Tinetti ME, Bogardus ST Jr, Agostini JV.Potential pitfalls of disease-specific guidelines for patients with multiple conditions.N Engl J Med.2004;351:2870–2874.
4. .Patient-centered care for older adults with multiple chronic conditions: a stepwise approach from the American Geriatrics Society.J Am Geriatr Soc.2012;60:1957–1968.
5. Boyd CM, Fortin M.Future of multimorbidity research: how should understanding of multimorbidity inform health system design?Public Health Rev.2011;32:451–474.
6. Tinetti ME, Studenski SA.Comparative effectiveness research and patients with multiple chronic conditions.N Engl J Med.2011;364:2478–2481.
7. Parekh AK, Goodman RA, Gordon C, et al..Managing multiple chronic conditions: a strategic framework for improving health outcomes and quality of life.Public Health Rep.2011;126:460–471.
8. Sox HC, Greenfield S.Comparative effectiveness research: a report from the Institute of Medicine.Ann Intern Med.2009;151:203–205.
9. Fortin M, Dionne J, Pinho G, et al..Randomized controlled trials: do they have external validity for patients with multiple comorbidities?Ann Fam Med.2006;4:104–108.
10. Smith SM, Soubhi H, Fortin M, et al..Managing patients with multimorbidity: systematic review of interventions in primary care and community settings.BMJ.2012;345:e5205.
11. Brookhart MA, Sturmer T, Glynn RJ, et al..Confounding control in healthcare database research: challenges and potential approaches.Med Care.2010;48supplS114–S120.
12. Maciejewski ML, Curtis LH, Dowd BE.Study design elements for rigorous quasi-experimental comparative effectiveness research.J Comp Effect Res.2013;2:159–173.
13. Berger ML, Dreyer N, Anderson F, et al..Prospective observational studies to assess comparative effectiveness: the ISPOR good research practices task force report.Value Health.2012;15:217–230.
14. Cox E, Martin BC, Van Staa T, et al..Good research practices for comparative effectiveness research: approaches to mitigate bias and confounding in the design of nonrandomized studies of treatment effects using secondary data sources: the International Society for Pharmacoeconomics and Outcomes Research Good Research Practices for Retrospective Database Analysis Task Force Report—Part II.Value Health.2009;12:1053–1061.
15. Dreyer NA, Schneeweiss S, McNeil BJ, et al..GRACE principles: recognizing high-quality observational studies of comparative effectiveness.Am J Manag Care.2010;16:467–471.
16. von Elm E, Altman DG, Egger M, et al..The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.Ann Intern Med.2007;147:573–577.
17. Tunis SR, Stryer DB, Clancy CM.Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy.JAMA.2003;290:1624–1632.
18. Larson EB.N-of-1 clinical trials. A technique for improving medical therapeutics.West J Med.1990;152:52–56.
19. Steinman MA, Lee SJ, John Boscardin W, et al..Patterns of multimorbidity in elderly veterans.J Am Geriatr Soc.2012;60:1872–1880.
20. Ioannidis JP, Haidich AB, Pappa M, et al..Comparison of evidence of treatment effects in randomized and nonrandomized studies.JAMA.2001;286:821–830.
21. Safford MM, Allison JJ, Kiefe CI.Patient complexity: more than comorbidity. the vector model of complexity.J Gen Intern Med.2007;22suppl 3382–390.
22. Shippee ND, Shah ND, May CR, et al..Cumulative complexity: a functional, patient-centered model of patient complexity can improve research and practice.J Clin Epidemiol.2012;65:1041–1051.
23. Schneeweiss S, Patrick AR, Sturmer T, et al..Increasing levels of restriction in pharmacoepidemiologic database studies of elderly and comparison with randomized trial results.Med Care.2007;45suppl 2S131–S142.
24. Foley FW, LaRocca NG, Sanders AS, et al..Rehabilitation of intimacy and sexual dysfunction in couples with multiple sclerosis.Mult Scler.2001;7:417–421.
25. Maciejewski ML, Farley JF, Parker J, et al..Copayment reductions generate greater medication adherence in targeted patients.Health Aff (Millwood).2010;29:2002–2008.
26. Quan H, Sundararajan V, Halfon P, et al..Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.Med Care.2005;43:1130–1139.
27. Elixhauser A, Steiner C, Harris DR, et al..Comorbidity measures for use with administrative data.Med Care.1998;36:8–27.
28. Chaudhry S, Jin L, Meltzer D.Use of a self-report-generated Charlson Comorbidity Index for predicting mortality.Med Care.2005;43:607–615.
29. Von Korff M, Wagner EH, Saunders K.A chronic disease score from automated pharmacy data.J Clin Epidemiol.1992;45:197–203.
30. Bayliss EA, Ellis JL, Steiner JF.Seniors’ self-reported multimorbidity captured biopsychosocial factors not incorporated into two other data-based morbidity measures.J Clin Epidemiol.2009;62:550–557e551.
31. Sangha O, Stucki G, Liang MH, et al..The Self-Administered Comorbidity Questionnaire: a new method to assess comorbidity for clinical and health services research.Arthritis Rheum.2003;49:156–163.
32. Linn BS, Linn MW, Gurel L.Cumulative illness rating scale.J Am Geriatr Soc.1968;16:622–626.
33. Hudon C, Fortin M, Vanasse A.Cumulative Illness Rating Scale was a reliable and valid index in a family practice context.J Clin Epidemiol.2005;58:603–608.
34. Fortin M, Hudon C, Dubois MF, et al..Comparative assessment of three different indices of multimorbidity for studies on health-related quality of life.Health Qual Life Outcomes.2005;3:74.
35. Bayliss EA, Ellis JL, Shoup JA, et al..Association of patient-centered outcomes with patient-reported and ICD-9-based morbidity measures.Ann Fam Med.2012;10:126–133.
36. Tanvetyanon T, Choudhury AM.Physician practice in the discontinuation of statins among patients with advanced lung cancer.J Palliat Care.2006;22:281–285.
37. Wolff JL, Starfield B, Anderson G.Prevalence, expenditures, and complications of multiple chronic conditions in the elderly.Arch Intern Med.2002;162:2269–2276.
38. Bottle A, Gnani S, Saxena S, et al..Association between quality of primary care and hospitalization for coronary heart disease in England: national cross-sectional study.J Gen Intern Med.2008;23:135–141.
39. Wu AW, Snyder C, Clancy CM, et al..Adding the patient perspective to comparative effectiveness research.Health Aff (Millwood).2010;29:1863–1871.
40. Kravitz RL, Duan N, Braslow J.Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages.Milbank Q.2004;82:661–687.
41. Farley JF, Lindquist J, Wansink D, et al..Medication adherence changes associated with value based insurance design.Am J Manag Care.2012;18:265–274.
42. Basu A.Economics of individualization in comparative effectiveness research and a basis for a patient-centered health care.J Health Econ.2011;30:549–559.
43. Greenfield S, Kravitz R, Duan N, et al..Heterogeneity of treatment effects: implications for guidelines, payment, and quality assessment.Am J Med.2007;120suppl 1S3–S9.
44. Kent DM, Hayward RA.Limitations of applying summary results of clinical trials to individual patients: the need for risk stratification.JAMA.2007;298:1209–1212.
45. Lagakos SW.The challenge of subgroup analyses—reporting without distorting.N Engl J Med.2006;354:1667–1669.
46. Kent DM, Rothwell PM, Ioannidis JP, et al..Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal.Trials.2010;11:85.
47. Greenfield S, Billimek J, Pellegrini F, et al..Comorbidity affects the relationship between glycemic control and cardiovascular outcomes in diabetes: a cohort study.Ann Intern Med.2009;151:854–860.
48. Wang V, Liu CF, Bryson CL, et al..Does medication adherence following a copayment increase differ by disease burden?Health Serv Res.2011;466 pt 11963–1985.
49. Kaplan SH, Billimek J, Sorkin DH, et al..Who can respond to treatment? Identifying patient characteristics related to heterogeneity of treatment effects.Med Care.2010;48supplS9–S16.
50. Rothwell PM, Warlow CP.Prediction of benefit from carotid endarterectomy in individual patients: a risk-modelling study. European Carotid Surgery Trialists’ Collaborative Group.Lancet.1999;353:2105–2110.
51. Kent DM, Hayward RA, Griffith JL, et al..An independently derived and validated predictive model for selecting patients with myocardial infarction who are likely to benefit from tissue plasminogen activator compared with streptokinase.Am J Med.2002;113:104–111.
52. Kent DM, Ruthazer R, Selker HP.Are some patients likely to benefit from recombinant tissue-type plasminogen activator for acute ischemic stroke even beyond 3 hours from symptom onset?Stroke.2003;34:464–467.
53. Hayward RA, Krumholz HM, Zulman DM, et al..Optimizing statin treatment for primary prevention of coronary artery disease.Ann Intern Med.2010;152:69–77.
54. Kent DM, Shah ND.Risk models and patient-centered evidence: should physicians expect one right answer?JAMA.2012;307:1585–1586.
55. Kiernan M, Kraemer HC, Winkleby MA, et al..Do logistic regression and signal detection identify different subgroups at risk? implications for the design of tailored interventions.Psychol Methods.2001;6:35–48.
56. Newcomer SR, Steiner JF, Bayliss EA.Identifying subgroups of complex patients with cluster analysis.Am J Manag Care.2011;17:e324–e332.
57. Bayliss EA, Blatchford PJ, Newcomer SR, et al..The effect of incident cancer, depression and pulmonary disease exacerbations on type 2 diabetes control.J Gen Intern Med.2011;26:575–581.
58. Jo B, Wang CP, Ialongo NS.Using latent outcome trajectory classes in causal inference.Stat Interface.2009;2:403–412.
59. Muthen B, Brown HC.Estimating drug effects in the presence of placebo response: causal inference using growth mixture modeling.Stat Med.2009;28:3363–3385.
60. Neelon B, O’Malley AJ, Normand SL.A Bayesian two-part latent class model for longitudinal medical expenditure data: assessing the impact of mental health and substance abuse parity.Biometrics.2011;67:280–289.
comparative effectiveness research; multimorbidity; chronic conditions; complexity; quasi-experiment; experiment; observational; treatment effect; outcomes; statistics
© 2014 by Lippincott Williams & Wilkins.
Highlight selected keywords in the article text.