Secondary Logo

Journal Logo

Original Article: Methods

Nested Randomized Trials in Large Cohorts and Biobanks

Studying the Health Effects of Lifestyle Factors

Ioannidis, John P. A.*†; Adami, Hans-Olov

Author Information
doi: 10.1097/EDE.0b013e31815be01c
  • Free

Abstract

Courses in introductory research methods have traditionally taught that trials evaluate interventions while epidemiologic studies address risk factors. However, the distinction becomes blurred when it comes to lifestyle and behavior. Epidemiologists have argued that these risk factors are preferably studied with observational studies.1 However, several randomized trials have been conducted in the last 2 decades on lifestyle interventions.2–6 We have also examples, such as the Women's Health Initiative (WHI), where both randomized trials and epidemiologic research have been conducted by the same team.2

Lifestyle could encompass a wide variety of interventions including advice, prescription of actions (eg, exercise), and even provision of food, supplements, or more sophisticated medical and other technology and health care. Both observational studies and randomized trials face limits in addressing lifestyle. On one hand, the cost of long-term follow-up trials is astronomical.2 WHI had a budget of 2 billion US dollars. Even with relatively long follow-up, major health events are few. Trials interested in cause-specific deaths are underpowered regardless of the intervention—lifestyle, drug, or other. In the finasteride trial,7 only 5 deaths due to prostate cancer occurred in each arm (total n = 18,882 men) over 7 years of follow-up. The ongoing Selenium and Vitamin E Cancer Prevention trial of selenium and vitamin E, with a budget of 176 million US dollars, is powered to address prostate cancer incidence, not prostate cancer deaths.8 It is extremely costly to test many lifestyle hypotheses with ordinary randomized trials.

On the other hand, randomized trials on lifestyle interventions have sometimes contradicted the results of observational research.2,3,9 While epidemiology has provided robust evidence on strong risk factors (eg, smoking), evidence on small everyday risks is more contentious because of confounding. Occasional contradictions10,11 create havoc and draw criticism.12–14 Without randomization, residual confounding cannot be excluded. Eventually, we tend to forget that observational studies and randomized trials have more to agree on and to share rather than disagree on and divide. The hypothetical border between epidemiologic and clinical research may largely be meaningless.

Lifestyle Versus Genetics: Uneven Partners in Measurement

A new thrust for revisiting the impact of everyday exposures is offered by the advent of genomics.15,16 Genetic measurements of potential disease determinants become rapidly better and cheaper.17–19 Genetic risk factors are measured massively with <0.5% genotyping error. Mendelian randomization20 may even offer considerable protection from confounding. Biases still exist,21,22 but may be overcome with careful design, standardization, and large-scale evidence.19

In contrast to genetics, methods of measurement of environmental exposures remain suboptimal. Selective reporting and publication biases are probably as large as on the genetic side.12,23 However, we need to measure the impact of environmental lifestyle exposures to incorporate, synthesize, and make sense of what we learn about the genetic component.24 Biobanks for large cohorts aim to meet exactly this goal: to measure and understand the effects of both genetic and acquired exposures in large populations. Biobanks offer the infrastructure for carrying out a cohort study in the population that has contributed the specimens and has agreed to some sort of follow-up. Several biobanks are being set up currently with the goal of evaluating large populations over several decades. Examples include the UK Biobank,25 the P3G coalition,26 and the Swedish LifeGene currently under development.

Nested Trials in Biobanks and Other Large Cohorts

We propose that lifestyle interventions can be studied with long-term, simple, factorial-design randomized trials nested in biobanks and other large cohorts, with reliable linkage to outcomes. The proposed randomized trials should be simple, tailored to personal preferences, and receptive to societal needs. These multiple lifestyle factorial experimental (multi-LIFE) designs combine characteristics of randomized trials and epidemiologic studies.

Multi-LIFE designs reverse the notion of an epidemiologic cohort or case-control study being nested in a randomized trial; instead, randomized trials are nested within a large cohort. These trials should ideally be built within newly recruited biobanks and cohorts with large anticipated sample size (eg, 500,000 participants). People who wish to participate in the biobank are also asked whether they wish to be randomized to any among several possible lifestyle interventions. The participants can choose from a long list of simple lifestyle randomization options. Many lifestyle interventions may be tested concurrently with factorial randomization, ie, a separate randomization is performed for each intervention. Participants for randomization are recruited among people who enroll in the cohort. Such trials may also be incorporated into an existing biobank or any large cohort that has already enrolled subjects, if new visits for participants can be accommodated. Some existing large cohorts have already collected large amounts of potentially useful information and biologic samples. Given this already available information, one could also identify strata of particular interest among whom to propose specific trials. The proposed design offers a clear advantage in the presence of established, reliable registries of outcomes that are linked to the cohort.

Each participant may choose interventions to which he or she wishes to be randomized, from among several randomization choices. For all the randomizations in the example discussed in Table 1, some epidemiologic evidence exists on potential benefits or harms to health, but the balance is not clear cut. For example, coffee has been associated with decreased risk of colorectal cancer and diabetes27,28 and increased bladder cancer risk.29 However, there is no strong evidence to recommend coffee consumption or abstaining from coffee for the general population. Tomatoes have been associated with reduced prostate cancer risk,30 but the evidence is too weak to assume that abstaining from tomatoes is an unethical recommendation. Even when randomized trials have been done, the evidence may be contentious or limited to short-term or surrogate outcomes.

TABLE 1
TABLE 1:
Example of Multiple Factorial Randomizations

For each randomization, the difference in exposure between the 2 arms should be considerable, but not extravagant, so as to allow many people to consider participation and be able to adopt the randomized lifestyle. We should acknowledge that when the proposed interventions do not allow a comparison of extremes of lifestyle, the power to find an effect may be smaller compared with studies of similar sample size where extremely different lifestyles are selected for comparison. However, this depends also on the shape of the (unknown) dose-response relationship, if there is one. Moreover, when comparing extreme lifestyles, the participation of volunteers is likely to decrease sharply, since few people would be willing to be randomized to extremes (eg, drink at least 50 cups of coffee per week vs. drink no coffee at all ever). Even among those who do accept to be randomized, drop-outs from the assigned extreme lifestyles are likely to be frequent. Finally, effects on extreme lifestyles may be less generalizable to the population at large. Therefore, one has to try to set the comparison at a level that would maximize participation and adherence and still have a reasonable contrast of different exposures in the 2 arms.

Even with nonextreme contrasts, it is possible that some participants will choose to adopt an extreme exposure. One may consider adding further collection of biospecimens or enhanced outcome tracking or both for adopters of extreme lifestyle options.

One of the major challenges in any randomized trial of lifestyle is the difficulty people face in making changes. However, in the proposed design, those willing to volunteer for randomization to either alternative are those who are likely to be most neutral toward the specific interventions and thus most likely to adopt the randomized assigned lifestyle.

Using the Cohort Data Collection and Outcome Linkage Machinery for Randomized Trials

Nesting randomized trials within ongoing large-scale cohorts would be advantageous, if we can exploit the machinery of the cohort on data collection and outcome linkage and if this machinery is complete and reliable. A major drawback of many randomized trials has been their difficulty in reliably capturing long-term outcomes. Losses to follow-up erode their credibility. Active follow-up of individuals and outcome ascertainment escalate their cost. Conversely, large-scale biobanks, such as the envisioned LifeGene in Sweden, may ascertain outcomes through linkage with reliable national registers, both for mortality and for specific diseases.31 They may also collect considerable information on lifestyle exposures, and some may do this with repeated measurements. Modern biobanks also explicitly collect blood samples to test genetic factors and other molecular markers.

Generating Randomization Laundry Lists and Recruiting Participants

While a typical randomized trial tests a limited number of interventions, multi-LIFE designs have long laundry lists of possible randomizations. What should be included in these lists? Scientists may propose a starter list, and then lists may evolve under input from the general public and participants themselves. Scientists can ensure that a systematic review of the evidence suggests equipoise for each new proposed intervention. A suggestive starter list is shown in Table 2.

TABLE 2
TABLE 2:
A Starter Randomization List for Multi-LIFE Designs

A pilot randomized phase may address whether there is enough interest for an intervention. Interventions that fail to elicit interest from a minimum number of participants within the first 2 months of becoming available can be dropped.32 If participation is satisfactory, the pilot may then continue for up to a year to further examine responsiveness to follow-up inquiry (either by visit or by electronic or mobile communication), short- to midterm adherence, and cross-overs. Interventions that also show strong adherence and limited cross-overs may then open for randomization to the wider population and long-term follow-up.

The whole process aims to recruit participants who are neutral about each specific intervention and would accept either randomization option. Participants who feel strongly about adopting or not adopting specific interventions should be discouraged from participating in the respective randomization. Randomized lifestyle contrasts are not extreme, to maximize participation.

It is also important to address how long the lifestyle modifications last. This can be answered if the cohort is designed to collect information on lifestyle with repeated measurements during follow-up. In addition to the usual methods (diaries, telephone calls), the advent of more efficient data collection methods that use Internet and mobile phone messages to communicate with participants should facilitate the process.

Citizen-Scientists Versus Subjects

In a multi-LIFE design, the participants are the experimenters who can propose, select, and adopt interventions for themselves. This takes motivated people who have genuine interest in the specific research question. Until now, randomized participants were subjected to the intervention according to a protocol others had designed for them. The term “subjects” mirrors this passive, dependent participation. With multi-LIFE designs, participants tailor the trial to their own interests. This may be more attractive in societies of health-conscious citizens.

Participation rates in traditional clinical trials have diminished alarmingly, and understanding of informed consent is often poor.33,34 Typical tested interventions consist of novel drugs and technologies about which even experts may have serious knowledge gaps. For lifestyle interventions, people can design their own custom-tailored trial on interventions that form part of their everyday lives.

Overall, multi-LIFE designs share the same philosophy of citizen participation in research as the practice of processing difficult computing problems across thousands of personal computers. This has become increasingly popular in computationally intensive fields in mathematics and physics. A brilliant component of the design of some early traditional cohorts (eg, the British Doctors Study or the Harvard cohorts of health professionals) was the targeting of individuals with professional interest to health. In the future, the citizen-scientist prototype may extend beyond professional boundaries when it comes to health issues.

Furthermore, the participants in each randomization may be considered to be the main researchers for that particular research question. Participants may decide on whether they wish to be specifically listed as contributors to the manuscripts derived from the data of the randomizations where they have participated. Wishes for anonymity will be respected. Long name lists could easily appear on the Web. The involvement of participants at all levels up to authorship is consistent with the concept of the citizen-scientists and this should be increasingly more appealing to people than the passive role of experimental subject. It will also reinforce the sense of civic responsibility in the population in promoting knowledge about health.

Issues to Discuss

The general principles presented above leave open several issues to discuss and settle appropriately (Table 3).

TABLE 3
TABLE 3:
Proposed Multi-LIFE Designs, Randomized Controlled Trials (RCTs) and Observational Epidemiologic Studies

Participation

Many people may choose no randomization at all. Some will not be convinced about the importance of self-experimentation. Others may be unwilling to modify their lifestyle. Only a minority may accept randomization to one or several lifestyles. Participation rates depend on the population, the ability to make the project popular, and available randomization choices.2

Nonparticipation may affect the generalizability of the results and may introduce volunteer bias. Volunteer bias is likely to be stronger than in observational cohorts, but less than in randomized trials. In classic randomized trials the percentage of patients who consent to randomization is very small.35 Multi-LIFE consent rates should be considerably higher, especially if the list of options includes simple interventions, and exclusion criteria would be minimal or even nonexistent for most interventions.

We should also acknowledge that individuals may opt to enter a particular intervention (or not) depending on their previous experience with that intervention, and the results it did or did not produce. This may tend to exclude individuals with extreme outcomes, both favorable and unfavorable, which would lead to greater homogeneity of treatment responses.

The proposed design may be more attractive to educated participants with higher socioeconomic status and better health at baseline. Conversely, it may enroll few participants among population strata that do not usually volunteer for research, eg, migrants. This applies to any type of clinical trial.

Number of Randomizations

Even with very simple interventions, participants may not remain loyal to the allocations, if they select too many. However, up to 3 or even 5 interventions per participant should be feasible. A pilot phase may examine the exact extent to which individuals can select multiple randomizations and what impact this may have on adherence to each randomization option selected.

Compliance/Adherence

Low compliance and adherence are problems for long-term studies, regardless of design. In the WHI trial of calcium and vitamin D, 69% of the women were taking calcium on their own by the end of the trial, and over a third were taking vitamin D. In the Physician's Health Study, one-third of the men initially enrolled were dropped during the run-in phase (before randomization) because of inadequate compliance. One option is a randomized pilot phase to ensure at least short-term adherence.

Poor adherence tends to underestimate treatment effects.36 The impact on multi-LIFE designs could be considerable. However, by their very nature, these studies would allow the participants to design their own trials, so compliance may be the best possible. Evidence from past lifestyle intervention studies37–40 suggests that people who are more health conscious and have better health are more likely to be compliant; compliance is also higher in people who are motivated and attend training sessions. Multi-LIFE designs are likely to be attractive particularly to such health-conscious, motivated citizens. This does not mean that compliance would necessarily be very high. Even the most motivated participants may abandon the assigned interventions. Importantly, the existing cohort and biobank machinery may be used to capture reliable information on adherence. This information may be used for structural-equation modeling of causal effects of the interventions, as discussed below.

Power and False-Negatives

Assume a cohort with 500,000 subjects. If there are 20 potential interventions from which to choose, and 30% of the participants agree to be randomized to an average of 4 interventions, the average sample size per intervention would be 30,000. With long-term follow-up, the event rate and resulting power would be sufficient to study major common outcomes of interest. For 80% power to identify a 20% relative risk reduction, 640 events are needed. If half of this effect is dissipated with poor adherence and cross-overs, 80% power for a 10% relative risk reduction requires 2900 events. We acknowledge that small effects may be missed in the presence of extensive nonadherence and cross-overs. However, small effects are extremely difficult, if not impossible, to study with classic observational studies in any event.41 Power in randomized trials would be eroded not only by poor adherence and cross-overs, but also by loss to follow-up. Loss to follow-up and long-term accrual of events are not a concern for biobanks and other cohorts with linkage to registries.

These power considerations follow the traditional intention-to-treat analysis based on the original randomization. However, if nonadherence and time variability in the intervention are considerable, we propose that analysis of these trials may also adopt causal inference techniques with structural equation modeling and instrumental variables.42–45 These techniques have already been proposed for both observational and randomized datasets and may be more suitable to quantify causal effects between an intervention and the outcome of interest, when exposure/intervention varies over time.

False-Positives

As in any study with multiple outcomes, false-positive findings are expected. For example, with α = 0.05, 2.5 false-positives are expected among 50 tested interventions. The same applies also to randomized trials and epidemiologic studies. However, these studies may suffer also from additional publication and selective analysis bias (reporting of specific analyses and outcomes that show the “best” results) that may spuriously inflate the proportion of reported statistically significant results in the literature.11,46 The new biobanks offer the opportunity to create an environment of high visibility and transparency where all protocols and analyses can be explicitly described and registered. Despite progress in trial registration, not all journals have adopted registration as a prerequisite for publication, and the majority of randomized trials are still not registered. Moreover, there are already many ongoing biobanks and others being designed and this could allow for replicate testing and even biobank-based meta-analyses of randomized data for specific interventions.

Composite Effects of Lifestyle Interventions

Lifestyle factors may sometimes interact as composites, ie, the whole may be greater than the sum of parts. For example, it has been argued that while each component of Mediterranean diet is not formally related to mortality risk, the composite Mediterranean diet score is.47 However, studying such composites in the epidemiologic setting is problematic. Most claims have showed a significant effect on some composite score, rather than proving that indeed the interaction effects explain significantly more than the independent marginal effects. Power would be limited to study complex interactions. However, the most powerful way to study interactions is factorial randomized designs.48 Nevertheless, we believe that interaction effects should be seen as secondary analyses. Unless some randomizations are very popular or the effect modification is very large, evaluation of effect modification would be largely exploratory.

If a very specific complex intervention made up of many different components is deemed important to study, this could also be done through a classic clinical trial. However, for lifestyle, it is difficult to select which out of thousands of complex combinations should be prioritized for testing.

Selection of Outcomes

Outcomes should be carefully selected. Much depends on the existing disease registries linked to the biobank or other large cohort and their reliability. Arguably, mortality should be an outcome in all these trials. Depending on availability of disease registers, one may also study specific diseases such as cancer, specific cancers, cardiovascular disease, and end-stage renal failure. For example, Scandinavian countries have highly reliable national registers for these conditions. Information on additional outcomes may be collected actively, but this would be cost-efficient mostly for short- or midterm outcomes.

Length of Follow-up and Monitoring

An advantage of biobanks and other large cohorts is their anticipated long-term follow-up. Information on major outcomes is collected until all participants die. There is no reason not to exploit this information.

One should discuss on a case-by-case basis the type of monitoring in these trials and whether any stopping rules are indicated.49 Stopping rules, if selected, should account for the multiplicity of comparisons involved, to avoid premature termination from false-positive results.

Masking and Allocation Concealment

Multi-LIFE designs are unmasked. It is unclear whether unmasked trials provide exaggerated treatment effects compared with double-blind trials.50,51 Bias should not be major for well-defined outcomes (eg, death or cancer). Masking would greatly increase complexity and would not allow studying many interventions concomitantly. However, if soft outcomes (eg, symptoms) are studied, then lack of masking may bias some results. For example, some trials of saw palmetto for urinary symptoms in prostatic hypertrophy yielded apparent benefits, but masking was difficult because saw palmetto has a strong aroma/taste; benefits were not verified in a large properly masked trial.52

Allocation concealment, an important aspect for any unbiased randomized trial, should be ensured for these trials. There is no reason why rigorous allocation concealment cannot be adopted. Similarly, while masking of participants will not be feasible for most interventions, blinding of data collectors, assessors, and analysts is attainable.

Age of Participants

Most current biobanks have recruited middle-aged adults. However, lifestyle factors may exert differential effects on health when adopted early in life.53,54 Classic randomized trials cannot study long-term hard outcomes, including mortality, in people enrolled at young ages. Follow-up, cost, and intensity of ascertainment are prohibitive. Biobanks have advantages in this regard: linkage to established registries should allow capturing outcomes even many decades after the original randomization. The latest consensus meetings of the Swedish LifeGene biobank currently being developed have shifted attention to enrolling a large portion of young participants. Depending on the existing data collection machinery, moreover, other important outcomes may also be studied in young populations that would not require long follow-up.

Confounding Due to False-Reassurance and Adoption of Complementary Lifestyles

Screening and prevention trials may be confounded by false-reassurance: participants offered the intervention may not seek proper medical attention, even when they develop symptoms and signs of the disease. Multi-LIFE designs share this concern. Yet for most interventions it is unlikely that the participants would feel that they are so reassuringly effective.

Adoption of a specific intervention may lead to changing some other complementary lifestyle: eg, people randomized to try to walk whenever they have a short distance to cover, may decide to cut all other physical activity. Such counter-acting lifestyle changes can be captured if the cohort uses repeated longitudinal measurements of exposures for its own purposes. Both intention-to-treat and structural equation modeling accounting for time-varying exposures could be used to make inferences in this situation. One may also examine whether effects for any epidemiological associations probed are different for participants who have selected also to be randomized than in those who have avoided randomization.

Cost

Large-scale cohorts are expensive enterprises.2 This is actually an argument in favor of nesting simple randomized trials in these designs. The additional cost should be minimal if these experiments simply use the data machinery of a registry-linked biobank. The cost would increase if the randomizations include active supplementation of interventions, eg, vitamins, hormones, and drugs. Cost would also escalate if the assigned behaviors need to be reinforced among the participants, for example, to maximize the difference in the levels of exposure between 2 randomized arms.

CONCLUSION

Neither randomized trials nor observational studies are able to provide conclusive answers regarding lifestyle factors. Hybrid designs that address questions of lifestyle can combine the strengths of both approaches without compounding their limitations (Table 3). We discuss the possibilities and limitations of a hybrid approach to stimulate further exploration of this underutilized option.

REFERENCES

1. Stampfer M. Observational epidemiology is the preferred means of evaluating effects of behavioral and lifestyle modification. Control Clin Trials. 1997;18:494–499; discussion 514–516.
2. Prentice RL, Pettinger M, Anderson GL. Statistical issues arising in the Women's Health Initiative. Biometrics. 2005;61:899–911; discussion 911–941.
3. The Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group. The effect of vitamin E and beta carotene on the incidence of lung cancer and other cancers in male smokers. N Engl J Med. 1994;330:1029–1035.
4. Howard BV, Van Horn L, Hsia J, et al. Low-fat dietary pattern and risk of cardiovascular disease: the Women's Health Initiative Randomized Controlled Dietary Modification Trial. JAMA. 2006;295:655–666.
5. Dansinger ML, Gleason JA, Griffith JL, et al. Comparison of the atkins, ornish, weight watchers, and zone diets for weight loss and heart disease risk reduction: a randomized trial. JAMA. 2005;293:43–53.
6. Province MA, Hadley EC, Hornbrook MC, et al. The effects of exercise on falls in elderly patients. A preplanned meta-analysis of the FICSIT Trials. Frailty and injuries: cooperative studies of intervention techniques. JAMA. 1995;273:1341–1347.
7. Thompson IM, Goodman PJ, Tangen CM, et al. The influence of finasteride on the development of prostate cancer. N Engl J Med. 2003;349:215–224.
8. Lippman SM, Goodman PJ, Klein EA, et al. Designing the Selenium and Vitamin E Cancer Prevention Trial (SELECT). J Natl Cancer Inst. 2005;97:94–102.
9. Yusuf S, Dagenais G, Pogue J, et al. Vitamin E supplementation and cardiovascular events in high-risk patients. The Heart Outcomes Prevention Evaluation Study Investigators. N Engl J Med. 2000;342:154–160.
10. Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005;294:218–228.
11. Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2:e124.
12. von Elm E, Egger M. The scandal of poor epidemiological research. BMJ. 2004;329:868–869.
13. Skrabanek P. Has risk-factor epidemiology outlived its usefulness? Am J Epidemiol. 1993;138:1016–1017.
14. Taubes G. Epidemiology faces its limits. Science. 1995;269:164–169.
15. Hirschhorn JN, Daly MJ. Genome-wide association studies for common diseases and complex traits. Nat Rev Genet. 2005;6:95–108.
16. Palmer LJ, Cardon LR. Shaking the tree: mapping complex disease genes with linkage disequilibrium. Lancet. 2005;366:1223–1234.
17. Collins FS. The case for a US prospective cohort study of genes and environment. Nature. 2004;429:475–477.
18. Collins FS, Manolio TA. Merging and emerging cohorts: necessary but not sufficient. Nature. 2007;445:259.
19. Ioannidis JP, Gwinn M, Little J, et al. A road map for efficient and reliable human genome epidemiology. Nat Genet. 2006;38:3–5.
20. Davey Smith G, Ebrahim S. Mendelian randomization: can genetic epidemiology contribute to understanding environmental determinants of disease? Int J Epidemiol. 2003;32:1–22.
21. Ioannidis JP, Ntzani EE, Trikalinos TA, et al. Replication validity of genetic association studies. Nat Genet. 2001;29:306–309.
22. Ioannidis JP. Genetic associations: false or true? Trends Mol Med. 2003;9:135–138.
23. Pocock SJ, Collier TJ, Dandreo KJ, et al. Issues in the reporting of epidemiological studies: a survey of recent practice. BMJ. 2004;329:883.
24. Botto LD, Khoury MJ. Commentary: facing the challenge of gene-environment interaction: the two-by-four table and beyond. Am J Epidemiol. 2001;153:1016–1020.
25. Ollier W, Sprosen T, Peakman T. UK Biobank: from concept to reality. Pharmacogenomics. 2005;6:639–646.
26. Public Population Project in Genomics. Accessed at: http://www.p3gconsortium.org.
27. Tavani A, La Vecchia C. Coffee and cancer: a review of epidemiological studies, 1990–1999. Eur J Cancer Prev. 2000;9:241–256.
28. van Dam RM, Hu FB. Coffee consumption and risk of type 2 diabetes: a systematic review. JAMA. 2005;294:97–104.
29. Sala M, Cordier S, Chang-Claude J, et al. Coffee consumption and bladder cancer in nonsmokers: a pooled analysis of case-control studies in European countries. Cancer Causes Control. 2000;11:925–931.
30. Etminan M, Takkouche B, Caamano-Isorna F. The role of tomato products and lycopene in the prevention of prostate cancer: a meta-analysis of observational studies. Cancer Epidemiol Biomarkers Prev. 2004;13:340–345.
31. Adami HO. A paradise for epidemiologists? Lancet. 1996;347:588–589.
32. Haidich AB, Ioannidis JP. Effect of early patient enrollment on the time to completion and publication of randomized controlled trials. Am J Epidemiol. 2001;154:873–880.
33. Wendler D. Can we ensure that all research subjects give valid consent? Arch Intern Med. 2004;164:2201–2204.
34. Flory J, Emanuel E. Interventions to improve research participants’ understanding in informed consent for research: a systematic review. JAMA. 2004;292:1593–1601.
35. Murthy VH, Krumholz HM, Gross CP. Participation in cancer clinical trials: race-, sex-, and age-based disparities. JAMA. 2004;291:2720–2726.
36. Michels KB. The women's health initiative—curse or blessing? Int J Epidemiol. 2006;35:814–816.
37. Kriska AM, Bayles C, Cauley JA, et al. A randomized exercise trial in older women: increased activity over two years and the factors associated with compliance. Med Sci Sports Exerc. 1986;18:557–562.
38. Irwin ML, Tworoger SS, Yasui Y, et al. Influence of demographic, physiologic, and psychosocial variables on adherence to a yearlong moderate-intensity exercise trial in postmenopausal women. Prev Med. 2004;39:1080–1086.
39. Courneya KS, Segal RJ, Reid RD, et al. Three independent factors predicted adherence in a randomized controlled trial of resistance exercise training among prostate cancer survivors. J Clin Epidemiol. 2004;57:571–579.
40. Women's Health Initiative Study Group.Dietary adherence in the Women's Health Initiative Dietary Modification Trial. J Am Diet Assoc 2004;104:654–658.
41. Shapiro S. Looking to the 21st century: have we learned from our mistakes, or are we doomed to compound them? Pharmacoepidemiol Drug Saf. 2004;13:257–265.
42. Robins JM. A new approach to causal inference in mortality studies with sustained exposure periods-application to control of the healthy worker survivor effect. Math Model. 1986;7:1393–1512.
43. Robins JM. Addendum to “A new approach to causal inference in mortality studies with sustained exposure periods-application to control of the healthy worker survivor effect”. Comput Math Appl. 1987;14:923–945.
44. Sterne JA, Hernan MA, Ledergerber B, et al. Long-term effectiveness of potent antiretroviral therapy in preventing AIDS and death: a prospective cohort study. Lancet. 2005;366:378–384.
45. Hernan MA. A definition of causal effect for epidemiological research. J Epidemiol Community Health. 2004;58:265–271.
46. Ioannidis JPA. Evolution and translation of research findings: from bench to where? PLoS Clin Trials. 2006;1:e36.
47. Trichopoulou A, Costacou T, Bamia C, et al. Adherence to a Mediterranean diet and survival in a Greek population. N Engl J Med. 2003;348:2599–2608.
48. Piantadosi S. Clinical Trials: A Methodological Approach. Wiley; 1997.
49. DeMets DL, Lan KK. Interim analysis: the alpha spending function approach. Stat Med. 1994;13:1341–1352; discussion 1353–1356.
50. Schulz KF, Chalmers I, Hayes RJ, et al. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408–412.
51. Balk EM, Bonis PA, Moskowitz H, et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA. 2002;287:2973–2982.
52. Bent S, Kene C, Shinohara K, et al. Saw palmetto for benign prostatic hyperplasia. N Engl J Med. 2006;354:557–566.
53. Ness AR, Maynard M, Frankel S, et al. Diet in childhood and adult cardiovascular and all cause mortality: the Boyd Orr cohort. Heart. 2005;91:894–898.
54. Frankel S, Gunnell DJ, Peters TJ, et al. Childhood energy intake and adult mortality from cancer: the Boyd Orr Cohort Study. BMJ. 1998;316:499–504.
© 2008 Lippincott Williams & Wilkins, Inc.