The Transatlantic Active Surveillance on Cardiovascular Safety of Nuvaring study was a large, multinational, controlled, prospective, observational, active surveillance study that studied the cardiovascular safety of a contraceptive vaginal ring that releases 15 mcg ethinylestradiol and 120 mcg etonogestrel daily. After insertion, it remains in the vagina for 21 days, followed by 7 ring-free days before a new ring is inserted. The mode of action is based primarily on ovulation inhibition and the contraceptive efficacy of the vaginal ring is similar to that of combined oral contraceptive pills (OCPs).
The cohort study started in September 2007 and last follow-up activities were finished in March 2012. The study followed-up new users of the vaginal ring and new users of marketed combined OCPs using a noninterference approach to provide standardized, comprehensive, and reliable information regarding these treatments in routine clinical practice. When the Transatlantic Active Surveillance on Cardiovascular Safety of Nuvaring study was planned, clinical experience at that time suggested that serious clinical outcomes associated with vaginal ring use were rare. Nevertheless, because venous thromboembolism and, to a lesser extent, arterial thromboembolism are the most relevant adverse clinical outcomes linked to the use of hormonal contraceptives that contain both estrogen and progestin, a sufficiently large study to detect such rare events was necessary. Thus, the main clinical outcomes of interest for the short-term and long-term follow-up were venous thromboembolism and arterial thromboembolism. Differentiating between the inherent background population risk and a potential incremental risk for outcomes of interest attributable to treatment is often challenging. Active safety surveillance using valid epidemiologic study designs has been proven to be a valid method to approach this endeavor.1,2
MATERIALS AND METHODS
The methodology of the Transatlantic Active Surveillance on Cardiovascular Safety of Nuvaring study is similar to that of the European Active Surveillance study of OCPs, which is described elsewhere.3 Therefore, methodologic details regarding questionnaires, follow-up procedures, and blinded adjudication are presented succinctly.
The primary objective of this study was to investigate and compare the risks of short-term and long-term use of the vaginal ring with those of marketed combined OCPs in a study population representative of the actual users of these preparations. The main clinical outcomes of interest for the short-term and long-term follow-up were venous and arterial thromboembolism, deep venous thrombosis, pulmonary embolism, acute myocardial infarction, and cerebrovascular accidents, with a particular focus on venous thromboembolism. Other objectives of the study included the drug utilization pattern and contraceptive effectiveness of the vaginal ring and combined OCPs. This study focuses on cardiovascular outcomes. Planning, conduct, and evaluation of the study were supervised by an independent Safety Monitoring and Advisory Council. The international members of the Council (epidemiologists, cardiologists, and gynecologists) were selected by the chairman (Samuel Shapiro, Cape Town, South Africa) and the principal investigator at the Berlin Center for Epidemiology and Health Research. The present publication was reviewed by the Council; all requested changes were implemented. Primary ethical approval was provided by the ethical committee of the physician's association in Berlin, Germany (Ethik-Kommission der Ärztekammer Berlin). The study is registered in the public clinical trials registry of the United States National Library of Medicine with the registration number NCT00524771.
Recruitment of the cohort members was conducted via a network of physicians who prescribe hormonal contraceptives. The combined cohort was planned to include more than 33,000 women recruited in the United States and in five European countries (Austria, France, Germany, Italy, and Russia). Study participants were women who had a new prescription for the vaginal ring or a combined OCP. The participating physicians discussed the study with the patient only after one of these preparations had been prescribed. Study centers that did not comply with this requirement were excluded. For every vaginal ring user recruited, the physician recruited the next combined OCP user who was willing to participate in the study. All women who were eligible for recruitment were asked to participate.
Participating women could have been starters (first-ever use of hormonal contraceptives), switchers (switch from one hormonal contraceptive to another), or restarters (restarted hormonal contraception after a break of at least 4 weeks). All women who fulfilled these criteria and signed the informed consent and data privacy form were enrolled in the study. More specific inclusion or exclusion criteria were not applied because of the noninterference approach of the study design. The objective was to avoid influencing the prescribing behavior while at the same time making significant efforts to ensure standardized, comprehensive, and reliable documentation of all baseline characteristics and adverse events during the follow-up period.3
Baseline data were recorded on a questionnaire that addressed the participant's state of health and potential risk factors. Participants provided their medical history, including medication history and history of hormonal contraceptive use. They also provided their addresses and telephone numbers, those of relatives or friends who could serve as back-up contacts, and those of their primary care physicians and gynecologists.3 Baseline questionnaires were completed in the physician's offices and checked by the physicians or their coworkers. Follow-up assessments for each woman were scheduled at 6 and 12 months after study entry, and then every year for up to 4 years. The self-administered follow-up questionnaires addressed the occurrence of adverse events. Detailed reasons for discontinuing hormonal contraception or switching to another preparation were requested if applicable. The questionnaires were reviewed for completeness, plausibility, and consistency of the responses. Missing or inconsistent information was clarified directly with the women by telephone. A low loss to follow-up rate was essential for the validity of the study. To minimize loss to follow-up, a comprehensive follow-up process was established, which is described elsewhere.3 Study participants received a small compensation at each follow-up for returning the questionnaires. Follow-up questionnaires that contained information about serious adverse events were immediately given to the medical reviewer group at the Berlin Center for Epidemiology and Health Research.3 All group members were medical doctors specializing in epidemiology, drug safety, and internal medicine. In case of unclear or missing information, the women were contacted by telephone, e-mail, or other means. For many events, it was necessary to contact the diagnosing or treating physician for clarification and validation of the information received from the patient.3 These physicians were compensated for the time needed to provide the requested information and medical records. All serious adverse events were classified as confirmed or not confirmed. Events that were confirmed by diagnostic measures with high specificity (eg, phlebography for deep venous thrombosis or cerebral magnetic resonance imaging for cerebrovascular accidents) or a clinical diagnosis supported by a diagnostic test with low specificity (such as D-dimer for venous thromboembolism) were considered confirmed. Events were considered not confirmed if the diagnosis reported by the patient was excluded by diagnostic measures, if a different medical condition was diagnosed by the attending physician, or if the reporting woman did not contact a health professional to clarify her symptoms and no diagnostic measures were performed that could have clarified the diagnosis.3
For the purpose of continuously monitoring safety data during the study, classification of reported serious adverse events was performed by the investigators at the Berlin Center for Epidemiology and Health Research. For the final analysis, classification of the primary outcomes of interest—venous and arterial thromboembolism—was verified by independent blinded adjudication. To minimize classification bias, the decisions made by the investigators were reassessed by three independent medical experts specializing in radiology and nuclear medicine, cardiology, internal medicine, and vascular diseases. These specialists reviewed all available information regarding the reported event. Brand names, dose, regimen, and composition of the hormonal contraceptives used by the study participants were rendered anonymous for this process.3
The final analyses included both an as-treated and an intention to treat analysis. The safety conclusions of the study are based on the as-treated analyses because the intention to treat approach potentially dilutes differences between treatments.3 In this study, however, conclusions based on intention to treat results did not differ from the conclusions based on as-treated results. Therefore, only the as-treated results are reported. Cox regression models were used for inferential statistics. The analyses were performed in accordance with the statistical analysis plan, which was approved by the Safety Monitoring and Advisory Council before the first inferential analysis.
Based on the rather small number of outcomes, adjustment for potential confounding was based on a priori defined expert models with a limited number of well-established covariates. For venous thromboembolism, the models included age, body mass index (BMI, calculated as weight (kg)/[height (m)]2), duration of current hormonal contraceptive use, and family history of venous thromboembolism, whereas for arterial thromboembolism the models included age, BMI, smoking, and treated hypertension. In addition, the effect of several predefined prognostic factors such as educational level, geographical region, user status (starter, switcher, restarter) and surgical interventions were evaluated. This approach was only performed for exploratory reasons. The results were almost identical to those of the expert model. Therefore, only the results of the expert model are reported.
The analyses focused on comparisons among the two primary cohorts—the vaginal ring and combined OCPs. A priori planned subanalyses also included the comparison of the vaginal ring with combined OCPs without desogestrel or gestodene cohorts because the effect of these progestogens on the venous thromboembolic risk of combined OCPs is still undergoing scientific debate.2 After the start of the Transatlantic Active Surveillance on Cardiovascular Safety of Nuvaring study, the results of a Danish cohort study4 suggested an increased risk for drospirenone-containing combined OCPs users. Therefore, the Safety Monitoring and Advisory Council recommended an additional post hoc analysis of the venous thromboembolic risk associated with combined OCPs without desogestrel, gestodene, or drospirenone. Overall, the study participants were divided into the following five subcohorts for the analysis: vaginal ring; combined OCPs; combined OCPs without desogestrel or gestodene; combined OCP without desogestrel, gestodene, or drospirenone; and one nonuser cohort (women who stopped hormonal contraception after study entry). In addition, regulatory authorities suggested a post hoc comparison of the venous thromboembolic risk associated with the vaginal ring and levonorgestrel-containing combined OCPs. The primary outcome of interest was the venous thromboembolic hazard ratios (HRs) of users of the vaginal ring and combined OCPs. The null hypothesis to be tested was HR of 2 or more (ie, the venous thromboembolic HR for the vaginal ring compared with combined OCPs is 2 or more). The alternative hypothesis was HR less than 2. All analyses were performed with the statistical software packages SAS 9 and StataES8.
Sample size calculations showed that 33,000 patients with a total observation time of approximately 70,000 woman-years should be sufficient to exclude a twofold risk of venous thromboembolism for vaginal ring users compared with users of combined OCPs. These calculations were based on a venous thromboembolic incidence in the general combined OCP user population of 9.1 events per 10,000 woman-years3 and a power of 90%. A noninferiority test for two exponential survival curves was chosen to construct the sample size estimate. Exposure periods of up to 4 years were analyzed to ensure that the venous thromboembolic risk estimate was representative for routine clinical hormonal contraceptive use. In addition, a subanalysis of the relative risk during different exposure periods (6 months or less, 7–12 months, more than 12 months) was conducted.
A total of 34,100 women were enrolled by 1,661 active study centers. Overall, 805 of these 34,100 women (2.4%) had to be excluded because they were enrolled two or more times by one or more study centers, continued to use their previous hormonal contraceptive (long-term user), or did not start using the vaginal ring or combined OCPs after study entry. The remaining 33,295 quality-controlled computerized datasets from the women with baseline information (one per woman) were analyzed. After study entry, 16,864 women used the vaginal ring, 16,431 used combined OCPs, and 13,811 used combined OCPs without desogestrel or gestodene. These data are summarized in Table 1.
Table 2 depicts the predefined user subcohorts (ie, vaginal ring, combined OCPs, and combined OCPs without desogestrel or gestodene), the number of women with baseline information, exposure, mean age with standard deviation, mean weight with standard deviation, and mean BMI with standard deviation. A comparison of the vaginal ring cohort with the combined OCPs cohort showed that vaginal ring users had somewhat shorter exposure than the users of combined OCPs. Detailed analyses revealed that two factors accounted for this difference. Some women (particularly obese women) stopped vaginal ring use because of problems inserting the ring, and there was a heterogeneous reimbursement situation for the vaginal ring in the United States. Reimbursement within individual health plans changed over time and reimbursement differed between different health plans.
Mean age was slightly older for the vaginal ring cohort compared with the combined OCPs cohort (approximately 1-year difference). The age distribution—as indicated by the minimum, 5th, 25th, 50th, 75th, and 95th percentiles, and the maximum values—corresponds to the typical age profile of hormonal contraceptive users. Mean weight and mean BMI also were slightly higher in the vaginal ring cohort compared with the combined OCP subcohorts. However, a geographical comparison showed that the differences between Europe and the United States were much more pronounced than the differences between the cohorts. The mean BMIs in the United States for the vaginal ring users and combined OCPs users were 26.8 and 26.3, respectively; the corresponding values in Europe were 22.9 and 22.7. The slight age difference between the cohorts is also reflected in the proportion of nulliparous women: 41% compared with 50% for the vaginal ring users and combined OCPs users, respectively. Regarding other gynecologic history parameters, the cohorts were similar, eg, 12.9 years compared with 12.9 years for age at menarche and 23.3 years compared with 23.3 years for age at first delivery.
The distribution of prognostic factors for cardiovascular outcomes of interest as well as the medical history of selected diseases are shown in Table 3. Major differences between the cohorts were not found for most of the risk factors examined. The proportion of current smokers was slightly higher among vaginal ring users compared with combined OCP users. However, the difference between geographical regions is more striking. The prevalence values for current smoking in the United States for vaginal ring and combined OCPs users were 16.9% and 14.6%, respectively; the corresponding values in Europe were 31.2% and 29.8%. Regular use of concomitant medication was similar across cohorts, 17% compared with 17% for the vaginal ring and combined OCPs cohorts, respectively. The prevalence in the United States was substantially higher compared with that of Europe (23% compared with 10%). Psychotrophics were the most widely used concomitant medication in the study population (Europe, 1.4%; United States, 11.8%). Educational levels of the study participants were similar across cohorts.
Overall, there were no major differences in baseline risks for the vaginal ring, combined OCPs, and combined OCPs without desogestrel or gestodene subcohorts. The vaginal ring cohort had a slightly higher baseline risk for cardiovascular outcomes compared with the two OCP subcohorts (slightly higher prevalence of family history of venous thromboembolism and obesity; P<.01). However, these differences are too small to have a substantial effect on the risk estimates for venous thromboembolism.
A total of 33,295 study participants were followed-up for 66,489 woman-years of observation. In sum, 965 out of 33,295 women, or 2.9%, were lost to follow-up during the 4-year follow-up period. A comparison of the loss to follow-up rates among cohorts and geographical regions showed only minor differences. The loss to follow-up per cohort was 3.1% for the vaginal ring cohort and was 2.6% combined OCPs cohort, respectively; the loss to follow-up rates for Europe and the United States were 3.2% and 2.6%, respectively.
A total of 57 venous thromboembolic events were observed, with a similar incidence in the vaginal ring and combined OCPs cohorts. The vaginal ring cohort included 19 cases and 8.3 events per 10,000 woman-years, the combined OCPs cohort included 26 cases and 9.2 events per 10,000 woman-years, and the no use cohort (women who stopped hormonal contraception after study entry) included 11 cases and 8.0 events per 10,000 woman-years (Table 4). One venous thromboembolic event occurred in a study participant who had switched from a combined OCP to a contraceptive patch; a meaningful analysis of a subcohort with such sparse data was neither possible nor was it intended. Therefore, a detailed analysis of this subcohort was not performed. Overall, the point estimates of the incidence rates of the main cohorts (vaginal ring, combined OCPs, no use) were similar, with a broad overlap of the 95% confidence intervals (CIs). Five out of 11 venous thromboembolic events in the no use cohort were associated with pregnancy and delivery. Exclusion of these cases resulted in a substantially lower incidence rate of 5.0 events per 10,000 woman-years. The corresponding point estimate of the pregnancy and delivery-related venous thromboembolic incidence rate was 29.0 events per 10,000 woman-years.
The well-established high venous thromboembolic risk during the first months of combined hormonal contraceptive use also was observed in this study. Overall, the venous thromboembolic incidence associated with new hormonal contraceptive use decreased from 13.0 events per 10,000 woman-years during the first 6 months to 5.2 events after the first year of use.
For 13 of the 57 venous thromboembolic events (23%), a pulmonary embolism was observed (vaginal ring cohort, three cases; combined OCPs cohort, eight cases; patch subcohort, one case; no use cohort, one case). The pulmonary embolism incidence rates for the vaginal ring cohort were lower compared with the combined OCPs cohort (vaginal ring 1.3, 95% CI 0.3–1.8; combined OCPs 2.8, 95% CI 1.2–5.6). However, the broad overlap of CIs does not allow robust conclusions.
A Cox regression analysis (Table 4) was performed in accordance with the statistical analysis plan. The crude HR for the vaginal ring cohort compared with that for the combined OCPs cohort was 0.9, with a 95% CI of 0.5 to 1.6. The adjusted HR was 0.8, with a CI of 0.5 to 1.5. Therefore, the null hypothesis (HR more than 2) can be rejected and a twofold higher risk of venous thromboembolism during vaginal ring use compared with combined OCP use can be excluded. Because vaginal ring users had a slightly higher baseline risk for venous thromboembolism compared with the combined OCPs users, adjustment for the predefined prognostic factors resulted in a minor reduction in the adjusted compared with the crude HR. However, this reduction was minimal (ie, approximately 0.1). A sensitivity analysis with other prespecified potential prognostic factors (see Materials and Methods) had no relevant effect on the adjusted HRs (ie, less than 0.1). Also, a stratified analysis of venous thromboembolism per user status (starters, switchers, and restarters of hormonal contraceptives) did not indicate a higher venous thromboembolic risk for the vaginal ring users compared with combined OCPs users for women with a particular user status. The adjusted HR was always 1.0 or less. This also was true for an analysis stratified by three exposure periods (less than 6 months, 6–12 months, more than 12 months). The early treatment discontinuation of obese vaginal ring users had a slight effect on the crude risk estimates for the first six months of exposure. This effect disappeared after adjustment for BMI. Furthermore, the region (United States and Europe) had no substantial effect on the venous thromboembolic incidence rates associated with hormonal contraceptive use (United States, 8.9 events per 10,000 woman-years; Europe 8.5 events per 10,000 woman-years). None of the 40 women with a personal history of venous thromboembolism at study entry (Table 3) had a venous thromboembolic event after start of hormonal contraception. Accordingly, exclusion of these women had no effect on the results.
In the validation process for venous thromboembolism, six potential venous thromboembolic events were identified that did not represent venous thromboembolism according to the criteria described. Because these cases were unanimously classified by the blinded adjudicators as not being venous thromboembolic events, the risk of misclassification seems low. To assess possible error, an additional evaluation was performed in which potential venous thromboembolic events were combined with confirmed venous thromboembolic events. This subanalysis yielded only slight deviations from the analysis of confirmed venous thromboembolic events (adjusted HR 0.9; 95% CI 0.5–1.5).
The venous thromboembolic incidence rates for the a priori subcohort combined OCPs without desogestrel or gestodene and for the post hoc defined subcohorts combined OCPs without desogestrel, gestodene, or drospirenone and levonorgestrel-containing combined OCPs were similar to those of the vaginal ring cohort. Crude and adjusted HRs were close to unity (Table 4).
A total of 17 arterial thromboembolic events (Table 4) were observed in the study (six acute myocardial infarctions, five ischemic strokes, five transient ischemic attacks, and one complete thrombosis of a peripheral artery). The arterial thromboembolic events among the subcohorts were as follows: vaginal ring, five cases; combined OCPs, eight cases; combined OCPs without desogestrel or gestodene, six cases; other hormonal contraceptives, one case; and no use, three cases. This corresponds to arterial thromboembolic incidences of 2.2 events per 10,000 woman-years for the vaginal ring cohort and of 2.8, 2.5, 6.2, and 2.2 for the combined OCPs, combined OCPs without desogestrel or gestodene, patch, and no use subcohorts, respectively. Again, a meaningful analysis of the patch subcohort was neither possible nor intended. Therefore, a detailed analysis of this cohort was not performed.
Cox regression analysis was performed in accordance with the statistical analysis plan, ie, HRs were calculated if a minimum of five confirmed events were available in each of the comparison groups. Arterial thromboembolic HRs could be calculated for vaginal ring cohort compared with combined OCPs cohort and for vaginal ring cohort compared with combined OCPs without desogestrel or gestodene cohort. The adjusted HRs for the vaginal ring users compared with combined OCPs users and vaginal ring users compared with combined OCPs without desogestrel or gestodene users were 0.7 and 0.8, respectively. The corresponding 95% upper confidence limits were 2.3 and 2.6. Therefore, a threefold higher risk of arterial thromboembolism during vaginal ring use compared with combined OCP use can be excluded. A sensitivity analysis with other prespecified potential prognostic factors (see Materials and Methods) had no relevant effect on the adjusted HRs (ie, less than 0.1).
The incidence rates for venous and arterial thromboembolism were similar for all subcohorts, and crude and adjusted HRs indicated similar risk levels for these subcohorts.
In nonexperimental studies like the Transatlantic Active Surveillance on Cardiovascular Safety of Nuvaring study, the possibility of bias and residual confounding can never be entirely eliminated, and the ability to infer causation is correspondingly limited.5 Valid information regarding potential sources of confounding and sophisticated statistical and epidemiologic methodology help to reduce the effect of bias and residual confounding.6 However, the difficulty remains unresolved when all that exists is a weak association.7,8 Relative risk estimates that are close to unity may not allow differentiation between causation, bias, and confounding.9,10 In general, it is difficult to interpret a relative risk of two or less in observational research.11,12 Neither largely nor slightly increased relative risks were found in this study. Given the discussed limitations, these findings may exclude large, but not small, relative risks.
In our judgment, selection bias was probably not a major issue in this study because inpatients and outpatients were included and the demographic characteristics of the participants are representative for adult hormonal contraceptive users.3 Also, misclassification bias probably had no substantial effect on the results because precise information regarding the exposure and the outcomes of interest were available. In addition, reliable information regarding duration of current use was available. Accordingly, the well-known increased venous thromboembolic risk during the first months of combined hormonal contraceptive use2,13,14 could be reproduced in this study. Furthermore, a low loss to follow-up rate of 2.9% was found in this study. Serious adverse events could lead to a break in contact with study participants. An advantage of the Transatlantic Active Surveillance on Cardiovascular Safety of Nuvaring study design, however, is that contact was not lost if the study participants changed their physicians.
In contrast, it was impossible to exclude diagnostic bias. Venous thromboembolism can present as dramatic symptoms, unspecific mild symptoms, or without any clinical symptoms.15–17 A high awareness of potential cardiovascular risks of combined hormonal contraceptive use might have led to more diagnostic procedures and, therefore, to more detected venous thromboembolic events. However, we assume that this potential bias had no differential effect on the cohorts of interest or led to an overestimate of the relative risk of the rather new vaginal ring.
A strength of the study was the availability of information regarding many important prognostic factors for the outcomes of interest. Nevertheless, because of the noninterventional character of the study, information regarding specific gene mutations was only available for cases of venous thromboembolism but not for the majority of study participants. This limitation was mitigated through information regarding family history of venous thromboembolism, which has a higher predictive value for venous thromboembolism compared with gene mutations.18
The fact that the HRs remained close to unity if combined OCPs containing desogestrel, gestodene, or drospirenone were excluded from the analysis shows that eventual differences between the cohort were not diluted by the inclusion of combined OCPs that are potentially associated with an increased risk of venous thromboembolism. This is also supported by the comparison of the vaginal ring and levonorgestrel-containing combined OCPs cohorts. Robust conclusions cannot be drawn from these comparisons because the study was not powered for these post hoc analyses. However, our data do not indicate a substantial difference in venous thromboembolic risk associated with these preparations.
Our results regarding venous thromboembolism are consistent with the results of a recently published United States study.19 In contrast, a Danish cohort study reported an approximately twofold increase in venous thromboembolic risk for the vaginal ring cohort compared with combined OCPs containing levonorgestrel or norgestimate cohort.20 This study linked several national registers in Denmark. Advantages and disadvantages of this methodologic approach compared with the methodology used in this study have been discussed extensively.1,21–24 The Danish register studies are undoubtedly larger than field studies like our study. However, the CIs in large observational studies are misleading because their calculation only takes into consideration random variation of data. It ignores the systematic errors, the biases, and confounders that will almost invariably overwhelm the statistical variation.11 In addition, specific limitations of the Danish register studies—such as sparse information regarding relevant prognostic factors and limited validity of information regarding exposure and clinical outcomes25—increase the effect of bias and confounding compared with the Transatlantic Active Surveillance on Cardiovascular Safety of Nuvaring study. Therefore, the Danish register study does not falsify our results.
In our judgment, the results of this study are valid within the general limitations of observational research. We conclude that vaginal ring use and combined OCP use are associated with a similar risk of venous and arterial thromboembolisms during routine clinical use.
1. Grimes DA. Epidemiologic research using administrative databases; author's reply. Am J Obstet Gynecol 2011;117:411.
2. Rabe T, Luxembourg B, Ludwig M, Dinger J, Bauersachs R, Rott H, et al.. A statement from the German Society for Gynecological Endocrinology and Reproductive Medicine (DGGEF e.V.) and the Professional Association of German Gynaecologists. J Reproduktionsmed Endokrinol 2011;8:126–67.
3. Dinger JC, Heinemann LA, Kühl-Habich D. The safety of a drospirenone-containing oral contraceptive: final results from the European Active Surveillance study on Oral Contraceptives based on 142,475 women-years of observation. Contraception 2007;75:344–54.
4. Lidegaard Ø, Løkkegaard E, Svendsen AL, Agger C. Hormonal contraception and risk of venous thromboembolism: national follow-up study. BMJ 2009;339:b2890.
5. Susser M. What is a cause and how do we know one? A grammar for pragmatic epidemiology. Am J Epidemiol 1991;133:635–48.
6. Rothman KJ, Poole C. A strengthening programme for weak associations. Int J Epidemiol 1988;17:955–9.
7. Khoury MJ, James LM, Flanders WD, Erickson JD. Interpretation of recurring weak associations obtained from epidemiologic studies of suspected human teratogens. Teratology 1992;46:69–77.
8. Shapiro S. Bias in the evaluation of low-magnitude associations: an empirical perspective. Am J Epidemiol 2000;151:939–45.
9. Shapiro S. Causation, bias and confounding: a hitchhiker's guide to the epidemiological galaxy. Part 2. Principles of causality in epidemiological research: confounding, effect modification and strength of association. J Fam Plann Reprod Health Care 2008;34:185–90.
10. Shapiro S. Causation, bias and confounding: a hitchhiker's guide to the epidemiological galaxy. Part 3: principles of causality in epidemiological research: statistical stability, dose- and duration response effects, internal and external consistency, analogy and biological plausibility. J Fam Plann Reprod Health Care 2008;34:261–4.
11. Taubes G. Epidemiology faces its limits. Science 1995;269:164–9.
12. Hill AB. The environment and disease: association or causation? Proc R Soc Med 1965;58:295–300.
13. Suissa S, Blais L, Spitzer WO, Cusson J, Lewis M, Heinemann L. First-time use of newer oral contraceptives and the risk of venous thromboembolism. Contraception 1997;56:141–6.
14. Suissa S, Spitzer WO, Rainville B, Cusson J, Lewis M, Heinemann L. Recurrent use of newer oral contraceptives and the risk of venous thromboembolism. Hum Reprod 2000;15:817–21.
15. Scurr JH, Machin SJ, Bailey-King S, Mackie IJ, McDonald S, Smith PD. Frequency and prevention of symptomless deep-vein thrombosis in long-haul flights: a randomised trial. Lancet 2001;357:1485–9.
16. Belcaro C, Geroulakos G, Nicolaides AN, Myers KA, Winford M. Venous thromboembolism from air travel: the LONFLIT Study. Angiology 2001;52:369–74.
17. Schwarz T, Siegert G, Oettler W, Halbritter K, Beyer J, Frommhold R, et al.. Venous thrombosis following long-haul flights. Arch Intern Med 2003;163:2759–64.
18. Bezemer ID, van der Meer FJ, Eikenboom JC, Rosendaal FR, Doggen CJ. The value of family history as a risk indicator for venous thrombosis. Arch Intern Med 2009;169:610–5.
19. Sidney S, Cheetham TC, Connell FA, Ouellet-Hellstrom R, Graham DJ, Davis D, et al.. Recent combined hormonal contraceptives (CHCs) and the risk of thromboembolism and other cardiovascular events in new users. Contraception 2013;87:93–100.
20. Lidegaard Ø, Nielsen LH, Skovlund CW, Løkkegaard E. Venous thrombosis in users of non-oral hormonal contraception: follow-up study, Denmark 2001-10. BMJ 2012;344:e2990.
21. Dinger J, Shapiro S. Combined oral contraceptives, venous thromboembolism, and the problem of interpreting large but incomplete datasets. J Fam Plann Reprod Health Care 2012;38:2–6.
22. Jensen JT, Trussell J. Communicating risk: does scientific debate compromise safety? Contraception 2012;86:327–9.
23. Reid RL, Westhoff C, Mansour D, de Vries C, Verhaeghe J, Boschitsch E, et al.. Oral contraceptives and venous thromboembolism consensus opinion from an international workshop held in Berlin, Germany in December 2009. J Fam Plann Reprod Health Care 2010;36:117–22.
24. Grimes DA. Epidemiologic research using administrative databases: garbage in, garbage out. Obstet Gynecol 2010;116:1018–9.
25. Severinsen MT, Kristensen SR, Overvad K, Dethlefsen C, Tjønneland A, Johnsen SP. Venous thromboembolism discharge diagnoses in the Danish National Patient Registry should be used with caution. J Clin Epidemiol 2010;63:223–8.