Secondary Logo

Journal Logo

Environmental and Occupational Epidemiology

Quantitative Bias Analysis for Collaborative Science

Weuve, Jennifera; Sagiv, Sharon K.b; Fox, Matthew P.a,c

Author Information
doi: 10.1097/EDE.0000000000000875

An investigation conducted by Forns and colleagues1 asks whether prenatal exposure to air pollution increases a child’s risk of developing attention-deficit/hyperactivity disorder (ADHD). ADHD is the most common developmental disorder, affecting an estimated 5%–7% of children2 (possibly more3), and confers a substantial burden in lost educational achievement,4 costs to individuals and families,5 and risk for developing other psychiatric conditions, including anxiety and substance use disorders.6 Exposure to air pollution is so widespread that if prenatal exposure to air pollution elevates ADHD risk, even to a small degree, this could represent a meaningful target for reducing the population burden of ADHD.

To address this question, Forns and colleagues assembled data from 13 cohorts encompassing 2,801 ADHD cases. Most of the summary odds ratios (ORs) were less than, although close to, 1. For example, per 10-μg/m3 increment in NO2 exposure, the summary OR for ADHD traits within the borderline or clinical range was 0.95 (95% CI = 0.89, 1.01). This OR had a confidence interval width (ratio of upper to lower limits) of 1.13, an improvement over the precision of the cohort-specific estimates (range in confidence interval widths, 1.2–13). The absolute magnitude of this association is similar to that of air pollution’s adverse association with other health outcomes, such as adverse birth and cardiometabolic outcomes, with relative risks ranging from 1.05 to 1.10.7,8 Although similar in magnitude, the association in the present study is consistent with small protective effects. Notably, although meta-analyzing these cohorts enhanced the precision of the estimate, it did not remove the systematic error in the contributing cohort estimates and likely left systematic error as the dominant source of study error. One way to explore the influence of systematic error on study results is through quantitative bias analysis, in which we quantify the influence of differential selection, confounding, and measurement error on the direction and magnitude of an association. Admirably, Forns and colleagues made efforts—beyond what many collaborative projects undertake—to adjust for confounding, selection bias, and information bias, but important questions remain.


As sample size increases, random error decreases, and concerns about systematic error dominate. Quantifying the influence of systematic bias is preferable to discussing it in qualitative terms, especially when the collaborative, institutional, and inferential stakes are high.9 We used the study by Forns and colleagues to outline how quantitative bias analysis can be applied to collaborative science projects. Our objective was to quantify the conditions necessary to yield the observed cohort-specific effect estimates in scenarios when (1) air pollution has no effect on ADHD risk or (2) air pollution increases the risk of ADHD. We examined three classes of bias: differential misclassification, differential selection, and uncontrolled confounding. Where possible, we based this analysis on reported data and putative mechanisms of bias specific to the subject matter. The eAppendix ( provides details on the analyses and a spreadsheet showing estimated cell counts.

Differential Misclassification

ADHD is highly prone to misclassification. With no established biomarker, diagnosis relies on subjective report of symptoms by parents and teachers. Furthermore, ADHD is an extremely labile and heterogeneous condition10,11 with symptoms that are viewed through a sociocultural lens.12 Forns and colleagues quantified the influence of nondifferential misclassification of ADHD in the 13-component cohorts on the observed ADHD OR using published reports of the ADHD measures’ sensitivity and specificity. The misclassification-adjusted results suggested an even stronger protective effect of NO2 exposure on ADHD risk. This is not surprising as nondifferential misclassification of a dichotomous exposure leads to the expectation of bias toward the null; adjustment should increase the magnitude of the already protective association.

Whereas nondifferential misclassification is an unlikely explanation for the protective association observed in the study by Forns et al.,1 if the effect is truly null or harmful, differential misclassification where misclassification of ADHD is related to exposure or its correlates is a possible explanation. One potential source of differential misclassification is diagnostic instability. In studies using in-depth multimodal clinical assessments, about 30% of children diagnosed with ADHD no longer meet diagnostic criteria after a few years.10,13 Socioeconomic disadvantage is a predictor of stability10 such that ADHD diagnoses may be less stable among more socioeconomically advantaged children. Socioeconomic advantage often corresponds to lower air pollution exposure. Therefore, differential misclassification could operate via disproportionate false positives (lower specificity) among those with lower exposure. Differential misclassification could also arise from differences in accuracy, by race and ethnicity, of parental report of ADHD symptoms.14,15 If predictors of higher air pollution exposure are related to diminished perception of ADHD traits, it would translate to disproportionately more false negatives (lower sensitivity) among children with higher exposure.

We explored in the study by Forns et al1 the extent to which ADHD misclassification could yield the observed findings from a true OR that is either null (no effect of NO2) or at least 1.2 (an adverse effect).

When diagnostic specificity differed by exposure, even modestly, observed ORs dramatically differed from the true OR (eTable 1; Furthermore, when specificity was lower (i.e., false positives more common) among the low-exposed group, consistent with the diagnostic instability mechanism, observed ORs were biased downward (from a true OR of 1.0 or larger). For example, as described by Forns et al,1 the Danish National Birth Cohort (DNBC) study used the Strengths and Difficulties Questionnaire (SDQ) to assess ADHD. Published estimates of the sensitivity and specificity of this test are 0.49 and 0.96 (eTable 19; in the study by Forns et al1). If, in the DNBC study, the SDQ exhibited differential specificity, a true OR of 0.99 could appear as an observed OR of 0.89 if the test specificity was 0.96 among high-exposed children and 0.95 among low-exposed children. If the specificity among high-exposed children was 0.97, the observed OR could have originated from a true OR of 1.21. Thus, the result is sensitive to small differences in specificity of outcome classification. In the 11 cohorts with unadjusted ORs <1.0, the observed ORs would correspond to bias-adjusted ORs ≥1.0 when the specificity among the low-exposed children was lower than among high-exposed children, by as little as 1–4 percentage points.

Differential Selection

We also explored the potential influence of differential selection. Many participants from the target cohorts, typically more than 40%, were excluded owing to attrition or missing data on exposure or ADHD. Retention was often a function of risk factors for exposure and ADHD. To adjust for selection bias, the authors applied inverse probability-of-selection weights to their analyses and resulting estimates attenuated toward the null. However, concerns about selection bias may remain. For example, downward bias from a truly null or adverse relation could emerge if there was disproportionately less participation from children with ADHD who were highly exposed to air pollution.16 Studies have documented that mothers of children with ADHD or more ADHD symptoms are less likely than mothers of children without ADHD to enroll in research studies and continue participating.17,18 Participation may also depend on factors related to exposure, particularly socioeconomic disadvantage.19–21

Under scenarios in which the (1) true OR = 1.0 and (2) true OR ≈1.2, we quantified conditions22 for each cohort in which selection bias could result in the observed OR. With only small reductions in selection among high-exposed children with ADHD, several cohorts’ unadjusted ORs <1 became selection-adjusted ORs ≈1 (eTable 2;

Uncontrolled Confounding

The third bias we evaluated was uncontrolled confounding. That is, conditional on the covariates in the analyses, was it plausible that children at all levels of exposure shared the same ADHD risk, except for their specific air pollution exposures? The investigators adjusted for an extensive suite of covariates. Fully adjusted summary estimates (without selection-weighting) were strikingly similar to estimates adjusted for just a few factors. Whereas measured confounding was minimal, uncontrolled confounding potentially remains.

To quantify the conditions under which confounding could result in the observed ADHD OR, for two underlying scenarios (when the true OR per 10-μg/m3 increment in NO2 is 1.0 and when the true OR is 1.2), we computed the “E-value” for each cohort using multivariable-adjusted and selection-weighted ORs.23

Under the scenario of no effect, the E-values for all but one cohort were 1.20 or larger (eTable 3; This means that the minimum exposure–confounder and confounder–outcome risk ratios would need to be at least 1.20 (or <0.83) to fully account for the observed OR, after adjusting for measured confounders. For observed ORs <1, this confounder would have to be associated with NO2 exposure and ADHD risk, but in opposite directions. For example, if the true OR was 1.0, the adjusted OR of 0.84 from the ABCD study could be explained by an unmeasured factor that was at least 70% more common per 10-μg/m3 increment in NO2 exposure (RRexposure-confounder = 1.70) and reduced the risk of ADHD by at least 41% (RRconfounder-ADHD = 0.59). In settings where the observed ORs are >1, the two confounder associations would have to point in the same direction. Socioeconomic disadvantage is an example of a confounder that could meet that requirement.


Using simple, publicly accessible quantitative bias analysis tools, we quantified how an association of air pollution exposure with ADHD risk, produced from 13 cohorts, could be susceptible to bias from differential misclassification of ADHD, differential selection, and uncontrolled confounding—in spite of its 2,801 cases and precision. These findings indicate that if specificity in ADHD classification varied by exposure, even slightly so, this could fully account for several cohort-specific crude ORs <1.0. Less success in retaining highly exposed participants with ADHD, compared with other participants, could also downwardly bias ORs. Finally, we found that for most cohorts, the minimum confounder–exposure and confounder–outcome associations would need to be modest or larger to fully account for the results.

To understand how likely these biases are to explain the results, we need to ask whether the conditions proposed to be producing the bias are realistic. Answering this question requires insight into the cohorts, tendencies of children (via their mothers) of different exposure levels and different ADHD risks to remain involved in a study, and performance of the ADHD measures across a variety of settings and child characteristics. Critically, the biases—their types, their sources, and their severity—could vary by cohort and should be evaluated at that level. One key element of bias in the cohort-specific estimates of the air pollution–ADHD association is socioeconomic disadvantage. Socioeconomic disadvantage is associated with higher ADHD risk, diagnostic stability, and lower likelihood of continuing in a research study. By contrast, socioeconomic disadvantage does not appear to be associated with higher exposure to air pollutants in all European settings.24 For example, air pollution exposure follows an inverse socioeconomic gradient in Valencia, Spain,25 but the association is reversed in Rome.26 In other locations, the association depends on the socioeconomic index, the pollutant, or whether the area is urban or rural.24,27 Where exposure tracks directly with socioeconomic disadvantage, confounding from unmeasured or mismeasured socioeconomic disadvantage could bias findings upward. But where the exposure-disadvantage association runs in the opposite direction, the ensuing confounding bias would push results downward. Understanding the combined interplay of socioeconomic disadvantage with ADHD classification, selection, and exposure in each cohort is clearly important to appreciating the possibility of bias in each setting.

This quantitative bias analysis had limitations, owing principally to our lack of access to the full data. Ideally, the results of each component analysis should be combined in a multiple bias analysis, with the outputs of the misclassification analysis informing the selection analysis, and so on.28 A coordinated analysis can be especially helpful for quantifying the extent to which countervailing biases might offset each other. We also made simplifying assumptions, such as estimating crude data counts and dichotomizing NO2 exposure (except for the confounding analysis), which could itself induce bias. We ignored differential errors in measuring exposure. Our unfamiliarity with the cohorts meant that we had less insight on the most plausible mechanisms of these biases, if they existed. Finally, we focused on the individual cohorts’ estimates. Further analysis might examine how bias in specific cohort estimates could influence the summary OR.

Quantifying systematic bias, in addition to random error, is relatively simple with methods that are accessible and freely available. Coupled with researchers’ substantive knowledge of mechanisms for these biases, quantitative bias analysis offers collaborative study initiatives tools to explore systematic error, allowing for better inference.


1. Forns J, Sunyer J, García-Esteban R, et al. Air pollution exposure during pregnancy and childhood behavioral disorder symptoms in eight European cohort studies. Epidemiology. 2018;25:618626.
2. Willcutt EG. The prevalence of DSM-IV attention-deficit/hyperactivity disorder: a meta-analytic review. Neurotherapeutics. 2012;9:490499.
3. Rowland AS, Skipper BJ, Umbach DM, et al. The prevalence of ADHD in a population-based sample. J Atten Disord. 2015;19:741754.
4. Loe IM, Feldman HM. Academic and educational outcomes of children with ADHD. J Pediatr Psychol. 2007;32:643654.
5. Le HH, Hodgkins P, Postma MJ, et al. Economic impact of childhood/adolescent ADHD in a European setting: the Netherlands as a reference case. Eur Child Adolesc Psychiatry. 2014;23:587598.
6. Spencer TJ, Biederman J, Mick E. Attention-deficit/hyperactivity disorder: diagnosis, lifespan, comorbidities, and neurobiology. J Pediatr Psychol. 2007;32:631642.
7. Brauer M, Lencar C, Tamburic L, Koehoorn M, Demers P, Karr C. A cohort study of traffic-related air pollution impacts on birth outcomes. Environ Health Perspect. 2008;116:680686.
8. Eze IC, Hemkens LG, Bucher HC, et al. Association between ambient air pollution and diabetes mellitus in Europe and North America: systematic review and meta-analysis. Environ Health Perspect. 2015;123:381389.
9. Lash TL, Fox MP, MacLehose RF, Maldonado G, McCandless LC, Greenland S. Good practices for quantitative bias analysis. Int J Epidemiol. 2014;43:19691985.
10. Law EC, Sideridis GD, Prock LA, Sheridan MA. Attention-deficit/hyperactivity disorder in young children: predictors of diagnostic stability. Pediatrics. 2014;133:659667.
11. Sagiv SK, Kalkbrenner AE, Bellinger DC. Of decrements and disorders: assessing impairments in neurodevelopment in prospective studies of environmental toxicant exposures. Environ Health. 2015;14:8.
12. Morrow RL, Garland EJ, Wright JM, Maclure M, Taylor S, Dormuth CR. Influence of relative age on diagnosis and treatment of attention-deficit/hyperactivity disorder in children. CMAJ. 2012;184:755762.
13. Harvey EA, Youngwirth SD, Thakar DA, Errazuriz PA. Predicting attention-deficit/hyperactivity disorder and oppositional defiant disorder from preschool diagnostic assessments. J Consult Clin Psychol. 2009;77:349354.
14. Hillemeier MM, Foster EM, Heinrichs B, Heier B; Conduct Problems Prevention Research Group. Racial differences in parental reports of attention-deficit/hyperactivity disorder behaviors. J Dev Behav Pediatr. 2007;28:353361.
15. Dwivedi KN, Banhatti RG. Attention deficit/hyperactivity disorder and ethnicity. Arch Dis Child. 2005;90(1 suppl):i10i12.
16. Munafò MR, Tilling K, Taylor AE, Evans DM, Davey Smith G. Collider scope: when selection bias can substantially influence observed associations. Int J Epidemiol. 2018;47:226235.
17. Oerbeck B, Overgaard KR, Aspenes ST, et al. ADHD, comorbid disorders and psychosocial functioning: how representative is a child cohort study? Findings from a national patient registry. BMC Psychiatry. 2017;17:23.
18. Ullebø AK, Posserud MB, Heiervang E, Obel C, Gillberg C. Prevalence of the ADHD phenotype in 7- to 9-year-old children: effects of informant, gender and non-participation. Soc Psychiatry Psychiatr Epidemiol. 2012;47:763769.
19. Howe LD, Tilling K, Galobardes B, Lawlor DA. Loss to follow-up in cohort studies: bias in estimates of socioeconomic inequalities. Epidemiology. 2013;24:19.
20. Weyde KV, Krog NH, Oftedal B, et al. Road traffic noise and children’s inattention. Environ Health. 2017;16:127.
21. Jacobsen TN, Nohr EA, Frydenberg M. Selection by socioeconomic factors into the Danish National Birth Cohort. Eur J Epidemiol. 2010;25:349355.
22. Lash TL, Fox MP, Fink AK. Spreadsheets from Applying Quantitative Bias Analysis to Epidemiologic Data. 2014. Available at: Accessed 23 February 2018.
23. VanderWeele TJ, Ding P. Sensitivity analysis in observational research: introducing the E-value. Ann Intern Med. 2017;167:268274.
24. Hajat A, Hsia C, O’Neill MS. Socioeconomic disparities and air pollution exposure: a global review. Curr Environ Health Rep. 2015;2:440450.
25. Llop S, Ballester F, Estarlich M, et al. Social factors associated with nitrogen dioxide (NO2) exposure during pregnancy: the INMA-Valencia project in Spain. Soc Sci Med. 2011;72:890898.
26. Forastiere F, Stafoggia M, Tasco C, et al. Socioeconomic status, particulate air pollution, and daily mortality: differential exposure or differential susceptibility. Am J Ind Med. 2007;50:208216.
27. Fecht D, Fischer P, Fortunato L, et al. Associations between air pollution and socioeconomic characteristics, ethnicity and age profile of neighbourhoods in England and the Netherlands. Environ Pollut. 2015;198:201210.
28. Greenland S. Multiple-bias modelling for analysis of observational data. J R Stat Soc Ser A Stat Soc. 2005;168:267306.

Supplemental Digital Content

Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved.