Within epidemiology, there is a large literature on the effects of nondifferential measurement error or misclassification of individual persons’ exposures on the magnitude of statistical association: such error usually biases an epidemiologic effect estimate to the null.1 For instance, in a study of the effect of personal income on body mass index (BMI; eg, Rundle et al),2 nondifferential measurement error in income assessment would decrease the apparent association between income and BMI. There is a smaller literature on the effects of exposure measurement error in cross-sectional ecologic studies, such as a study relating the proportion of residents living below poverty across counties to the proportion of people in those counties who are obese.3,4 For this type of study, data on individual poverty status are aggregated to determine the proportion of residents in each county who live in poverty; if there is misclassification of individual poverty status, effect estimates are biased away from the null.4
However, there is almost no literature on how nondifferential measurement error or misclassification of exposure affects multilevel studies of neighborhood health effects on individual persons’ health outcomes. Multilevel neighborhood-health-effect studies typically include individual-level outcome measures in a study population (eg, BMI), individual-level predictors measured from the study participants (eg, age, race/ethnicity, sex, income), and neighborhood-level contextual predictors derived from Census data (eg, neighborhood poverty rates). One of the goals of such studies is to quantify the effect of a neighborhood contextual feature (eg, neighborhood poverty) on individual outcomes after adjusting for individual covariates. Neighborhood contextual measures are often computed by aggregating data collected from individual people during the US Decennial Census, American Community Survey, or other large social surveys. Sensitive information, including income, is likely to be measured with error in these surveys.5 A detailed review has suggested that underreporting and overreporting are about equally common6; for many outcomes, researchers might reasonably expect such error to be nondifferential. It has previously been argued that nondifferential measurement error in survey data causes bias away from the null for effect estimates relating outcome variables measured at the individual level in a study population to a neighborhood contextual variable created by aggregating individual-level survey data.7 However, the potential magnitude of the bias has not been documented, nor has the quantitative relation between the extent of measurement error and bias.
For a given neighborhood context measure created by aggregating data collected from individuals, an investigator may have several options as to how to create a contextual variable.8 For instance, “median household income” and “proportion of residents living in poverty” are both commonly used neighborhood contextual variables derived from the same individual-level Census data on income. Similarly, the investigator has several options for categorizing neighborhoods by a contextual variable.9 For example, an investigator may choose to compare neighborhoods with fewer than 5% of residents living in poverty with neighborhoods where more than 20% of residents live in poverty or may choose to compare the highest quintile to the lowest quintile of neighborhoods ranked by the percent of residents living in poverty. The effects of these choices on the extent of bias in calculated effect estimates when the underlying data include measurement error have not been documented.
Here, we demonstrate the bias that occurs in the effect estimate for the influence of neighborhood contextual variables on individual-level outcomes when there is measurement error or misclassification in the individual-level data that are used to create the contextual variable. We demonstrate how choices an investigator makes in the creation of contextual variables, such as the use of proportions rather than means, affect bias in the effect estimate, and how further manipulating the data to express continuous measures as categories affects bias.
We use a combination of health survey data, Census data, mathematical derivation, and simulations to explore bias in effect estimates for neighborhood contextual variables derived from the aggregation of individual-level data that are likely to have been measured with error. We use the association between survey respondent’s BMI and zip code-level socioeconomic status (SES) as our model, accounting for individual-level covariates, as assessed in the New York City Community Health Survey, a population-based health survey of residents of New York City conducted by the New York City Department of Health and Mental Hygiene that collected self-reported data on height and weight and zip-code.10–13 We simulate error in the Census data, then assess the bias created by that error given several study design options: (1) form (continuous or dichotomous) of outcome variable; (2) form (continuous or dichotomous) of the household-level variable aggregated by the Census to create the zip-code SES estimate; (3) modeling technique; (4) population size of neighborhoods and number of neighborhoods used in the study; and (5) categorization of neighborhoods (none, quantile-based, or external cut-point) for final effect estimation.
Nondifferential misclassification of a neighborhood-level variable aggregated from dichotomous measures can be simulated without needing to access underlying per-household Census data; analyses focused on zip-code poverty rate were performed using the Community Health Survey dataset linked to zip-code-level Census data. However, because simulations of measurement error of a neighborhood-level variable aggregated from an underlying continuous measure (eg, zip-code median household income) require the individual-level continuous data, we created completely simulated datasets to study the effects of measurement error in continuous data. These datasets were simulated to resemble the Community Health Survey dataset in general distribution of variables, although population sizes were typically smaller than zip codes to avoid computational overload (New York City zip codes vary widely, but average about 40,000 residents). Our simulation was adapted from work by Clarke et al14; SAS code for the simulation is included in eAppendix 1 (http://links.lww.com/EDE/A793). Details of the scenarios investigated are provided in the Table.
Survey Data Source
We used data from the 2002 to 2004 NYC Community Health Survey and the 2000 US Census to provide a baseline estimate of the association of neighborhood poverty rates with body mass index and with obesity status.12,13 Using the residential zip-code, individual-level outcome and covariate data were linked to Census data on the proportion of zip-code residents living below the poverty line. Analyses of BMI, individual demographic variables, and appended neighborhood characteristics were approved by the Columbia University Medical Center Institutional Review Board.
The BMI of Community Health Survey respondents was computed from self-reported height and weight data, and obesity was defined as BMI of 30 or more.15 The respondents also reported their age, sex, and race, which for this exercise was coded as black or nonblack. The Community Health Survey measures respondent’s SES as the ratio of reported family income to the US poverty threshold, as reported by the US Census.
Contextual Variables Aggregated from Dichotomous Individual-Level Census Measures
To assess the effects of nondifferential misclassification error in the Census data used to estimate the proportion of residents in each zip-code living in poverty, we first used mixed models to assess the association between zip-code poverty rate and respondents’ individual-level BMI after adjusting for individual age, sex, race, and poverty status. Treating the actual Census data as the truth, we created datasets in which in the poverty rate (PR) was assumed to have been measured with error using the formula:
where Se is sensitivity and Sp is specificity for the determination of whether an individual Census respondent lived in poverty. We modeled the association between neighborhood poverty rate observed with varying amounts of misclassification in the underlying Census data and the respondent’s individual-level BMI after adjusting for individual age, sex, race, and poverty status. Analyses were conducted for sensitivity and specificity ranging from 50% to 100%, and effect estimates calculated from zip-code-level data that incorporated error were compared with the effect estimate at 100% sensitivity and specificity to calculate magnitude of bias. Analyses were repeated with obesity status rather than BMI as the outcome, and using approaches based on generalized estimating equations, with clustering on zip codes and robust standard error estimation16 rather than mixed models. Finally, we repeated analyses with neighborhood poverty rates further transformed into categories based on quintile cut-points across the distribution of zip codes.
Researchers sometimes use externally defined cut-points to categorize neighborhoods (eg, neighborhoods with a greater than 20% poverty rate are classified as high poverty). Because misclassification of individual-level measures can result in recategorization of neighborhoods with respect to the cut-points, the impact of misclassification on neighborhood effect estimates is dependent on the distribution of neighborhoods with respect to the cut-points. For example, a neighborhood whose underlying poverty rate is 19.9% is more likely to be recategorized above a 20% cut-point than 1 whose underlying poverty rate is 10%. Because we did not want our analysis to be affected by the specifics of New York City zip codes, we analyzed this scenario using simulated datasets rather than the Community Health Survey. Each dataset had 200 neighborhoods with 4,000 residents each, for a total of 800,000 observations, and persons followed approximately the same distribution of sociodemographic predictors and associations between BMI and age, race, sex, and income as in the Community Health Survey Data. Using this framework, we simulated 100 datasets. Within each dataset, we varied sensitivity and specificity in the individual poverty measurements, aggregated individual measurements that reflected varying degrees of error to create estimates of neighborhood poverty rates, and computed the estimated effect on BMI of living in a neighborhood with a poverty rate measured at over 20% compared with living in a neighborhood with a poverty rate measured under 5%. Finally, we compared estimates calculated using neighborhood poverty rates aggregated from erroneous individual-level measures to estimates calculated using perfectly measured individual-level measures to estimate magnitude of bias owing to measurement error.
Contextual Variables Aggregated from Continuous Individual-Level Census Measures
Publicly available Census data do not include individuals’ or households’ incomes, and so simulations of measurement error in reported income could not be conducted directly using Census data. Therefore, we estimated the effects of measurement error in income data within a simulated dataset of 200 neighborhoods with 4,000 residents each; we then sampled 200 persons from each neighborhood to serve as study subjects. Within the simulated dataset, we used mixed models to assess the effect of neighborhood mean income on individual-level BMI after adjusting for individual age, sex, race, and income to poverty line. We introduced random measurement error into the measure of income in the simulated Census data as follows:
We recomputed mean income within each neighborhood with the erroneous measure and modeled the association between neighborhood mean income and BMI after adjusting for individual-level age, sex, race, and income to poverty line. We repeated the introduction of random error for varying degrees of error 50 times at each level from 1% to 50% noise and compared the resulting effect estimate to the estimate computed at 0% noise. To assess the effect of categorization of neighborhood mean income, we repeated the introduction of measurement error but computed effects of mean income on BMI, comparing neighborhoods categorized as the highest and lowest quintiles of observed mean income. To assess the effect of neighborhood population size from which neighborhood mean income had been estimated, we simulated datasets with 20% noise and 100 neighborhoods and varied the number of residents per neighborhood from 20 to 4,000. Finally, to assess the effect of number of neighborhoods on magnitude of bias owing to random measurement error, we simulated datasets with individual income measured at 20% noise although number of study neighborhoods ranged from 5 to 400.
All analyses were conducted using SAS 9.2, using PROC MIXED for mixed models with BMI as an outcome, PROC GLIMMIX using a logit link for mixed models with obesity as an outcome, and PROC GENMOD for all generalized estimating equation models. All models within the Community Health Survey dataset were clustered on zip-code, and all models in the simulated datasets were clustered on the simulated neighborhood designation.
Effect of Misclassification when Dichotomous Individual-Level Data Are Used to Create Contextual Variables
When analyzed using a mixed model, the proportion of zip-code residents living in poverty was positively associated with higher BMI among Community Health Survey respondents, after adjustment for individual-level covariates. As increasing amounts of nondifferential misclassification of individual-level poverty status were factored into the Census-data-derived measure of the proportion of zip-code residents living in poverty, the estimated association between neighborhood poverty rate and individual-level BMI increased (Figure 1), as did the standard error of the estimate. The extent of inflation in both the group-level parameter estimate and the estimate’s standard error was
times its true value, where Se and Sp are sensitivity and specificity of the designation of an individual person in the Census data living in poverty. The algebraic basis for this formula is presented in eAppendix 2 (http://links.lww.com/EDE/A793).
When obesity status was used as the outcome variable, misclassification in the underlying Census data similarly caused a bias away from the null. Analyses of the BMI and obesity outcomes using generalized estimating equation models resulted in the same extent of bias as analyses using mixed models. The model intercept estimate was unaffected by variation in sensitivity but was biased away from the null under conditions of imperfect specificity for both generalized estimating equation and mixed models. Parameter estimates for other variables in the model were not affected by measurement error in the neighborhood contextual variable. Because inflation in the parameter estimate was matched by an inflation in the standard error, significance tests were unaffected by the bias.
In many studies, neighborhoods are categorized for analysis either using cut-points based on the distribution of the data itself (such as quintiles of poverty rate) or using externally defined cut-points (such as above or below 20% poverty). When the data were analyzed categorizing neighborhoods into quintiles of poverty rate, misclassification in the underlying Census data used to estimate poverty rate did not cause bias in estimates of the effect of neighborhood poverty on BMI. That is, associations between quintiles of zip-code poverty rate and BMI under circumstances of misclassification in the individual-level Census data were the same as for associations between quintiles of zip-code poverty rate measured with no misclassification in the underlying Census data. The algebraic basis for this result is presented in eAppendix 3 (http://links.lww.com/EDE/A793). When neighborhoods were categorized based on externally defined cut-points, bias was generally away from the null. Figure 2 shows bias in effect estimates comparing individual BMI in neighborhoods with 5% or fewer households living in poverty to neighborhoods with 20% or more households living in poverty, for varying levels of sensitivity in classifying persons with regard to poverty status in the underlying Census data.
Effect of Measurement Error when Continuous Individual-Level Data Are Used to Create Contextual Variables
Simulation analyses showed that nondifferential measurement error in the underlying individual-level income data used to estimate mean neighborhood income did not consistently bias the estimated effect of mean neighborhood income on BMI—both bias toward the null and bias away from the null occurred across simulations. In simulations in which measurement error in the individual-level income data caused an expansion in the range of the zip-code-level mean income data, bias toward the null occurred; conversely when the range was compressed, bias away from the null occurred. However, nondifferential measurement error in the underlying income data did not systematically expand or compress the range of the zip-code-level mean income values.
Generally, the magnitude of bias was positively correlated with the extent of noise in the individual-level data used to estimate mean neighborhood income; inversely correlated with the number of neighborhoods included in the analysis; and inversely correlated with number of individuals used to estimate neighborhood mean income. When the number of persons used to create the neighborhood estimate exceeded 4,000 (the approximate number of residents in a US Census tract), estimates were never biased more than 0.5% toward or away from the null. However, studies estimating neighborhood context from sources less comprehensive than the Census may be more substantially biased. For example, in a study with 15 neighborhoods for which neighborhood income is estimated from only 30 surveyed residents each, noise levels at 25% resulted in effect estimates biased as high as 100% above and as low as 100% below the true value. eFigures 1 to 3 (http://links.lww.com/EDE/A793) illustrate these 3 relation over repeated simulations. Categorizing neighborhoods in quintiles for comparison did not remove the bias.
This article explores the effects of nondifferential misclassification and measurement error in individual-level data that are used to create estimates of neighborhood-level contexts for neighborhood health effects studies. It is common for investigators to use Census data to create measures of neighborhood income/poverty, educational attainment, and ethnic composition and then to assess whether the neighborhood-level contextual variable predicts individual-level outcomes in a study population. Owing to the wealth of data available from the Census, the researcher has many options as to the form these contextual measures might take. Blakely and Woodland7 previously asserted that measurement error in the individual-level data that underlie these types of contextual measures would cause bias away from the null in the contextual variable’s β-coefficients. Here, we document the extent of the bias described by them, and we additionally show the following: the presence of bias in the beta coefficient is determined by the form of the contextual variable; standard errors of beta coefficients are also affected by this type of nondifferential measurement error; the model intercept can also be affected by this type of nondifferential measurement error; and that the bias is identical for continuous and dichotomous outcomes and when data are analyzed with generalized estimating equations and mixed models. Figure 3 summarizes the effects of the various choices a researcher might make in constructing neighborhood contextual measures when there is measurement error or misclassification in the data being aggregated to create the contextual measure.
The form a researcher selects for a neighborhood-level contextual variable has implications for how measurement error in the individual-level data that underlies the contextual measure affect the effect estimates derived from a study. When the neighborhood-level variable is expressed as a proportion of residents having a dichotomous individual-level state, the beta coefficient for the contextual measure will be biased away from the null in the presence of misclassification error for the individual-level state.17 If a contextual measure expressed as a proportion is further transformed into a quantiled categorical variable (eg, quintile categories across the observed distribution of proportion of residents living in poverty),8,18 nondifferential misclassification error in the underlying individual-level Census data does not produce bias in the estimated association between the dependent variable and increasing quantiles of the contextual variable. However, when neighborhoods are categorized using predefined cut-points (eg, neighborhoods with >20% poverty classified as high poverty and neighborhoods with ≤ 20% poverty classified as low poverty),8 nondifferential misclassification in the underlying Census data will usually cause a bias away from the null. Analyses using internal versus external cut-points to categorize neighborhoods behave differently in the presence of nondifferential misclassification in the underlying Census data, because misclassification causes recategorization of neighborhoods when external cut-points are used but not when internal cut-points are used. Suppose that, if ranked by true poverty rate, the 20th percentile neighborhood would have a 5% poverty rate and the 80th percentile neighborhood would have a 20% poverty rate, but the Census actually measures individual poverty status with 90% sensitivity and 100% specificity. With this imperfect measure, the first quintile cut-point will be 4.5% and the fifth quintile will be 18%, but the rank order of neighborhoods by poverty rate is unchanged by the measurement error. Thus, misclassification of individual poverty status in the Census data does not alter which neighborhoods will be classified as being in the lowest and highest quintile of poverty rate. However, when neighborhoods are defined as low poverty if their poverty rate is <5% and high poverty if their poverty rate is >20%, misclassification of individuals’ poverty status in the Census data alters some neighborhood’s classifications: a neighborhood whose true poverty rate is 5.1% will be included in the low-poverty group and a neighborhood whose true poverty rate is 21% will be excluded from the high-poverty group. This shuffling of neighborhoods across categories defined by externally set cut-points can cause bias toward or away from the null; bias away from the null was more common in our simulations (Figure 2).
Measurement error in individual-level Census data expressed as a continuum (eg, income) and aggregated to a contextual variable also expressed as a continuum (eg, mean household income) does not cause a consistent direction of bias in the beta coefficient relating the neighborhood contextual variable to the dependent variable. Random error in the underlying individual-level Census income data produces random error in the neighborhood-level mean income data for each neighborhood, but when the number of residents sampled to derive the neighborhood mean income measure is large, the random error in the neighborhood estimate is small. As a result, the magnitude of bias in the resulting effect estimate is likely to be negligible if the number of measurements per neighborhood aggregated to create the neighborhood measure is as large as a Census tract.
Our analysis focused on the simplified view of measurement error that investigators frequently hope for: that measurement error is nondifferential with respect to true exposure level, true outcome level, and error in measured outcome. We show that, even under these assumptions, bias away from the null can occur. We note, however, that such assumptions often do not hold in reality; measurement error may be correlated with the true value (eg, income may be disproportionately underreported by those with higher incomes) or with measurement error in the outcome.19 Such scenarios may cause bias toward or away from the null and thus warrant more complex simulations than we have presented here.
Our simulations of misclassified dichotomous individual-level data assumed deterministic misclassification across all neighborhoods. If misclassification is treated as random error (ie, the misclassification rate may vary between neighborhoods, although the variability is unrelated to the outcome), as explored in the simulations of Jurek et al,20 reordering of neighborhoods may occur; however, because the proportion of individuals misclassified is related to the true proportion of individuals with a trait, rank-order changes are minimal and thus quantiling still minimizes the effects of nondifferential misclassification.
In addition to the study of neighborhood health effects, several other areas of research commonly use these types of contextual variables created through the aggregation of individual-level data. For example, no individual-level income information is available in the Surveillance, Epidemiology, and End Results database, and so it has been suggested that measures derived from Census data on the patient’s Census tract of residence are the best available indicator of socioeconomic status.21 Although such analyses frequently assess the resulting tract measures in quantiles (eg, Gomez et al22), analyses that do not do so may be affected by nondifferential measurement error in the underlying Census data. It is likely that the validity of studies of educational and health outcomes in schools in which classroom- or school-level contextual measures are created by aggregating student-level data is threatened by the measurement issues identified here, as well. The socioeconomic context of a school or classroom is often of interest as a predictor of outcomes and is often operationalized as the proportion of students receiving free or reduced-price school lunch.23,24 This measure has been criticized as including measurement error.25 Within a classroom, only a small number of students may contribute to continuously distributed contextual measures (eg, mean family income26 and these studies often study only a modest number of classrooms or schools).27 Thus, such studies fall within the design parameters where measurement error in the estimation of a contextual variable will produce considerable bias.
In conclusion, measurement error in the individual-level data that underlie the construction of neighborhood-level contextual variables can dramatically influence estimates of neighborhood effects on individual-level health outcomes. This bias can be reduced by carefully choosing the form of the contextual variable or by transforming the contextual data into quantiles.
We thank colleagues at the New York City Department of Health and Mental Hygiene for making Community Health Survey Public Use Data available.
1. Greenland S. The effect of misclassification in the presence of covariates. Am J Epidemiol. 1980;112:564–569
2. Rundle A, Field S, Park Y, Freeman L, Weiss CC, Neckerman K. Personal and neighborhood socioeconomic status and indices of neighborhood walk-ability predict body mass index in New York City. Soc Sci Med. 2008;67:1951–1958
3. Brenner H, Greenland S, Savitz DA. The effects of nondifferential confounder misclassification in ecologic studies. Epidemiology. 1992;3:456–459
4. Brenner H, Savitz DA, Jöckel KH, Greenland S. Effects of nondifferential exposure misclassification in ecologic studies. Am J Epidemiol. 1992;135:85–95
5. Krieger N, Williams DR, Moss NE. Measuring social class in US public health research: concepts, methodologies, and guidelines. Annu Rev Public Health. 1997;18:341–378
6. Moore JC, Stinson LL, Welniak EJ. Income measurement error in surveys: a review. J Official Stat.. 2000;16:331–362
7. Blakely TA, Woodward AJ. Ecological effects in multi-level studies. J Epidemiol Community Health. 2000;54:367–374
8. Krieger N, Chen JT, Waterman PD, Soobader MJ, Subramanian SV, Carson R. Choosing area based socioeconomic measures to monitor social inequalities in low birth weight and childhood lead poisoning: the Public Health Disparities Geocoding Project (US). J Epidemiol Community Health. 2003;57:186–199
9. Rothman KJ, Greenland S, Lash TL Modern Epidemiology. 2008 Philadelphia, PA Wolters Kluwer Health/Lippincott Williams & Wilkins
10. Stark JH, Neckerman K, Lovasi GS, et al. Neighbourhood food environments and body mass index among New York City adults. J Epidemiol Community Health. 2013;67:736–742
11. Rundle A, Field S, Park Y, Freeman L, Weiss CC, Neckerman K. Personal and neighborhood socioeconomic status and indices of neighborhood walk-ability predict body mass index in New York City. Soc Sci Med. 2008;67:1951–1958
12. Freeman L, Neckerman K, Schwartz-Soicher O, et al. Neighborhood walkability and active travel (walking and cycling) in New York City. J Urban Health. 2013;90:575–585
13. Lovasi GS, Bader MD, Quinn J, Neckerman K, Weiss C, Rundle A. Body mass index, safety hazards, and neighborhood attractiveness. Am J Prev Med. 2012;43:378–384
14. Clarke P, Wheaton B. Addressing data sparseness in contextual population research—using cluster analysis to create synthetic neighborhoods. Sociol Methods Res.. 2007;35:311–351
15. World Health Organization.. Obesity and overweight. Available at: http://www.who.int/mediacentre/factsheets/fs311/en/
. Accessed 17 July 2013.
16. Hubbard AE, Ahern J, Fleischer NL, et al. To GEE or not to GEE: comparing population average and mixed models for estimating the associations between neighborhood risk factors and health. Epidemiology. 2010;21:467–474
17. McGrath JJ, Matthews KA, Brady SS. Individual versus neighborhood socioeconomic status and race as predictors of adolescent ambulatory blood pressure and heart rate. Soc Sci Med. 2006;63:1442–1453
18. Robert SA, Strombom I, Trentham-Dietz A, et al. Socioeconomic risk factors for breast cancer: distinguishing individual- and community-level effects. Epidemiology. 2004;15:442–450
19. Wacholder S. When measurement errors correlate with truth: surprising effects of nondifferential misclassification. Epidemiology. 1995;6:157–161
20. Jurek AM, Greenland S, Maldonado G, Church TR. Proper interpretation of non-differential misclassification effects: expectations vs observations. Int J Epidemiol. 2005;34:680–687
21. Bach PB, Guadagnoli E, Schrag D, Schussler N, Warren JL. Patient demographic and socioeconomic characteristics in the SEER-Medicare database applications and limitations. Med Care. 2002;40(8 uppl):IV–19
22. Gomez SL, O’Malley CD, Stroup A, Shema SJ, Satariano WA. Longitudinal, population-based study of racial/ethnic differences in colorectal cancer survival: impact of neighborhood socioeconomic status, treatment and comorbidity. BMC Cancer. 2007;7:193
23. Hanushek EA, Kain JF, Markman JM, Rivkin SG. Does peer ability affect student achievement? J Appl Econometr.. 2003;18:527–544
24. Okpala CO, Okpala AO, Smith FE. Parental involvement, instructional expenditures, family socioeconomic attributes, and student achievement. J Educ Res.. 2001;95:110–115
25. Harwell M, LeBeau B. Student Eligibility for a Free Lunch as an SES Measure in Education Research. Educational Res.. 2010;39:120–131
26. Singer JD. Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. J Educ Behav Stat.. 1998;23:323–355
27. O’Dwyer LM, Russell M, Bebell D, Seeley K. Examining the relationship between Students’ mathematics test scores and computer use at home and at school. J Technol Learn Assess.. 2008;6:4–46