How Healthy Are Survey Respondents Compared with the General Population?

Using Survey-linked Death Records to Compare Mortality Outcomes

Abstract

Background: 

National surveys are used to capture US health trends and set clinical guidelines, yet the sampling frame often includes those in noninstitutional households, potentially missing those most vulnerable for poor health. Declining response rates in national surveys also represent a challenge, and existing inputs to survey weights have limitations. We compared mortality rates between those who respond to surveys and the general population over time.

Methods: 

Survey respondents from 20 waves of the National Health Interview Survey from 1990 to 2009 who have been linked to death records through 31 December 2011 were included. For each cohort in the survey, we estimated their mortality rates along with that cohort’s mortality rate in the census population using vital statistics records, and differences were examined using Poisson models.

Results: 

In all years, survey respondents had lower mortality rates compared with the general population when data were both weighted and unweighted. Among men, survey respondents in the weighted sample had 0.86 (95% confidence interval = 0.853, 0.868) times the mortality rate of the general population (among women, RR = 0.887; 95% confidence interval, 0.879, 0.895). Differences in mortality are evident along all points of the life course. Differences have remained relatively stable over time.

Conclusion: 

Survey respondents have lower death rates than the general US population, suggesting that they are a systematically healthier source population. Incorporating nonhousehold samples and revised weighting strategies to account for sample frame exclusion and nonresponse may allow for more rigorous estimation of the US population’s health.

Nationally representative surveys serve multiple purposes for health promotion and planning.^1–3 Reliable estimates of the population prevalence of health-related behaviors are critical to public health infrastructure for multiple purposes, including formulating and evaluating policies aimed at improving and maintaining population health and well-being, national guidelines, percentiles, and clinical decision-making. Two developing dynamics warrant reconsideration of the role of surveys purporting to be nationally representative in describing the nation’s health.

First, validity of inference from survey data is dependent on their representativeness of the general population, yet the sampling frame of such surveys is typically restricted to noninstitutionalized households and so excluding homeless and institutionalized populations known to have high rates of poor health.⁴ For example, the population of the US state and federal prisons was less than 200,000 from the 1950s through the 1970s when many national surveys began. Since then, both the number and the proportion of individuals in the United States experiencing incarceration has increased dramatically. The number in state and federal prisons doubled by 1983, reached over a million by 1994, and reached 1.6 million by 2010.⁵ As of 2015, approximately 1 in 218 Americans was incarcerated in a state or federal prison,⁶ and the burden of incarceration falls disproportionately on Black and low-income Americans, as well as men.⁷ These trends imply that an increasing number of Americans—particularly those concentrated in certain demographic groups—fall outside the sampling frame of US surveys, which may obscure our ability to assess health disparities by race and socioeconomic status.

Second, survey response has been declining for the last decade.^8–11 For example, the US National Health and Nutrition Examination Survey achieved household response levels between 80% and 90% throughout the 1980s and 1990s⁸ but have been less than 80% since 2007, and in 2011–2012, the response was 72.6%.¹² The National Health Interview Survey’s (NHIS) household response level fell below 80% in 2010 from over 90% in the 1990s and decreased to 77.6% in 2012.¹³

There is growing concern about the resulting nonrepresentativeness of national surveys, including whether use of households as a sampling frame is appropriate and the ensuing potential for unreliable inference to the general population.^11,14 Two issues are particularly salient to this discussion. First, undercoverage of the general population may result in prevalence estimates that are not an accurate snapshot of the nation’s health. Second, associations between demographic characteristics and risk behaviors may not reflect these associations in the general population. For example, we may over- or underestimate the extent of health disparities by such factors as sex and race if there are differential associations by inclusion status. Such potential for inferential bias is nontrivial. Estimates of both smoking and excessive alcohol consumption indicate decreases over the past 10 years.¹⁵ Given that nonrespondents and the incarcerated are more likely to be smokers and excessive alcohol consumers than those who respond,^10,16–20 at least part of the reported prevalence decreases over time could be attributable to growing nonresponse bias and misrepresentation of the nation based on the sampling frame. As data sources from which population health estimates are generated become larger and with the rise of “big data,”²¹ understanding representativeness, or lack thereof, is growing in importance for epidemiologic investigation.

The most commonly used post hoc method to address nonresponse bias is to use population demographics and selection probabilities and reweight the samples using inverse probability weighting. However, the factors selected for weighting may not be adequate to account for differential selection into the sample if they do not capture or adjust for differences between respondents and nonrespondents in terms of health aspects, such as those related to smoking or alcohol consumption.^22,23 Further, survey weighting methods applied to most national surveys do not account for factors associated with living in a household (e.g., wealth and criminal justice involvement). Poststratification weighting uses demographically stratified population totals from the US census, which does include those residing outside of households, and as such, poststratification sample weighting should in theory aid in estimating prevalences and associations that are more representative of the general population, despite excluding the institutionalized in the sampling frame. The extent to which weighting may be insufficient to render whole population-representative estimates is unknown, given that weighted estimates are difficult to validate: using weighting, we can ensure that the distributions of the factors for which there is population-level data at the household level (e.g., sex, age, race) match between population and sample, but this does not necessarily resolve representativeness for unmeasured factors.

Increasingly, national surveys are being individually record-linked to routine mortality data,^24,25 providing the ability to assess death rates among survey respondents. Because we also have the information for the entire population, we can compare death rates between survey samples and the general population. If there is no nonresponse bias and household dwellers are representative of the whole population, these estimates should be equivalent, especially when the data are weighted for demographics, as well as nonresponse. To the extent that the survey respondents are healthier, however, we may conclude that current approaches are not capturing individuals mostly likely to have important risk factors for illness. Such survey to general population death comparisons has revealed insufficiencies in survey coverage and weighting in various countries,^22,26–32 but to date, no attempt has been made, to our knowledge, to assess whether such results generalize to the United States where survey recruitment, sampling strategies, and population size and characteristics differ greatly. The present study compares weighted and unweighted mortality rates in US survey samples to comparable cohorts in the general population.

METHODS

Data Sources

National Health Interview Survey

The NHIS is an annually conducted in-person household survey comprising a multistage probability sample of noninstitutionalized respondents. Total household response rates ranged from ≈82% (2009) to 96% (1991 and 1992). We used data on adults in the NHIS surveys from 1990 to 2009 that have been individual-level record-linked to the National Death Index (NDI) through 31 December 2011, with successful linkage of ≈94% of eligible respondents (N = 1,309,449). Study respondents were probabilistically^24,33 linked to the NDI through at least one of seven matching criteria, including some combination of social security number, first and last name, middle initial, date of birth, and father’s surname. Those respondents who were 18+ at the time of interview were eligible to be linked; the present study includes those eligible respondents aged 18 to 79 years (ceiling coding in the NHIS precluded precise designation of birth cohort for those older than 80 years). Table 1 provides the weighted and unweighted sample size for each year of the NHIS, and the proportion of each sample decreased by 31 December 2011. The Columbia Institutional Review Board approved analyses of the NHIS-linked mortality data.

TABLE 1: NHIS Sample Sizes by Year and the Number and Percentage of Those Participants Who Died as of December 31, 2011

Details of the NHIS sampling frame and the estimation of sample weights are found elsewhere.^34,35 Briefly, NHIS uses a multistage area probability design, which divides the United States into geographically defined Primary Sampling Units and groups those into strata to ensure broad geographic representation and, within those Primary Sampling Units, area and permit segments. The data on individuals for each NHIS sample has a set of sample weights that are based on the design, nonresponse, and poststratification adjustments. Poststratification adjustments are based on age, sex, and race/ethnicity such that the distributions of these demographics in the weighted sample approximate the US census population for the closest year of Census data collection. In the NHIS Linked Mortality Files, the sampling weights are further adjusted based on ineligibility or insufficient identifying data for linkage. Of the 1,953,298 survey respondents from 1990 to 2009, 1,309,449 (67%) were eligible for linkage, 547,290 (28%) were under age 18 years and not available for public release, and 96,559 (5%) were ineligible. Respondents >79 years old at the time of survey (N=48,923) were removed from the analytic sample because of differences across years in age categorization. For the present analysis, we utilized the sample weight for the linked mortality files that incorporates these weights.³⁶

Vital Statistics

The National Center for Health Statistics maintains the National Vital Statistics System for all deaths in the United States, coordinating and processing US Standard Death Certificates obtained from each state registrar.³⁷ Death data are collected and reported by trained medical certifiers. The present study examines all-cause mortality.

US Census

Population totals stratified by sex and age are derived from the National Cancer Institute’s Surveillance, Epidemiology, and End Results (SEER) Program; they represent a modification of the intercensal and Vintage 2014 annual time series of July 1, county population estimates by age, sex, race, and Hispanic origin produced by the US Census Bureau’s Population Estimates Program, in collaboration with the National Center for Health Statistics, and with support from the National Cancer Institute through an interagency agreement.³⁸ SEER population totals provide the most granular population estimates that are publicly available. Population census estimates are not directly linked to vital statistics regarding death.

Analytic Strategy

The general approach was to first estimate mortality numerators and denominators for the linked NHIS sample and for comparably aged individuals in the general population. We determined the mortality among individuals in the entire United States who would have been the same age at the time of the NHIS survey in any given year. This figure comprises those who would have been eligible for the survey (including nonrespondents) and those ineligible but living in the United States (e.g., the institutionalized). We detail our process for these estimations below.

Among adult survey respondents aged 18 to 79 years who were linked (N = 1,260,526), we first determined the numbers in each survey cohort who had died by 31 December 2011, overall and by age and sex. We then estimated the person time at risk among both those who died and those who did not die to inform the mortality rate. Among those in NHIS who died during follow-up, person time was estimated as survey year subtracted from death year. For those who had not died, person time was estimated as survey year subtracted from 2012 (given that respondents were linked through 31 December 2011).

We then obtained corresponding information on death and person-time at risk in the population. We did so by using age of death from vital statistics to determine the age of a deceased person in each year in which they would have been, based on age, eligible to be in the NHIS. For each death, we calculated a “pseudo age at survey” for each year in which the deceased individual would have been eligible and used that pseudo-age to estimate their person-time as the difference between death age and pseudo-age. For example, an individual who died at age 70 years in 2005 would have been eligible to be in the 1990 NHIS survey at age 55, and eligible for the 1991 survey at age 56, and so forth. Thus, the pseudo age at survey is 55 years in 1990 and 56 years in 1991. We then calculated the person-time for each decadent by subtracting would-be survey year from real death year, for each year they would have been eligible for survey in. We then take the complement of this person time by subtracting it from the total possible person time—2012 minus the survey year. Then we collapse the data, summing the complement of person time variable by single year age and survey year. Next, we merge these data with the census population totals, by year and age. To calculate the total synthetic person time sum at each year and age, we multiply the census population by the person time for those who did not die (2012 minus year of potential survey) and then subtract the complement of the person time from the mortality data—that is, person time lost to death.

Once the data on deaths and person-time were collected, we then combined survey and population data and estimated age-standardized, as well as age-specific, mortality rates and rate ratios from Poisson models. Race (for those listed as White and Black, as other racial groups did not have sufficient sample size in the NHIS to produce reliable estimates) and sex stratifications were performed given well-documented mortality differences and potential for differential exclusion from the sampling frame due to institutionalization as well as nonresponse, with existing evidence of sex-specific effects.²² Age-standardization was conducted with reference to the US 2000 Standard Population, as recommended by the US Centers for Disease Control and Prevention.^39,40 These models estimated predicted mortality rates overall and by year, rate ratios comparing NHIS participants to the general population, and corresponding 95% confidence intervals (CI). We additionally included single year of age as a covariate in these models to account for possible differences in age distribution within 5-year age groups. Mortality rates in the NHIS and the general population were estimated in a single model with a dichotomous indicator for survey versus population.

We also estimated the mortality rate for each single year of age in the data from Poisson models. To estimate age-specific mortality rates by year, we included a three-way interaction between survey/population (1 versus 0), age (in single years), and year. We then used the marginal predicted rates from this model to estimate the mortality rate from the survey and the mortality rate from the population by age by year.

Modeling was done both without sample weighting for NHIS and including the NHIS sample weight for participants in the death record linkage (general population weights were assigned the value 1). Modeling was conducted in Stata Version 13. Full SAS and Stata code for these analyses are included as an online supplement, and data files are available by request to the corresponding author.

RESULTS

Figure 1 shows the mortality rate per 100,000 for NHIS respondents through 2014, by year, using both weighted and unweighted NHIS samples, as well as the estimated mortality rates for comparably aged contemporaneous general population cohorts. Results are stratified by sex. Confidence intervals for these estimates are provided in eTable 1; https://links.lww.com/EDE/B288. Sample weighting had little effect on the mortality estimates, and in all years, mortality among NHIS respondents was lower than the general population. Mortality declined over time as more recent survey respondents were followed for a shorter time and, therefore, over a younger age span than the respondents to earlier surveys.

FIGURE 1: Comparison of age-standardized mortality rates in National Health Interview Survey respondents versus the population, among men (square markers) and among women (circular markers).

Table 2 shows the age-standardized mortality rate ratios and confidence intervals between survey respondents and the general population for men and women, respectively. Across all years, among men, survey respondents in the weighted sample have 0.86 times the rate of mortality as the general population (95% CI = 0.853, 0.868). Among men, the rate ratio by year ranged from 0.798 (95% CI = 0.759, 0.838) in 2007 to 0.893 (95% CI = 0.874, 0.911) in 1995. Across all years, among women, survey respondents in the weighted sample have 0.887 times the rate of mortality as the general population (95% CI = 0.879, 0.895). Among women, the rate ratio by year ranged from 0.724 (95% CI = 0.66, 0.787) in 2009 to 0.925 (95% CI = 0.912, 0.938) in 1994.

TABLE 2: Rate Ratios Comparing Death Rates in NHIS to Comparably Aged Cohorts in the General Population by Year, by Sex

Figure 2 shows the predicted 5-year age group–specific mortality rate ratios for each year from a Poisson model with survey/population by age by year interaction; confidence intervals are provided in eTable 2; https://links.lww.com/EDE/B288 (P < 0.001 for the interaction, both when age was considered in single years and in 5-year age groups). Mortality was consistently lower for survey respondents than for the general population across all ages through 2002, with those at younger ages consistently showing more divergence in mortality between survey and population than those in older ages; after 2002, there was more variation in the correspondence between survey and general population samples with mortality, likely due to smaller numbers of deceased for the more recent years.

FIGURE 2: Mortality rate ratios comparing National Health Interview Survey (NHIS) to the general population by age group and year.

Supplmentary Analyses

In eTables 3 and 4; https://links.lww.com/EDE/B288, we show the associations between sex (women compared with men) and race (Black compared with White), respectively, for each year in the NHIS and the general population. In NHIS, rate ratios for the mortality difference between women and men ranged from 0.556 (95% CI = 0.494, 0.618) in 2009 to 0.702 (95% CI = 0.684, 0.72) in 1990. In the general population, mortality rates among women were 0.657 times those of men in 1990 and increased to 0.633 times those of men by 2009. Generally, the trend toward growing disparity in mortality favoring women over the survey years was largely consistent between NHIS and the general population.

Comparing mortality rates among Black and White individuals in NHIS, mortality rate ratios ranged from 1.195 in 1991 to 1.767 in 2003, tending to be larger for more recent survey years. In contrast, in the general population, there was little evidence for trends over time in rate ratios, which ranged from 1.431 to 1.459 in all years from 1990 to 2009.

DISCUSSION

The present study documents that NHIS survey respondents have lower mortality rates than the general population. We draw this conclusion by using information on survey respondents from a 20-year period who have been linked to death records. We compared their mortality rates to the estimated mortality rates among the contemporaneously aged general population. Overall, survey respondents die at 0.86 (men) to 0.887 (women) times the rate of the general population, and survey weighting did not have an impact on the results. The differential was evident for older and younger respondents, though with smaller magnitude in the earlier years.

Further, there is little evidence for systematic change in the relative magnitude of differences between survey respondents and the general population over time. As survey response levels have declined,^8–11 we might have expected to see growing differences between survey respondents and the general population, but this is not evident at present. Continued surveillance of these mortality rates as younger cohorts age, as well as examinations of subgroup differences, will be important next steps in survey research.

Finally, our results indicate that the association between sex and mortality is similar in survey respondents and the general population, indicating that even if prevalence estimates from surveys by sex may not represent the general population, associations with mortality are consistent. On the other hand, associations between race (Black versus White) and mortality increased over time in the NHIS participants but did not change through the course of this study in the general population, indicating that surveys may be increasingly misrepresenting racial disparities in health in the United States as a whole. Other work suggests that disparities in health between Black and White Americans are decreasing nationally, though studies have not followed pseudo-cohorts in a similar methodology as the present study,⁴¹ suggesting that a greater focus on longitudinal research would benefit overall understanding of disparities.

One source of this difference is known and well described: as outlined earlier, the sampling frame of US surveys is most often households.^25,42 The omission of individuals who are incarcerated, otherwise institutionalized, homeless, or serving in the military, provides an over-optimistic picture of our nation’s health. Further, exclusion of institutionalized populations from the sample frame of surveys under-represents the extent of health disparities by race.⁷ Such exclusion may be increasingly problematic, as a disproportionate number of African American men continue to be incarcerated at high rates. At least a portion of the difference between survey respondents and general population documented in our analyses is certain to be attributable to incarcerated and institutionalized individuals being systematically excluded from national survey sampling frames.

Other sources of this difference are less well understood. While men as well as racial/ethnic minorities (e.g. African American and Hispanics) are less likely to respond to surveys,^11,19,43 such nonresponse should, in theory, be accounted for with sample weighting. While oversampling of harder-to-reach groups may enable nonresponse weighting to perform better, currently the design of NHIS does not overample racial/ethnic minorities except those over 65 years.²⁵ Heavy alcohol consumers and smokers, on the other hand, are also less likely to respond to surveys,^10,16–19 and while such health behaviors are associated with demographics, standard weighting schemes cannot account for variation in these health behaviors between respondents and nonrespondents within demographic subgroups. Given that heavy alcohol consumers and smokers have higher mortality rates than the general population,^1,2,27 such health-related selection likely accounts for a proportion of the differences observed in the present study as well.

Differences between survey respondents and the general population have implications not only for estimates of prevalence but also for estimates of associations and disparities. It is well established that women consume less alcohol and cigarettes than men^44,45 but are more likely to respond to surveys compared with men.^11,19,43 Given that heavy alcohol and cigarette consumers are also less likely to respond to surveys,^10,16–19 heavy alcohol consuming and heavy smoking men may be those least likely to respond to surveys across gender and substance use subgroups, and those who do respond may be atypical. If this is the case, surveys are underestimating the gender disparities in the general population. Conversely, African Americans and Hispanics in the United States are less likely to consume alcohol and smoke compared with non-Hispanic whites^46,47 but are also less likely to respond to surveys.^11,19,43 Thus, racial/ethnic disparities in alcohol consumption and smoking based on surveys may be overestimates of the disparity in the general population. There is evidence of such overestimation in our analysis, as racial disparities in mortality tend to be higher and growing over time in NHIS participants compared with the general population. Development and implementation of survey weighting or imputation procedures³² to reflect potential health-related disparities are necessary.

The present study should be considered in light of limitations. We did not have direct estimates of the mortality rates of those individuals who were eligible for NHIS but did not participate. Rather, we compared responders to the general population, estimating their mortality rates using vital statistics and census population totals. Thus, individuals who were not in the sampling frame of NHIS are included in the general population estimate, and we cannot disentangle the proportion of the difference in the NHIS mortality rates and the general population that are due to nonresponse versus institutionalization. Population estimates may not be accurate as the census can miss hard to reach individuals such as homeless and transient populations among whom mortality rates are high. These population estimates should be interpreted with those cautions in mind, although such undercounting of our general population denominators may actually be inflating the mortality rates; thus, the actual differences between survey respondents and the general population could be lower than those we have identified. Further, a small proportion of survey respondents (6% of eligible sample) could not be linked to the NDI, and thus, mortality follow up is incomplete. Additionally, linkages were done probabilitistically, and there may be error in the linkage. To the extent that these errors may favor certain sociodemographic groups is unknown. All-cause mortality provides one important metric of health but certainly not the only metric; information on underlying and contributing causes of death are also available on this cohort, and continued analyses will allow for testing of specific causes of death that may differentiate respondents and the general population. Finally, death record–linked NHIS through 2009 with follow-up to 2011 have been released to date; updated analyses with more recent surveys and years of linkage are important to continue to assess potential divergence between general population and survey representation.

In conclusion, the issue of survey weighting will continue to gain importance in epidemiology. The rise of “big data” and electronic health records promise masses of data,^21,48,49 many of which are highly likely to be quite unrepresentative of underlying population distributions of health and illness. The potential for misrepresentation of population distributions is not necessarily realized, though inferential bias is increasingly being identified across a number of surveys.^{22,28,50–53} However, the potential could be carefully assessed. The issues we highlight in the present article, including systematic exclusion of nonhousehold dwelling individuals from sampling frames, and continuing declines of response rates, are among the multiple challenges that are faced by survey researchers. Continued innovation in methods including sample design and imputation techniques³² or sample weight formulation that allows for broader representation in ways that are fully replicable and valid may improve the utility of large data sources for understanding population health. While some suggest that representative sampling is unnecessary for epidemiologic inquiry,^54,55 the present results and others^56,57 continue to demonstrate that our understanding of the distribution and determinants of disease and other health outcomes vary by population characteristics, and attention to the source populations from which our cases are drawn will likely grow in importance in coming years rather than diminish.

ACKNOWLEDGMENTS

We acknowledge the Columbia University Calderone Prize for Junior Faculty, which provided funding for this project to K.M. Keyes. F. Popham and L. Gray are funded by the Medical Research Council, UK, and Chief Scientist’s Office, Scottish Government (MC_UU_12017/13 and SPHSU13), as part of the core funding for the MRC/CSO Social & Public Health Sciences Unit, University of Glasgow, UK.