Secondary Logo

Research Methodologies for Total Worker Health®

Proceedings From a Workshop

Tamers, Sara L., PhD, MPH; Goetzel, Ron, PhD, MA; Kelly, Kevin M., PhD; Luckhaupt, Sara, MD, MPH; Nigam, Jeannie, PhDc, MS; Pronk, Nicolaas P., PhD; Rohlman, Diane S., PhD; Baron, Sherry, MD, MPH; Brosseau, Lisa M., ScD; Bushnell, Tim, PhD, MPA; Campo, Shelly, PhD; Chang, Chia-Chia, MPH, MBA; Childress, Adele, PhD, MSPH; Chosewood, L. Casey, MD, MPH; Cunningham, Thomas, PhD; Goldenhar, Linda M., PhD, MS; Huang, Terry T.-K., PhD, MPH; Hudson, Heidi, CDR, MPH; Linnan, Laura, ScD; Newman, Lee S., MD, MA; Olson, Ryan, PhD; Ozminkowski, Ronald J., PhD; Punnett, Laura, ScD; Schill, Anita, PhD, MPH, MA; Scholl, Juliann, PhD; Sorensen, Glorian, PhD, MPH

Journal of Occupational and Environmental Medicine: November 2018 - Volume 60 - Issue 11 - p 968–978
doi: 10.1097/JOM.0000000000001404

Objective: There is growing interest in the NIOSH Total Worker Health® program, specifically in the process of designing and implementing safer, health-promoting work and workplaces. A Total Worker Health (TWH) Research Methodology Workshop was convened to discuss research methods and future needs.

Methods: Twenty-six experts in occupational safety and health and related fields reviewed and discussed current methodological and measurement issues and those showing promise.

Results: TWH intervention studies face the same challenges as other workplace intervention studies and some unique ones. Examples are provided of different approaches and their applications to TWH intervention studies, and desired developments in the TWH literature.

Conclusions: This report discusses and outlines principles important to building the TWH intervention research base. Rigorous, valid methodologic, and measurement approaches are needed for TWH intervention as well as for basic/etiologic, translational, and surveillance research.

Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, Washington, DC (Dr Tamers, Ms Chang, Dr Childress, Dr Schill); Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, Atlanta, GA (Dr Chosewood); Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, Cincinnati, OH (Dr Luckhaupt, Ms Hudson, Dr Cunningham, Dr Bushnell, Dr Scholl, Ms Nigam); Johns Hopkins University, Baltimore, Maryland (Dr Goetzel); IBM Watson Health, Bethesda, Maryland (Dr Goetzel, Dr Ozminkowski); University of Iowa, Iowa City, Iowa (Dr Kelly, Dr Rohlman, Dr Campo); HealthPartners Institute, Minneapolis, Minnesota (Dr Pronk); City University of New York, New York City, New York (Dr Baron, Dr Huang); University of Illinois at Chicago, Chicago, Illinois (Dr Brosseau); Center for Construction Research and Training, Silver Spring, Maryland (Dr Goldenhar); University of North Carolina–Chapel Hill, Chapel Hill, North Carolina (Dr Linnan); Center for Health, Work & Environment, Colorado School of Public Health, University of Colorado, Colorado (Dr Newman); Oregon Health and Science University, Portland, Oregon (Dr Olson); University of Massachusetts Lowell, Lowell, Massachusetts (Dr Punnett); Harvard University, Cambridge, Massachusetts (Dr Pronk, Dr Sorensen).

Address correspondence to: Sara L. Tamers, PhD, MPH, Centers for Disease Control and Prevention, Washington DC (

Dr Goetzel, Dr Kelly, Dr Luckhaupt, Ms Nigam, Dr Pronk, Dr Rohlman have contributed equally and are listed alphabetically.

Dr Baron, Dr Brosseau, Dr Bushnell, Dr Campo, Ms Chang, Dr Childress, Dr Chosewood, Dr Cunningham, Dr Goldenhar, Dr Huang, Ms Hudson, Dr Linnan, Dr Newman, Dr Olson, Dr Ozminkowski, Dr Punnett, Dr Schill, Dr Scholl, and Dr Sorensen: authors listed alphabetically.

Funding support: (CDC/NIOSH Cooperative Agreement No. U19OH008868, Dr Diane Rohlman, Dr Kevin Kelly, Dr Shelly Campo; CDC/NIOSH Cooperative Agreement No. OH009762, Dr Linda Goldenhar; CDC/NIOSH Cooperative Agreement No. U19 OH008857, Dr Laura Punnett; CDC/NIOSH Cooperative Agreement No. U19 OH008861, Dr Glorian Sorensen; CDC/NIOSH Cooperative Agreement No. U19 OH 011227, Dr Lee Newman; CDC/NIOSH Cooperative Agreement No. U19OH011232, Dr Lisa Brosseau; CDC Workplace Health Research Network No. 3U48DP005008-01S2, Dr Sherry Baron; CDC/NIOSH Cooperative Agreement No. U19 OH010154, Dr Ryan Olson; CDC Worksite Health ScoreCard Evidence Review and Update Contract Number GS-00-F-0092P-200-2015-87943, Dr Ron Goetzel).

The authors report no conflicts of interest.

This is an open access Written work prepared by employees of the Federal Government as part of their official duties is, under the U.S. Copyright Act, a “work of the United States Government” for which copyright protection under Title 17 of the United States Code is not available. As such, copyright does not extend to the contributions of employees of the Federal Government.

Back to Top | Article Outline


Defining Characteristics of Total Worker Health®

The National Institute for Occupational Safety and Health (NIOSH) defines Total Worker Health (TWH) as policies, programs, and practices that integrate protection from work-related safety and health hazards with promotion of injury and illness prevention efforts to advance worker well-being. The paradigm expands upon the previous definition that emphasized integration of health protection and health promotion1 to encourage integration across a wider set of workplace efforts that support safety, health, and well-being. Integration can occur through collaboration and coordinated programming around organizational leadership and commitment; supportive organizational policies and practices; accountability and training; management and employee engagement strategies; supportive benefits and incentives; and integrated real-time evaluation and surveillance that leads to corrective action where needed.1,2 Several frameworks and models—such as those developed by Sorensen et al, Pronk, and Punnett et al—have been proposed to help explain what integration looks like in practice.3–7 Differences among these models reflect the variety of ideas emerging about how to implement TWH approaches and the evolving nature of this relatively new field.

The TWH definition explicitly reflects the priority placed on changes to improve physical, organizational, and psychosocial factors in the work environment. This approach is distinctly different from interventions that only address or place sole responsibility on the worker to manage the risks. In other words, a workplace intervention must consider how to redesign the work environment to reduce the risk (rather than only providing personal protective equipment for the worker) and address how the work itself may be stress-inducing (rather than instituting a wellness program that only promotes stress management). Programs with a primary motivation of cost savings, rather than improved worker safety, health, and well-being, also do not fit into the TWH framework.

Although the TWH research base has grown,8 it is still an emerging field that can benefit from further examination of how current and new research methods can be used to solidify the evidence base and advance the field. A literature review of TWH intervention research identified only 17 articles on the evaluation of occupational safety and/or health plus wellness or well-being outcomes.8 Promisingly, most of these studies showed an improvement and demonstrated that the gain achieved was greater than when either the safety and/or health or wellness intervention was evaluated alone; this has also been noted in previous literature.9,10 The review further highlighted current gaps in the TWH field, including a need to focus on methodological and measurement issues.

Back to Top | Article Outline

The TWH Research Methodology Workshop

On March 7 and 8, 2017, the University of Iowa's College of Public Health and Healthier Workforce Center of the Midwest hosted a TWH Research Methodology Workshop (“Workshop”). Planning and implementation support for the Workshop was provided by an executive planning committee consisting of several TWH experts and NIOSH personnel. The invited participants were leading researchers in the field of TWH and other pertinent areas.

The impetus for the Workshop was two-fold. The first was to respond to two of the eight recommendations put forth by the Independent Panel of the Pathways to Prevention 2015 Meeting (Total Worker Health®: What's Work Got to Do With It?): (1) expand research and evaluation design options to include a range of rigorous methodologies and (2) develop a core set of measures and outcomes that are incorporated into all integrated intervention studies.11 The second was to respond to one of the research intermediate goals: “to apply and develop rigorous, standardized methods for studying TWH interventions,” supported by several activity/output goals (Sections 1.3 and 1.3.2–1.3.6 in reference [12])12 in NIOSH's National Total Worker Health® Agenda (National TWH Agenda).

Although the TWH field can and should be advanced by means of other research methods (basic/etiologic and surveillance), the Workshop, and thus this paper, concentrate primarily on intervention and (to a lesser extent) translational research.13,14 This is because intervention research, in particular, accounts for the majority of the TWH literature to date and was emphasized in the recommendations and goals by the Independent Panel of the Pathways to Prevention 2015 Meeting11 and the National TWH Agenda.12

The Workshop planners selected a few TWH-related research methodological and measurement focus areas that were then reviewed by participants who discussed limitations to currently-used approaches and described others that show promise for advancing the TWH scientific evidence base. This report summarizes both the 2-day Workshop and subsequent working group discussions. It is intended to highlight the TWH methodological and measurement approaches currently in use, describe the range of possibilities, and suggest promising research practices with the potential to advance the field, through rigorous and repeatable TWH intervention research. The authors also refer to approaches from other research fields that may be applicable to the TWH field. Though some of the issues raised and points for consideration may not be new to occupational safety and health or wellness research, the Workshop participants agreed that it was useful to highlight these in order to strengthen and advance this rather new field's knowledge base. The Workshop topics are as follows:

  • Study design considerations for measuring the effectiveness of TWH programs.
  • Potential selection bias, attrition, and unanticipated changes to the work environment that could affect TWH research.
  • Qualitative and mixed-methods approaches in TWH research.
  • Case study design in TWH research.
  • Measures and outcomes relevant to TWH research.

The expectation is that the summary of these Workshop topic discussions will be relevant not only to seasoned and new TWH and occupational safety and health researchers, but also to others in related disciplines, and to practitioners and stakeholders who read and interpret the TWH literature.

Back to Top | Article Outline


Study Design Considerations for Measuring the Effectiveness of TWH Programs

General Considerations for TWH Programs

The range of TWH research approaches allows for a research design that best fits the research question. Social epidemiological research is important for understanding how working conditions shape safety and health outcomes, whereas dissemination and implementation research helps reveal what and how such research is being translated and delivered in practice. Surveillance research can provide additional information and data on current TWH efforts, to further inform other research processes and practice.

Regardless of research study design type, the measurement and evaluation of interventions can pose several challenges. For instance, TWH interventions are, by definition, multi-factorial; therefore, it is difficult to isolate any given element and compare it to another element found at an organization or worksite not implementing TWH programs. In addition, workers and employers may view data collection as intrusive, burdensome, and/or not central to their work requirements. This difficulty is further complicated when there is a need to collect data from control group workers who will not benefit immediately from the intervention being evaluated; this also touches on the interrelationship between methodological possibilities or decisions made and Institutional Review Board considerations.

Workplace intervention implementation is largely controlled by the employer, in terms of content, duration, and intensity and manipulation of specific variables. Organizational changes, including those in leadership and the work environment, can introduce significant challenges. Fidelity of the intervention and consistency across worksites may prove difficult to control, especially when a TWH program is tailored to a specific workplace and delivered for different exposure periods. Furthermore, a long-time horizon may be needed to realize changes from TWH interventions. This is true especially when a change in safety or health culture is the foundation for other program features and when the outcomes of interest are relatively rare (such as work accidents, incidents of disease, or changes in health habits) and/or involve long latency from exposure to onset.

Another consideration is that potentially sensitive financial and personnel data may be needed to achieve buy-in for investing in organizational redesigns to improve worker safety, health, and well-being.15 The business case may address the cost consequences of not addressing the problem; the resources needed to support a TWH solution, for both the short and long term; the cost-effectiveness of interventions, or value-on-investment (VOI); and the potential cost-benefit of a TWH approach, and its return-on-investment (ROI). An organization may be reluctant to provide such information to investigators, even though employers themselves typically ask for such results before committing themselves to participate. Also, upon implementation, workers may not wish to provide information to employers, a possibility that should be considered in assessing the practicality of such an interaction to be useful.

Back to Top | Article Outline

Study Design Options

Studies for measuring TWH program impacts are of three basic designs, each of which has its strengths and limitations. The first is pre-experimental, relying primarily on post-hoc surveys, pre/post measurements, case studies, or time series studies where interventions are modulated over time. These measurements are applied to only the group receiving the TWH intervention; there is no comparison or control group. In these pre-experimental studies, the employee and sometimes the business serve as their own control, when there is a lag period of observation before implementing the intervention.

The second is a quasi-experimental design, meaning an observational study in which a comparison group is used to answer the question, “What would have happened were there no intervention?” Comparison groups can be drawn from within the organization—either as nonparticipant or unexposed populations—or from outside the organization in the form of another company or group of companies providing a normative database. Quasi-experimental studies may include evaluations of interventions initiated by employers, such as comparisons of change between intervention sites and comparison sites not randomly assigned to the intervention condition.16 Because of the nonrandom selection, sophisticated statistical techniques like multivariate analysis, propensity score matching/weighting, coarsened exact matching, instrumental variable analysis, and other adjustment methods are used to control for confounders.17–19

The third option, a randomized controlled trial (RCT), is a true experimental design, which requires random assignment of workers to a TWH program or a control condition and then comparison of the experience of treatment versus control subjects. A useful variation is the cluster randomized trial which uses a multi-level design; usually there are two levels, the individual participant and the cluster (organization, department, or other group), where the cluster is the unit of randomization.20 Although there can be challenges to conducting RCTs in workplaces, experimental designs for TWH research are encouraged when these conditions exist: organizational leaders allow it21; a long enough time horizon is available to record short- and long-term sustainable results; few inclusion or exclusion criteria are mandated, a circumstance which improves the generalizability of results; and there is minimal worry about spillover effects across sites, which would affect intervention fidelity—the degree to which the intervention is implemented as intended.

On the other hand, the RCT design was not intended for organization-level interventions and would be virtually impossible to apply, at the very least because so many organizations would be required that the research costs would be prohibitive. Further, an aversion by management to experimental study designs is understandable because, by definition, some interventions would be offered to a certain segment of workers while other workers would be denied access to potential safety and health improvements. However, this does not need to be the case if a delayed intervention control is used, such as in the stepped wedge design. For a discussion of ways to address common challenges to all workplace safety and health program evaluations, the authors refer readers to several publications that offer thoughtful consideration of the issues raised here.19,22–26

Each of these designs for studying intervention effectiveness can be supplemented with in-depth examination of the intervention process using qualitative and/or quantitative data to document intervention feasibility and the reasons for success or failure, as well as long-term sustainability.27

Back to Top | Article Outline

Selecting Comparison Groups in Non-Randomized Designs

A quasi-experimental design is generally preferred to pre-experimental because it introduces a reference group against which the experience of intervention subjects can be gauged. However, because participants (whether workers or workplaces) are not assigned randomly to the intervention and control conditions, selection bias may result. Therefore, quasi-experimental designs should draw comparison subjects from a group of individuals who are as alike as possible to the intervention subjects. Comparison subjects may be employees recruited from inside the organization or from other organizations that are similar in size, industry, region of the country, and workforce composition. In either case, treatment subjects can be matched or “twinned” with comparison subjects, or data from each group can be weighted so that, on average, the groups look the same on measurable factors. Matching or weighting variables may include socio-demographics, region, job type, industry, union/nonunion, hourly/salaried, health plan, plan generosity, medical costs, healthcare service use, adequacy and completeness of the program, and readiness to adopt safe and healthy habits. The variables chosen for matching or weighting would include forces likely to influence program participation, as well as the outcomes of interest aside from participation.

Quasi-experimental designs are sometimes achieved by leveraging large, integrated databases that facilitate detection of small differences among groups; however, those differences, while statistically significant, may not be meaningful in terms of program impact. That said, smaller datasets may prove equally problematic, as the magnitude of effect needed to show a positive result may then be less likely, given the intervention. This further demonstrates the need to consider effect size and the population sample size as part of the study design. Quasi-experimental designs may also be applied retrospectively to determine what occurred following a given TWH intervention and what the cumulative impacts might be. This approach may offer an advantage over prospective trials, which may require a long-time horizon for data collection and for the intervention to take root within an organization. Indeed, to document the benefits of a TWH program, years may be needed to fully realize the advantages of one approach over another. However, the authors recognize that this is not always feasible, and it may also introduce other challenges (such as attrition, a topic discussed later on). In some instances, assessing more intermediate outcomes may be an achievable compromise, as discussed in the measurement portion of this paper.

Back to Top | Article Outline

Participatory Approaches in TWH

A key distinguishing feature among intervention studies—whether in the workplace or in the community—is whether or not the individuals who are the subjects of the study actually participate in designing and carrying out the intervention and/or the evaluative data collection and analysis (ie, the research). Such methods were pioneered in community-based participatory research and have since been adopted by some workplace investigators.28 For TWH programs, workers’ insights into the interactions between work and nonwork hazards are particularly valuable, because they can shape the selection of the intervention itself as well as how it is delivered and communicated. Participatory involvement is critical for all stakeholders and participants at all levels, including executives and senior leaders, middle management, labor representatives, and workers. A number of intervention research studies have relied on worker participation and emphasize that participation is vital at all stages of program design, development, and evaluation.10,29

Prioritizing worker involvement is critical to ascertain the culture and climate of a worksite, including which type of intervention is most appropriate. Another continual part of the process is creating communication plans through a loop-back effect, whereby information and results are disseminated to labor representatives, workers, and employers.

Back to Top | Article Outline

Data Reporting

It is important to be transparent in describing methods and analysis and providing data summaries that are adequate to understand the study sample. This includes delineating the studied population in an attrition table and describing numerically how many individuals were dropped because they did not satisfy specified inclusion or exclusion criteria. Creating tables containing raw numbers is vital to the credibility of the study. Descriptive data tables should include the number of individuals studied (“n”) and values such as group means, medians, standard deviations, ranges, and outliers. Researchers should explain the steps taken to clean and normalize the data by providing cut-off values for nonvalid responses, explaining how negative values or missing data were handled, and describing any transformation techniques or other methods used to address skewed data. Finally, supplying formulas applied in multivariate analyses, along with the rationale for including or excluding key variables in the model, is key for meaningful interpretation as well as for replication and comparisons across studies.

Back to Top | Article Outline

Key Conclusions

There are many points to bear in mind when selecting study designs to measure the effectiveness of TWH programs. Each intervention is necessarily carried out in a rich natural environment and whose characteristics may be determinative of the outcome. Balancing feasibility and process insights with potential threats to validity and rigor is always a challenge, no less in TWH. Within the context of study design selection and planning, the determination of appropriate comparison groups and inclusion of involved stakeholders using participatory approaches are important considerations. Lastly, in order to understand multi-component interventions fully, transparent and detailed data-reporting processes are central.

Back to Top | Article Outline

Potential Selection Bias, Attrition, and Unanticipated Changes to the Work Environment That Could Affect TWH Research

Selection Bias

As in all observational research, sample type and size as well as participation rates may be influenced by selection bias, which may then impact program outcome estimates. These concerns are particularly relevant to TWH research, given that TWH interventions occur in the workplace rather than a laboratory. In addition, any intervention that leads to modification of the job and/or workplace might be more susceptible to subject refusal or withdrawal from participation, potentially leading to selection bias.

Selection bias is an inclination or partiality introduced by the selection of individuals, groups (such as workers or worksites), or data for analysis in such a manner that accurate randomization is not realized.30 The effect is that the sample acquired may not fully represent the intended study population, because program participants and nonparticipants may not be similar enough to make valid comparisons of outcomes or adequately signify the eligible population. As a result, the representation of the sample may be compromised and conclusions may be limited or even wrong.

Selection bias might occur at the organizational level as well as the worker level. If a study includes multiple worksites, some may be willing to participate but others not, and willingness might be associated with size, readiness for change, innovation, or leadership's commitment to worker safety and health. Similarly, individual workers within those worksites who volunteer to participate may differ from their counterparts who decline to participate (as in demographics, baseline well-being, risk behaviors, or communication styles). Hence, it is best to use a control group or workplace that is equally willing to participate, as determined by the investigators in preliminary screening for eligibility and willingness. This reduces the likelihood of confounding by those distinguishing characteristics. Although not always possible, using a RCT design with all participants recruited via standard protocol may help alleviate potential selection bias and other biases.

Next, obtaining a large, diverse sample of workers will likely aid in achieving a more accurate representation of the population of interest, along with a more accurate analysis of how participating workers differ from those who do not. There are several econometric approaches and techniques that can help address and control for selection bias due even to unobservable determinants of the outcomes of interest.31 Terza et al32 describe an extension of these techniques to work in health-related program evaluations.

Nonetheless, despite best attempts, selection bias may be inevitable. Studies with substantial selection bias may still prove useful by highlighting the issue of effectiveness (in the “real world”), even if they do not address the issue of efficacy (in an ideal situation). If participation is less than expected, using qualitative methods to better understand the reasons for nonparticipation can add to the study findings and improve future interventions. Finally, another strategy to help counter this potential inevitability is a series of replication studies in different settings and populations that demonstrate that selection bias had minimal influence on study outcomes.

Back to Top | Article Outline


Attrition occurs when individuals or participating worksites are lost from a sample over time30 and may, in fact, produce selection bias if there is differential loss to follow-up. Although not exclusive to the TWH field, attrition can occur at multiple levels and thus has particular applicability here. At both the organizational and worker levels, potential factors include changes in priorities, mergers, downsizing, and turnover, as well as type of work environment and work itself. In fact, some work settings, such as contingent, seasonal or precarious work, are more prone to attrition than others. Moreover, support (or lack thereof) at the organizational and management levels is a vital factor that can influence workers’ ability to participate and remain in a study, thereby impacting both recruitment and attrition.

Although a longer study timeframe is often most ideal, strategies to address attrition might include a more compact study timeframe, especially in an industry prone to high turnover rates. While shorter-term studies limit the types of outcomes that can be assessed, they do tend to suffer less from attrition. Having solid leadership buy-in at the outset of any study is highly encouraged to address this issue, as is having strong champions, incentives, and continuous active and strategic communication between participants and those implementing the TWH program. Understanding the patterns of attrition, whenever possible, is an added but useful effort. For example, it would be valuable to know whether participants—at either the organizational or worker level—are missing at random or not. If an organization drops out of a study, investigators should attempt to conduct a qualitative “exit interview” to fully understand why. Although some researchers take this approach, such interviews are not widely performed and the information gleaned is not always shared in publications. In an emerging field such as TWH, publishing these findings and lessons learned can be particularly beneficial.

Back to Top | Article Outline

Unanticipated Changes to the Work Environment

In all work environments, unexpected changes during a research study can occur at multiple levels and with different implications. These changes can include external ones, such as political, legal, social, and economic, as well as internal ones within the work environment (structure, leadership, management, and workforce). Although the impact of some changes may be easier to ascertain and quantify than that of others, researchers should consider using an array of assessment tools to establish the baseline status of the organization and to track any changes in relation to potential unexpected external and internal issues that might impact the organization, its workers, and existent TWH endeavors. For instance, an external change impacted by the state of the economy might influence an organization's hiring, turnover rates, layoffs, morale, and working conditions. All of these factors might be applicable internally, to workers’ safety, health, and well-being outcomes. Although some changes can positively or negatively affect an organization's current TWH efforts, it is wise to try to salvage and continue efforts and to report changes accordingly. Even if an unexpected change may affect the nature of integration, the participating parties may still benefit and valuable reportable information for the scientific community and other stakeholders can be obtained.2

Back to Top | Article Outline

Key Conclusions

Selection bias, attrition, and unanticipated changes to the work environment are not unique to the field of TWH. Nonetheless, we highlight them here given that they are ubiquitous and potentially problematic at both the organizational and worker level. When the “real-world” setting creates such limitations for the chosen method and plan, researchers are encouraged to describe those in their publications and to consider all available statistical approaches to handle these challenges.

Back to Top | Article Outline

Qualitative and Mixed-Methods Approaches in TWH Research

Why Use Qualitative or Mixed-Methods Approaches?

As noted earlier, study design and research methods used in an investigation should always align with the research question(s) at hand. Increasingly, researchers have been gathering qualitative data—particularly useful when studying new and complex areas or questions, hard-to-measure constructs, and experimental settings—to help describe, understand, and explain their quantitative findings. Despite this usefulness and recommendations for using a mixed-methods (quantitative and qualitative) approach in intervention research, few TWH studies have done so, especially in RCTs. In 2014, O’Cathain et al33 reviewed all of the health research proposals funded in the United Kingdom from 2001 to 2010 that involved RCTs. They found that only 2% of the funded proposals also involved qualitative methods, and those provided only brief descriptions of the qualitative methods used.

The evidence base would be more actionable if researchers used qualitative or mixed-methods research approaches alongside or independent of quantitative methods. For instance, case studies (see next section), focus groups, Photo Voice,34 and peer interviews can empower workers to share their insights and experiences and learn from one another.35 Online social media forums with images and photography sharing, toolboxes, and active online community discussions are other approaches. These strategies may be used throughout the research process and combined with quantitative assessments to confirm scientific accuracy and provide metrics around participation. These metrics and other measures of dissemination and implementation13,14 would offer supplementary information on the process and strategies used, why workers or employers participate or not, and the makeup of the intended audience.

Whereas using only qualitative research methods can provide the researcher with useful information about the effectiveness of a TWH intervention, some circumstances may call for a mixed-methods approach to more comprehensively answer the what, why, and how of the research issue being examined.36 Mixed-methods allow for the inclusion of mental schemes into the same area of analysis, to enable a more collective understanding of a phenomenon.37

Back to Top | Article Outline

Additional Qualitative or Mixed-Methods Research

The Experience Sampling Method (ESM) is a qualitative research method that provides insight into workers’ experiences by frequently sampling their behaviors or perceptions before, during, or even after the study period.38 Often the sampling is conducted at randomly selected times throughout 1 or more days.39 The data gathered in this “real life” monitoring activity is then reviewed with the workers through in-depth interviews to better understand the factors that might have influenced the study-related behaviors or work patterns. Ecological Momentary Assessment (EMA) is another qualitative approach that provides real-time and repeated qualitative and quantitative data.40 EMA gathers qualitative data by using study participants’ mobile phones to ascertain their emotions, cognition, and behaviors, producing data on safety and health outcomes within the work and community environment at different times in a day. Regardless of the qualitative or mixed-methods approach used, assessments must be rigorous and tailored for the particular work organization and workforce. In this context, Table 1 provides some examples of how qualitative and mixed-methods approaches can be used at various stages of a TWH intervention.



Back to Top | Article Outline

Reporting of TWH Results From Mixed-Methods Approaches

Publishing studies that use qualitative or mixed-methods approaches can be challenging, especially when the approaches vary in types of results and research questions, or they produce conflicting results.41 The authors of this paper strongly encourage journal editors to consider enlisting more reviewers who are well-versed in mixed-methods and then in publishing such findings to help advance the field of TWH more rapidly. Additional dissemination channels that researchers might consider include webinars, social media, and other web and communication outlets. These channels can improve dissemination, enhance visibility to more diverse audiences, and draw attention to mixed-methods research in TWH studies. This dissemination reach can be measured with different tools, such as altmetrics.42 Also known as “alternative metrics,” they measure influence and reach of academic research and other outputs by following traffic to and conversations about them across online and social media outlets.43 Such dissemination channels and metrics hold the potential to improve adoption of research via targeted communication to practitioners and occupational safety and health professionals who might not have access to paid peer-review journal articles and/or also use other channels to glean the latest research findings.

Back to Top | Article Outline

Key Conclusions

Relevant stakeholders should make a greater case for—and support—studies using qualitative and mixed-methods approaches, using both more commonplace and innovative techniques. Studies that are more inductive enable researchers to discover new patterns and relationships among variables. They can build on prior findings and existing frameworks as well as raise questions that lead to new hypotheses tests. Researchers have noted that the big advances in science come from induction and not from deduction, which merely confirms the correctness of prior explanations but adds little to new knowledge.44 Finally, in addition to traditional publication formats, authors might also consider using alternative ones such as online forums and social media to draw more attention to such methods (eg, case studies) that are not as widely reported and, as a result, may be overlooked and/or under-appreciated.

Back to Top | Article Outline

Case Study Design in TWH Research

Why Case Studies?

Case studies are a promising research method in the developing field of TWH, as they may be applied to clarify9 highly complex issues and concepts such as integration, leadership, worker engagement, culture, confidentiality and privacy of workers, and worker well-being. One of the greatest strengths of a case study here is the opportunity to gather information on the context and rationales for TWH efforts, their reach and depth vis-à-vis the extent of participation among eligible members of the target audience, their specific details, and their perceived impact. Indeed, the ability to query in detail the why and how lies at the heart of the appeal of case studies.45 There is a need to document interventions in terms of how they are delivered as well as how they are received, to gain a fuller understanding of not only the many ways that the concept of TWH can support worker safety, health, and well-being but also the obstacles that might arise in implementing TWH programs.

Case studies are in-depth investigations of a single person, group, event, or community.45 Typically, data are gathered from a variety of sources and by using several different methods (such as direct observations, key informant interviews, focus groups, worker surveys, quantitative data collection, or interviews) to understand complex social phenomena. Case studies describe and investigate contemporary real-life phenomena through detailed contextual analysis of a limited number of events or conditions and their interactions and relationships. They are routinely used in social and business research46 and may be retrospective, focus on current activities, or be conducted longitudinally.

An example of a case study representative of TWH efforts is the LifeWorks@TURCK program (Minneapolis, MN).47 This case study presents the 10-year experience of a medium-sized manufacturing facility that intentionally addressed the safety, health, and well-being of its workers, their families, and the community and in the process fundamentally changed its corporate culture. The case study provides context, is organized according to a set of best practice program design principles, and includes both quantitative assessment of program performance as well as qualitative perspectives and viewpoints.

Back to Top | Article Outline

Approach to the Case Study Design

Ideally, a case study follows a set of principles and characteristics that generate a well-rounded understanding of the cultural systems of action at play (ie, the interrelated activities operative in the TWH program at hand).48 Case study designs need to be appropriate to the research question, record systematically a “web of evidence” using qualitative, quantitative, or mixed-methods, and be linked to a theoretical framework.

Case study designs might be single-case or multiple-case. A single-case design is adopted when no other cases are available for replication; that is, it pertains to a single unique event. A multiple-case design must follow a replication rather than a sampling logic, using a rigorous and standardized research framework.49 A case study may also be categorized as exploratory, descriptive, or explanatory.50–52 These three categories of case studies may be seen to provide different levels of depth that satisfy the three tenets of the qualitative method to describe, understand, and explain.

More specifically, exploratory case studies involve efforts prior to defining research questions or hypotheses and may be considered a prelude to social research. It is assumed that a conceptual study framework created a priori guides this effort, and pilot projects are useful in determining the final protocols to be used. Case selection for exploratory studies should maximize the opportunity to learn about the situation.

Descriptive case studies cover the depth and scope of the case being studied, since the overall aim is to form hypotheses of cause–effect relationships. This may include study and in-depth description of multiple conditions or situations that may be compared with each other and to an idealized hypothetical or theoretical construct or pattern. Pattern-matching techniques (comparisons of two patterns to determine whether they are the same or different) may be used in analyses of complex and multivariate cases and are defined as the linkage of data to propositions as well as criteria for interpreting the study findings.53

Explanatory case studies are designed to address causality. They describe and examine data closely at both a surface level and a deep level to explain phenomena observed in the data. Often, explanatory cases will involve complex and multivariate situations. Here, just as in descriptive cases, pattern-matching may be used in the analysis.53

Back to Top | Article Outline

Advantages and Disadvantages of Case Studies

Advantages of case studies include the fact that examination of the data is conducted within the context of its use. This is in direct opposition to what happens in experimentation, where there is an attempt to deliberately isolate a phenomenon from its context and focus on a limited number of variables. Case studies also allow for both quantitative and qualitative analyses of data, along with detailed qualitative accounts that help to explain complexities of real-life situations, a promising consideration for TWH research.

One of the key uses of case studies is to develop an understanding of the ways in which employers and organizations select, adopt, and sustain certain TWH approaches. This use is particularly important given that a complete realization of the TWH approach must be built through stages. It is a learning process that requires development of managerial and worker perspectives and practices. Different employers are likely to take diverse paths in their response to the particular nature of safety, health, and well-being issues they confront and in their relationship to other organizational goals. Sometimes TWH interventions come as a packaged program prepared by others, and sometimes they may be more homegrown. In either case, it is useful to know who the key leaders and facilitators are, the resources they rely upon, the origins of their thinking, and the problems and priorities they are addressing. This is useful not only for devising ways to promote the TWH approach but also for gaining insight into variations in paths to TWH, for developing guidance concerning a pathway, and even for supporting the case for TWH by documenting that many of its components are logically compelling responses to immediate problems and opportunities. Any management innovation may be adopted with attention to its specific fit to problems at hand or, instead, based on a perception that it is universally effective, regardless of context. Case studies may help researchers better understand the different pathways to adoption and how these might translate into various contexts. Although TWH approaches may be broadly considered as ideal practice everywhere, documenting instances of the situation-specific and well-considered adoption of its elements can help make the case for wider adoption and adaptation. Finally, case studies in mixed and/or triangulation methods similarly use more than one approach to explain and investigate a research question more comprehensively by exploring it from different angles. The use of either or both of these may confirm validity of the overall process, findings, or conclusions, and recommendations offered by a TWH program.54

Disadvantages of case studies include the perception that they lack rigor. They may be labeled as being too long or difficult to conduct and are thought to produce massive amounts of documentation requiring extensive time and resources. At times, they are perceived as providing little basis for scientific generalization because they use only one or a few subjects. A contrary perspective is that in any single application in a work setting, there typically is little or no emphasis on generalization. The localized approach is meant to improve the situation at that particular organization at that particular time, regardless of the generalizability of the TWH intervention, how it occurred, and its results.

Context, in terms of the research questions that the TWH study is designed to address, may be most important for determining whether a case study approach is preferable. For example, although a case study may be less effective in assessing efficacy where causality and rigor are central, the use of it may be more suitable for generating hypotheses or examining possible mechanisms of how an approach might work within certain types of settings.

Back to Top | Article Outline

Key Conclusions

Though there are both advantages and disadvantages to using a case study approach, it is a valid method to further the TWH evidence base and complements more quantitative comparisons. The techniques delineated above can be used to support case studies individually and also to pool their results. Using case studies may even prove to be a more attractive learning approach and buy-in option for organizations that place attention on what similar competitor organizations are doing to assure a safer and healthier workforce.

Back to Top | Article Outline

Measures and Outcomes Relevant to TWH

TWH Measures and Outcomes

TWH approaches have holistic applicability for wide-ranging worker safety, health, and well-being issues across occupations and industries. This is exhibited in the numerous and varied outcomes and measures reported across TWH studies.6,8,9 Although a diversity of applicable measures and outcomes can be beneficial for new learning, it is challenging to assess and compare findings across studies that include different measures and outcomes. Indeed, Feltner et al10 reviewed 15 studies on the effectiveness of TWH interventions and were unable to determine the strength of evidence across the studies because of the diversity of measures as well as methodological differences and limitations.

As is the case with all emerging areas, the definition and scope of the TWH field continue to evolve on the basis of scientific research findings and input from stakeholders.12 For example, the revision of NIOSH's definition of TWH in 2015 better reflects TWH priorities and helps to differentiate the TWH paradigm from traditional workplace wellness programs that do not integrate worker protection fundamentals.12 This revision had implications for the TWH-related measures and outcomes selected and studied subsequently. That said, studies based on the earlier definition of TWH were pivotal in building the evidence base and foundational in shaping the eventual refining and refocusing of the TWH paradigm.2,5,6,8,10,55–57

The Pathways to Prevention 2015 Meeting recommended that common TWH measures were needed to compare results across research studies.11 TWH measures include both independent and dependent variables. The independent variables are the baseline characteristics of workers and workplaces, as well as the interventions themselves, meaning the workplace policies, practices, and programs that are intended to ensure safety and grow health. For instance, relevant measures for small businesses or contingent workforces may differ from those addressing issues faced by workers in large organizations or more stable workforces. The dependent variables, or outcomes, similarly include organizational changes in these same workplaces, along with health, behavioral, attitudinal, and other characteristics of the workers that might demonstrate the impact of the intervention.58 Measures may be obtained at the organizational level from a worksite representative or at the worker level through an employee survey. Outcome measures focused on the organizational level may include new worksite programs, policies, and practices. At the worker level, they may include perceived working conditions and supervisor support, for example.

The authors recognize and endorse that the wide variety of possible TWH programs will necessarily require different independent and outcome variables, in line with the types of interventions being evaluated and their hypothesized or desired changes from baseline. Therefore, each set of investigators will inevitably select specific measures that are tailored to their research questions. It is true that comparability of findings would be facilitated by the adaptation of core or standard TWH measures. For the outcomes, this might entail assessment of broad, multi-dimensional well-being in addition to the specific health conditions targeted by the intervention. While the authors concur with prior work that steps toward achieving a core set of relevant measures and outcomes is a useful endeavor, in this report, the authors outline areas of measures and provide examples to illustrate them instead of listing specific measures that should only be used. The authors contend that doing otherwise at this juncture may not only constrain the TWH field but also confirm such measures and outcomes without the lengthy base of research evidence needed to validate them.

Table 2 provides a broad overview of examples of TWH measures and outcomes that fall into several categories, span multiple levels, and have short-term and/or long-term implications, as described in the TWH literature and referenced throughout this document. All structural and process measures are assessed at the organization level, but outcome measures can also be assessed at the worker level. In this sample table we first delineate program structure and process measures. Structural measures assesses the programs, policies, and environmental supports and degree of integration and can be guided as well as evaluated by established and validated tools such as the CDC Worksite Health ScoreCard,59 Leading by Example,60 the CPH-NEW Program Evaluation Rating Sheet,61 and similar organizational assessment tools.62 Here, also, the plurality and evolving nature of available measures has led to inconsistency among investigations; standardization may become possible in the future. Process measures consist of these factors: (1) the degree to which the program (as a whole or specific elements) is implemented as planned (its fidelity), (2) the extent to which all aspects of the interventions are put in place with appropriate intensity, frequency, and duration (its dose), and (3) the degree of awareness, participation, and satisfaction with various program components and the program overall (the dose received and program reach).



Next, the table illustrates six key categories of outcomes for consideration at the organization level: leadership, policy, occupational injury and illness rates, working conditions and environment, safety culture, and productivity. At the worker level, we highlight six relevant categories: safety practices, health (both short- and long-term), job satisfaction, well-being, health care utilization, and personal health behaviors. It may also be useful to ascertain the same particular measure(s) at both organization and worker levels to compare perceptions of the employers versus employees. Table 2 is not meant to provide an exhaustive list of TWH measures and outcomes; rather, it offers several examples for consideration that have been and continue to be used in TWH studies (both intervention and nonintervention).

Back to Top | Article Outline

Emerging TWH Measures

Many of the measures listed above are not specific to the evaluation of TWH interventions (although the integrated use of them is), but some of the most relevant measures may only just be emerging in the literature. These assess the degree to which the intervention integrates protection from work-related safety and health hazards with promotion of injury and illness prevention efforts. As reflected in the table, structural measures include program elements—addressing both work and nonwork risks and opportunities, and coordination of activities—and the degree of integration by design.2,3,29,63 Process measures that may be considered include recruitment methods and statistics (such as eligible organizations/workers, those deemed ineligible and why, those enrolled, number completing measurement waves, and reasons for dropout), fidelity of intervention implementation and integrity, and dose (adoption and participation rates).

The social validity of TWH interventions is another valuable category of process measurement and includes participant ratings of the acceptability of the goals, procedures, and outcomes of an intervention. Finally, investigators should also consider assessing transferability—the successful application of the TWH intervention to diverse organizations, occupations, industries, and workers.

Back to Top | Article Outline

What to Consider When Selecting TWH Measures and Outcomes

As noted above, the unique worksite and workforce will determine the research question(s) and study design, which in turn will guide the measure and outcome selection process. Each measure should be considered in terms of whether it requires single, repeated, short-term, and/or long-term assessment. The conceptual causal model that underlies the study should also specify whether each outcome is considered intermediate or lagging. Examples of intermediate outcomes include eliminated or reduced hazards, workplace safety practices, and personal health behaviors. Lagging outcomes may include incidence of musculoskeletal disorders; disability related to injuries, illness, or chronic disease; and overall mortality rates.

Well-being, in particular, is a broad concept that extends beyond traditional conceptions of occupational safety and health and is an outcome that can be impacted by TWH interventions.64 Defining and measuring well-being can be complex, but many validated measures are available that address such facets as functional status, quality of life, and stress or distress. Recently, NIOSH and RAND published an article on the development of a worker well-being framework to expand upon current understanding of well-being in the occupational safety and health literature.65

Outcomes measured objectively, especially if sequential or longer-term, or those obtained from data collection efforts that are other than self-reported, are highly desirable. Examples of objective outcomes include organizational records of recordable injuries, absenteeism, turnover, and production. It should be noted, though, that some investigators do not view these as truly objective because they rely fundamentally on the behavior of a worker and/or employer representative. Organizations may already be collecting such data for other purposes (for instance, occupational injury and illness cases recorded on OSHA logs) and be willing to share them with researchers for use as outcome measures in a study. Objective measures can also include researcher-assessed environmental/exposure measurements (vibration, noise, particulate, or chemical, eg), biomarkers (such as blood, urine, saliva, or hair samples), anthropometric measurements, physiological assessments, and direct observations of work conditions or practices. Obtaining workers’ permission to collect, analyze, and report data and observing confidentiality and privacy are paramount.

Back to Top | Article Outline

Available Resources for Consideration

It is key to continue to develop reliable, validated, evidence-based TWH resources, assessment tools, scales, and surveys with both quantitative and qualitative approaches to comprehensively assess and evaluate measures and outcomes, including those relevant to integration. In addition to screeners, questions, and checklists about the presence or absence of specific outcomes (such as injuries or disease diagnosis and treatment), investigators should consider the benefits and challenges of brief, simple measures (a single question to measure self-rated safety and health,66 for instance) and more broad measures (multiple questions used to measure a single construct,67 eg), which collect deeper data but may be more costly and time-consuming to administer and more complicated to analyze. Relevant resources to aid in this process and guide intervention efforts are available on the NIOSH TWH website and elsewhere.3,62,68,69

Back to Top | Article Outline

Key Conclusions

It is hoped that, over time, a core set of relevant measures and outcomes will be adopted by TWH researchers. Meanwhile, it is appropriate and even valuable to have a variety of independent and dependent variables under study in this still emerging and wide-ranging discipline. Ultimately, focusing on the specific issues to be solved should govern the selection of metrics and measures needed to respond to those very issues. The prime focus of this section is on organizational level and worker level outcomes and measures; however, the authors recognize that there are implications of addressing these that go beyond the scope of the worksite, working environment, and work itself, not underscored in great depth in this report. Future TWH studies might consider including measures and outcomes capturing these additional levels of influence, using some of the emerging resources highlighted.

Back to Top | Article Outline


Final Thoughts and Next Steps

This Workshop proceedings summary of methodological and measurement considerations in TWH research is intended to highlight existing approaches, their advantages and disadvantages, as well as propose additional options that hold promise for advancing the field. Many of these principles are understood and articulated in other sources but may not be consistently observed or practiced in research reported in the TWH literature. Comparability of scientific findings is an important focus to solidify any field but cannot be sought at too early a stage. The TWH field has broad applicability, making a narrowing of its scope by the prescription of specific recommendations, methods, measures, and outcomes that must be used potentially too limiting and disadvantageous.

Given its nascency and complexity, further building of the TWH scientific evidence base is essential. Future forums should build upon these Workshop topics by explicitly and directly connecting the methodological best practices offered here to specific methodological limitations of the TWH literature and studies, as the field continues to grow. Finally, additional exploration of TWH research methodological issues is necessary for not only intervention research but also basic/etiologic, translational, and surveillance research, and concrete examples are needed of how challenges can be overcome to drive research practices in the field of TWH.8,10

Back to Top | Article Outline


The authors express their thanks to all those who participated in the planning and execution of the TWH Research Methodology Workshop. In particular, the authors thank the following: the University of Iowa's College of Public Health and Healthier Workforce Center of the Midwest; the Workshop co-chairs, Dr. Diane Rohlman and Dr. Kevin Kelly, and their staff; the Workshop moderators, Dr. Ron Goetzel, Dr. Kevin Kelly, Dr. Sara Luckhaupt, Ms. Jeannie Nigam, Dr. Nico Pronk, and Dr. Diane Rohlman; the Workshop external planning committee; Dr. Steve Sauter, Sarah Mitchell, Reid Richards, and Seleen Collins for manuscript review and editing; and the National Institute for Occupational Safety and Health.

The findings and conclusions in this paper are those of the authors and do not necessarily represent the official position of the National Institute for Occupational Safety and Health, Centers for Disease Control and Prevention.

Back to Top | Article Outline


1. Schill AL, Chosewood LC. The NIOSH Total Worker Health program: an overview. J Occup Environ Med 2013; 55:S8–S11.
2. Sorensen G, McLellan D, Dennerlein JT, et al. Integration of health protection and health promotion: rationale, indicators, and metrics. J Occup Environ Med 2013; 55:S12–S18.
3. Lee MP, Hudson H, Richards R, et al. Fundamentals of total worker health approaches: essential elements for advancing worker safety, health, and well-being. Cincinnati: OH. U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health. DHHS (NIOSH) Publication No. 2017-112; 2016.
4. Institute of Medicine. Integrating Employee Health: A Model Program for NASA; 2005.
5. Punnett L, Cherniack M, Henning R, Morse T, Faghri P, Team CNR. A conceptual framework for integrating workplace health promotion and occupational ergonomics programs. Public Health Rep 2009; 124 (suppl):16–25.
6. Pronk N. Integrated worker health protection and promotion programs: overview and perspectives on health and economic outcomes. J Occup Environ Med 2013; 55:S30–S37.
7. National Institute for Occupational Safety and Health. The Hierarchy of Controls Applied to Total Worker Health; 2015. Available at: Accessed October 12, 2017.
8. Anger WK, Elliot DL, Bodner T, et al. Effectiveness of total worker health interventions. J Occup Health Psychol 2015; 20:226–247.
9. National Institute for Occupational Safety and Health. Research compendium; The NIOSH Total Worker Health (Program: seminal research papers) 2012. Cincinnati, OH: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, DHHS (NIOSH) Publication No. 2012-146; 2012.
10. Feltner C, Peterson K, Palmieri Weber R, et al. The effectiveness of total worker health interventions: a systematic review for a National Institutes of Health Pathways to Prevention Workshop. Ann Intern Med 2016; 165:262–269.
11. National Institutes of Health and National Institute for Occupational Safety and Health. Pathways to Prevention Workshop: Total Worker Health: What's Work Got to Do With It; 2015? Available at: Accessed September 7, 2017.
12. National Institute for Occupational Safety and Health. National occupational research agenda (NORA)/National Total Worker Health agenda (2016–2026): A national agenda to advance Total Worker Health (research, practice, policy, and capacity). Cincinnati, OH: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, DHHS (NIOSH) Publication 2016-114; 2016.
13. Schulte PA, Cunningham TR, Nickels L, et al. Translation research in occupational safety and health: a proposed framework. Am J Ind Med 2017; 60:1011–1022.
14. Dugan A, Punnett L. Dissemination and Implementation Research for Occupational Safety and Health. Occup Health Sci 2017; 1:29–45.
15. Rohlman D, Campo S, Hall J, Robinson E, Kelly K. What could Total Worker Health® look like in small enterprises? Ann Work Exposur Health. In press.
16. Dennerlein JT, O’Day ET, Mulloy DF, et al. Lifting and exertion injuries decrease after implementation of an integrated hospital-wide safe patient handling and mobilisation programme. Occup Environ Med 2017; 74:336–343.
17. Pourhoseingholi MA, Baghestani AR, Vahedi M. How to control confounding effects by statistical analysis. Gastroenterol Hepatol Bed Bench 2012; 5:79–83.
18. Stuart EA. Matching methods for causal inference: a review and a look forward. Stat Sci 2010; 25:1–21.
19. Schelvis RM, Oude Hengel KM, Burdorf A, Blatter BM, Strijk JE, van der Beek AJ. Evaluation of occupational health interventions using a randomized controlled trial: challenges and alternative research designs. Scand J Work Environ Health 2015; 41:491–503.
20. Campbell MK, Piaggio G, Elbourne DR, Altman DG, Group C. Consort 2010 statement: extension to cluster randomised trials. BMJ 2012; 345:e5661.
21. Carr LJ, Leonhard C, Tucker S, Fethke N, Benzo R, Gerr F. Total worker health intervention increases activity of sedentary workers. Am J Prev Med 2016; 50:9–17.
22. Ozminkowski R, Goetzel R. Getting closer to the truth: overcoming research challenges when estimating the financial impact of worksite health promotion programs. Am J Health Promot 2001; 15:289–295.
23. Kristensen TS. Intervention studies in occupational epidemiology. Occup Environ Med 2005; 62:205–210.
24. Goldenhar LM, LaMontagne AD, Katz T, Heaney C, Landsbergis P. The intervention research process in occupational safety and health: an overview from the National Occupational Research Agenda Intervention Effectiveness Research team. J Occup Environ Med 2001; 43:616–622.
25. Griffiths A. Organizational interventions: facing the limits of the natural science paradigm. Scand J Work Environ Health 1999; 25:589–596.
26. Olsen O, Albertsen K, Nielsen ML, Poulsen KB, Gron SM, Brunnberg HL. Workplace restructurings in intervention studies - a challenge for design, analysis and interpretation. BMC Med Res Methodol 2008; 8:39.
27. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 2013; 14:15.
28. O’Toole TP, Aaron KF, Chin MH, Horowitz C, Tyson F. Community-based participatory research: opportunities, challenges, and the need for a common language. J Gen Intern Med 2003; 18:592–594.
29. Punnett L, Warren N, Henning R, Nobrega S, Cherniack M, Team CNR. Participatory ergonomics as a model for integrated programs to prevent chronic disease. J Occup Environ Med 2013; 55:S19–S24.
30. Lavrakas P, ed. Encyclopedia of Survey Research Methods. Washington D.C.: Sage Publications; 2008.
31. Vella F. Estimating models with sample selection bias: a survey. J Hum Resourc 1998; 33:127–169.
32. Terza JV, Basu A, Rathouz PJ. Two-stage residual inclusion estimation: addressing endogeneity in health econometric modeling. J Health Econ 2008; 27:531–543.
33. O’Cathain A, Thomas KJ, Drabble SJ, Rudolph A, Goode J, Hewison J. Maximising the value of combining qualitative research and randomised controlled trials in health research: the QUAlitative Research in Trials (QUART) study – a mixed methods study. Southampton (UK): NIHR Journals Library (Health Technology Assessment, No 1838). Available at: https://wwwncbinlmnihgov/books/NBK262060/2014. Accessed November 20, 2017.
34. Flum MR, Siqueira CE, DeCaro A, Redway S. Photovoice in the workplace: a participatory method to give voice to workers to identify health and safety hazards and promote workplace change-a study of university custodians. Am J Ind Med 2010; 53:1150–1158.
35. Given L, ed. The Sage Encyclopedia of Qualitative Research Methods. Beverly Hills, CA: Sage Publishing; 2008.
36. Creswell JW, Klassen AC, Plano Clark VL, Smith KC. Best practices for mixed methods research in the health sciences. Office of Behavioral and Social Sciences Research, National Institutes of Health; 2011.
37. Greene J. Mixed Methods in Social Inquiry. San Francisco:Jossey-Bass; 2007.
38. Zirkel S, Garcia J, Murphy M. Experience-sampling research methods and their potential for education research. Educ Res 2015; 44:7–16.
39. Pronk NP, Katz AS, Lowry M, Payfer JR. Reducing occupational sitting time and improving worker health: the Take-a-Stand Project, 2011. Prev Chronic Dis 2012; 9:E154.
40. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol 2008; 4:1–32.
41. Patton M, ed. Qualitative Research and Evaluation Methods (4th ed.). Thousand Oaks, CA: Sage; 2015.
42. Priem J, Taraborelli D, Groth P, Neylon C. Altmetrics: A Manifesto; 2010. Available at: Accessed January 17, 2018.
43. Priem J, Piwowar HA, Hemminger BM. Altmetrics in the wild: Using social media to explore scholarly impact; 2012. arXiv:1203.4745v1.
44. Spector P. The lost art of discovery: the case for inductive methods in Occupational Health Science and the Broader Organizational Sciences. Occup Health Science 2017; 1:11–27.
45. Yin R. Case Study Research: Design and Methods. 5th ed.Beverly Hills, CA:Sage Publishing; 2014.
46. Mills A, Durepos G, eds. Case Study Methods in Business Research. Beverly Hills, CA: Sage Publishing; 2013.
47. Pronk NP, Lagerstrom D, Haws J. Lifeworks@TURCK: a best practice case study on workplace wellness program design. ACSM's Health Fitness J 2015; 19:43–48.
48. Feagin J, Orum A, Sjoberg G, eds. A Case for Case Study. Chapel Hill, NC: University of North Carolina Press; 1991.
49. Tellis W. Introduction to case study. Qual Rep 1997; 3:1–14.
50. Yin R. Applications of Case Study Research. Beverly Hills, CA:Sage Publications; 1993.
51. Yin R, Moore G. The use of advanced technologies in special education. J Learn Disabil 1987; 20:60.
52. Pyecha J. A Case Study of the Application of Noncategorical Special Education in Two States. Chapel Hill, NC:Research Tringale Institute; 1988.
53. Mills A, Durepos G, Wiebe E, eds. Encyclopedia of Case Study Research: Pattern Matching. Beverly Hills, CA: Sage Publishing; 2010.
54. Lewis-Beck M, Bryman A, Liao TF. The Sage Encyclopedia of Social Science Research Methods: Triangulation. Beverly Hills, CA:Sage Publishing; 2004.
55. Institute of Medicine. Promising and Best Practices in Total Worker Health: Workshop Summary. Washington, DC: The National Academies Press; 2014. Available at: Accessed August 25, 2017.
56. Sorensen G, McLellan DL, Sabbath EL, et al. Integrating worksite health protection and health promotion: a conceptual model for intervention and research. Prev Med 2016; 91:188–196.
57. Bradley CJ, Grossman DC, Hubbard RA, Ortega AN, Curry SJ. Integrated interventions for improving total worker health: a panel report from the National Institutes of Health Pathways to Prevention Workshop: total worker health-what's work got to do with it? Ann Intern Med 2016; 165:279–283.
58. Pronk NP, McLellan DL, McGrail MP, et al. Measurement tools for integrated worker health protection and promotion: lessons learned from the safewell project. J Occup Environ Med 2016; 58:651–658.
59. Roemer EC, Kent KB, Samoly DK, et al. Reliability and validity testing of the CDC Worksite Health ScoreCard: an assessment tool to help employers prevent heart disease, stroke, and related health conditions. J Occup Environ Med 2013; 55:520–526.
60. Della LJ, DeJoy DM, Goetzel RZ, Ozminkowski RJ, Wilson MG. Assessing management support for worksite health promotion: psychometric analysis of the leading by example (LBE) instrument. Am J Health Promot 2008; 22:359–367.
61. Dugan AG, Farr DA, Namazi S, et al. Process evaluation of two participatory approaches: Implementing total worker health(R) interventions in a correctional workforce. Am J Ind Med 2016; 59:897–918.
62. National Institute for Occupational Safety and Health. Total Worker Health Tools: Let's Get Started; 2016. Available at: Accessed August 30, 2017.
63. Sorensen G, Sparer E, Williams J, et al. Measuring best practices for workplace safety, health and wellbeing: the workplace integrated safety and health assessment. Int J Occup Environ Med 2018; 60:430–439.
64. Schulte PA, Guerin RJ, Schill AL, et al. Considerations for incorporating “Well-Being” in public policy for workers and workplaces. Am J Public Health 2015; 105:e31–e44.
65. Chari R, Chang CC, Sauter SL, et al. Expanding the paradigm of occupational safety and health: a new framework for worker well-being. J Occup Environ Med 2018; 60:589–593.
66. Luckhaupt SE, Alterman T, Li J, Calvert GM. Job characteristics associated with self-rated fair or poor health among U.S. workers. Am J Prev Med 2017; 53:216–224.
67. Karasek R, Brisson C, Kawakami N, Houtman I, Bongers P, Amick B. The Job Content Questionnaire (JCQ): an instrument for internationally comparative assessments of psychosocial job characteristics. J Occup Health Psychol 1998; 3:322–355.
68. Health Enhancement Research Organization and Population Health Alliance. The Program Measurement and Evaluation Guide: Core Metrics for Employee Health Management; 2015. Available at: Accessed October 12, 2017.
69. Center for the Promotion of Health in the New England Workplace. The CPH-NEW Healthy Workplace Participatory Program (HWPP) Toolkit; 2011. Available at: Accessed January 14, 2018.

health; research methodologies; safety; Total Worker Health; well-being; worker

Copyright © 2018 by the American College of Occupational and Environmental Medicine