Unhealthy and overweight employees show elevated sickness absence levels and lower productivity levels, resulting in higher costs for employers.1,2 Many worksite health promotion programs (WHPPs) have been developed to improve employee health and to reduce overweight and associated higher sickness absence rates.1,2 Reviews have indicated that WHPPs can be effective in changing lifestyle behaviors.3–7 Nevertheless, because of the emphasis on outcome evaluations, it often remains unclear which problems organizations encounter when implementing a multicomponent lifestyle intervention. Understanding what happens during the implementation of a WHPP and how that affects the impact of a program is an essential step in opening the intervention “black box.”8,9
Although researchers have started to acknowledge the importance of process evaluations over the past decade, only 7% of the WHPPs evaluating a lifestyle intervention actually have performed a process evaluation. Furthermore, the content, approach, and quality of these process evaluations differed greatly between studies.10 The BRAVO@Work project attempted to remedy this by conducting a controlled trial alongside a comprehensive process evaluation to further our understanding of the natural course of program implementation. BRAVO@Work is a WHPP in which multiple lifestyle interventions (related to physical activity, smoking, alcohol use, nutrition, and relaxation) were implemented at two different worksites using a 7-step implementation strategy that facilitated structured implementation by the organizations themselves.11
Process evaluations are an important tool for studying the underlying working mechanisms of WHPPs and factors affecting implementation.12,13 They provide information on the degree of implementation of a WHPP by giving insight into whether an intervention has been delivered as planned and the extent to which the program was received by participants.14,15 This information can be used by researchers and practitioners for program improvement and optimization to further future implementation of these programs in daily practice.8 The WHPPs often report a limited impact on outcome measures such as physical activity, healthy nutrition, and obesity, and so process data may teach us more about how program implementation and reception by participants are linked to outcomes.8,15,16 This study, therefore, systematically evaluated the implementation of a multicomponent lifestyle intervention at two different worksites by (1) studying the extent and quality of program delivery and maintenance; (2) looking at employee recruitment, reach, participation, and satisfaction levels; and (3) evaluate the effectiveness of the program in a quasi-experimental controlled trial conducted alongside the process evaluation.
This process evaluation is part of the systematic formative evaluation of the BRAVO@Work project. It focused exclusively on evaluating the implementation of multiple lifestyle interventions among employees at two worksites. A researcher on site (referred to as the “embedded” researcher) continuously monitored the natural course of implementation but emphatically tried not to actively intervene in the development and implementation of the interventions. Details of the study design have been described elsewhere.11 The study protocol was approved by the Medical Ethics Committee of the University Medical Centre of Utrecht.
Study Design and Population
The study was designed as a systematic formative evaluation with a quasi-experimental controlled trial on the side, involving an Academic Hospital and a University of Applied Sciences (hereinafter “the hospital” and “the university”) in Utrecht (Netherlands). One intervention department was recruited at both organizations—a department of gynecology at the hospital with 662 employees and the health faculty of the university with 546 employees—in 2009 to 2010 through the personal network of the researchers. The management of the two intervention departments contacted a comparable control department and faculty (a cardiovascular and lung diseases department with 501 employees at the hospital and the society and law faculty with 484 employees at the university). The upper management of the intervention groups signed a letter of intent, stating that they were willing to participate in the study and agreed to the financial and organizational consequences of participating in BRAVO@Work. The control groups were not allowed to implement any interventions during the course of the study. All employees were eligible to participate in the study.
The BRAVO@Work Study
In the BRAVO@Work study, a 7-step implementation strategy was used by both intervention departments to develop, implement, and maintain interventions targeting multiple lifestyle behaviors. The principal feature of this study is that the intervention departments had sole responsibility for the development, implementation, and continuation of the interventions; the researchers acted solely as embedded observers throughout the study. An external advisor informed the project groups of both intervention departments about the 7-step strategy during the first project meetings and supplied them with a list of environmental and individual interventions and evidence-based interventions of varying complexity.3–6,17 The external advisor was present during most project meetings to answer questions and to provide guidance, but he was specifically briefed to refrain from taking the lead at any time during the project. The seven-step strategy consists of the following steps: (1) creating solid support, (2) formation of a project structure, (3) performing a needs assessment, (4) development of interventions, (5) implementation of interventions, (6) evaluation, and (7) maintenance. Detailed information about the use and interpretation of the specific steps of the strategy can be found elsewhere.11
In short, the upper management of both participating organizations was required to form their own steering committee and to appoint a project leader. The project leader was then required to establish a project group and was advised to include the following relevant stakeholders: managers and employees from different teams, a communications officer, a human resources officer, a facility manager, and a company physician. Managers and employees were eligible for project membership if they were working at the intervention department. With guidance from the external advisor, the project members conducted a needs assessment among all employees in their intervention department, resulting in a list of possible lifestyle interventions fitting the department and the needs of employees. This list was combined with the list of evidence-based interventions referred to earlier. The project groups needed to choose on a consensus basis the most appropriate and feasible interventions matching employees’ needs. The project leader then needed to draw up a project plan with information about the desired changes and project goals, an intervention template, a timeline, a budget plan, and a list of involved persons, including their tasks and responsibilities. After approval had been given by the steering committees, the project members were required to implement the interventions during the following 12 months. After the intervention year, the project groups needed to evaluate the project. This evaluation needed to be the basis for the go/no-go decision about whether to continue the program of specific interventions.
The Process Evaluation
To establish a picture of the implementation process of the interventions, eight process components were assessed: recruitment, reach, dose delivered, dose received, fidelity, satisfaction, maintenance, and contamination (Table 1).11 The frameworks of Steckler and Linnan, the RE-AIM framework, and the framework of Baranowski and Stables were combined for this purpose.14,18,19 The complete theoretical framework has been described in more detail elsewhere.11 To determine whether the interventions were implemented as planned and which interventions were maintained within the organizations, information was obtained from project members about dose delivered, fidelity, and maintenance. The process components “recruitment,” “reach,” “dose received,” and “satisfaction” were used to determine employee exposure to the interventions. More detailed information on the definitions and used methods is presented in Table 1.
Data Collection Procedures
This process evaluation was conducted within the intervention departments only, except for the component contamination (for which data were gathered at the control departments). Data on the other seven process components were collected on two levels: project group level (ie, project members from intervention departments) and participant level (ie, employees from the intervention departments). The data presented in this article were collected with:
- Web-based questionnaires distributed among all employees to assess:
- The implementation process at 9 months (T1; n = 622) and 15 months (T2; n = 613) after start of the implementation of the program at the hospital. At the university, the questionnaire was distributed 6 months (T1; n = 504) and 12 months (T2; n = 489) after start of the implementation of the program. The process questionnaire addressed awareness and satisfaction levels with the implemented interventions and the recruitment methods. The employees were asked to state in the questionnaire whether they were willing to participate in an additional semistructured interview to supplement the results.
- The effectiveness of the implemented interventions at baseline (T0) and 12- or 15-month follow-up (T2) at the university and hospital, respectively, in the employees in the intervention group (hospital T0 n = 662, T2 n = 663; university T0 n = 546, T2 n = 489) and control group (hospital T0 n = 501, T2 n = 462; university T0 n = 484, T2 n = 327). Self-reported physical activity levels, food intake, smoking status, alcohol use, and vitality were assessed.
- Semistructured interviews with a random convenience sample of employees at T1 (hospital n = 7; university n = 10) and at T2 (hospital n = 7; university n = 5). In addition, most project group members were interviewed at baseline (hospital n = 8; university n = 8), at T1 (hospital n = 6; university n = 11), and at T2 (hospital n = 3; university n = 7). The interviews were designed to address all nine process components listed earlier relating to the quality and extent of implementation of the interventions. At 18-month follow-up, the project leaders of both organizations were contacted by telephone to assess program maintenance. The principal investigator (D.W.) conducted all interviews during a face-to-face or telephone meeting at a time and location convenient for the participants. Before the start of the interview, all participants were informed about the purpose of the study and provided with reassurance about confidentiality, and all participants granted oral permission. All participants were selected by convenience sampling and were invited by e-mail to participate in the interviews. Interviews with employees lasted an average of 24 minutes (range, 8 to 45 minutes) and those with project members lasted 28 minutes on average (range, 10 to 56 minutes).
- On-site monitoring: Throughout the study period, an embedded researcher (D.W.) continuously monitored the implementation process by documenting relevant e-mail communications, minutes of project meetings, and observations in predefined spread sheets that were based on the conceptual framework developed before the commencement of the study to:
- further understand the context in which the project was implemented;
- learn about issues that project members were unaware of or that they were unwilling or unable to discuss candidly in the semistructured interviews; and
- assess whether or not the project or aspects of the project were delivered and operated as planned.
Effectiveness of BRAVO@Work
To assess employees’ physical activity levels, three questions from the validated “Injuries and Physical Activity in the Netherlands” questionnaire were used to measure whether employees had met the physical activity guidelines of the CDC/ACSM, which states that adults should engage in physical activity of at least moderate intensity for at least 30 minutes a day at least 5 days a week. To assess commuting activity (walking and cycling), one domain from the validated Short Questionnaire to Assess Health Enhancing Physical Activity was used.20,21 In addition, questions were added to assess self-reported stair use, elevator use, and how often employees spent their lunch break in an active manner. Employees’ food intake was measured as the intake of fruit and vegetables. They were asked on how many days in a normal week they consumed vegetables and fruit and to report the average number of servings (50 g) of vegetables and the number of pieces of fruit they consumed in a day. To assess employee vitality, the sum score of four questionnaire items from the Copenhagen Psychosocial Questionnaire was used.22 In addition, employees were asked to report their body height and body weight, as well as their smoking behavior and alcohol consumption.
Employees aged 18 years or older and with a contract for at least 10 months at the start of the intervention were eligible for inclusion in the analysis. Descriptive statistics (ie, percentage, mean, and standard deviation) were used to analyze recruitment, reach, dose delivered, dose received, and satisfaction of employees with the interventions and the overall program. Differences over time for these process variables were tested for intervention group employees. When variables were continuous, independent t-testing was used, with paired t-testing being conducted for differences over time in the intervention group. Chi-square testing was performed when a variable was dichotomous.
To determine effectiveness at 12-month follow-up, linear and logistic regression analyses were conducted with the variable of interest as the outcome, and group allocation (intervention or control) as the independent variable, adjusted for the baseline value of the outcome of interest and relevant covariates (company, sex, educational level, and contract hours). The effect analysis included only participants for whom data were present for both time points. Data analyses were also performed to determine significant relationships between the compliance of workers with the interventions and the study outcomes. Compliance with the interventions was defined as (1) low compliance: two or more interventions and (2) high compliance: more than two interventions based on the mean number of interventions in which employees had participated. Linear regression analyses were performed to test the differences between these compliance groups. Finally, data about contamination were collected in the process questionnaire at 12-month follow-up from employees in the control departments (hospital n = 462; university n = 327).
All recorded interviews were transcribed verbatim. Transcripts were then read to establish a general picture of the concepts being studied and of the dynamics of the interviews. Using MAXQDA version 11 (VERBI GmbH, Berlin, Germany), transcripts were marked with open codes (descriptive codes within the immediate domain of the interview questions) and axial codes (analytic codes that represent emerging and overarching themes).23,24 All codes were then grouped into central concepts related to all process components (such as satisfaction, recruitment, and fidelity). One meeting was organized with the researchers (D.W., L.E., and P.E.) during the data analysis stage, during which they identified codes, concepts, and themes, and discussed interpretations of the data to enhance the validity of the interpretation of the findings. In all cases, consensus was reached through discussion.
Questionnaire Respondents’ Characteristics
In the hospital intervention department, 215 of all the eligible employees (34.6%) filled out the process questionnaire at T1 and 177 employees (28.9%) completed the questionnaire at T2. The university response rates were 216 (42.9%) and 174 (35.6%) at T1 and T2, respectively. The demographic characteristics of the completers of the questionnaires in the two organizations were comparable at both time points (P > 0.05) (Table 2).
The response rates to the T2 questionnaire for both control worksites were 28.4% (n = 131) and 37% (n = 121) for the hospital and the university, respectively.
Implementation of the Interventions
Both organizations developed the mandatory project plan. Despite the presence of a project plan at the hospital, delays and incompletion of the execution of some interventions could be observed, which could be related to (1) the lack of ownership for the project by project members and (2) the fact that the project plan contained only a list of the general activities, desired changes, overall budget, and goals of the project whereas a detailed intervention template, communication plan, budget specification, and a list of involved persons with their tasks and responsibilities was lacking. For instance, no interventions were delivered between September and December (2011) and between January and April 2012. Because of the length of time between the interventions, most interviewed employees stated that they often thought that the project had already ended and as a result their willingness to participate was impaired: “I feel that the amount of time between the interventions is too long. So I know there are some more coming. But it's been so long since there has been something that you wonder whether it's already finished or still ongoing” (Academic Hospital Employee 1).
Interviews with project members at the university revealed that because of the presence of detailed project plan and a project leader who was assigned 16 hours a week solely for the implementation, program delivery was structured throughout the academic year (2011 to 2012) without any delays in the delivery of the intended interventions. In addition, it should be noted that at the university, the project members took 1 year for writing the project plan and preparing the project.
Eventually, at the hospital, 12 of 14 of the planned interventions (85.7%) were delivered to the employees. At the university, 21 of 25 planned interventions (84.0%) were delivered (Table 3). Main reasons for not delivering an intervention were (1) lack of approval from upper management due to rules and regulations related to the use of the building, (2) time constrains of project members, and (3) excessive costs. Continuous monitoring showed that both project groups mainly developed and implemented relatively simple interventions that did not require active employee participation (such as distribution of free fruit and posters). Most interventions chosen were environmental and educational interventions that could be implemented easily at low costs and effort. For example, one interviewed project member stated: “It (assigning a sports field) looks so simple but then it turns out not to be so simple at all due [rules and regulations regarding] the place and housing and things like that, despite low costs” (University of Applied Science Project Member 1).
Eventually, at the hospital, project members only choose to integrate the topic of vitality in employees’ annual performance interviews because the majority of managers expressed a positive attitude toward this change. This resulted in a pilot project with new funding, which will train managers in the use of the tools and abilities they need to conduct the appropriate interviews with a focus on sustainable productivity related to employees’ lifestyle. If this project proves successful, it will be implemented hospital-wide in early 2014. At the university, a human resources officer was instructed in December 2012 to embed some of the interventions in the organization's general health policy at the intervention faculty (Table 3). The coaching program, yearly bicycle check, and stair massages are ongoing. The distribution of the free fruit was stopped 1 year later (in Fall 2013) because of excessive costs. The structural environmental interventions (like the enlarged bicycle shed, changes to staircases, changes to the routing, sitting balls, standing tables, and the stricter smoking policy) were also maintained. Interviews with project members and the field notes revealed that the maintenance decisions were mainly based on the satisfaction rates with an intervention among project members and employees, low intervention costs, perceived success, and ease of implementation.
Employee Recruitment for Exposure to, and Satisfaction with, the Interventions
Employees from both organizations were recruited for participation in the project by postcards with general project information at the launch of the implementation sent to employees’ home addresses (employee awareness T1: 64.3%, n = 247). In addition, at the hospital, a kickoff was organized in the form of health checks with an information stand and, at the university, the project was launched during the annual opening ceremony of the academic year after a health week with workshops and an information stand (Table 4). During the project, both organizations sent informative e-mails (awareness T2: 76.0%, n = 256) and launched an official project Web site on the organizations’ intranet (awareness T2: 51.2%, n = 171). In addition, other methods were used, such as placing project banners at the entrances of the intervention departments at the hospital and a monthly e-mail update after 6 months, with specific project information at the university. The interviews with employees from both organizations revealed that most employees were satisfied with the recruitment methods. Nevertheless, some interviewees said that a personal approach would have been better because they did not read the information on the Web site and already receive a large number of e-mails every day.
Of all respondents, 89.5% (n = 383) were aware of the project at T1 and 96.6% (n = 338) at T2 (Table 5). Of the latter respondents, 76.6% (n = 258) stated that they were also aware of the project goals. Frequently mentioned project goals by the respondents and interviewees at both organizations included creating awareness of the benefits of a healthy lifestyle and promoting a healthy lifestyle to reduce sickness absence rates and to improve employee performance. In addition, 75.1% (n = 386) of the respondents of both organizations had participated in at least one intervention (excluding environmental interventions and posters). Nonparticipating respondents worked an average of 4.7 hours less per week than participating respondents at T1 (P < 0.05). At the university, nonparticipating respondents worked an average of 24.1 hours in 3.3 days and participating respondents had worked an average of 28.6 hours in 3.8 days (P < 0.05). Nevertheless, this difference was observed only at T1.
On average, the hospital employees participated in 2.2 ± 1.6 (range, 0 to 8) of 10 interventions that were eligible for participation and university employees in 3.0 ± 1.7 (range, 0 to 9) of 11 interventions eligible for participation. Employees of both organizations who participated in at least one intervention during the study period and responded to both questionnaires (n = 164) rated the overall project as moderate at both T1 (6.7 ± 1.6) and T2 (6.8 ± 1.1), with no significant change over time (P > 0.05) (Table 5). Interviewed employees were mainly satisfied with the overall program because they perceived the program to be a good initiative that showed that their employer acknowledged the value of employee health and lifestyles, especially because they are working in the health sector. Other reasons mentioned by a majority of the interviewees were “the program generated awareness about the importance of a healthy lifestyle,” and “most interventions were perceived to be ‘pragmatic’ and thought to facilitate employees.” Nonetheless, respondents were also very critical about the need for some of the developed interventions and said that, in some cases, a long time interval between interventions had had a negative effect on interest in the project. In addition, interviewed employees commented that future implementation of health programs in their organization should focus more on comprehensive interventions (like the coaching activities at the university) embedded in the company's health policy. Other suggested improvements were that more efforts should be made to specifically target unhealthy employees and that one employee should be appointed in each department team as an ambassador for the project to raise awareness and participation levels in the organization.
The highest participation and satisfaction rates in both organizations were found for the distributed free fruit. In general, employees were most aware of and participated in easily accessible interventions such as environmental interventions and interventions that were part of the existing meetings or events in the organization that stimulated social interaction with colleagues (Table 4). The interviews with employees revealed that employees did not use the materials or actively participated in interventions mainly when the intervention was not easily integrated into their working life (high work demands or inconvenient time and location of intervention) or when the intervention was not perceived as interesting because it did not fit their needs. Interviewed employees were more inclined to be positive about an intervention when they perceived the intervention as relevant, easily accessible, and matching their needs.
Awareness of the project was confirmed by 31.3% of the hospital control respondents and 21.7% of the university control respondents. The questionnaire results showed that almost no employees (a maximum of 2) at the two control worksites were aware of some of the interventions and had also participated in some interventions. Nevertheless, at the university, the human resources department was so enthusiastic about the free fruit that they decided to implement this intervention at all faculties of the university during the intervention year, despite the objections of the research group. Consequently, 36.8% (n = 42) employees of the control group reported that they received the free fruit.
Effectiveness of BRAVO@Work
From 145 respondents, data were available from baseline as well as 12-month follow-up and were included in the analyses. Regression analyses (Table 6) identified an intervention effect on the number of days of fruit consumption per week (β = 0.44 days/week, 95% CI: 0.02 to 0.85) in favor of the intervention group. No other significant effects were found. In both the intervention and control groups, the number of pieces of fruit a day rose from baseline to 12-month follow-up (+1.65 and +1.67 pieces/day, respectively), with no significant differences between the groups (β = −0.042 pieces/day, 95% CI: −0.291 to 0.208). For the relationship between the outcome measures and participation levels (low participation 2 or less and high participation 2 or more), logistic regression analysis showed a significant relationship between days of vegetable intake and high participation in the intervention group (β = −0.358, 95% CI: −0.691 to −0.026), as compared with low participation in favor of the low participation group.
The main aim of this study was to evaluate the implementation of a multicomponent lifestyle intervention at two different worksites. To our knowledge, this is the first published process evaluation looking at the natural course of implementation of lifestyle interventions coordinated and implemented by employees and managers without active involvement of the researchers, who acted as embedded observers only.
The first aim of this study was to study the extent and quality of program delivery and maintenance. High rates for dose delivered (84.0% and 85.7%) in both intervention departments were observed. Although in this study the project groups at both organizations were responsible for the implementation, the dose delivered rates we found compare well with other WHPPs in which the researchers were responsible, with dose delivered ranging from 72% to 86%.25,26 Hence, rates greater than 80% can, therefore, be considered good and are not dependent on who delivers the interventions. Nevertheless, it should be noted that even though the hospital implemented 12 of 14 interventions, they did not implement any interventions during two periods of three consecutive months, which questions the feasibility of a user-driven approach in the hospital. Both organizations implemented a mix of environmental, educational, and individual interventions. Nevertheless, both project groups mostly opted for relatively simple-to-implement and “fun” educational and environmental interventions like the free fruit and posters and did not pay much attention to the evidence base of the interventions that were chosen. The interventions chosen are more likely to create awareness among employees rather than change behavior. This raises the question whether this is an implication of the formative design of this study or that this is an implication of the fact that the project groups were in charge of developing and implementing the lifestyle interventions. Using a formative design enables researchers to gain insight into program implementation over time and understand the strengths and weaknesses of the chosen strategy.27,28 In the chosen formative design, the researchers tried not to intervene with the implementation process as we were primarily interested in gaining insight into the implementation of evidence-based interventions in daily practice whereby the organizations themselves were in charge. So for that aim, the chosen study design was fitting. Nevertheless, the “simple and fun” interventions implemented by the project groups in combination with the lack of effects on changing lifestyle behavior suggest that the project groups by themselves do not have the correct tools at hand to implement more comprehensive individual interventions. This is also reflected by the lack of ownership and feasibility of the study at the hospital. Overall, at this point in time organizations need more intervening from researchers at the start of the program to make sure a planned approach is used and to give guidance in developing and implementing more intensive individually based interventions.
The results indicated that adherence to the original project plan (fidelity) at the university was high, whereas in the hospital more difficulties with program delivery were experienced. A comparison of the two organizations indicated that program implementation may be facilitated when, and forehand, a detailed project plan is developed including the desired changes and goals, an intervention template and a budget per intervention, a description of the explicit responsibilities of each project member, and a timeline. This may have helped management staff take informed decisions when they allocated budgets to the interventions they wanted to have implemented. By contrast, the more ad hoc approach in the hospital has the risk of a decline in participation and satisfaction rates among employees, because of delayed decision making and the failure to deliver interventions on time despite the rapid launch of the project. This ad hoc approach could also be the reason that none of the interventions in the hospital relating to physical activity, nutrition, or habits were maintained after the project had ended. They opted only for the continuation of vitality in employees’ yearly performance interviews that, interestingly, was the only intervention that was well thought out and the only intervention that the majority of managers and project members were satisfied with. Funding for a pilot project that focuses on training managers in conducting these performance interviews was obtained. If successful, it is expected that, by early 2014, all managers at the hospital will be required to discuss sustainable productivity (and lifestyle-related factors) in annual performance interviews.
The university continued three interventions (chair massages, coaching program, and the bicycle check) after the project. This decision was also mainly based on employees’ and managers’ satisfaction with the interventions. So, the sustainability of “simple and fun” worksite lifestyle interventions, with the exception of environmental interventions, at the university (ie, the adjusted building routes and standing tables) can be regarded as low if these do not closely fit employees’ needs or if the costs are excessive. So, the main driver of intervention uptake in daily practice is, therefore, ease of implementation, degree of satisfaction, and a match to employees’ needs, regardless of how effective the intervention was in terms of changing the targeted behavior.
With respect to the second aim—establishing a picture of employee recruitment, reach, participation, and satisfaction levels—we can conclude that the majority of employees were aware of the program (96.6%) and had participated in at least one intervention (75.1%) during the program implementation. So the methods used by both organizations for recruiting employees to the program were apparently effective. Nevertheless, employees generally reported a preference for a more personal approach, such as an introduction to the project in their monthly team meetings, rather than a dedicated project Web site and e-mails that were poorly read. Such a personal approach encourages interaction and would enable employees to discuss and clarify any ambiguities beforehand. This could have facilitated their participation in and furthered their satisfaction with the program as a whole. Employees also stressed that more efforts should be made in the future to target relatively unhealthy employees. Nevertheless, the high awareness rates among respondents in this study suggest that both organizations successfully communicated the program through the organization. This can be considered a necessary first step in obtaining high participation rates and, ultimately, in changing employee lifestyle behavior.29 Furthermore, the participation rates (75.1%) in this study for the program as a whole can be considered high because, in general, participation levels in WHPPs vary widely (10% to 76%).29,30 Nevertheless, it should be noted that at the hospital, employees participated in 2 of 10 interventions and at the university in 3 of 11 interventions. The participation in the overall project compared with other studies can still be considered high but a closer look to the amount of interventions that employees participated in show that the levels were moderate. A possible explanation for the high overall participation rates in this study is the offering of multiple interventions as shown by a systematic review by Robroek et al.30 By addressing multiple lifestyle themes, a program reaches employees with different needs relating to different lifestyle aspects, and this leads to higher overall participation compared with programs that implement physical activity interventions only.30 Furthermore, both organizations included a mental health component focusing on work stress and relaxation at work (chair massages, mindfulness sessions, peer group counseling) because the results of previous work satisfaction surveys at both organizations had shown that employees perceived work stress levels as high. Because their employer acknowledged this problem by specifically addressing mental health, employees possibly were more appreciative of the entire program and more inclined to participate. Focusing exclusively on lifestyle aspects such as physical activity, nutrition, and smoking may lead employees to feel that the employer is intervening too much in the personal lives of the employees. By contrast, adding a mental health component allowed the organizations to show that management cared about the well-being of employees and their working conditions.
Differences were found in terms of actual program participation rates between the two organizations over time and for the specific interventions. Participation rates at the hospital declined over time, possibly because of a very limited project plan, whereas at the university, rates inclined, possibly because of the longer preparation time they took for writing the plan. Despite the decline in participation at the hospital and the rise in participation at the university, employee satisfaction with the program as a whole was moderate (6 or more and less than 7.5), even though satisfaction levels for the specific interventions were found to be good. Nevertheless, in a more comprehensive and individually based worksite lifestyle intervention by Strijk et al26 that included yoga, workout sessions, and a personal vitality coach, satisfaction with the program was good (≥7.5).26 This suggests that although employees were satisfied with the specific interventions, overall program satisfaction could have been higher if more individually based interventions had been implemented.
Finally, the effectiveness of the entire program was evaluated in a quasi-experimental controlled trial conducted alongside the process evaluation. Although multiple lifestyle interventions targeting different behaviors were implemented, only a favorable change over time was observed in the number of days of self-reported fruit consumption but not in the number of pieces of fruit per day. It should be noted that the free fruit was also distributed to the control group at the university due to an overenthusiastic HRM department. Despite this contamination, an effect was still found in favor of the intervention group. This may be explained by the results on dose received that show that employees from the intervention group (84.3%) had made more use of the fruit than the participants in the control group (36.8%). Because of the lack of any effect on other primary outcome measures, only the distribution of the free fruit could be described as effective. The lack of effects on other lifestyle behaviors could possibly be explained by the type of the implemented interventions, as explained earlier. In addition, this study was conducted among a relatively healthy sample of employees. Maybe if we specifically targeted a high-risk population, for example, based on a high body mass index, the interventions would be more effective. Nevertheless, it was chosen not to do this because the organizations wanted to focus on prevention in general and create a positive environment without excluding anybody who wanted to participate. Another explanation for the lack of effects could be the lack of power to detect possible effects due to low response rate to the questionnaire, due to the extensive length of the questionnaire, technical problems with filling out the questionnaire, and employee suspicion toward their employer about the privacy of the data. The issue of whether it was possible with our chosen study design (quasi-experimental controlled trial) to detect any difference in lifestyle behavior between the intervention and control groups should be considered also. Although the results did not find any major differences in the demographic characteristics between the two groups, from a methodological perspective, it is better to investigate the effectiveness of an intervention in a randomized controlled trial (RCT), with the organizations being randomized. Nevertheless, an RCT design is not the most ideal design in studies like these where the primary focus is on the natural implementation process of an intervention. An RCT takes place in controlled conditions (research driven), whereas implementation in “real life” practice requires flexibility in the conditions that can be controlled. Nevertheless, the results of the latter probably say more about the generalizability.
Strengths and Limitations
A major strength of this process evaluation is that we evaluated the implementation process on the basis of a comprehensive framework that was developed prior to the start of the implementation. As a result, data were collected continuously from the start of the implementation, which resulted in detailed information about the real-time implementation of the individual interventions. Because data were collected from all relevant stakeholders by combining quantitative and qualitative evaluation methods, more in-depth information was obtained about the reasons for not implementing an intervention participating in an intervention, and about the quality of intervention delivery. The qualitative evaluation, therefore, helped interpret the results from the questionnaires.31 The fact that the “embedded” researcher monitored the implementation process could be a strength, but it may also be a limitation. By being present in the organization, the researcher may unintentionally affect the implementation process, even when, as here, the researcher does not actively intervene in the process. For example, at the university, the project would probably have terminated prematurely if the researchers and external advisor had not been present. As a result of their presence, the project members made the project a priority and did not allow the project to get side-tracked because of high work pressure. Nevertheless, the chosen design produced valuable information about the implementation process, reasons for program adaptation, and the failure to deliver some interventions that might not have been obtained in interviews, during which project members may have given social desirable answers.
A limitation in this study is the possibility of selection bias because not all employees completed the questionnaires or participated in the interviews. Nevertheless, by a random convenience sampling based on employees’ time constraints and work demands for recruiting employees to the interviews, we tried to ensure that we interviewed employees with different participation rates and satisfaction levels. Another strength of this evaluation was the proportion of process components included in this study, which allowed us to acquire a full overview of implementation at all levels. This is a change from other process evaluations, which often focus primarily on dose received.10
Conclusions and Implications for Future Research
This study indicates that the primary precondition for a WHPP in a nonresearch context is not that an intervention has to be evidence-based, but that it should be easy to implement, with low costs and minimal effort. The intervention departments in this study, and probably most organizations in the Netherlands, are not eager to spend a large amount of their budget and time on implementing more comprehensive “evidence-based” individual interventions from the list of known effective interventions. Nevertheless, this does not seem to have had an effect on employee awareness, participation, and satisfaction because the results in these areas were comparable to, if not better than, other more comprehensive lifestyle interventions. As most known evidence-based worksite lifestyle interventions are developed and implemented with coordination from academic researchers, our findings raise questions about the practical applicability of most of these interventions. There is little or no practical information available for organizations about these interventions that they can use as a guide to adopt and implement an evidence-based, complex, lifestyle intervention. This means that they will opt for other, more simple, interventions. Our results suggest that, if interventions are to be implemented in daily practice, they should be visible and easy to implement, require low-cost and minimal effort, and match the needs of the organization and the employee. Furthermore, employees will be more inclined to participate in more simple interventions that do not require active participation and in which they can participate when that is convenient for them, as in the case of the free fruit. Nevertheless, these simple interventions are not continued and in this project only the more elaborate interventions were maintained (ie, discussing vitality in employees’ yearly performance interviews, chair massages, and the coaching trajectory). The main challenge for upcoming implementation research will be to strike a balance between implementing evidence-based interventions that are known to be effective and to leave the coordination of the implementation to organizations’ employees and managers.
The authors thank Helene Tuinman and Anna van der Steen for transcribing the interviews. The authors also thank all project members, managers, and employees who were involved in this study.
1. van Duijvenbode DC, Hoozemans MJ, van Poppel MN, Proper KI. The relationship between overweight and obesity, and sick leave: a systematic review. Int J Obes (Lond). 2009;33:807–816.
2. Goetzel RZ, Ozminkowski RJ. The health and cost benefits of work site health-promotion programs. Annu Rev Public Health. 2008;29:303–323.
3. Anderson LM, Quinn TA, Glanz K, et al. The effectiveness of worksite nutrition and physical activity interventions for controlling employee overweight and obesity. A systematic review. Am J Prev Med. 2009;37:340–357.
4. Jepson RG, Harris FM, Platt S, Tannahill C. The effectiveness of interventions to change six health behaviours: a review of reviews. BMC Public Health. 2010;10.
5. Ni Mhurchu C, Aston LM, Jebb SA. Effects of worksite health promotion interventions on employee diets: a systematic review. BMC Public Health. 2010;10:62.
6. Groeneveld IF, Proper KI, Van Der Beek AJ, Hildebrandt VH, Mechelen WV. Lifestyle-focused interventions at the workplace to reduce the risk of cardiovascular disease—a systematic review. Scand J Work Environ Health. 2010;36:202–215.
7. Verweij LM, Coffeng J, van Mechelen W, Proper KI. Meta-analyses of workplace physical activity and dietary behaviour interventions on weight outcomes. Obes Rev. 2011;12:406–429.
8. Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract. 2005;6:134–147.
9. Grimshaw JM, Zwarenstein M, Tetroe JM, et al. Looking inside the black box: a theory-based process evaluation alongside a randomised controlled trial of printed educational materials (the Ontario Printed Educational Message, OPEM) to improve referral and prescribing practices in primary care in Ontario, Canada. Implement Sci. 2007;2:38.
10. Wierenga D, Engbers LH, van Empelen P, Duijts S, Hildebrandt VH, van Mechelen W. What is actually measured in process evaluations for worksite health promotion programs: a systematic review. BMC Public Health. 2013;13:1190.
11. Wierenga D, Engbers LH, van Empelen P, Hildebrandt VH, van Mechelen W. The design of a real-time formative evaluation of the implementation process of lifestyle interventions at two worksites using a 7-step strategy (BRAVO@Work). BMC Public Health. 2012;12:619.
12. Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks PA. The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med. 2004;27:3–12.
13. Bartholomew LK, Parcel GS, Kok G, Gottlieb NH. Planning Health Promotion Programs: An Intervention Mapping Approach. San Francisco, CA: Jossey-Bass; 2006.
14. Steckler A, Linnan L. Process Evaluation for Public Health Interventions and Research. San Francisco, CA: Jossey-Bass; 2002.
15. Oakley A, Strange V, Bonell C, Allen E, Stephenson J; RIPPLE Study Team. Process evaluation in randomised controlled trials of complex interventions. BMJ. 2006;332:413–416.
16. Rongen A, Robroek SJW, van Lenthe FJ, Burdorf A. Workplace health promotion: a meta-analysis of effectiveness. Am J Prev Med. 2013;44:406–415.
17. Chau JY, der Ploeg HP, van Uffelen JG, et al. Are workplace interventions to reduce sitting effective? A systematic review. Prev Med. 2010;51:352–356.
18. Baranowski T, Stables G. Process evaluations of the 5-a-day projects. Health Educ Behav. 2000;27:157–166.
19. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–1327.
20. de Hollander EL, Zwart L, de Vries SI, Wendel-Vos W. The SQUASH was a more valid tool than the OBiN for categorizing adults according to the Dutch physical activity and the combined guideline. J Clin Epidemiol. 2012;65:73–81.
21. Wendel-Vos GC, Schuit AJ, Saris WH, Kromhout D. Reproducibility and relative validity of the short questionnaire to assess health-enhancing physical activity. J Clin Epidemiol. 2003;56:1163–1169.
22. Kristensen TS, Hannerz H, Høgh A, Borg V. The Copenhagen Psychosocial Questionnaire—a tool for the assessment and improvement of the psychosocial work environment. Scand J Work Environ Health. 31:438–449.
23. Pope C, Ziebland S, Mays N. Qualitative research in health care. Analysing qualitative data. BMJ. 2000;320:114–116.
24. Glaser BG. The constant comparative method of qualitative analysis. Soc Problems. 1965;12:436–445.
25. Verweij LM, Proper KI, Hulshof CT, van Mechelen W. Process evaluation of an occupational health guideline aimed at preventing weight gain among employees. J Occup Environ Med. 2011;53:722–729.
26. Strijk JE, Proper KI, van der Beek AJ, van Mechelen W. A process evaluation of a worksite vitality intervention among ageing hospital workers. Int J Behav Nutr Phys Act. 2011;8:58.
27. Patton MQ. Evaluation of program implementation. Eval Stu Rev Annu. 1979;4:318–345.
28. Stetler CB, Legro MW, Wallace CM, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21:S1–S8.
29. Glasgow RE, McCaul KD, Fisher KJ. Participation in worksite health promotion: A critique of the literature and recommendations for future practice. Health Educ Q. 1993;20:391–408.
30. Robroek SJ, van Lenthe FJ, van Empelen P, Burdorf A. Determinants of participation in worksite health promotion programmes: a systematic review. Int J Behav Nutr Phys Act. 2009;6:26.
31. Mays N, Pope C. Qualitative research in health care: assessing quality in qualitative research. BMJ. 2000;320:50–52.