Eventually, at the hospital, project members only choose to integrate the topic of vitality in employees’ annual performance interviews because the majority of managers expressed a positive attitude toward this change. This resulted in a pilot project with new funding, which will train managers in the use of the tools and abilities they need to conduct the appropriate interviews with a focus on sustainable productivity related to employees’ lifestyle. If this project proves successful, it will be implemented hospital-wide in early 2014. At the university, a human resources officer was instructed in December 2012 to embed some of the interventions in the organization's general health policy at the intervention faculty (Table 3). The coaching program, yearly bicycle check, and stair massages are ongoing. The distribution of the free fruit was stopped 1 year later (in Fall 2013) because of excessive costs. The structural environmental interventions (like the enlarged bicycle shed, changes to staircases, changes to the routing, sitting balls, standing tables, and the stricter smoking policy) were also maintained. Interviews with project members and the field notes revealed that the maintenance decisions were mainly based on the satisfaction rates with an intervention among project members and employees, low intervention costs, perceived success, and ease of implementation.
Employee Recruitment for Exposure to, and Satisfaction with, the Interventions
Employees from both organizations were recruited for participation in the project by postcards with general project information at the launch of the implementation sent to employees’ home addresses (employee awareness T1: 64.3%, n = 247). In addition, at the hospital, a kickoff was organized in the form of health checks with an information stand and, at the university, the project was launched during the annual opening ceremony of the academic year after a health week with workshops and an information stand (Table 4). During the project, both organizations sent informative e-mails (awareness T2: 76.0%, n = 256) and launched an official project Web site on the organizations’ intranet (awareness T2: 51.2%, n = 171). In addition, other methods were used, such as placing project banners at the entrances of the intervention departments at the hospital and a monthly e-mail update after 6 months, with specific project information at the university. The interviews with employees from both organizations revealed that most employees were satisfied with the recruitment methods. Nevertheless, some interviewees said that a personal approach would have been better because they did not read the information on the Web site and already receive a large number of e-mails every day.
Of all respondents, 89.5% (n = 383) were aware of the project at T1 and 96.6% (n = 338) at T2 (Table 5). Of the latter respondents, 76.6% (n = 258) stated that they were also aware of the project goals. Frequently mentioned project goals by the respondents and interviewees at both organizations included creating awareness of the benefits of a healthy lifestyle and promoting a healthy lifestyle to reduce sickness absence rates and to improve employee performance. In addition, 75.1% (n = 386) of the respondents of both organizations had participated in at least one intervention (excluding environmental interventions and posters). Nonparticipating respondents worked an average of 4.7 hours less per week than participating respondents at T1 (P < 0.05). At the university, nonparticipating respondents worked an average of 24.1 hours in 3.3 days and participating respondents had worked an average of 28.6 hours in 3.8 days (P < 0.05). Nevertheless, this difference was observed only at T1.
On average, the hospital employees participated in 2.2 ± 1.6 (range, 0 to 8) of 10 interventions that were eligible for participation and university employees in 3.0 ± 1.7 (range, 0 to 9) of 11 interventions eligible for participation. Employees of both organizations who participated in at least one intervention during the study period and responded to both questionnaires (n = 164) rated the overall project as moderate at both T1 (6.7 ± 1.6) and T2 (6.8 ± 1.1), with no significant change over time (P > 0.05) (Table 5). Interviewed employees were mainly satisfied with the overall program because they perceived the program to be a good initiative that showed that their employer acknowledged the value of employee health and lifestyles, especially because they are working in the health sector. Other reasons mentioned by a majority of the interviewees were “the program generated awareness about the importance of a healthy lifestyle,” and “most interventions were perceived to be ‘pragmatic’ and thought to facilitate employees.” Nonetheless, respondents were also very critical about the need for some of the developed interventions and said that, in some cases, a long time interval between interventions had had a negative effect on interest in the project. In addition, interviewed employees commented that future implementation of health programs in their organization should focus more on comprehensive interventions (like the coaching activities at the university) embedded in the company's health policy. Other suggested improvements were that more efforts should be made to specifically target unhealthy employees and that one employee should be appointed in each department team as an ambassador for the project to raise awareness and participation levels in the organization.
The highest participation and satisfaction rates in both organizations were found for the distributed free fruit. In general, employees were most aware of and participated in easily accessible interventions such as environmental interventions and interventions that were part of the existing meetings or events in the organization that stimulated social interaction with colleagues (Table 4). The interviews with employees revealed that employees did not use the materials or actively participated in interventions mainly when the intervention was not easily integrated into their working life (high work demands or inconvenient time and location of intervention) or when the intervention was not perceived as interesting because it did not fit their needs. Interviewed employees were more inclined to be positive about an intervention when they perceived the intervention as relevant, easily accessible, and matching their needs.
Awareness of the project was confirmed by 31.3% of the hospital control respondents and 21.7% of the university control respondents. The questionnaire results showed that almost no employees (a maximum of 2) at the two control worksites were aware of some of the interventions and had also participated in some interventions. Nevertheless, at the university, the human resources department was so enthusiastic about the free fruit that they decided to implement this intervention at all faculties of the university during the intervention year, despite the objections of the research group. Consequently, 36.8% (n = 42) employees of the control group reported that they received the free fruit.
Effectiveness of BRAVO@Work
From 145 respondents, data were available from baseline as well as 12-month follow-up and were included in the analyses. Regression analyses (Table 6) identified an intervention effect on the number of days of fruit consumption per week (β = 0.44 days/week, 95% CI: 0.02 to 0.85) in favor of the intervention group. No other significant effects were found. In both the intervention and control groups, the number of pieces of fruit a day rose from baseline to 12-month follow-up (+1.65 and +1.67 pieces/day, respectively), with no significant differences between the groups (β = −0.042 pieces/day, 95% CI: −0.291 to 0.208). For the relationship between the outcome measures and participation levels (low participation 2 or less and high participation 2 or more), logistic regression analysis showed a significant relationship between days of vegetable intake and high participation in the intervention group (β = −0.358, 95% CI: −0.691 to −0.026), as compared with low participation in favor of the low participation group.
The main aim of this study was to evaluate the implementation of a multicomponent lifestyle intervention at two different worksites. To our knowledge, this is the first published process evaluation looking at the natural course of implementation of lifestyle interventions coordinated and implemented by employees and managers without active involvement of the researchers, who acted as embedded observers only.
The first aim of this study was to study the extent and quality of program delivery and maintenance. High rates for dose delivered (84.0% and 85.7%) in both intervention departments were observed. Although in this study the project groups at both organizations were responsible for the implementation, the dose delivered rates we found compare well with other WHPPs in which the researchers were responsible, with dose delivered ranging from 72% to 86%.25,26 Hence, rates greater than 80% can, therefore, be considered good and are not dependent on who delivers the interventions. Nevertheless, it should be noted that even though the hospital implemented 12 of 14 interventions, they did not implement any interventions during two periods of three consecutive months, which questions the feasibility of a user-driven approach in the hospital. Both organizations implemented a mix of environmental, educational, and individual interventions. Nevertheless, both project groups mostly opted for relatively simple-to-implement and “fun” educational and environmental interventions like the free fruit and posters and did not pay much attention to the evidence base of the interventions that were chosen. The interventions chosen are more likely to create awareness among employees rather than change behavior. This raises the question whether this is an implication of the formative design of this study or that this is an implication of the fact that the project groups were in charge of developing and implementing the lifestyle interventions. Using a formative design enables researchers to gain insight into program implementation over time and understand the strengths and weaknesses of the chosen strategy.27,28 In the chosen formative design, the researchers tried not to intervene with the implementation process as we were primarily interested in gaining insight into the implementation of evidence-based interventions in daily practice whereby the organizations themselves were in charge. So for that aim, the chosen study design was fitting. Nevertheless, the “simple and fun” interventions implemented by the project groups in combination with the lack of effects on changing lifestyle behavior suggest that the project groups by themselves do not have the correct tools at hand to implement more comprehensive individual interventions. This is also reflected by the lack of ownership and feasibility of the study at the hospital. Overall, at this point in time organizations need more intervening from researchers at the start of the program to make sure a planned approach is used and to give guidance in developing and implementing more intensive individually based interventions.
The results indicated that adherence to the original project plan (fidelity) at the university was high, whereas in the hospital more difficulties with program delivery were experienced. A comparison of the two organizations indicated that program implementation may be facilitated when, and forehand, a detailed project plan is developed including the desired changes and goals, an intervention template and a budget per intervention, a description of the explicit responsibilities of each project member, and a timeline. This may have helped management staff take informed decisions when they allocated budgets to the interventions they wanted to have implemented. By contrast, the more ad hoc approach in the hospital has the risk of a decline in participation and satisfaction rates among employees, because of delayed decision making and the failure to deliver interventions on time despite the rapid launch of the project. This ad hoc approach could also be the reason that none of the interventions in the hospital relating to physical activity, nutrition, or habits were maintained after the project had ended. They opted only for the continuation of vitality in employees’ yearly performance interviews that, interestingly, was the only intervention that was well thought out and the only intervention that the majority of managers and project members were satisfied with. Funding for a pilot project that focuses on training managers in conducting these performance interviews was obtained. If successful, it is expected that, by early 2014, all managers at the hospital will be required to discuss sustainable productivity (and lifestyle-related factors) in annual performance interviews.
The university continued three interventions (chair massages, coaching program, and the bicycle check) after the project. This decision was also mainly based on employees’ and managers’ satisfaction with the interventions. So, the sustainability of “simple and fun” worksite lifestyle interventions, with the exception of environmental interventions, at the university (ie, the adjusted building routes and standing tables) can be regarded as low if these do not closely fit employees’ needs or if the costs are excessive. So, the main driver of intervention uptake in daily practice is, therefore, ease of implementation, degree of satisfaction, and a match to employees’ needs, regardless of how effective the intervention was in terms of changing the targeted behavior.
With respect to the second aim—establishing a picture of employee recruitment, reach, participation, and satisfaction levels—we can conclude that the majority of employees were aware of the program (96.6%) and had participated in at least one intervention (75.1%) during the program implementation. So the methods used by both organizations for recruiting employees to the program were apparently effective. Nevertheless, employees generally reported a preference for a more personal approach, such as an introduction to the project in their monthly team meetings, rather than a dedicated project Web site and e-mails that were poorly read. Such a personal approach encourages interaction and would enable employees to discuss and clarify any ambiguities beforehand. This could have facilitated their participation in and furthered their satisfaction with the program as a whole. Employees also stressed that more efforts should be made in the future to target relatively unhealthy employees. Nevertheless, the high awareness rates among respondents in this study suggest that both organizations successfully communicated the program through the organization. This can be considered a necessary first step in obtaining high participation rates and, ultimately, in changing employee lifestyle behavior.29 Furthermore, the participation rates (75.1%) in this study for the program as a whole can be considered high because, in general, participation levels in WHPPs vary widely (10% to 76%).29,30 Nevertheless, it should be noted that at the hospital, employees participated in 2 of 10 interventions and at the university in 3 of 11 interventions. The participation in the overall project compared with other studies can still be considered high but a closer look to the amount of interventions that employees participated in show that the levels were moderate. A possible explanation for the high overall participation rates in this study is the offering of multiple interventions as shown by a systematic review by Robroek et al.30 By addressing multiple lifestyle themes, a program reaches employees with different needs relating to different lifestyle aspects, and this leads to higher overall participation compared with programs that implement physical activity interventions only.30 Furthermore, both organizations included a mental health component focusing on work stress and relaxation at work (chair massages, mindfulness sessions, peer group counseling) because the results of previous work satisfaction surveys at both organizations had shown that employees perceived work stress levels as high. Because their employer acknowledged this problem by specifically addressing mental health, employees possibly were more appreciative of the entire program and more inclined to participate. Focusing exclusively on lifestyle aspects such as physical activity, nutrition, and smoking may lead employees to feel that the employer is intervening too much in the personal lives of the employees. By contrast, adding a mental health component allowed the organizations to show that management cared about the well-being of employees and their working conditions.
Differences were found in terms of actual program participation rates between the two organizations over time and for the specific interventions. Participation rates at the hospital declined over time, possibly because of a very limited project plan, whereas at the university, rates inclined, possibly because of the longer preparation time they took for writing the plan. Despite the decline in participation at the hospital and the rise in participation at the university, employee satisfaction with the program as a whole was moderate (6 or more and less than 7.5), even though satisfaction levels for the specific interventions were found to be good. Nevertheless, in a more comprehensive and individually based worksite lifestyle intervention by Strijk et al26 that included yoga, workout sessions, and a personal vitality coach, satisfaction with the program was good (≥7.5).26 This suggests that although employees were satisfied with the specific interventions, overall program satisfaction could have been higher if more individually based interventions had been implemented.
Finally, the effectiveness of the entire program was evaluated in a quasi-experimental controlled trial conducted alongside the process evaluation. Although multiple lifestyle interventions targeting different behaviors were implemented, only a favorable change over time was observed in the number of days of self-reported fruit consumption but not in the number of pieces of fruit per day. It should be noted that the free fruit was also distributed to the control group at the university due to an overenthusiastic HRM department. Despite this contamination, an effect was still found in favor of the intervention group. This may be explained by the results on dose received that show that employees from the intervention group (84.3%) had made more use of the fruit than the participants in the control group (36.8%). Because of the lack of any effect on other primary outcome measures, only the distribution of the free fruit could be described as effective. The lack of effects on other lifestyle behaviors could possibly be explained by the type of the implemented interventions, as explained earlier. In addition, this study was conducted among a relatively healthy sample of employees. Maybe if we specifically targeted a high-risk population, for example, based on a high body mass index, the interventions would be more effective. Nevertheless, it was chosen not to do this because the organizations wanted to focus on prevention in general and create a positive environment without excluding anybody who wanted to participate. Another explanation for the lack of effects could be the lack of power to detect possible effects due to low response rate to the questionnaire, due to the extensive length of the questionnaire, technical problems with filling out the questionnaire, and employee suspicion toward their employer about the privacy of the data. The issue of whether it was possible with our chosen study design (quasi-experimental controlled trial) to detect any difference in lifestyle behavior between the intervention and control groups should be considered also. Although the results did not find any major differences in the demographic characteristics between the two groups, from a methodological perspective, it is better to investigate the effectiveness of an intervention in a randomized controlled trial (RCT), with the organizations being randomized. Nevertheless, an RCT design is not the most ideal design in studies like these where the primary focus is on the natural implementation process of an intervention. An RCT takes place in controlled conditions (research driven), whereas implementation in “real life” practice requires flexibility in the conditions that can be controlled. Nevertheless, the results of the latter probably say more about the generalizability.
Strengths and Limitations
A major strength of this process evaluation is that we evaluated the implementation process on the basis of a comprehensive framework that was developed prior to the start of the implementation. As a result, data were collected continuously from the start of the implementation, which resulted in detailed information about the real-time implementation of the individual interventions. Because data were collected from all relevant stakeholders by combining quantitative and qualitative evaluation methods, more in-depth information was obtained about the reasons for not implementing an intervention participating in an intervention, and about the quality of intervention delivery. The qualitative evaluation, therefore, helped interpret the results from the questionnaires.31 The fact that the “embedded” researcher monitored the implementation process could be a strength, but it may also be a limitation. By being present in the organization, the researcher may unintentionally affect the implementation process, even when, as here, the researcher does not actively intervene in the process. For example, at the university, the project would probably have terminated prematurely if the researchers and external advisor had not been present. As a result of their presence, the project members made the project a priority and did not allow the project to get side-tracked because of high work pressure. Nevertheless, the chosen design produced valuable information about the implementation process, reasons for program adaptation, and the failure to deliver some interventions that might not have been obtained in interviews, during which project members may have given social desirable answers.
A limitation in this study is the possibility of selection bias because not all employees completed the questionnaires or participated in the interviews. Nevertheless, by a random convenience sampling based on employees’ time constraints and work demands for recruiting employees to the interviews, we tried to ensure that we interviewed employees with different participation rates and satisfaction levels. Another strength of this evaluation was the proportion of process components included in this study, which allowed us to acquire a full overview of implementation at all levels. This is a change from other process evaluations, which often focus primarily on dose received.10
Conclusions and Implications for Future Research
This study indicates that the primary precondition for a WHPP in a nonresearch context is not that an intervention has to be evidence-based, but that it should be easy to implement, with low costs and minimal effort. The intervention departments in this study, and probably most organizations in the Netherlands, are not eager to spend a large amount of their budget and time on implementing more comprehensive “evidence-based” individual interventions from the list of known effective interventions. Nevertheless, this does not seem to have had an effect on employee awareness, participation, and satisfaction because the results in these areas were comparable to, if not better than, other more comprehensive lifestyle interventions. As most known evidence-based worksite lifestyle interventions are developed and implemented with coordination from academic researchers, our findings raise questions about the practical applicability of most of these interventions. There is little or no practical information available for organizations about these interventions that they can use as a guide to adopt and implement an evidence-based, complex, lifestyle intervention. This means that they will opt for other, more simple, interventions. Our results suggest that, if interventions are to be implemented in daily practice, they should be visible and easy to implement, require low-cost and minimal effort, and match the needs of the organization and the employee. Furthermore, employees will be more inclined to participate in more simple interventions that do not require active participation and in which they can participate when that is convenient for them, as in the case of the free fruit. Nevertheless, these simple interventions are not continued and in this project only the more elaborate interventions were maintained (ie, discussing vitality in employees’ yearly performance interviews, chair massages, and the coaching trajectory). The main challenge for upcoming implementation research will be to strike a balance between implementing evidence-based interventions that are known to be effective and to leave the coordination of the implementation to organizations’ employees and managers.
The authors thank Helene Tuinman and Anna van der Steen for transcribing the interviews. The authors also thank all project members, managers, and employees who were involved in this study.
1. van Duijvenbode DC, Hoozemans MJ, van Poppel MN, Proper KI. The relationship between overweight and obesity, and sick leave: a systematic review. Int J Obes (Lond). 2009;33:807–816.
2. Goetzel RZ, Ozminkowski RJ. The health and cost benefits of work site health-promotion programs. Annu Rev Public Health. 2008;29:303–323.
3. Anderson LM, Quinn TA, Glanz K, et al. The effectiveness of worksite nutrition and physical activity interventions for controlling employee overweight and obesity. A systematic review. Am J Prev Med. 2009;37:340–357.
4. Jepson RG, Harris FM, Platt S, Tannahill C. The effectiveness of interventions to change six health behaviours: a review of reviews. BMC Public Health. 2010;10.
5. Ni Mhurchu C, Aston LM, Jebb SA. Effects of worksite health promotion interventions on employee diets: a systematic review. BMC Public Health. 2010;10:62.
6. Groeneveld IF, Proper KI, Van Der Beek AJ, Hildebrandt VH, Mechelen WV. Lifestyle-focused interventions at the workplace to reduce the risk of cardiovascular disease—a systematic review. Scand J Work Environ Health. 2010;36:202–215.
7. Verweij LM, Coffeng J, van Mechelen W, Proper KI. Meta-analyses of workplace physical activity and dietary behaviour interventions on weight outcomes. Obes Rev. 2011;12:406–429.
8. Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract. 2005;6:134–147.
9. Grimshaw JM, Zwarenstein M, Tetroe JM, et al. Looking inside the black box: a theory-based process evaluation alongside a randomised controlled trial of printed educational materials (the Ontario Printed Educational Message, OPEM) to improve referral and prescribing practices in primary care in Ontario, Canada. Implement Sci. 2007;2:38.
10. Wierenga D, Engbers LH, van Empelen P, Duijts S, Hildebrandt VH, van Mechelen W. What is actually measured in process evaluations for worksite health promotion programs: a systematic review. BMC Public Health. 2013;13:1190.
11. Wierenga D, Engbers LH, van Empelen P, Hildebrandt VH, van Mechelen W. The design of a real-time formative evaluation of the implementation process of lifestyle interventions at two worksites using a 7-step strategy (BRAVO@Work). BMC Public Health. 2012;12:619.
12. Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks PA. The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med. 2004;27:3–12.
13. Bartholomew LK, Parcel GS, Kok G, Gottlieb NH. Planning Health Promotion Programs: An Intervention Mapping Approach. San Francisco, CA: Jossey-Bass; 2006.
14. Steckler A, Linnan L. Process Evaluation for Public Health Interventions and Research. San Francisco, CA: Jossey-Bass; 2002.
15. Oakley A, Strange V, Bonell C, Allen E, Stephenson J; RIPPLE Study Team. Process evaluation in randomised controlled trials of complex interventions. BMJ. 2006;332:413–416.
16. Rongen A, Robroek SJW, van Lenthe FJ, Burdorf A. Workplace health promotion: a meta-analysis of effectiveness. Am J Prev Med. 2013;44:406–415.
17. Chau JY, der Ploeg HP, van Uffelen JG, et al. Are workplace interventions to reduce sitting effective? A systematic review. Prev Med. 2010;51:352–356.
18. Baranowski T, Stables G. Process evaluations of the 5-a-day projects. Health Educ Behav. 2000;27:157–166.
19. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–1327.
20. de Hollander EL, Zwart L, de Vries SI, Wendel-Vos W. The SQUASH was a more valid tool than the OBiN for categorizing adults according to the Dutch physical activity and the combined guideline. J Clin Epidemiol. 2012;65:73–81.
21. Wendel-Vos GC, Schuit AJ, Saris WH, Kromhout D. Reproducibility and relative validity of the short questionnaire to assess health-enhancing physical activity. J Clin Epidemiol. 2003;56:1163–1169.
22. Kristensen TS, Hannerz H, Høgh A, Borg V. The Copenhagen Psychosocial Questionnaire—a tool for the assessment and improvement of the psychosocial work environment. Scand J Work Environ Health. 31:438–449.
23. Pope C, Ziebland S, Mays N. Qualitative research in health care. Analysing qualitative data. BMJ. 2000;320:114–116.
24. Glaser BG. The constant comparative method of qualitative analysis. Soc Problems. 1965;12:436–445.
25. Verweij LM, Proper KI, Hulshof CT, van Mechelen W. Process evaluation of an occupational health guideline aimed at preventing weight gain among employees. J Occup Environ Med. 2011;53:722–729.
26. Strijk JE, Proper KI, van der Beek AJ, van Mechelen W. A process evaluation of a worksite vitality intervention among ageing hospital workers. Int J Behav Nutr Phys Act. 2011;8:58.
27. Patton MQ. Evaluation of program implementation. Eval Stu Rev Annu. 1979;4:318–345.
28. Stetler CB, Legro MW, Wallace CM, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21:S1–S8.
29. Glasgow RE, McCaul KD, Fisher KJ. Participation in worksite health promotion: A critique of the literature and recommendations for future practice. Health Educ Q. 1993;20:391–408.
30. Robroek SJ, van Lenthe FJ, van Empelen P, Burdorf A. Determinants of participation in worksite health promotion programmes: a systematic review. Int J Behav Nutr Phys Act. 2009;6:26.
Copyright © 2014 by the American College of Occupational and Environmental Medicine
31. Mays N, Pope C. Qualitative research in health care: assessing quality in qualitative research. BMJ. 2000;320:50–52.