In the United States, fires and burns are among the leading causes of unintentional injury deaths. The majority of these fire- and burn-related deaths occur in homes. In 2006, a total of 2704 individuals died in an unintentional home fire.1 Older adults, generally defined as persons 65 years and older, experience a disproportionate burden of these home fire deaths. In fact, older adults represented only 12.5% of the US population that year, yet they accounted for 35% of home fire deaths. With a residential fire mortality rate of 2.5 per 100 000, older adults are 3.7 times more likely to die in home fires than the rest of the population. Also, the risk increases with age, with persons older than 85 years being 5.8 times more likely to die in a home fire than is the rest of the population.1 Moreover, older adults are one of the fastest growing segments of the US population. Estimates indicate that by the year 2050 they will become the nation's largest age group—approximately 86 million persons or approximately 20% of the population.2 Older adults are vulnerable to home fire injuries for several reasons, including age-associated sensory and cognitive impairments. Because some older, homebound adults are challenged with disabilities and mobility limitations, they are at even greater risk for home fire injuries and death than are their nonhomebound counterparts.3
Smoke alarms are effective, reliable, and inexpensive devices that provide early warning during home fires. Smoke alarms decrease the risk of death in a home fire by up to 50%.4 However, though most households have at least 1 smoke alarm,5 nearly one-quarter lack working smoke alarms.4,6 This is a problem because 23% of all home fire deaths occur in homes with inoperable smoke alarms.4 As an added concern, older adults are more likely than is the general population to have smoke alarms that are beyond the recommended 10-year functioning life span.4 In this regard, one generally accepted fire safety goal is to increase the number of working smoke alarms among high-risk households (eg, homes with older adults) by having trained persons such as firefighters install smoke alarms.7,8
Another approach to reducing fire-related injuries and deaths is fire safety education. The traditional delivery approach has been to reach older adults with fire safety education in community settings (eg, residential complexes) and at special events (eg, senior fairs). These are effective ways to reach large audiences of older adults, but older, homebound adults have limited access to them.
One alternative is to extend fire prevention efforts to homebound older adults by partnering with organizations connected to older adult communities. The Meals On Wheels Association of America (MOWAA) is the largest national organization involved in older adult nutrition programs. Its members include local Meals On Wheels (MOW) programs in communities across the United States. Through home delivery of meals, these local programs have regular contact with homebound older adults. MOW staff and volunteers are a known and trusted source of information for these hard-to-reach and often-isolated older adults.
In 2006, the Department of Homeland Security awarded MOWAA an Assistance to Firefighters Grant to develop and put into practice the Residential Fire Homebound Elderly Lifeline Project (Fire H.E.L.P.). MOWAA proposed developing Fire H.E.L.P. to promote fire safety among homebound older adults, with the goal of reducing injuries and loss of property and life due to fire and fire-related hazards. Now complete, Fire H.E.L.P. has 3 major components: (1) conduct home assessments; (2) provide education on fire risk factors, the importance of smoke alarms, and escape planning; and (3) partner with local fire departments to install free smoke alarms with long-life lithium batteries (Figure 1).
In 5 Texas communities beginning in December 2007, the MOWAA launched a pilot run of Fire H.E.L.P. via local MOW programs. The following 5 programs were selected as the pilot sites for Fire H.E.L.P.:
- Meals On Wheels of Tarrant County, Inc, Fort Worth
- Christian Senior Services, San Antonio
- Meals On Wheels of Abilene, Abilene
- Meals for the Elderly, San Angelo
- Meals On Wheels of Johnson and Ellis counties, Cleburne
These sites were selected because they had the infrastructure in place to support the Fire H.E.L.P. project. In addition, they were chosen to give a mix of rural, urban, and suburban communities.
As part of its commitment to reducing fire-related injuries and deaths, the Centers for Disease Control and Prevention (CDC) collaborated with the MOWAA during the development of the overall Fire H.E.L.P. program. CDC also took on primary responsibility for developing an accompanying tool kit for the Fire H.E.L.P. program. The Fire H.E.L.P tool kit includes instructional curricula on how to implement Fire H.E.L.P., a home assessment tool to determine a residence's smoke alarm needs, and education materials to increase the client's fire safety knowledge. The tool kit also uses a train-the-trainer approach, in which selected MOW representatives are trained in how to implement Fire H.E.L.P. and, in turn, train their staff members.
The following is a description of how CDC used formative evaluation during the development, testing, and initial implementation of the tool kit. We will also identify how we built on these efforts to provide non-MOW programs with the necessary tools to implement a smoke alarm installation and fire safety education program.
Formative evaluation is often used to collect continuous feedback from participants in an ongoing program and then to revise the program elements (eg, materials or procedures) as needed.9 During the development stage of a prevention program, formative evaluation is often used to strengthen the implementation process10 and to understand how the program will work.11 In addition to improving the quality of program materials and their delivery, formative evaluation has been shown to demonstrate some impacts (eg, increased knowledge).11 During the development, testing, and implementation stages of tool kit creation, we used formative evaluation methods as shown in Figure 2 and as described in more detail later.
Stage 1: Development
We applied a broad health education and health communication approach to the development of the tool kit to ensure its use and effectiveness in MOW programs. We started the development process with an environmental scan of fire safety education materials and best practices for smoke alarm installation programs. The environmental scan identified evidence-based and practical approaches to starting similar programs.
CDC also conducted an audience analysis, which included 30-minute telephone interviews with local coordinators and staff from each of the 5 MOW sites participating in the pilot study. The semistructured interviews focused on learning about each MOW client population. The interviews explored settings, channels, and activities best suited to reach intended audiences and clarified logistical program issues, such as:
- Who will implement the trainings and project?
- What are the staff training needs? and
- What are the important factors to consider when delivering programs to the MOW clients?
To supplement the interview data, CDC reviewed information from MOW Web sites and from archival documents.
The findings contributed to the development of a health communication plan and to the tool kit itself. Extensive feedback from a Partnership Team comprised of CDC subject matter experts, MOWAA, 5 MOW pilot sites in Texas, and the International Association of Fire Chiefs furthered tool kit development. During content development, the Partnership Team critically examined the program materials (including designs and concepts) for scientific and technical accuracy as well as for sequencing.
Stage 2: Testing
We pilot-tested the tool kit with a group of 12 local coordinators and staff from the 5 MOW pilot sites. The pilot test assessed tool kit administration, collected feedback about the tool kit's contents, and evaluated potential changes in participants' knowledge about fire safety and program components.
Before the training session began, we administered a 13-question pretest with questions related to the training content (eg, client eligibility criteria, fire safety education messages, delivery process). Throughout the session, we documented participant feedback. After the session, we administered a posttest that contained questions identical to the pretest. Participants also completed a 7-question evaluation form that asked for their views about the usefulness of the overall training and specific training components, their confidence in implementing various program components, their assessments of the trainers' effectiveness, and their suggestions for training improvement.
Stage 3: Implementation
After a review of the pilot study findings, we modified the tool kit, and the Partnership Team reviewed final versions of the revised materials. A final version of the tool kit was distributed to the MOW pilot sites. We then made follow-up conference calls to each of the pilot sites to discuss the changes to the materials since the pilot training session, each site's plans for conducting local training sessions, and any questions or concerns about using the tool kit.
In December 2007, the local MOW sites used the tool kit to train their personnel on how to conduct home assessments and client education visits and how to make referrals to fire departments for smoke alarm installations. We then held a joint conference call with all of the sites to see whether any tool kit issues had arisen during the site training sessions.
Stage 1: Development
There were several key findings from the formative evaluation of the tool kit development, including the following: First, we determined that client education and assessments could not take place during meal delivery because of time limitations. Instead, these activities would occur during intake screenings or follow-up visits. Second, MOW staff shared their preference for limiting training sessions to 1 day because volunteers have limited time, and most of the staff members need to travel some distance to the training site. Third, MOW staff also indicated the need for tool kit materials to be concise because of the limited attention span of some clients.
During the development process, the train-the-trainer instructional curriculum was structured to focus on the 3 main elements of the Fire H.E.L.P. project: home assessment, client education, and smoke alarm referral. These elements, detailed in Figure 3, correspond with the core components of CDC's Smoke Alarm Installation and Fire Safety Education (SAIFE) program.8
CDC also identified 5 topics the educational messages needed to cover:
- Smoke alarm installation and testing
- Escape planning
- Alternative heat sources
The Partnership Team guided the resulting fire safety messages using a combination of best practices, evidence found in the scientific literature, and the audience analysis findings.
The final version of the tool kit includes material for training, home assessment, and client education, which are detailed in Figure 4.
Stage 2: Testing
Pilot testing the tool kit and train-the-trainer session highlighted areas for improvement and demonstrated materials and training effectiveness. For example, the intent of the role-playing exercises is to provide the MOW staff members with the opportunity to develop facility using the client education tool. Depending on the specific risk factors that a client has, he or she will receive different educational messages. A successfully completed exercise would entail the staff member delivering the correct messages for a given scenario. However, observations of the exercises revealed that MOW staff would often paraphrase the educational messages, which sometimes resulted in incorrect information being delivered. This led us to change the training to emphasize that educational messages should be delivered verbatim. Another area that needed improvement was a skip pattern in the client education tool that participants identified as difficult to use.
Participants correctly answered 43% of the pretest questions and 80% of the posttest questions. While the small sample size and nature of the pilot test (ie, materials and delivery refinement) precluded a statistical analysis of score changes, the results provided some evidence that the materials and training influenced participant knowledge. In addition, all 12 pilot testing participants reported that attending the train-the-trainer session provided them with the necessary information and tools to use Fire H.E.L.P effectively. Participants indicated high levels of confidence that they could practice the skills outlined in the training objectives.
Stage 3: Implementation
The MOW pilot sites reported no problems related to using the tool kit. The sites indicated that the educational flyers worked well with the clients. They also confirmed that it was nice to have copy-ready materials to use because they do not normally have the resources to develop them.
By the end of the Assistance to Firefighters Grant project period for Fire H.E.L.P. (July 2008), evaluation results reported by MOWAA showed that 2251 MOW clients in Texas received fire safety education and 4453 smoke alarms were installed (Table 1).
Fire H.E.L.P. is an innovative approach to promoting community health among older, homebound adults at risk for home fires and injuries. The formative evaluation process used to create the Fire H.E.L.P. tool kit allowed CDC to identify and address barriers to development and use and therefore improve the tool kit's utility and impact on program activities. The lessons learned during the tool kit's development might be helpful while creating health education initiatives to address other healthy home topics.
As part of the development process, evaluators should consider startup and use issues to make sure identified activities are feasible in light of any program context and staff capacity limitations. One such challenge may be how to gain access to the target population. Reaching reclusive or shut-in populations can be difficult. Some homebound older adults, relatives, or caregivers may be distrustful of people showing up at the door to offer “help.” Formative research with the target population can assist in identifying these issues as well as train potential agents, such as natural or “lay” helpers12 or community ambassadors,13 who might provide access to hard-to-reach populations or deliver prevention messages themselves. While MOW had routine access to homebound older adults, some constraints remained on how much time staff who delivered meals could spend in each home—a set number of meals had to be delivered within a given amount of time. The formative evaluation process revealed that time limitations would not allow meal deliverers to carry out this aspect of the program. Yet, by engaging the MOW staff early on to identify a solution, we were able to determine that these activities could occur during intake screenings or follow-up visits. We incorporated this change into the final version of the tool kit, ensuring more realistic program delivery expectations.
Another challenge identified during the tool kit development process was the limited time MOW staff had to attend training sessions. Training is essential to the success of prevention activities because it develops skills and increases competence related to program goals, content, and strategies. Also, the more individuals who are trained to help, the greater the potential program reach. As we found during our formative evaluation, however, participants may have limited time to attend training sessions. This problem was compounded by the difficulty of effectively communicating a large amount of new information in a short period of time. Therefore, using input from MOW and MOWAA staff, we created streamlined training curricula that can be delivered in 1 day, 4 hours, or even 2 hours. Pilot testing the training materials with the end users allowed us to refine those materials, ensuring that they were well developed, easy to follow, and effective.
Older adults, especially the homebound, have unique needs and considerations. These must be taken into account when developing prevention programs and materials. In fire safety education for older adults, research shows that one element of program success involves building rapport with older adult audiences.14 In terms of Fire H.E.L.P., many of the MOW staff members have through meal delivery developed a base rapport with the older adult clients. When creating the tool kit, we included role-playing exercises that could enhance the communications skills of the client educator, and, as a result, augment educator-client rapport. Having MOW staff role-play the educational and home assessment procedures also provides each of them the opportunity to insert their personality and style into the process, further enhancing these exchanges. The tool kit provides an appendix of “Tips for Communicating with Older Adults” that also aims to enhance the client-educator rapport.
Formative evaluation can identify cognitive, ambulatory, and sensory (eg, hearing, visual) issues that affect older adult clients. Many of the fire safety strategies that are included in the tool kit—such as having working smoke alarms installed on every level of your home, testing smoke alarms monthly, and knowing 2 ways out of every room—are appropriate for the general population. However, MOW and MOWAA staff felt that some of the clients might not be able themselves to test and change their smoke alarm batteries or escape from a fire on their own. On the basis of this input, we have included some adaptations to the materials. For example, MOW programs can choose to have their volunteers test the alarms each month rather than having the client doing so. With regard to escape planning, because not all clients are easily mobile, the tool kit includes recommendations that a phone and emergency numbers are kept near the client's sleeping area. In the event that a client cannot get out during a fire, he or she should get as low to the ground as possible. In addition, the educational tool is designed to give the MOW volunteers the flexibility to deliver a range of messages on the basis of the needs and abilities of the resident.
Smoke alarms are effective and reliable devices that can decrease the risk of death in a home fire by up to 50%.4 Smoke alarm installation programs have shown the potential to reduce fire-related injuries and deaths.8 The results reported by MOWAA (ie, 2251 clients receiving fire safety education and 4453 installed smoke alarms) reinforce our confidence that the tool kit was an important component of the Fire H.E.L.P. program. In the participating Texas communities, the MOW programs that used the Fire H.E.L.P. program and tool kit made substantial progress toward reducing injuries and loss of property and life due to fire and fire-related hazards.
Traditionally, firefighters deliver smoke alarm installation and fire safety education programs. The Fire H.E.L.P. approach suggests that nontraditional organizations with access to high-risk populations and with the capacity to deliver fire safety education can effectively reach older adults in home settings as well.
CDC used formative evaluation in the development of the Fire H.E.L.P. tool kit. The process for developing the tool kit has been described in detail to enable other programs to learn from and potentially replicate CDC's approach, resulting in time savings and enhanced results.
With the success of Fire H.E.L.P., CDC has developed a generic version of the Fire H.E.L.P. tool kit that other organizations can use for a smoke alarm installation and fire safety education program targeting older adults. Called Fire Safe Seniors, the tool kit has an implementation guide with information for planning and running a comprehensive fire safety program for older adults. It also has 3 different training curricula and tools for conducting the home assessments, client education, smoke alarm installations, and process evaluation. Access the tool kit online at http://www.cdc.gov/HomeandRecreationalSafety/Fire-Prevention/index.html.
When using the formative evaluation approach to create health education initiatives and to address other healthy home topics, the lessons learned during the tool kit's development should be helpful.