Perceptions of Program Effectiveness
Perceptions of program effectiveness by facility.
The mean score of the service delivery subscales was examined for each facility (Fig. 1). In all three facilities, teaching parents about child development and behavior problems was a reported weakness. All facilities had a mean that was relatively low in this area (facility 1, 2.85; facility 2, 2.50; and facility 3, 2.58). One explanation for these low scores may be related to the interpretation of the item assessing the value of the child spending time with children without disabilities. Six of the parents (33.3%) and three of the professionals (27%) felt they valued the time a child spent with any other child, whether the other child was developing typically or not. The intent of the item was to elicit beliefs about the integration of children with disabilities with children who exhibit normal developmental milestones to assess the positive effect on socialization skills and incidental learning.
Parents’ perception of program effectiveness.
Descriptive statistics were used to examine parent satisfaction with service delivery. Responses were categorized as either agree or disagree (Fig. 2). Of the parents surveyed, 16 parents (87%) were satisfied with their EI programs (range, 61%-100%). Parents reported less satisfaction in the areas of goal setting and developing strategies to help their child (66% agree), learning strategies to discipline and set limits for their child (75% agree), and learning about community resources and agencies available to assist them (61% agree).
A Wilcoxon rank sum test was used to compare parents’ responses in each subscale according to race and level of education. A slight difference was found by race with regard to parents’ perceptions of program and staff responsiveness. In this case, white parents were more likely to report staff responsiveness than nonwhite parents (median, 3.63 and 3.01, respectively;p = 0.03). No differences were found with regard to level of education.
Providers’ perceptions of program effectiveness.
Providers’ overall satisfaction with the program's effectiveness was slightly higher than that reported by parents (99%; range, 70%–100%). Areas where providers felt they needed more development included building parent networks and helping parents value the time their child spends with children without developmental delays.
Parent vs provider perception of program effectiveness.
A Wilcoxon rank sum test was used to examine the differences in the subscale scores between parents and providers (Table 4). Although not statistically significant, the providers’ perception of the program's and staff's effectiveness in providing services to promote growth in understanding normal child development differed from the parents’ rating of the providers. Providers tended to overrate their effectiveness in promoting parents’ growth in understanding normal child development as compared with parents’ rating of the providers’ abilities in this area (median, 1.61 and 1.35, respectively;p = 0.07). This was also an area that parents were least satisfied with as compared with other service areas.
Overall rating of program effectiveness at meeting family needs.
We examined the frequency of responses to identify specific service areas that may require improvement (Fig. 2). Responses were collapsed into two groups, agree vs disagree, to determine program strengths and weaknesses. Perceived areas of weakness included parents’ lack of awareness on how to set goals for their children, how to elicit the cooperation of their children, parents’ need for more knowledge about community resources, greater parental involvement as a team member, and parents’ beliefs that the child's strengths are being discussed.
According to these findings the strengths of the programs included the ability of the staff to give useful and clear information, parents’ increased ability to look at their child and see what he or she is learning to do, parents’ increased confidence in their ability to help their child and family, staff's willingness to help families and friends with questions and concerns, and parents’ confidence that their child is receiving services that combine the expertise of all staff members (Table 5).
Providers’ Perceptions of Barriers to Building Collaborative Relationships with Families
Seven of the 11 providers (64%) completed the open-ended questionnaire. The four providers from facility 3 did not answer this section of the survey. The providers who completed the open-ended questions believed parent contributions were important to a child's outcome and listed parent satisfaction as a major goal of the program. However, only four of seven (59%) expressed confidence in working with families. Two providers stated that their confidence in working with parents was dependent on the characteristics of the family. Some providers attributed their lack of comfort in working with families to a lack of formal training in family assessment and communication skills and their personal observation that some parents do not want the services the EI program could provide. Providers’ confidence in working with parents seemed to be related to their years of experience in EI and their belief that families are usually responsive to EI services.
Providers were also asked to describe the role they believed parents should play in EI service provision. Among the roles they reported were observer to primary teacher, therapist, “active” participant, expert on their child's development, advocate, and a team member. Although providers were able to list a variety of roles that would actively engage the parents, they felt a number of barriers prevented the formation of collaborative relationships. These barriers included nonvoluntary referrals, active substance abuse, families with priorities or concerns other than their children, race, language, scheduling difficulties, differing views on what EI services are and what they can do for the family, and the parents’ view of outside agencies (eg, the Department of Social Services) with respect to family function.
These providers reported that they incorporated a variety of techniques to reduce barriers and promote active family involvement in care. Some of these techniques include involving the families at each step of the process, respecting their priorities and concerns, providing information on child development and resources, listening to what parents want, keeping an open dialogue, discussing parents’ goals, being flexible, and offering positive reinforcement to parents.
This study examined parents’ and providers’ attitudes regarding the effectiveness of family-centered EI services and identified providers’ perceived barriers to forming collaborative relationships with families. Our results are consistent with past studies in this area and suggest that the attitudes of parents and professionals regarding EI program effectiveness are fairly similar. 12,19 Both parents and professionals reported high levels of satisfaction with EI service delivery, although providers felt more confident working with children than with families because of a perceived lack of formal training. Greater confidence was reported by the more experienced EI professionals. Formal training in family assessment would provide EI staff with the knowledge, skills, and confidence necessary to communicate information at an appropriate level and to encourage parent participation.
All three facilities reported a need to increase parents’ knowledge and understanding of normal child development and the potential problems that may arise as a child grows and matures. Educating parents about child development is perhaps a new role for EI service providers and is an area for future growth.
A common concern reported by parents was the feeling that their child's strengths were not discussed. One parent reported that she felt that she continually had to point out to the service provider what her child was doing. Parents also recognized the need to learn to set goals and strategies for their children. Only 66% of the parents felt they had improved their knowledge regarding their ability to develop strategic goals and plans for their child, whereas 100% of the providers felt they had taught the parents how to accomplish this objective.
A 1996 study 12 researched mothers’ perceptions of the effects of physical and occupational therapy services on their caregiving competency. The results showed the services helped to augment the maternal caregiver's confidence in caring for her child. Furthermore, although the therapists’ technical skills were much appreciated, the relationship between the therapist and mother and their communication with each other had a greater effect on the mother's perceptions of the therapists’ caregiving skills.
A final area of concern was the parents’ lack of awareness of community agencies and programs that could help their child and family manage their needs when they are no longer receiving EI services. Only 61% of the parents, as compared with 91% of the providers, believed that this information had been shared. These results are consistent with the findings of Lanners and Mombaerts, 13 who examined parental perceptions of EI services. Lanners and Mombaerts found that parents were generally satisfied with their EI services; however, the degree of satisfaction depended on the dimension of early intervention. 13 For example, parents were mostly satisfied with the EI services they were receiving, yet they gave low scores to the three dimensions concerning social support networks. This study also identified factors associated with parent satisfaction with EI services. Parent satisfaction with the program was associated with demographic features of the parents. White parents seemed to be more satisfied with services than nonwhite parents. In general, white healthcare professionals are providing services for families with diverse cultural backgrounds. A review of the literature shows that of 24 studies on the congruence of parent-professional perceptions about the development of young children with disabilities, 22 were based on white, middle-class families. 5 Researchers suggest that intervention programs do not always meet the needs of families from diverse backgrounds. 4 The differences seen between parents’ responses may be indicative of poor matching between parents and providers with respect to their racial and cultural backgrounds and beliefs. It is important that providers evaluate their own cultural views and see how these beliefs may enhance or present barriers to meeting the needs of the family seeking EI services. Family assessments are one method that providers can use to identify the cultural characteristics of families. As one study 16 suggests, cultural sensitivity should be considered in both design and implementation of EI programs. Cultural sensitivity training would enhance providers’ awareness of cultural differences, enabling them to recognize and respond to differences in parents’ perception of the impact of disabling conditions and their beliefs about the value of EI. 16
The collection of information regarding parents’ attitudes about EI service delivery is a distinctive aspect of this study. The Massachusetts Department of Public Health supports the inclusion of parents in all phases of EI program activity and project development. 3 McNaughton 20 agrees that including parental attitudes in program evaluations can be useful in delivering more effective family-centered EI services.
Furthermore, parents participating in a parent group may be more involved than other parents and may therefore be an important source of information for other parents. Targeted as “ideal” parents, they can provide valuable insight on areas where the child and family's interests need to be met. For example, despite being in a parent group, only 82% of parents and 80% of providers believed that parents increased their support of other parents through the program. One facility reported that they already involve parents in program evaluations through their own satisfaction survey. This facility (facility 1) had slightly higher mean scores on the service delivery subscales (Table 4). These higher ratings may reflect the parents’ greater sense of control over EI program offerings as a result of the incorporation of parent feedback. The other two facilities involved in this study believed that parent involvement in EI program evaluations was a worthwhile idea. One provider reported that parent satisfaction with EI program effectiveness should be assessed more often throughout a parent's involvement in the program.
The limitations of this study include the potential for selection bias because facilities with less satisfied parents and staff may have chosen not to participate. Johnson et al 19 acknowledge that the development and implementation of program evaluations in EI has not been satisfactorily done. They note that service providers are often resistant to program evaluations and view them as cumbersome. Johnson et al 19 recommend demonstrating the benefits of program evaluation and the impact of feedback on enhancing service delivery to decrease staff resistance to evaluations. McNaughton 20 suggests ways to improve response rate: endorsement letters, follow-up calls, using a variety of data collection tools, and sampling the opinions of nonrespondents to determine if they differ from those who responded. Another limitation of this study was the inability to match parents with their service providers so that both participants could rate services provided by the case manager. However, because of issues of confidentiality, this strategy could not be implemented. Finally, this study used a small sample of the provider and parent groups and was limited in geographic area.
The strengths of the study should also be mentioned. We used the Project Dakota assessment, a valid and reliable instrument for assessing program effectives, 4 along with open-ended questions to elicit further information from provider and parents. McNaughton 20 recommends the use of parent and provider surveys in conjunction with a variety of other tools (open-ended questionnaires, interviews, and child outcome measures) because parents’ and providers’ evaluations of EI programs do not solely reflect program quality and effectiveness. We also examined demographic features of parents and providers to determine the impact of these factors on program satisfaction and perceived effectiveness of service delivery. Consistent with the findings of McWilliam et al, 14 we found that both parents and providers believed services could be better, families hold strong beliefs about the need and quantity of therapy, and barriers such as amount of control over service intensity affect satisfaction with EI service delivery.
Overall parents and providers were similar in their views of EI program effectiveness, with parents ranking categories slightly lower than providers. The biggest difference was seen in the area of goal setting and developing strategies to help the child. Although providers reported that were performing very well in this area, parents’ reports resulted in considerably lower scores. Satisfaction with EI services did differ, though not significantly, on the basis of racial and cultural characteristics of the parents.
The modified Project Dakota survey is an effective tool for program evaluation, providing useful information on the strengths and weaknesses of EI programs sampled from metropolitan area of Massachusetts. Parents and providers who participated seemed to be very positive about using this instrument as a means of improving EI service delivery. The open-ended questionnaire provided additional useful information on potential barriers to providing family-focused services that would not have been obtained by the structured survey alone. Assessing the impact of family-focused EI service delivery is more accurately represented by collecting data from both parents and providers. Involving parents early in the assessment process of EI service delivery may facilitate more active parent participation in the child's care.
EI programs and state agencies are becoming increasingly concerned about the effectiveness of family-focused services to prove that earlier intervention constitutes best practice. This study has suggested several effective, cost-efficient ways of evaluating EI services through both quantitative and qualitative measures. Future research is warranted to determine the effect of cultural diversity and provider training in family assessment and communication skills on the perceived effectiveness of program delivery and satisfaction with care.
Ms Poulin Shimmel was a graduate student at Northeastern University and participated in the project as part of her requirements for a master's degree in physical therapy.
1. Effgen SK, Chiarello LA. Physical therapist education for service in early intervention. Infants Young Child. 2000; 12: 63–76.
2. McLean M, Bailey D, Wolery M. Assessing Infants and Preschoolers with Special Needs. Englewood Cliffs, NJ: Prentice-Hall Inc; 1996.
3. Family Participation Task Group of Massachusetts Early Intervention Integracy Council with collaboration of the Massachusetts Department of Public Health. Working Together. 1997.
4. Kjerland L, Kovach J. Family-staff collaboration for tailored infant assessment. In: Gibbs E, Teti D, eds. Interdisciplinary Assessment of Infants: A Guide for Early Intervention Professionals. Baltimore, MD: Paul Brookes; 1990: 287–297.
5. Bailey DB Jr, McWilliam RA, Darkes LA, et al. Family outcomes in early intervention: a framework for program evaluation and efficacy research. Exceptional Child. 1998; 64: 313–328.
6. Gavidia-Payne S, Stoneman Z. Family predictors of maternal and paternal involvement in programs for young children with disabilities. Child Dev. 1997; 68: 701–717.
7. Dinnebeil L, Hale H, Rule S. Early intervention program practices that support collaboration. Top Early Child Special Educ. 1999; 19: 225–244.
8. Holloway E. Parent and occupational therapist collaboration in the neonatal intensive care unit. Am J Occup Ther. 1994; 48: 535–538.
9. Sexton D, Lobman M, Constans T, et al. Early interventionists’ perspectives of multicultural practices with African-American families. Exceptional Child. 1997; 63: 313–328.
10. Peterander F. The best quality cooperation between parents and experts in early intervention. Infants Young Child. 2000; 12: 32–45.
11. Masin H. Perceived maternal knowledge and attitudes toward physical therapy during early intervention in Cuban-American and African-American families. Pediatr Phys Ther. 1995; 7: 118–123.
12. Washington K, Schwartz I. Maternal perceptions on the effects of physical and occupational therapy services on caregiving competency. Phys Occup Ther Pediatr. 1996; 16: 33–54.
13. Lanners R, Mombaerts D. Evaluation of parents’ satisfaction with early intervention services within and among European countries: construction and application of a new parent satisfaction scale. Infants Young Child. 2000; 12: 61–70.
14. McWilliam RA, Young H, Harville K. Therapy services in early intervention: current status, barriers, and recommendations. Top Early Child Special Educ. 1996; 16: 348–374.
15. O'Neil ME, Palisano RJ, Wescott SL. Relationship of therapists’ attitudes, children's motor ability, and parenting stress to mothers’ perceptions of therapists’ behaviors during early intervention. Phys Ther. 2001; 81: 1412–1424.
16. McGonigel M. Guidelines and Recommended Practices for the Individualized Family Service Plan. Bethesda, Md: Association for the Care of Children's Health; 1991.
17. SAS Institute. SAS System for Windows. Cary, NC: SAS Institute Inc: 1996.
18. Glaser BG. Theoretical Sensitivity: Advances in the Methodology of Grounded Theory. Mills Valley, CA: Sociology Press; 1978
19. Johnson LJ, Gallagher RJ, Lamontagne MJ. Meeting Early Intervention Challenges: Issues From Birth to Three. Baltimore, MD: Paul Brookes; 1994: 1–13, 185–271.
20. McNaughton D. Measuring parent satisfaction with early childhood intervention programs: current practice, problems, and future perspectives. Top Early Child Special Educ. 1994; 14: 26–48.
Professional Survey Questions
- I listen and respond to the concerns, questions, and ideas of caregivers.
- In meetings with staff (for assessments, conferences, monthly updates, etc), I feel I encourage caregivers to be an active member of the team and not just a listener.
- Although one staff member mainly serves a child, I feel that I seek the expertise of other staff.
- I give information that is clear and useful to caregivers.
- I feel I create programs for a child that include what is important to caregivers.
- I feel I create programs that meet the individual needs of a child.
- Overall I am satisfied with the progress children make with this program.
- I try to make sure the help I offer fits into the family's routines and activities.
- I respect the limits a family puts on the time and energy they have for their child's program.
- I inform caregivers of a variety of choices for how their child could be served.
- I enable caregivers to look at their child and see what he/she is learning to do.
- I enable caregivers to learn about helping their child.
- I enable caregivers to enjoy their child more.
- I enable caregivers to know what their child needs to learn.
- I enable caregivers to be aware of how to help their child's learning and development.
- I enable caregivers to feel more confident about how they and their family are helping their child.
- I enable caregivers to be more aware of how to help their child's development.
- I enable caregivers to have a clearer picture of their child's special needs at this time.
- I enable caregivers to feel satisfied that their child's strengths are being discussed.
- I enable caregivers to know more about how to set goals and strategies for their child.
- I enable caregivers to more strongly value their child spending time with children who don't have developmental delays.
- I enable caregivers to be more aware of how their child is like other children.
- I enable caregivers to know more ways to get their child to cooperate.
- I enable caregivers to get the help they need to learn about handling their child's behavior.
- I enable caregivers to know more about community agencies, services, and programs that can help their child or family.
- I enable caregivers to get help from staff when they want other programs to people to work with them, their child, or family.
- I enable caregivers to have contact with services and programs or people to work with them, their child, or their family.
- I enable caregivers to be satisfied with the communication between their child's team and community resource persons involved in their child's program.
- I enable caregivers to be able to get information that is important to the health and happiness of their family and child.
- I enable caregivers to involve their partner/family more in their child's learning.
- I enable caregivers to have more friends or other children helping caregivers help their child.
- I enable caregivers to help the people they know be more caring and understanding of their child.
- I help caregivers get to know other people who are caring and understanding.
- I help caregivers get support from other parents.
- I enable caregivers to feel less alone as the parents of their child.
- I enable caregivers to help their family and friends when they have concerns or questions about their child.
Parent Satisfaction Survey Questions
- The staff listens and responds to my concerns, questions, and ideas.
- In my meetings with staff (for assessments, conferences, monthly, updates, etc.), I feel I am an active member of the team and not just a listener.
- Although one staff member mainly serves my child, I feel that we receive the expertise of other staff.
- The staff gave me information that is clear and useful to me.
- I feel the program for my child includes what is important to me.
- My child's program meets my child's needs.
- I am satisfied with my child's progress since beginning this program.
- The help I get fits into our family routines and activities.
- The staff respects the limits my family puts on our time and energy for our child's program.
- I am informed of a variety of choices for how my child could be served.
- I am more able to look at my child and see what he/she can do.
- I have learned about helping my child.
- I enjoy my child more.
- I know what my child needs to learn.
- I am aware of how ordinary activities are part of my child's learning and development.
- I feel more confident about how my family and I are helping our child.
- I am more aware of how to help my child's development.
- I have a clearer picture of my child's special needs at this time.
- I feel satisfied that my child's strengths are being discussed.
- I know more about how to set goals and strategies for my child.
- I more strongly value my child spending time with children who don't have developmental delays.
- I am more aware of how my child is like other children.
- I know more ways to get my child to cooperate.
- I am getting the help I need to learn about handling my child's behavior.
- I know more about community agencies, services, and programs that can help my child or my family.
- I get help from staff when I want other programs or people to work with my child, my family, or me.
- I now have contact with services and programs or people to work with me, my child, or my family.
- I am satisfied with the communication between my child's team and community resource persons involved in my child's program.
- I am able to get information that is important to the health and happiness of my family and child.
- My partner/family are more involved in my child's learning.
- I have more friends or other children helping me with my child.
- The staff helped the people I know be more caring and understanding of my child.
- The staff helped me get to know other people who are caring and understanding.
- I have gotten support from other parents.
- I feel less as the parent of my child.
- The staff is willing and able to help my family and friends when we have concerns or questions about my child.
Keywords:© 2003 Lippincott Williams & Wilkins, Inc.
infant; child; professional-family relations; early intervention (education); outcome and process assessment; attitude of health personnel; questionnaires