Secondary Logo

Journal Logo

Evaluating Model Programs to Support Dissemination: An Evaluation of Strengthening the Developmental Surveillance and Referral Practices of Child Health Providers

McKay, Kathleen Ph.D.

Journal of Developmental & Behavioral Pediatrics: February 2006 - Volume 27 - Issue 1 - p S26-S29
SECTION II. OVERCOMING BARRIERS THROUGH TRAINING AND EDUCATION
Free
SDC
Back to Top | Article Outline

KEY POINTS

  • Evaluation of developmental programs should consider process, outcome, and impact measures.
  • Brief, in-office training for community-based child health providers was successful in influencing provider and practice behavior.
  • Training increased the identification of children with developmental and behavioral concerns and referrals to the Child Development Infoline (CDI).

The project Strengthening the Developmental Surveillance and Referral Practices of Child Health Providers was designed to educate Connecticut's community-based child health providers in early detection and identification of childhood developmental and behavioral concerns and in the use of the Help Me Grow referral system. The training program provided in-office education and discussion on developmental surveillance and on the use of a statewide triage and referral system for community-based primary care practices. Project staff traveled to each practice and offered a brief training intervention to providers and office staff.

Back to Top | Article Outline

EVALUATION DESIGN

The evaluation, supported by The Commonwealth Fund, included process measures, which tracked program activities; impact measures, which examined whether the training changed developmental surveillance and referral patterns within participating practices; and outcome measures, which assessed provider and staff satisfaction with the training and changes in attitudes toward developmental surveillance and referral. The impact evaluation was designed to determine whether the intervention was effective, using 3 questions:

  1. Is the rate of developmental surveillance for intervention practices higher after training?
  2. Is developmental surveillance more comprehensive in intervention practices after training (i.e., detecting children with a wider range of concerns, detecting more at-risk children, and detecting children at earlier ages)?
  3. Are intervention practices more likely to refer to Help Me Grow following training?

The impact evaluation consisted of a chart review study and an analysis of provider referrals. The chart review compared provider behavior before and after the intervention and compared trained and untrained practices. Charts were examined for evidence that developmental surveillance had occurred at the last well-child visit, defined as the presence in the chart of any of the following: notes on development; a completed surveillance instrument; or evidence of solicitation of parental concerns. "Completed instrument," as defined in the chart review, included any of: a validated instrument (e.g., PEDS,1 Denver II,2 Ages and Stages3), a milestone checklist, a hearing tracking tool, a growth chart, and various practice-specific forms designed to track development. The charts were also reviewed to determine if a developmental concern had been identified at the last well-child visit.

The second impact evaluation study examined providers' referral behavior using data from Child Development Infoline (CDI). CDI operates a statewide telephone number that receives all calls related to Help Me Grow, Birth-to-Three (Early Intervention), Children with Special HealthCare Needs, and Preschool Special Education and provides referral to the appropriate programs and services. Details of the impact evaluation designs are shown in the Appendix.

The outcome evaluation assessed provider and staff knowledge and attitudes using the results of questionnaires distributed at the end of each training intervention. The questionnaires were designed to assess providers' confidence in their ability to conduct developmental surveillance, the practice staff's confidence in its ability to use Help Me Grow, and the practice staff's intent to use Help Me Grow. The questionnaires also assessed barriers to surveillance and referral and the perceived importance of those barriers.

Back to Top | Article Outline

RESULTS

Process Evaluation

During the study period, 141 of 300 Connecticut community-based pediatric and family medicine practices received the training intervention (47% of all practices), compared to the project goal of 50%. Practices receiving training included a combined staff of 1301 (330 physicians, 454 other primary care providers, and 517 other staff). Of these, 59% attended the training, including 53% of primary care providers and 67% of office staff.

Back to Top | Article Outline

Impact Evaluation

Chart reviews:

Chart reviews were conducted, on average, 7 weeks after the practice intervention. Reviewers examined 629 charts from the practices that received the intervention (309 before and 320 after intervention) and 289 charts from the control practices. Over 90% of charts contained some evidence of developmental surveillance, both in the control group and in the intervention group pre- and post-training (Table 1). There were no statistically significant differences between the intervention and control groups.

The factors defined as "evidence of developmental surveillance" did not appear at different rates in intervention group charts, before vs. after training (Table 2).

Table 1

Table 1

Table 2

Table 2

Trained practices identified significantly more developmental or behavioral concerns, with 18% of reviewed charts noting a concern, compared to 9% in those same practices before training (odds ratio: 2.34; 95% confidence interval: 1.42, 3.85; p = .001) (Table 3). The increase in children identified occurred across most practices trained.

Table 3

Table 3

In those charts containing some evidence of developmental surveillance, there were no differences in the child's age, sex, insurance status, or type of delay in trained vs. untrained practices. Of those children where a developmental concern had been identified, there were also no differences in these factors by practice training status.

Back to Top | Article Outline

Provider calls to Child Development Infoline (CDI).

There were 1217 calls from community-based practices to CDI over the study period. Of these, 33% were from practices trained by the date of the call, while 67% of calls were from practices that were not or had not yet been trained. Calls to CDI from intervention practices increased over the training period. By the end of the training period, 44% of practices had been trained, but 57% of calls came from trained practices.

The average age of children referred to CDI from practices that had received the intervention was 23.0 months, compared to an average age referred from non-trained practices of 20.9 months (p = .006). Fourteen percent of calls from trained practices were for children over 3, compared to 6.4% for untrained practices (p < .0001). Among the 1106 calls for children ages 0 to 36 months, the average age of children referred from trained practices was 18.1 months, compared to 19.0 months (p = .11) from untrained practices.

There were significant differences in the conditions for which young children were referred in trained vs. untrained practices. Behavioral conditions were involved in 4.2% of referrals from trained practices, compared to 1.4% for untrained practices (p = .005). Four percent of referrals from trained practices were for parent support, compared to 0.8% from untrained practices (p = .0002). Trained practices referred relatively less frequently for health and communication issues. There were no differences by practice training status in the sex, language spoken at home, and Medicaid status of referred children.

Back to Top | Article Outline

Outcome Evaluation

Questionnaires were collected from 517 staff members from 105 practices, representing 85% of the recorded attendance at those trainings. Seventy percent of providers reported that they expected a very significant or significant change in their practice of developmental surveillance attributable to the training. Providers reported various barriers to conducting developmental surveillance, including lack of time, with 26.9% describing lack of time as a significant or very significant barrier. Other reported barriers to conducting developmental surveillance were lack of medical office staff support (15.5%), inadequate reimbursement (11.3%), lack of confidence in ability to conduct developmental surveillance (5.5%), and unspecified barriers (23.6%). Ninety-three percent of providers and staff reported that they definitely or probably intended to use the Help Me Grow program in their practices. Fifteen percent of providers and staff reported that the overall effect of all barriers to using Help Me Grow would be significant or very significant.

Back to Top | Article Outline

DISCUSSION

This brief, in-office training program for community-based pediatric and family medical practices was successful in influencing provider and practice behavior. Among practices receiving the intervention, the identification of children with developmental or behavioral concerns doubled. Practices that received the intervention referred children to CDI in greater numbers than control practices. The training increased referrals to CDI for older children, and for children with behavioral issues or a family need for parent support, although the chart review study did not find any differences in age or types of need in children identified with concerns. Trained practices may be differentially referring to CDI, using the CDI more often for certain groups of children, perhaps where there are fewer available alternatives.

The model used in developing the evaluation design assumed that the training program would lead to more consistent use by the medical practices of certain techniques of developmental surveillance (i.e., chart instruments, solicitation of parental concerns, and consistent monitoring of the child's development) and that this would be expressed in a greater frequency of chart notations and completed instruments. The model further assumed that as a result of this consistent use, more children with developmental and behavioral concerns would be identified. Although more children with developmental and behavioral delays were identified in the trained practices, it is unclear what changes in provider and staff practices, knowledge, or attitudes resulted in this change. There were no changes in charted evidence of developmental surveillance after training.

The rates of developmental surveillance noted in the charts were very high, over 90% in both intervention and control practices. This is much higher than recent surveys conducted by American Academy of Pediatrics4 and is contrary to current understanding of the state of developmental surveillance in pediatric and family practice. The high rates of surveillance may have been driven by a liberal definition of a "completed instrument" as one type of evidence of surveillance. Other types of evidence of surveillance were present in the charts with less frequency, including a note on development (45% of charts prior to intervention) and a note on parental concerns (30% of charts).

Further research on training pediatric and family providers in the use of developmental surveillance and referral may wish to investigate:

  1. Medium- and long-term effects of this program-How long does the increased rate of identification of children last? What is the longer-term effect of the training on calls to CDI?
  2. Mechanism-What changes in provider behavior or attitudes have been effected by the intervention? What is the mechanism leading to greater identification of children with developmental delays? How can these techniques or changes in attitude be more effectively transmitted to providers?
  3. Attendance-In practices that were trained, only half of the providers attended the training program. Since the primary effect of the program is on provider behavior, how can this attendance rate be improved in future training, and what effect will it have?

All of these questions are potential future issues for study that can build on the strong base of the program Strengthening the Developmental Surveillance and Referral Practices of Child Health Providers, which successfully influenced provider surveillance and referral behaviors for children in primary care practices across Connecticut.

Back to Top | Article Outline

Acknowledgement.

We are grateful to The Commonwealth Fund for the generous support of the evaluation of the training program and to Edward Schor, MD for his guidance and encouragement.

Back to Top | Article Outline

REFERENCES

1. Glascoe FP. Parents' evaluation of developmental status: how well do parents' concerns identify children with behavioral and emotional problems? Clin Pediatr. 2003;42:133-138.
2. Frankenburg WK, Dodds J, Archer, et al. The Denver II: a major revision and restandardization of the denver developmental screening test. Pediatrics. 1992;89:91-97.
3. Squires J, Lawanda P, Bricker D. The ASQ User's Guide. 2nd ed. Baltimore, MD: Paul H. Brookes; 1999.
4. American Academy of Pediatrics, Division of Health Policy Research. Identification of children <36 months at risk for developmental problems and referral to early identification programs. Executive Summary. April 2003. Available on online at http://www.aap.org/research/periodicsurvey/ps53exs.htm.
Back to Top | Article Outline

APPENDIX: IMPACT EVALUATION DESIGN

CHART REVIEWS

The chart review study was conducted among a group of practices enrolled in the ProHealth physicians' network. Ten intervention practices of varying sizes and patient populations provided access to their charts for review of well-child visits both before and after the intervention. Ten practices that did not receive training were matched on practice characteristics with the first ten, and served as a comparison group. For each outcome of interest, results from each practice that received training were compared to results in two comparison groups-the same practice prior to training, and a matched practice not receiving training.

Back to Top | Article Outline

REFER CENTER DATA

This study examined records of calls from primary care providers to Child Development Infoline (CDI) during the one-year training period. In the CDI data, the practice of the referring primary care provider was matched to the program database to determine whether the call came from a practice receiving or not receiving training, and whether the call occurred prior to or following the training session. Monthly trends in calls during the training period were examined to determine if there were increases in the number of calls to Help Me Grow and whether trained practices were disproportionately represented among practices calling or referring to Help Me Grow.

© 2006 Lippincott Williams & Wilkins, Inc.