DR. EDWARD SCHOR: You made a point of training not just the physicians but, in fact, the office practices and the staff. I wonder if you have any sense of the different responsiveness of those two parties and their subsequent role in making referrals? I subscribe to the idea that involving the staff is probably critical because they are the ones who are going to make some of this happen. But I don't have any data for that and I wonder what your experience has been?
MS. SUSAN VATER: The people who attended the training tended to be really self-selective. When they knew what the training was going to be, they chose who was going to sit in on it. There was always a discussion about who was the decision-maker about making the referral and then who actually made the call.
DR. EDWARD SCHOR: Who actually made the referrals? Who made the referrals happen? Was it because you had the staff involved? Were more referrals happening when you had greater staff involvement than when it was just the doctor who showed up for the training? You don't have that exact statistic, but I wonder if you have a feel for that in any of the calls that have been made or follow up interviews you're doing?
MS. SUSAN VATER: In terms of what they said they were going to do, the people at the training session were the decision-makers and the people who made the call. Whether that actually happened, we're not sure.
DR. CLIFFORD O'CALLAHAN: Help Me Grow contacted me and we had the training in our residency program. I had already been using the PEDS for several years after a presentation at a Pediatric Academic Society meeting by Frances Glascoe. I had incorporated it within our residency because I'm working with family practitioners. I had previously done so in the Indian Health Service in the Northwest Territories because their family practitioners take care of many of our children. I cannot teach family practice residents in four months what I learned in three years of pediatric training, especially around child development.
Staff training is important, because it's the staff who put the PEDS forms in our charts. I'm doing this within our three offices as a Q.I. project to prepare my residents for practice and to help them be good at screening for development. Until the nurses accepted the concept of why we're doing this, by using examples of some of our children, they didn't bother putting the PEDS in charts. Once they understood, there was buy in and the percentage has increased. At our staff meetings, I am using the National Initiative for Children's Healthcare Quality (NICHQ) method of giving feedback and it has created friendly competition among our practices. In Middletown, Connecticut, we've gone beyond the practice setting. We believe that changing physician behavior is difficult. Anyone who touches the life of a child should get this training. So we've trained Woman, Infants, and Children program (WIC) workers, parent aides, the home visitors in the Nurturing Family Program, and day care providers in the school readiness programs. They love to hear about this. They are doing their version of screening and some of the programs already use Ages and Stages. If something is identified, they now know to whom to refer.
The buy in from other people is almost stronger than for physicians. As a practitioner, I may be more influenced by another practitioner. If somebody who is doing it can say, "I have the experience and it is possible," there's more likelihood of implementing change.
DR. KATHLEEN MCKAY: When we did the evaluations after the training, we knew whether the respondent was the primary care provider or other office staff. By and large, the other office staff was less positive about the training than the primary care providers. The providers were confident in both their own ability and that of their practice to change. Although the staff evaluations were also very positive, they were notably less positive. They had more doubts and less confidence that the practice was going to be able to change. So I think that the training program can do more to try to reach the non-primary care provider staff. The high-volume practices had one person making the calls over and over again. This was often a nurse. Those were the practices who were cranking out the referrals. When there was one nurse in charge of referrals, the volume was increased. So I think there are efficiencies in assigning that responsibility to one staff member.
MS. JUDY SHAW: I think it's easy to change provider behavior, whether pediatrician, family practitioner, or nurse practitioner. I've worked not just in Vermont, but in inner-city Boston for 20 years and I've worked in primary care, so I've got that perspective, as well. I believe that all providers want to deliver the absolute best care possible, so they are receptive if you show up in a practice with a way for them to provide better care. I've heard in Vermont, "preventative services, immunization, and lead screening are great, Judy, but what I really care about and why I come to work everyday is to help parents raise healthy and happy children. Can you do something about child development? I'm really interested in that."
If you can give them a proven way, or at least some hope about how to implement developmental screening in their practice in a way that is efficient, that eliminates some of the barriers to doing it, and you can use a systems approach and help them to do a better job and give them the resources, it's easy to change behavior. So often we go in with an unfunded mandate, you throw things at them. In North Carolina, they've mandated developmental screening, but what they are trying to do is couple that with how to improve the system. It's hard to change behavior if it's an unfunded mandate or it's "You must do this, but we're not going to tell you how to do it and how to make it work in your practice." But if you couple it with a Help Me Grow program and how to think about implementing screening and surveillance in your practice with supports, you have a much better chance of improving performance.
DR. JUDITH MEYERS: I'm very interested in efficacy as one of funders, but we might need to actually broaden the conversation. We have been focusing on EPIC and the evaluation of how it's influencing behavior to get to the end, the use of developmental outcomes for children. We also need to think about how well the whole system is working and how we evaluate and document it. The challenges are going to be similar. This is one piece of the larger system that we are all supporting and trying to grow.
We use evaluation data in many ways and one is to help us improve the system. We make the case as to why this is worth supporting as we bring data to policy makers and others who are going to make decisions about whether to reimburse or fund for these kind of services. At some point, can we also get people thinking about what else we need to look at and study and collect data to show how Help Me Grow is working and how we make the case for why this model makes sense?
DR. NEAL HALFON: To me, the unexpected was the low level of physician uptake in the number of referrals. It's a doubling, but I actually thought it was going to be more than what you found. The information about the family practitioners is interesting in the sense that the family practitioners don't basically understand the early childhood development system and have a hard time making referrals. So there is an interesting and obvious educational intervention.
We have been thinking about improving how the pediatric office works and doing it with a Help Me Grow kind of system by also engaging the childcare sector in the developmental surveillance process. We are now exploring having childcare centers actually use the PEDS. So that when the child shows up in the pediatrician's office, the PEDS has already been done and the childcare center has sent a referral: "We're with this kid a lot of the day. We're concerned about this. The parents are concerned about this. Can you sort of look at this?" So that, in a sense, there is some momentum coming into the pediatric office. There is an additional flow of information and interesting concerns, so that the pediatrician has more to deal with and potentially use. The surveillance happens outside of the office, the screening actually happens within the office, and then, if an additional assessment needs to be done, they refer the child to a mid-level assessment center.
We are particularly interested in the developmental services pathway. What role does the childcare sector play in the surveillance piece, the pediatric sector in the screening piece, and whoever else is going to do assessment and evaluation? We have broken it down into those four areas. In Rhode Island, they have three areas and are looking at it in a similar way. Can the childcare centers be mobilized, especially when early education child care is a real partner? Then the issues become how do you do the hand offs, how do you get the information to them?
We are going to be piloting, hopefully in about six months, a virtual private network in which you have childcare providers and pediatricians actually linked together, so you can send information electronically, similar to a system in Baltimore that Ray Sterner and Barbara Howard have created. They have created an automated system also for parent input. We see childcare centers doing similar things.
DR. PAUL DWORKIN: How do we optimally capture the effectiveness or lack of effectiveness of training? What are the measures that best document the effectiveness of these sorts of programs? And how do we gather the data that would support expansion, replication, and dissemination?
MS. AMY FINE: The pharmaceutical companies go back and visit every few weeks. It's not a "one shot deal." Something in their data tells them that it is cost effective for them, and this goes back to the relationship piece, to come back on a regular basis and convince them that they should buy the product. I am wondering if we can look at how they marshall their data and think about how we can measure impact over time, and whether we should think about these recurring visits. We are realizing that we need to re-immunize folks after a certain amount of time. We should think both in terms of practice and in beginning to collect data.
DR. KATHLEEN MCKAY: The training part is one small piece, but how do we evaluate the whole Help Me Grow system? It may be obvious that perhaps the place to start is to what extent is the system penetrating into the group of at-risk kids? We might make estimates of how many of those Connecticut children are we trying to capture? Who is our at-risk population? There are various ways of estimating what that population is and then looking at the volume of calls to Help Me Grow.
For instance, when I was working on ChildServ, we estimated that it about one percent of children per year. We did not have an estimate of our target group, what percent of the total pediatric population that was, but this was a basic start to looking at the system. How many might we expect to reach? What are estimates of the major groups, such as behavioral concerns? Does our penetration vary by geographic area? Start with the basics. How effective is the system at reaching into the group of at-risk kids, and does it vary by urban, rural, and other ways? You can go on from there, but that is the essential question to start with.
DR. ROCHELLE MAYER: In pharmaceutical detailing, there is a goal to increase sales. Some number is being produced by these visits. I wonder if, on a practice by practice basis, you can estimate when doing the training that you should be finding about 10 percent, or whatever the target, and if you are finding two percent, it means you are missing children.
In addition to identification, you are looking at referrals. How many children should be referred? Then you should be looking at the intervention. How many receive services? Population-based projections on what should be achieved would be a useful measure.
DR. WILLIAM HOLLINSHEAD: We decided not to call this process detailing, although pharmaceutical representatives help train our staff to do it. You can reach all of the practices in an environment our size every month or so with a staff of about two full-time individuals through extensive training. They are promoting more than developmental screening. They are also focusing on vaccines, and lead screening, and other topics.
Real- time access to a database helps for just the reasons that Rochelle has mentioned. Just as the pharmaceutical representative apparently has access to the latest sales data from the drug wholesaler, we can go into a practice and say, we notice that we could help to do more to do lead screening, or whatever it is. If done diplomatically (and with food), the practice rallies round and becomes a partner in this work with the responsible state agencies. Now that we have the Head Start and the large childcare centers and also the school nurses tapped into Kids Net, the same person details those users, who each have rather different needs and attitudes. The staff person becomes abridge between the medical home and the community partners, who share an information base and a set of responsibilities. So if you can afford it and sustain it, that real time, direct eye-contact, detailing-type function turns out to be very helpful, at least in our environment.
DR. PAUL DWORKIN: To use a publishing parlance, what Rochelle and, to a degree, what Kathleen mentioned would really be the kind of "impact factor" of the program, if you will. What percent of children are being reached? How many are being detected? How many are being linked to programs and services? To one degree or another, they are process measures. How critical is it from the standpoint of expanding, disseminating, replicating, that we actually be able to demonstrate a positive impact on children's development and/or behavior? I raise that question cautiously, fully recognizing the multitude of factors that are simultaneously impacting on development.
In my own experience while pursuing grant support for research projects, funders, particularly public government funders at the federal level, are not very sympathetic to those research projects that use very proximate measures that we know impact on children's development. For example, when we designed a study to look critically at the impact of anticipatory guidance on developmental outcomes, we proposed to use parent/infant interaction as an outcome measure, knowing that the child development literature is replete with research studies showing the positive correlation between parent/infant interaction and infant developmental outcomes. We had no success in securing funding because we were not directly measuring developmental outcomes.
Many factors impact on development. So what are the most meaningful and feasible outcome measures for developmentally oriented programs?
DR. RICHARD ANTONELLI: We are talking about the target audience and the children in the families. But in terms of training, we are talking about target audiences being the office structures. We were part of two different studies that characterized what care coordination is at the level of a practice and who does it? The second study we did was national in scope and purposefully stratified by practice types. We had rural, we had inner city, we had primarily private pay, and public pay.
Results were very interesting. The staff who actually do the lion's share of care coordination, which is a surrogate for our discussion of referrals, was driven by the type of payer mix for that community. In one center that actually had a paid care coordinator, a nurse worked from her kitchen table in her home. She only coordinated care for children with autism and Down syndrome, based on grant funding. A particular practice was paid fee for service. So whenever a family needed anything, they would be brought into the office, see the doctor, the doctor would essentially give the 800 number, record an office visit, collect a fee, and the parent would leave. At the other end of the spectrum, practices with care coordinators would allow the family to call in and receive care from home. In terms of training and efficacy, what is going to give "the biggest bang for the buck?"
The people that do most of the care coordination across the country, in fact, are the nurses. And our second, national study showed that indeed the RN population of care coordinators made the biggest impact on the bottom line, specifically the number of emergency room and office visits that were avoided. Think about what other business model could get you kudos for not selling your service? In the medical home, when you are doing a good job, it is very antithetical to surviving.
In talking about spread, you need to look at the model of the practice that is influenced by many factors and that is where you need to target your training. You should always get buy-in from the doctor. If you don't have senior leadership buy-in, things usually fall apart.
Speaking from my own practice, cross training brings broad endorsement of quality improvement initiatives. Our secretaries take great pride in the fact that they are engaged in trying to secure difficult referrals. Our nurses take great pride in the fact that they will interrupt their time on the triage line because a particular family with whom they have an ongoing relationship calls in. In a practice that is functioning on a shoestring, i.e., a primary care pediatrics practice that is focused on children and youth with disabilities, for nurses to feel that they are contributing more than returning 100 phone calls a day is a huge incentive.
We should be measuring provider satisfaction, as broadly defined, and also their ability to make these referrals. So we should not just be counting referrals, but also asking those nurses, "Do you feel in power? Are you part of the team? Are you contributing to these referrals?"
DR. ROCHELLE MAYER: Two issues are potential measures. One is the children who are being "kicked out" of daycare. So one measure could be whether the retention rates in daycare are improving. Another is the age by which children are first identified for early intervention. Are those ages becoming less with these interventions?
DR. PAUL DWORKIN: There are certain dramatic behavioral outcomes that are best avoided and there is also the possibility of looking at age of early identification. The leap that we are making is our belief that early intervention works.
MS. JUDY SHAW: How important are the outcome measures? If they are important, who is going to pay for them? For those of us who are in the research world, so often only five percent of our budget is evaluation. So not only do we need to think about building a bigger component for evaluation, but we must think about how are we going to come up with the measures, develop the measures, test the measures, and then implement and use the measures? And very few funders will support us to do that.
An additional challenge is that we don't have good measure for many of the child health outcomes. We do not know the baseline at what we are aiming and there are not great outcome measures to really assess how we are doing. I feel like the work that I am doing is jumping from project to project: this ones finishes, the next one comes at me. I would love to take a look at the big picture and really understand what we are doing and the impact of it but, too often, I am not funded to do that. This goes back to the Legislature. It goes back to the funders. It goes back to those people that are paying us to think through this system of care. How can we do a better job of understanding the impact, before we just move onto the next thing?