Gorter, Hetty PT, MSc; Lucas, Cees PhD; Groothuis-Oudshoorn, Karin PhD; Maathuis, Carel MD, PhD; van Wijlen-Hempel, Rietje MD, PhD; Elvers, Hans RI, MSc
Telemedicine is widely used in various health care fields, and recently the development of more sophisticated technologies has generated wider interest among health care providers. Despite the widespread use of telemedicine in most major areas of health care, only in a few areas does strong evidence exist that the diagnosis and management decisions provided by telemedicine are comparable with those made face to face. The paucity of evidence was well established by the systematic review of Hersch et al.1 Their review addressed the efficacy of telemedicine in diagnostic and management decisions. The results showed that the overall methodological quality of most studies was low and that the best evidence for diagnostic efficacy was found in the areas of psychiatry and dermatology. In addition, reasonable evidence shows that the sensitivity and specificity of general medical history and physical examinations performed using telemedicine are relatively good.1 Craig et al2 demonstrated that the neurological examination of adults using telemedicine is at least as good as face-to-face examination. The results of the study of Salih et al3 indicated that the use of video clips to identify an abnormal gait in elderly patients with a variety of gait abnormalities was probably as good as live examination, even when supporting clinical information was not supplied to the observers. Little is known, however, about the application of teleconsultation in the field of pediatrics.1
In pediatric rehabilitation and pediatric physical therapy, children with motor problems are often referred at preschool age. However, assessment at preschool age is notoriously difficult due to the time needed to ensure the child's cooperation.4 A good solution to achieve the child's cooperation for the assessment of motor abilities is playing in a standardized free-field situation. This situation fits well into the setting of pediatric rehabilitation and pediatric physical therapy, where observation of motor behavior is one of the screening tools for preschool children with movement disorders.
Video recordings of such “move and play” sessions may form a good basis for teleconsultation. Our medical team's experience and clinical expertise inform us that video recording is a good way to communicate about the quality of movement. This idea was the starting point for a new telecommunication service for health care providers in pediatric rehabilitation and pediatric physical therapy.5 As images say more than words, the main added value consists in the moving image, which helps when assessing the child's movement disorders.
Therefore, the primary aim of this teleconsultation study was to investigate the reliability and validity of motor assessment on the basis of short video clips summarizing a free-field “move and play” session, which may be used in teleconsultation. Reliability was evaluated by assessing agreement between observation of a child on the full video recording and his/her observation on short video clips sampled from the full video recording. How the video clips were selected from the full video recording is reported in the Methods section. We have done this to determine whether the observers' opinion was influenced by the selection of the video clips. Validity was evaluated by assessing the agreement between live observation of a child and his/her observation on the full recording.
A secondary aim of the study was to explore the validity of the clinical classification “typical development” (“typical”) and “suspect for atypical development” (“suspect”) based on the short video clips for the prediction of neuromotor developmental disorder.
Twenty-seven children (14 boys and 13 girls), aged 18 to 48 months (mean ± SD: 33 ± 4 months), participated in the teleconsultation study. They had been referred to the rehabilitation center between August 2009 and July 2010 on the basis of neuromotor concerns. Another 30 children (17 boys and 13 girls), also aged 18 to 48 months (mean ± SD: 32 ± 8 months), were included in the explorative validity portion of this study focused on the clinical classification based on video clips. Twenty-one of the 30 children represented referrals to the rehabilitation center in the period October 2001 to December 2009 based on neuromotor concerns. At the time of their first observation in the rehabilitation center, the children's diagnoses had not yet been made. After the initial observation, a full clinical workup was completed, including—on indication—a magnetic resonance imaging of the brain, a muscle biopsy or DNA analysis and their final diagnoses were then established. This means that the children in the explorative validity study of the clinical classification were included retrospectively on the basis of their diagnoses and for this research the recording of the Hempel screening (see later) in the first assessment was used. The diagnoses of the 21 children included cerebral palsy or other central neurological disorder, with level I or II classification on the Gross Motor Function Classification System6 (n = 8), neuromuscular or mitochondrial disorder (n = 7) or syndrome with problems with neuromotor development (n = 6). Nine children with typical development, recruited through colleagues, participated in the neuromotor assessment for the explorative validity study of the clinical classification.
The Medical Ethics Committee of Medical Spectrum Twente considered the study as a part of usual care and did not give additional approval. All parents provided written informed consent for the use of the videos for research purposes.
The Hempel examination technique can be used for the evaluation of neurological dysfunction.7,8 The interrater reliability and construct validity in terms of correlation with perinatal and neonatal risk factors of the Hempel examination are satisfactory.7
Neuromotor behavior of all the children was assessed by means of Hempel screening, which was developed from the neurological examination technique of Hempel.7 This screening is based mainly on the observation of spontaneous motor behavior and is a suitable tool for the assessment of qualitative changes in motor functions during this age period, thus making it suitable for teleconsultation. During this age-specific neurological assessment, the children performed various motor tasks while playing in a standardized free-field situation, which lasted about 15 minutes (Figures 1 and 2); in addition, traditional neurological items such as muscle tone, reflexes and cranial nerve function were evaluated. An essential feature of the screening is the evaluation of the quality of motor behavior in terms of variation and variability.4,9 Typical human motor development is characterized by variation and the development of adaptive variability. Hadders-Algra9 described the possible applications of variation and variability in diagnostics in children with or at risk of a neuromotor developmental disorder.
In the full Hempel examination, neuromotor function is evaluated in 5 domains: fine motor function, gross motor function, posture and muscle tone, reflexes, and visuomotor function. Qualitative aspects of motor functions (ie, prehension, sitting, crawling, standing up, standing, and walking), muscle tone, reflexes, and visuomotor function can be assessed. The classification of neurological dysfunction is based on specific criteria for single items in the various domains. Items are scored in qualitative categories, denoting the type and/or severity of deviancy using a precoded ordinal measurement scale.7,8
The Hempel screening7,8 consists of 21 items from the full neurological examination technique of Hempel. In the current project, only the items dealing with spontaneous motor behavior (n = 18) were taken into account. The sum of these 18 items resulted in the total screening score, with a range of measures from 16 to 58. The sequence of the items of the screening is more or less fixed, depending on the cooperation of the child. The interrater reliability of this screening is satisfactory.7,8 Only limited data are available on the validity of the screening.8 In a small group of children (n = 42) at risk of developing neuromotor disorders, the discriminating potential of this instrument was evaluated.7 The predictive value of the screening has not been tested.
The screening also results in the clinical classification “typical development” (“typical”) or “suspect for atypical development” (“suspect”) based on Gestalt perception. This is more subjective than the judgement of the items of the screening in accordance with the protocol.
Agreement between live observation of a child and his/her observation on a video recording using Hempel screening has not been studied.
The live observation was carried out by 2 physiotherapists (C.H. and G.V.), who observed the screening through a 1-way screen (Figure 3). Clinical details of the child were unknown to the observers. The full screening was recorded on video. For the assessment of agreement between the live observation and the observation based on the full video recording, both assessors reassessed the children after an interval of 2 months on the basis of the full recording. In the study of Heineman,10 the observers evaluated the video recordings of the same child twice, with an interval of 4 weeks for evaluating intrarater agreement of the Infant Motor Profile. Heineman's study, however, only used observation of a video recording. In our teleconsultation study, the interval between the 2 assessments (live observation of the child and observation of the full video recording) was chosen on the personal advice of Hadders-Algra (Professor of Developmental Neurology, University Medical Center Groningen, the Netherlands). Her advice was to wait at least 6 weeks between the live observation and that of the video recording, to avoid recall bias. This interval was sometimes longer for practical reasons (availability of the observers) and to ensure that there was no possibility of recall bias.
A short video summary was produced from the full video recordings of the screening of all the children from the teleconsultation study group by sampling 6 one-minute video clips from the full video recording, which includes the items of the screening (“6act”), and another short video summary, which shows the child at 6 one-minute intervals (2nd, 5th, 7th, 10th, 12th, and 15th minute (“6time”). Agreement was determined for both the total screening scores and the clinical classification “typical” or “suspect” between the full video recording and the short video clips (“6act”) and between the full video recording and the other short video clips (“6time”). All recordings were presented randomly (both the child and the type of recording) and were assessed by 3 observers: 1 pediatric physiatrist (C.M.) and 2 trainees in pediatric physiatry (M.H. and A.E.), with an interval of a minimum of 6 weeks. The clinical details of the children were unknown to the observers, except their ages.
Also agreement was determined between live observation of the child and observation on the full video recording for both the total screening scores and the clinical classification “typical” or “suspect.”
Exploration of Validity of the Clinical Classification Based on the Short Video Clips
The 21 children from the explorative validity study of the clinical classification also underwent Hempel screening the first time they came to our center for observation, which was videotaped. The 9 children with typical development also underwent Hempel screening. From the video recording of these 30 children, short summaries were sampled in a similar way to the teleconsultation study group. Agreement was determined between the clinical classification “typical” or “suspect” and the child's final diagnosis, which was determined at a later stage and after a full clinical workup, and typical development. The video clips were assessed by 2 pediatric physiatrists (C.M. and R.W.), who classified the neuromotor development of the children as “typical” or “suspect” (Figure 4). The clinical details were unknown to the observers, except their ages. The video clips were shown at random, with a period of 2 to 3 months between 2 observations.
The choice of observers fits with the practical situation of our center, in which the pediatric physical therapist assesses the children by means of Hempel screening and questions about suspect neuromotor development are put to medical specialists. An important factor is that there was no funding for this research, and thus it had to be carried out with the observers available.
Agreement between the total screening scores of the full video recording and the video clips was measured with the intraclass correlation coefficient (ICC). Agreement between the total screening scores of the live observation and the full video recording was measured with the ICC. Agreement in the clinical classification “typical” or “suspect” between the various types of assessment (live observation, full video recording, and both types of video clips) was determined with Cohen's kappa.
In the explorative validity study of the clinical classification agreement between the clinical classification based on the short video clips sampled from the full video recording of the first observation, or the full video recording of the screening of the children with typical development, and the presence of a final diagnosis or classification as typical development, was determined with Cohen's Kappa.
Statistical analyses were performed using SPSS version 16. Statistical significance was considered achieved at a α value less than .05. The kappa value indicates the strength of agreement, where 1 is complete agreement and 0 is chance agreement.11–13
The international terms for agreement (almost perfect, substantial, moderate, fair, slight, poor)11 are related to the associations between groups and are not reducible to individuals.
Agreement between the total screening scores based on observation of the child on the full video recording (“film15”) and his/her observation on video clips including the items of the screening (“6act”) was almost perfect; the ICC was 0.83 (Figure 5). The mean difference was −0.04 (SD = ±2.69); the range of measures was 16 to 58. The Bland Altman plot (Figure 6) shows that 95% of the score difference was between −5.31 and 5.24. Agreement between the clinical classification “typical” or “suspect” based on observation of the full video recording (“film15”) and observation of the same child on video clips including the items of the screening (“6act”) was substantial (Cohen's kappa: 0.63).
Agreement between the total screening scores based on observation of the child on the full video recording (“film15”) and his/her observation on video clips showing the child at 6 one-minute intervals (“6time”) was substantial; the ICC was 0.80 (Figure 7). The mean difference was 0.15 (SD = ±2.99); the range of measures was 16 to 58. The Bland Altman plot (Figure 8) shows that 95% of the score difference was between −5.71 and 6.01. Agreement between the clinical classification “typical” or “suspect” based on observation of the full video recording (“film15”) and his/her observation on video clips showing the child at 6 one-minute intervals (“6time”) was moderate (Cohen's kappa: 0.56).
Agreement between the total screening scores based on live observation of the child and observation of the full video recording (“film15”) was almost perfect; the ICC was 0.87 (Figure 9). The mean difference was −1.28 (SD = ±3.07); the range of measures was 16 to 58. The Bland-Altman plot (Figure 10) shows that 95% of the score differences was between −7.30 and 4.74.
Agreement between the clinical classification “typical” or “suspect” based on live observation and on the full video recording (“film15”) was substantial (Cohen's kappa: 0.66). Perfect agreement occurred for 91% of the children (in 49 of 54 cases), but the prevalence of the classification “typical” was low, which reduced Cohen's kappa.
In the explorative validity study of the clinical classification agreement between the clinical classification “typical” or “suspect,” based on the short video clips, and the presence of a neuromotor developmental disorder was substantial (Cohen's kappa: 0.75).
The present teleconsultation study suggested a satisfactory agreement between evaluation of a child's motor behavior on the basis of spontaneous motor behavior assessed with Hempel screening and evaluation on the basis of short video clips sampled from this screening.
There was a stronger agreement between the total screening scores of the full video recording of the screening and of both video clips than between the clinical classification “typical” or “suspect” based on the full video recording and on both video clips. An explanation might be that the criteria of the clinical classification “typical” or “suspect” of Hempel screening are based on Gestalt perception, which is more subjective than the use of clear criteria for the items of Hempel screening.
The results of this study are in line with the results of Craig et al,2 which showed that neurological examination on video recording compared favorably with face-to-face examination, and the study of Salih et al,3 which showed that the use of video clip examination is probably as good as live examination. Both studies were conducted with adults.
Evaluation of the scores in our teleconsultation study indicated that the items of Hempel screening had been scored more strictly for the video recording than for the live observation. This is in line with the results of Craig et al,2 showing that examination using video recording is more sensitive in detecting abnormalities than face-to-face examination.
The explorative validity study of the clinical classification showed that agreement on the clinical classification “typical” or “suspect” based on the short video clips and the presence of a neuromotor developmental disorder was substantial. This is in line with the findings of the systematic review of Hersch et al,1 describing relatively good sensitivity and specificity of physical examinations performed with teleconsultation, that is, short video clips. The explorative validity study of the clinical classification showed that screening with short video clips could have predictive validity and may form a good basis for teleconsultation.
A strength of this study lies in the fact that the children participating in both study groups were a representative group of children who were potentially eligible for teleconsultation. The diagnoses of the children in the explorative validity of the clinical classification portion of the study indicate a neuromotor developmental disorder. In most cases, this is not immediately obvious at birth but becomes apparent in the first few years. This group of children was thus eligible for teleconsultation. Other strengths were that the children's clinical details and final diagnoses were known to none of the observers and that the children were examined in an age-appropriate way and with a method applicable in pediatric rehabilitation.
The study also has several limitations. First, only limited data on the validity of Hempel screening were available. As the discriminative validity was tested in a small and heterogeneous group of children who were considered to be at risk of developmental disorders, interpretation of the differences in the quality of motor behavior with that of the children with typical development had to be done carefully. Second, data from the Hempel screening were limited and information on the construct validity of the screening was lacking. A third limitation is that the study group of the explorative validity study of the clinical classification was small and selective, because the children were included retrospectively on the basis of their diagnoses. Therefore, the results cannot be generalized to the population of children referred for teleconsultation. This also implies that sensitivity and specificity could not be determined. Further research with larger samples is needed to support the predictive validity of the video screening.
In the future, we hope that children with a suspected neuromotor developmental disorder can be detected earlier through teleconsultation.
Assessments based on short video clips of the spontaneous motor behavior part of the Hempel screening, as used in teleconsultation, agree in a satisfactory way with assessments based on the entirely spontaneous motor behavior part of the Hempel screening. The results of this teleconsultation study indicate that Hempel screening enables reliable and valid observation of video clips of spontaneous movement behavior, for which video clips including the items of the screening are preferred.
The explorative validity study of the clinical classification suggested that the assessment of short video clips may be a valid indicator of the presence of a neuromotor developmental disorder. This suggestion certainly needs to be tested in larger samples.
The authors thank their colleagues in the Division of Physical Therapy for Children at the Roessingh Rehabilitation Center (Cécile Hudepohl and Gerben Visser) and at the University Medical Center Groningen (Miriam Helmus and Agnes Elema) for performing part of the assessment; and Jitse Jagersma and Peter Kroon for their technical assistance with information and communication technology.
© 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins and the Section on Pediatrics of the American Physical Therapy Association.