Kloos, Anne D. PhD, PT, NCS1; Bello-Haas, Vanina Dal PhD, PT2; Thome, Rachel MS, PT3; Cassidy, Jan PT4; Lewis, Lisa PT5; Cusma, Tricia PT6; Mitsumoto, Hiroshi MD, DSc7
Amyotrophic lateral sclerosis (ALS), the most common adult-onset motor neuron disease, is a progressive disease of unknown etiology, characterized by degeneration of the motor neurons in the spinal cord, brainstem, and cerebral motor cortex.1 Damage to upper motor neurons results in clinical signs and symptoms of spasticity, hyperreflexia, and pathological reflexes, whereas lower motor neuron involvement leads to muscle weakness and atrophy, fasciculations, and hyporeflexia.1 As the muscle weakness progresses, individuals with ALS experience functional losses of speech, swallowing, mobility, and activities of daily living (ADLs).2 Amyotrophic lateral sclerosis is slightly more common in men than in women, and the average age of onset is 55.1,3 There is no cure for ALS and, although some individuals may live relatively long lives, most individuals with ALS die within 5 years of diagnosis.1,4
As individuals with ALS progress through Sinaki and Mulder's 6 stages of the disease5,6 (Table 1), they experience increasing postural instability and gait deviations due to muscle strength and coordination impairments, and joint range of motion limitations. Balance impairments often result in falls and/or injuries.1,7,8 Thirty-three percent of 234 patients with ALS treated at our ALS Center during a 2-year period reported incidents of falling.8 In the early stages of the disease (Stages I and II), falls from tripping may occur due to lower extremity muscle weakness leading to foot-drop or footdrag during the swing phase of gait.1 Falls also may be caused by delayed and/or uncoordinated responses to postural perturbations or changes due to spasticity affecting the extensor leg muscles that result in ‘stiffening,’ or loss of flexion movements at the hip and knee joints, and Achilles tendon shortening.1 Thus, stretching of tight lower extremity muscles and use of orthotics and/or assistive devices may be prescribed at these early stages to prevent falls. In Stage III falls may occur because severe muscular weakness in some muscle groups may prevent adequate responses to postural disturbances. Fall prevention interventions such as stretching of lower extremity muscles, transfer and gait training with orthotics and/or assistive devices, and use of adaptive equipment such as grab bars and handrails become necessary.5,6 Individuals become increasingly unable to stand in Stages IV through VI due to severe muscle weakness in many lower extremity muscle groups, thus requiring them to change from ambulating with assistive devices to using a wheelchair as their main means of mobility.
To identify individuals with ALS who are at risk of falling, therapists must use examination tools that objectively measure functional mobility and balance.7 A variety of clinical balance tests, including the balance subscale of the Tinetti Performance Oriented Mobility Assessment (POMA), the Berg Balance Scale, the Timed Up and Go Test, and the Functional Reach Test can be used to assess an individual's balance status (for review of tests, see references 13 and 14).9-12 At our multidisciplinary ALS Clinic at the Cleveland Clinic Foundation (Cleveland, Ohio), physical therapists must evaluate and treat each individual with ALS within a time period of 30 to 45 minutes and within the confines of average-sized examination rooms. Given these time and environmental constraints, physical therapists at the ALS Clinic use the balance subscale of the POMA, also referred to as the Tinetti Balance Test, because it provides useful information about an individual's ability to safely perform a variety of functional tasks, requires minimal equipment and very little space, is easy and inexpensive to administer, and takes approximately 5 to 10 minutes to complete.13,14 The decision to use the Tinetti Balance Test over other clinical balance tests also was influenced by its ability to provide information about all aspects of an individual's postural control, including steady state, reactive, and anticipatory control.
Previous studies have established the reliability and validity of the Tinetti Balance Test for measuring balance in the elderly.9,15–19 Assessment of the Tinetti Balance Test on 15 ambulatory residents of a long-term care facility by 2 observers resulted in more than 90% agreement on all items.9 Cipriany-Dacko et al15 reported fair to excellent interrater reliability (kappa range 0.40-1.00) of scores obtained by 12 raters on 24 hospital inpatients and 31 residents of a skilled nursing facility. Assessments of 31 ambulatory elderly individuals revealed good to excellent correlations between Tinetti Balance Test scores and scores obtained on the Berg Balance Scale (r=.91), Barthel Mobility Subscale (r=.76), and the Timed Up and Go Test (r=.74).10
The Tinetti Balance Test has been shown to predict falls among elderly individuals.16–19 Tinetti et al16 found that deficits in 4 balance items (unsteady sitting down, unable to stand on one leg unsupported, unsteady turning, unsteady when nudged) and 3 gait items (increased trunk sway, inability to pick up walking pace, and increased path deviation) increased the risk of falling by 50% to 80% in 336 elderly community-dwelling individuals. Elderly individuals scoring < 10 out of 16 on the Tinetti Balance Test have a high risk of falling (relative risk = 5.4).17 Robbins et al18 developed a regression equation using hip weakness, balance and gait items on the POMA, and number of prescribed medications that predicted fallers in 149 nursing home residents with 89% sensitivity and 60% specificity. The Tinetti Balance Test was found to have 68% sensitivity and 78% specificity compared to a physical therapist's evaluation at a cutoff score of 14 when used as a screening test to detect elderly individuals with balance and mobility impairments who needed referral for detailed physical therapy evaluation and possible intervention.19
With the current move towards evidence-based physical therapy practice encouraged by professional organizations and third-party payers, therapists need valid and reliable balance measures for individuals with ALS to establish a baseline measure of functional performance and to document the stages of disease progression and the efficacy of fall prevention and other rehabilitative interventions. Previously, we reported that low total Tinetti Balance Test scores, indicating impaired balance, were moderately to strongly related to lower extremity muscle weakness and functional disabilities in individuals with ALS.7 The inter- and intrarater reliability of the Tinetti Balance Test has never been determined for the population of individuals with ALS. Therefore, the purpose of this study was 2-fold: (1) to determine the interrater reliability of Tinetti Balance Test scores during on-site rating of individuals with ALS, and (2) to determine the intrarater reliability of Tinetti Balance Test scores from videotaped performances of individuals with ALS. Physical therapist and physical therapy student raters were included to determine whether there were differences in the reliability of scores between raters with different amounts of clinical experience and education. We believed that establishment of the reliability of the Tinetti Balance Test was imperative for health professionals to accept it as a clinical tool to identify and monitor fall risk in this population.
Part 1: Interrater Reliability
Twenty-one individuals with a diagnosis of ALS [mean age = 62.81 ± 14.20 SD; 13 males (62%), 8 females (38%)] who attended the Cleveland Clinic Foundation ALS Clinic (Cleveland, Ohio) voluntarily participated in this portion of the study. Subjects were admitted to the study if they were in Stages I, II, or III of the disease. This criterion was chosen to select individuals who were ambulatory and capable of performing test maneuvers. When greater amounts of physical assistance are required for mobility, individuals receive ‘0’ scores on most or all test maneuvers, producing a floor effect. Thus, we believe that Tinetti Balance Test scores are of limited usefulness for assessing an individual's balance performance during later stages of the disease. An informed consent was obtained from each subject prior to participation. Six of the subjects were in Stage I, 8 were in Stage II, and 7 were in Stage III of the disease. Total ALS Functional Rating Scale (ALSFRS) scores ranged from 13 to 39 out of 40 (mean= 29.0 ± 6.47 SD) and ambulation and stair function ALSFRS scores ranged from 2 to 8 out of 8 (mean = 4.71 ± 2.33 SD). Higher scores on the ALSFRS indicate better function.
One experienced physical therapist, who is a member of a multidisciplinary team at the Cleveland Clinic Foundation ALS Clinic, and 2 physical therapy students were the raters. Rater 1 had 8 years of experience treating individuals with neurological diagnoses. Raters 2 and 3 were in the second year of a 2-year master's degree program in physical therapy. All raters received verbal instructions on scoring the Tinetti Balance Test by a faculty member prior to the study. Rater 1 had some experience administering the Tinetti Balance Test on a few patients prior to the initiation of the study, whereas the students had practiced the test several times on healthy individuals.
Many studies have compared the reliability of ratings between students and experienced clinicians for a variety of tests to determine whether clinical experience has an effect on reliability.15,20–24 Student raters may have lower reliability compared to experienced clinicians because of greater uncertainty about how to rate a subject's performance, inconsistent application of the test, or because they are more influenced by conditions or distractions in the testing environment. Alternatively, reliability might be higher for the student raters because they are more attentive to details than experienced clinicians. Knowledge about differences in reliability on outcome measures between student or novice and experienced clinician raters is important for clinicians working with patients, for researchers designing clinical trials, and for clinical instructors evaluating student performance on clinical experiences. A limitation of combining student and experienced clinician raters in this study is that the small numbers of raters in each group decreases the generalizability of the results to the entire populations of physical therapy students and physical therapists.
Rater 1 was designated the administering rater (AR), as he instructed the subjects on how to perform the maneuvers, guarded the subjects, and scored each subject's per-formance.15 Raters 2 and 3 were designated the observing raters (ORs) as their role was to watch at a distance of less than 8 feet from the subject and score each subject's per-formance. The ORs had a lateral view of the subject and the AR never stood between the ORs and the subjects during testing. Raters did not speak to each other during testing. This design was used because many subjects could not tolerate performance of repetitive tests on a single day due to excessive fatigue. Testing subjects on different days was not considered a viable option due to potential effects of disease progression and because of the time, expense, and physical demands required for many subjects to travel to and from a tertiary care center.
During their regularly scheduled visit to the Cleveland Clinic Foundation ALS Clinic, subjects performed one trial of each of the 9 maneuvers of the Tinetti Balance Test: sitting balance (SIT), arises (ARS), attempts to arise (AR), immediate standing balance (ISB), standing balance (SB), sternal nudge balance (SNB), eyes closed balance (ECL), turning 360° balance (TUR), and sitting down balance (STD).9,15 Each item of the Tinetti Balance Test was scored using a scale of 0 to 2 with a score of 0 indicating an abnormal response, 1 indicating an adaptive response, and 2 indicating a normal response, except for the sitting and eyes closed maneuvers in which a score of 1 indicates a normal response.9 For example, the rating of a subject's response to sitting down could be ‘safe, smooth motion (normal),’ ‘uses arms or not a smooth motion (adaptive),’ or ‘unsafe (misjudged distance, falls into chair) (abnormal).’ The total possible score on the Tinetti Balance Test is 16 points, with higher scores indicative of better balance.9
The entire testing procedure took approximately 10 minutes. Subjects were permitted to use any assistive and/or orthotic device.9 Rest intervals were provided for the subjects between maneuvers if necessary. Subjects were instructed that they could refuse to perform any maneuver if they felt unsafe. If the subject refused or was unable to perform any maneuver, each rater assigned a score of 0 for that maneuver.
Part 2: Intrarater Reliability
Eleven individuals with a diagnosis of ALS [mean age = 57.36 ± 10.71 SD;7 males (64%) and 4 females (36%)] who attended the Cleveland Clinic Foundation ALS Clinic (Cleveland, Ohio) voluntarily participated in this portion of the study. None of the subjects were participants in Part 1. Inclusion criteria for admission to the study were the same as described for Part 1. All subjects completed both an informed consent and a videotape release form prior to participating in the study. Three of the subjects were in Stage I, 5 were in Stage II, and 3 were in Stage III of the disease. To tal ALSFRS scores ranged from 18 to 37 out of 40 (mean= 29.0 ± 7.80 SD) and ambulation and stair function ALSFRS scores ranged from 2 to 8 out of 8 (mean= 4.36 ± 1.69 SD).
Two experienced physical therapists and 4 physical therapy students were the raters. Raters 1 through 4 were in the last year of a bachelor's degree program in physical therapy and received training on scoring the Tinetti Balance Test by a faculty member prior to the study. Rater 5 had 20 years of experience treating individuals with neurological diagnoses, 7 of which were spent treating primarily individuals with ALS. Rater 6 had treated individuals with neurological diagnoses for 16 years, 6 of which were spent treating primarily individuals with ALS. Raters 5 and 6 each had at least 8 years of experience administering the Tinetti Balance Test prior to the initiation of the study.
During their regularly scheduled visit to the Cleveland Clinic Foundation ALS Clinic, subjects performed each of the 9 maneuvers of the Tinetti Balance Test as described in Part 1. Test sessions were videotaped by one of the student raters using a video camera. The camera was hand-held and was placed approximately 5 feet away from the subject in the frontal plane.15 Viewer ratings of videotaped performances of subjects have been used frequently in reliability studies.15,22,25–28 Advantages of using videotapes are that they allow raters to view movement patterns repeatedly without inducing subject fatigue, and tapes can be slowed or stopped by raters to allow greater precision of observa-tions.25 To determine intrarater reliability, each of the raters viewed and rated the videotaped test sessions for each of the 11 subjects on Day 1 and then repeated the same process 1 week later.24
Data was collected from all raters and analyzed using SPSS Version 11.0 software. The interrater reliability of the individual maneuver scores on the Tinetti Balance Test among Raters 1, 2, and 3 was analyzed by calculating kappa coefficients. The kappa statistic measures the percentage of agreement between raters with a correction factor for chance agreement.29 The strength of agreement associated with the kappa statistics was classified as follows: ±0.80 = almost perfect; 0.61 - 0.80 = substantial; 0.41 - 0.60 = moderate; 0.21 - 0.40 = fair; 0.00 - 0.20 = slight; < 0.00 = poor.30 A mean kappa statistic for all 3 raters was calculated by converting kappa coefficients for individual pairs by arc sine transformation. In order to get the mean kappa values, it is necessary to convert the kappa correlation coefficients that are on a nonlinear scale into a linear scale by arc sine transformation. The transformed values are then averaged, and the mean value obtained is converted back to a correlation coefficient.31 Because of the large number of possible scores, the interrater reliability for total Tinetti Balance Test scores was analyzed by calculating intraclass correlation coefficients (ICCs) using model 3 (ICC3,1).29 Model 3 was chosen because the raters were not randomly selected.29 The inferred reliability from the ICC values was classified as follows: ± 0.85 = excellent;0.75 - 0.85 = good; < 0.75 = fair.31 Intrarater reliability among scores obtained on Day 1 and 1 week later (Day 8) for each rater and interrater reliability between pairs of raters was analyzed using similar procedures as for Part 1.
All subjects completed the 9 maneuvers of the Tinetti Balance Test without falling to the ground or complaining of fatigue in both Parts 1 and 2.
Part 1: Interrater Reliability
Substantial to almost perfect agreement (kappa range 0.62 - 0.84) was found across all raters for the individual scores on all 9 maneuvers of the Tinetti Balance Test, except for eyes closed (0.44, moderate agreement) (Table 2). When comparing agreement of scores between pairs of raters, the highest kappa values were found between observing raters (ORs) (0.61 -1.00), followed by the administering rater (AR) and OR1 (0.43 -0.83), and lastly between the AR and OR2 (0.25 -0.76).
The mean of the total Tinetti Balance Test scores for all subjects was 11.08 and ranged between values of 0 to 16. The ICC value for the total scores across all raters was excellent [r = 0.95;95% confidence interval (CI) 0.91,0.98], indicating high reliability. The highest ICC values for the total scores were found between ORs (r = 0.98; CI 0.95, 0.99), followed by the AR and OR1 (r = 0.96; CI 0.91, 0.98), and lastly between the AR and OR2 (r = 0.92;CI 0.81, 0.97).
Part 2: Intrarater Reliability
Fair to perfect agreement (kappa range 0.40-1.00) was found for each of the 6 raters for the individual scores on all 9 maneuvers of the Tinetti Balance Test recorded on Day 1 and 1 week later, except for attempts to rise for Raters 3 and 6 (0.39 and 0.30, fair range) and turning 360° for Rater 6 (0.31) (Table 3). The mean of the total Tinetti Balance Test scores for all subjects was 11.77 and ranged between values of 4 to 16. The ICC values for the total scores for raters in order from lowest to highest was rater 2 (r = 0.92; CI 0.73, 0.98), raters 1 and 4 (r = 0.96; CI 0.85, 0.99), and raters 3, 5, and 6 (r = 0.97; CI 0.89, 0.99). Comparison of mean kappa coefficients for individual maneuver scores between all possible pairings of the 6 raters revealed that interrater agreement was generally highest between pairs of physical therapy students (Raters 1- 4; 0.94 – 0.98), followed by the experienced physical therapists (Raters 5 & 6; 0.89), and lastly between students and experienced physical therapists (0.71 – 0.86) (Table 4).
This study is the first to examine inter- and intrarater reliability of Tinetti Balance Test scores when administered by physical therapists and physical therapy students in a population of individuals in the early to moderate stages of ALS. Our findings are similar to previous reports of high reliability between raters on total Tinetti Balance Test scores when the test was administered in an elderly population.9,15 The Tinetti Balance Test is well-known and widely used by a diversity of health care professionals including physicians, nurses, occupational therapists, and physical therapists.32 Despite a paucity of objective evidence for its usefulness, the Tinetti Balance Test has been used clinically to: (1) objectively measure balance performance during static and dynamic functional activities, (2) predict falls, (3) set objective goals, (4) design intervention programs, (5) measure change in balance performance over time, and (6) measure the outcome of an intervention program in elderly popula-tions.9,14,33
Physical therapists at our ALS Clinic report that the Tinetti Balance Test provides useful information that can be used to establish a baseline and stage the progression of the disease, to assist in decision-making regarding treatment goals and interventions to prevent falls, and to document the efficacy of fall prevention programs.7 For example, a physical therapist may be treating a patient with ALS who is resistant to using prescribed assistive devices or using a wheelchair for long distances. The physical therapist could explain the Tinetti Balance Test results to the patient to identify specific tasks that increase the patient's risk of falling, rather than just saying that the patient needs to use a walker because he or she is unsteady. If 3 months ago a patient's score was 15 and now it is 7, because he or she is unsteady or unsafe with 4 of the items, then the therapist knows that the patient is presently at higher risk for falling compared to 3 months ago. The large decline in the patient's scores in only 3 months demonstrates that the disease progression is fairly rapid and would alert the physical therapist that he or she needs to start planning for the patient to use motorized mobility in the near future.
The relatively low kappa value obtained across all raters for the eyes closed maneuver found in Part 1 may be due to unclear definitions of ratings. Difficulty in understanding specific definitions such as ‘steady’ and ‘unsteady’ used to rate the eyes closed maneuver may have contributed to the low reliability.15 The amount of trunk sway required to rate a person as ‘unsteady’ may vary between raters. Expansion of the rating scale to include categories similar to item #6 (SNB), where 0 is begins to fall, 1 is staggers, grabs, catches self, and 2 is steady would provide more differentiation among what is considered unsteady for the rater, ie, almost falls versus able to catch self.
No substantial differences were found in the ICC values of total Tinetti Balance Test scores between physical therapist and physical therapy student raters, suggesting that more clinical experience or education does not necessarily improve reliability. While there was a tendency for kappa coefficients to be lower for student and physical therapist rater pairs compared to pairs of students or physical therapists, the student and physical therapist raters had moderate to perfect agreement on the majority of individual Tinetti Balance Test items (Tables 2 and 4). Previous studies that compared the reliability between student or novice clinicians and experienced clinicians for a variety of tests reported similar results.15,20–25,34 Although our study was limited to physical therapist and physical therapy student examiners, we anticipate that with training other health care professionals with varying amounts of clinical experience and education could also reliably administer the Tinetti Balance Test to individuals with ALS.
The greater amount of agreement between the scores of the ORs, as compared to between the AR and ORs in Part 1 (Table 2) may be due to the less complex role of the ORs compared to the AR during testing sessions. The ORs had fewer tasks to perform compared to the AR, as they simply observed the subjects' performances and rated them immediately on each maneuver. In contrast, the AR was required to give the subjects directions, guard them to prevent falls, and remember their scores for later documentation upon completion of the entire test. The greater number of tasks performed, as well as the longer time period between the actual observation of each subject's performance and the recording of the scores for the AR, may have caused the AR's scores to be slightly different than the ORs.
In Part 2, the low kappa values for the attempts to rise and turning 360° maneuvers for some of the raters may be due to uncertainties about ratings, unclear videotaped images, or inability to view subjects from more than one plane.15 The raters may have been uncertain about choosing a particular rating on the first viewing of the videotape and changed their decisions when they viewed it the second time. Distortion of the subjects' motions from the 2-dimensional videotaped image may make it difficult for raters to clearly view all of the subjects' movements during maneuvers. Stopping the tape and reviewing small portions of the tape where the movements were unclear was allowed by the raters, but the option to slow the tape down or view the whole tape several times was not provided. The review of specific parts of videotapes is similar to the clinical practice of the examiner asking the patient to repeat a test maneuver that he or she is unsure of. Since videotaped observations only allow raters the opportunity to view subjects from one plane, it is possible that the raters may miss some of the subjects' positions or movements.
One limitation of this study is that it tested rater reliability on subject performance of the Tinetti Balance Test, and not on administering the test. As discussed previously, the reliability for raters that both administer the test and simultaneously assess subject performance may be different because of the greater number of tasks performed. Differences in the way that the test is administered such as verbal instructions to the patient, chair style or height, and how the nudge is applied could affect the patient's test performance. Patients may use their hands to push up from the chair whether they need to or not unless specifically instructed by the examiner not to use their arms if possible. If different styles or heights of chairs are used, it could substantially change the patient's ability to arise from the chair. Examiners may apply the nudge with different amounts of force, time between applications (ie, rapid succession or pause between nudges to allow the patient to recover), or rate of applications (ie, quickly versus slow).
There are several limitations to the use of the Tinetti Balance Test for assessing individuals with ALS. Based on our experience, the Tinetti Balance Test has limited usefulness for evaluating the balance status of individuals in the later stages (ie, Stages IV-VI) of the disease due to a floor effect in scores. The sensitivity of the Tinetti Balance Test to detect changes in balance performance over time is not currently known and needs to be investigated in individuals with ALS. Clinicians also should be aware that an individual's performance on the Tinetti Balance Test in a clinical setting does not necessarily indicate how the person will function at home or in the community.9 To obtain a complete assessment of an individual's balance status, clinicians must observe the person performing many different activities in as many environments as possible.
Studies are needed to determine whether the Tinetti Balance Test, either alone or in conjunction with the gait component of the POMA, can be used to predict falls in the ALS population and to determine its usefulness as an outcome measure for intervention studies. The reliability and validity of the Tinetti Balance Test should be tested in other patient populations. We believe that the reliability of the Tinetti Balance Test would be enhanced if researchers established more precise operational definitions to describe the scoring of some maneuvers. However, any revisions to the Tinetti Balance Test would need to be tested and published for reliability prior to clinical use.
This study examined the inter- and intrarater reliability of physical therapists' and physical therapy students' scores using the Tinetti Balance Test from both live and videotaped performances of individuals with ALS. The results of this study suggest that the total score of the Tinetti Balance Test is a reliable tool for assessing the balance status of individuals in the early to middle stages of ALS (Stages I-III). The results further suggest that raters with differing amounts of clinical experience or education are equally reliable at administering the Tinetti Balance Test. The Tinetti Balance Test is a valuable examination tool that can be used by physical therapists to assist in decision-making regarding prognostic, diagnostic, and intervention aspects of the clinical management of individuals with ALS.
We are grateful to Daniel Kelly, George Sibel, and Rob Wenzler for their help in the data collection and data analysis phases of this project.
This research, in part, was presented at the annual International Symposium on Amyotrophic Lateral Sclerosis/ Motor Neuron Disease, November 1999, Vancouver, Canada.
1 Mitsumoto H, Chad DA, Pioro EK. Amyotrophic Lateral Sclerosis. Philadelphia, Pa: FA Davis Co; 1998.
2 Dal Bello-Haas V, Kloos AD, Mitsumoto H. Physical therapy for a patient through six stages of amyotrophic lateral sclerosis. Phys Ther. 1998;78:1312–1324.
3 McGuire V, Longstreth WT, Koepsell TD, et al. Incidence of amyotrophic lateral sclerosis in three counties in western Washington state. Neurology. 1996;47:571–573.
4 Sorenson EJ, Stalker AP, Kurland LT, et al. Amyotrophic lateral sclerosis in Olmsted County, Minnesota, 1925 to 1998. Neurology. 2002;59:280–282.
5 Sinaki M, Mulder DW. Rehabilitation techniques for patients with amyotrophic lateral sclerosis. Mayo Clin Proc. 1978;53:173–178.
6 Sinaki M. Rehabilitation. In: Mulder DW, ed. The Diagnosis and Treatment of Amyotrophic Lateral Sclerosis. Boston, Mass: Houghton Mifflin Co; 1980:171–193.
7 Kloos A, Dal Bello-Haas V, Burton K, et al. Validity of the Tinetti Balance Assessment in individuals with amyotrophic lateral sclerosis. Proceedings of the 9th International Symposium on ALS/MND, Munich, Germany: November 1998, pg. 149.
8 Dal Bello-Haas V, Andrews-Hinders D, Richer CB, et al. Development, analysis, refinement, and utility of an interdisciplinary amyotrophic lateral sclerosis database. ALS Motor Neuron Disorders. 2001;2:39–46.
9 Tinetti ME. Performance-oriented assessment of mobility problems in elderly patients. J Am Geriatr Soc. 1986; 34:119–126.
10 Berg, KO, Wood-Dauphinee SL, Williams JI, Maki B. Measuring balance in the elderly: validation of an instrument. Can J Public Health. 1992; 83(2 Suppl):S7–11.
11 Podsiadlo D, Richardson S. The timed “Up & Go” A test of basic functional mobility for frail elderly persons. JAGS. 1991;9:142–148.
12 Duncan PW, Weiner DK, Chandler J, et al. Functional reach: A new clinical measure of balance. J Gerontol. 1990;45:M192–197.
13 Russo SG. Clinical balance measures: literature resources. Neurol Report. 1997;21:29–36.
14 Whitney SL, Poole JL, Cass SP. A review of balance instruments for older adults. Am J Occup Ther. 1998; 52:666–671.
15 Cipriany-Dacko LM, Innerst D, Johannsen J, Rude V. Interrater reliability of the Tinetti Balance Scores in novice and experienced physical therapy clinicians. Arch Phys Med Rehabil. 1997;78:1160–1164.
16 Tinetti ME, Speechly M, Ginter SF. Risk factors for falls among elderly persons living in the community. N Eng J Med. 1988;319:1701–1707.
17 Tinetti ME, Williams TF, Mayewski R. Fall risk index for elderly patients based on number of chronic disabilities. Am J Med. 1986;80:429–434.
18 Robbins AS, Rubenstein LZ, Josephson KR, et al. Predictors of falls among elderly people: results of two population-based studies. Arch Intern Med. 1989;149:1628–1633.
19 Harada N, Chiu V, Damron-Rodriguez J, et al. Screening for balance and mobility impairment in elderly individuals living in residential care facilities. Phys Ther. 1995; 75:462–469.
20 Mann M, Glasheen-Wray M, Nyberg R. Therapist agreement for palpation and observation of iliac crest heights. Phys Ther. 1984;64:334–338.
21 Lord S, Sawyer B, Pond D, et al. Interrater reliability of computer-assisted scoring of breathing during sleep. Sleep. 1991;12:550–558.
22 Knudson D. Validity and reliability of visual ratings of the vertical jump. Percept Mot Skills. 1999;89:642–648.
23 Fritz JM, Delitto A, Vignovic M, et al. Interrater reliability of judgments of the centralization phenomenon and status change during movement testing in patients with low back pain. Arch Phys Med Rehabil. 2000;81:999–1000.
24 Gregson JM, Leathley M, Moore P, et al. Reliability of the Tone Assessment Scale and the Modified Ashworth Scale as clinical tools for assessing poststroke spasticity. Arch Phys Med Rehabil. 1999;80:1013–1016.
25 Eastlack ME, Arvidson J, Snyder-Mackler L, et al. Interrater reliability of videotaped observational gait-analysis assessments. Phys Ther. 1991;71:465–472.
26 Keenan AM, Bach TM. Video assessment of rearfoot movements during walking: a reliability study. Arch Phys Med Rehabil. 1996;77:651–655.
27 Jeng SF, Yau KI, Chen LC, et al. Alberta infant motor scale: reliability and validity when used on preterm infants in Taiwan. Phys Ther. 2000;80:168–178.
28 Slagle J, Weinger MB, Dinh MT, et al. Assessment of the intrarater and interrater reliability of an established clinical task analysis methodology. Anesthesiology. 2002;96:1129–1139.
29 Portney L, Watkins M. Foundations of Clinical Research: Applications to Practice. Norwalk, Conn: Appleton & Lange; 1993.
30 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33: 159–174.
31 Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates; 1988:180.
32 Shrout PE, Fleiss JL. Intraclass correlation: uses in assessing rater reliability. Psychol Bull. 1979;86:420–428.
33 Lewis C. Balance, gait test proves simple yet useful. Phys Ther Bulletin. 1993;32:40.
34 Brooks D, Thomas J. Interrater reliability of auscultation of breath sounds among physical therapists. Phys Ther. 1995;75:1082–1088.