Advances in the design, control, application, and provision of the upper limb prostheses in recent years have combined with pressure on resources to require more objective justification of the costs involved in providing these services. This has intensified interest in objective measures of performance and use of artificial arms. This need covers the entire cycle of prosthetic development from initial design, to use of a prosthesis in the home and community. At each stage, the measurement of the factors influencing prosthetic use and performance becomes key to decisions made concerning the continued development, treatment or provision of the devices and systems. Thus, it is clear that objective measures are central to this process. Discussion between professionals and a review of the literature shows that there are differences in the language and expectations of different professions. The same words can be used with very different implied meanings. For example, occupational therapists have implied meanings for terms such as “function” and “activity,” which are markedly different to what an engineer would infer from the same words. Function for a therapist is the ability for a person to accomplish an activity in their daily lives. Prosthetists and engineers may see function as the technical performance of a device (such as grip speed or force). So without a common terminology, there is a plenty of opportunity for misinterpretation.
Faced with the need to communicate more, a group of like-minded professionals came together and ultimately formed a special interest group to promote the idea of a standard within the profession to facilitate this goal. Through a series of meetings and workshops (starting in 2005), the concepts were developed and refined. A more extensive description of the process can be found in a study by Hill et al.1 The underlying method was to ascertain what tools and methods were already used, then attempt to see what questions the different professionals needed answers to, before identifying the most effective tools, and finally seeking a way to provide such information in a clear and concise way. At all points, the group aimed to create consensus through consultation with the rest of the profession. The resulting group, the Upper Limb Prosthetic Outcome Measures (ULPOM) Group presents the background to the problem, the methods they used, and the results of this process.
THE UNDERLYING PROBLEM
It was observed that within the literature, there are many tools apparently designed to assess the performance of the upper limb,2–5 as well as tools designed to measure the prosthetic control and use. Although some tools have been used for many years, it seemed that they may still have uncertain psychometric properties. In addition, only parts of particular standardized measures were being used, or the tools were being modified to fit the needs of a particular centre or culture, invalidating their design.2 Still other tests were validated for use with a different population, i.e., for neurological disorders, and have then been used in prosthetics. The regrettable conclusion from this is that the results from these tests cannot be compared between centers and cannot be trusted as they may be incomplete or invalid. Therefore, it is almost impossible for members of the same profession to unambiguously communicate between centers or countries. Additional barriers to communication include the differences in meaning between professions, (as discussed earlier), as well as cultural or linguistic limitations, with the tools not being translated into other languages, or relying on ideas or devices that are particular to one culture or another. Without clear definitions and a standardized approach, it is impossible for results in one place to be comprehended in another.
DEVELOPMENT OF THE STANDARDIZED APPROACH
Through a series of meetings regarding outcome measures, the group identified factors that contribute to the process of assessment and sought a means to create a standard, effective approach.
INTERNATIONAL CLASSIFICATION OF FUNCTIONING, DISABILITY AND HEALTH
The starting point to create a standard approach was to refer to the World Health Organization's International Classification of Functioning, Disability and Health (ICF)6 (Figure 1). The ICF model was intended to serve as a common language across health disciplines and was recommended as the framework for organizing outcome measure selection and goal setting.
ICF has three related components: body structures and functions, activity, and participation. However, according to the definitions of the components,6 prostheses are regarded as assistive devices and, hence, belong to the “environmental factors.” If it is interpreted this way, it limits the model for prosthetics and ignores the experience of many prosthesis users. Based on this experience, it is clear that for many, the prosthesis is genuinely an extension of the user's body, and so it becomes a replacement of “body function/body structure” not merely an assistive device as the ICF model would have it. The ULPOM group interpreted the model as follows:
- “The ICF model, as it relates to the provision of prosthetic devices and treatment/care of upper limb amputees, provides the following interpretation for the components.
- Body structures and functions relates to performance of the prosthesis and can capture measures of grasp, release, and abilities of the prosthesis.
- The activity component refers to the carrying out of tasks, which relates to the assessment of function with daily life activities with a prosthesis.
- Participation relates to prosthetic involvement in real-life situations, which is reflected in the assessment of the impact of the prosthesis function from a user's perspective in his/her own life situations.”
The ICF-related components reflect the aspects of the provision and the mode of treatment or care required. Critically, each component has its own set of particular requirements, and so the process of assessing progress or function within each component necessarily has to be different. The bonus of looking at it this way is that it becomes apparent that previous attempts to measure function without such insight are more likely to prove ineffective, as the measurement may not be focused on the correct component, or it may use the wrong method to measure it.
LIFE CYCLE OF PROSTHESIS DEVELOPMENT
A new prosthesis design begins with research and development before moving into the field through the clinic and into daily use. Any design or development needs to be monitored to ensure it fulfills the requirements that were set for it. To monitor the progress requires measurement or assessment (Figure 2, black lines).
The process falls into four major areas:
- Research—where the ideas for new devices and techniques are considered and created.
- Development—where the ideas are matched to the demands of production and field application.
- Clinical use—the customized fitting of the device to the person and training of the device's use.
- Home—the daily use of the device by the target population.
At each phase, the results of innovation or change needs to be measured and fed back to the preceding phases, thus changes can be iterative. Critically, the information to be captured is specific to the area within which it is being tested. For example, during research and development, important technical information about how a prosthetic component works need to be measured. Therefore, if a new powered hand is being created, prehension speed, grip force, etc must be measured, which would use a particular set of tools. The designers also need input from prosthetists about how the hand mechanism will fit within a complete prosthesis and how it is to be controlled, necessitating different measurement tools. Information on how it works within real-life settings and tasks will ultimately be required and fed back to the designer and prosthetist, likely by the occupational therapist (OT) or the user. Vitally, prosthesis users need to be able to convey how the hand functions in their own life situations, to the clinical and design teams. Means to evaluate each area are needed.
There are two main types of assessment tools currently in use: observational measures and self-rated measures. Observational measures may be timed tests or tests using a rating scale. Self-rated measures are generally questionnaires, which may be rated by the patient or by a parent or a guardian. Placed in this context, it is likely that it would not be possible to cover all the related components within the ICF model using only one assessment tool or one type of measure. For a complete analysis, knowledge of each component is necessary. A single measure can only serve some of the range so to cover all components will require multiple tests. The approach adopted by the ULPOM was to identify enough tests and their appropriate ICF-related components (Figure 2, light grey) to cover the entire range. Similarly, throughout the development lifecycle of a prosthesis, the different stakeholders (i.e., prosthetists, therapists, engineers, physicians, and prosthesis users) will have different levels of priority to the process just as different assessment techniques must dominate or operate.
Thus, after the development of a prosthesis, using an example from Ref. 1: The design engineer needs to know if a prosthetic hand design can open wide enough to grasp objects or close fast enough to be practical. So, functional/technical tests can be applied.
Once the device is more advanced, the assessment moves on to activity-based measures. Can the device pick up household objects? Can it hold on to them and move them about? This information is important to the engineer, but now the input from the clinical team becomes important as their insight into its long-term use becomes relevant. Early fittings in the clinic also need observational-based or self-rated measures of prosthetic use as the clinicians need to compare the device with others and to monitor the progress using a specific device.
Finally, the device moves on to the home, and the activities may revolve around tasks specific to the needs of that user. At this stage, the participation outcome measure may tell the clinician about the functional capabilities of the device or of the person's ability and something of the person's motivations. When the therapist wants to know more about how the user feels about their device and how it integrates into their daily lives, then the information will more likely to be obtained through a questionnaire.
Hence, it can be seen that at each stage a different tool is used to obtain information, and that each piece of information is important and provides different insight. Some techniques overlap into different ICF-related components, but the full picture will be clear only with multiple assessments. From this, it can be seen that different existing tools cover different areas of the continuum.
An important aspect of any test is its psychometric properties, i.e., the validity and reliability of the test. This ensures that the test is measuring what it is intended for, that the measures are repeatable and do not depend on who conducts the test or when or where it is performed.
Thus, what was proposed was a common, unified approach to outcome measures. The ULPOM group set out to identify the appropriate tools to meet the needs of the various stakeholders. It was appreciated that if such a set of tools that could cover all the ICF-related components was identified, it was likely that one test might cover two overlapping areas (such as the development and the clinical application sphere). However, it was unlikely that a single tool could cover the entire range.
BACKGROUND TO THE PROPOSED APPROACH
The process of achieving consensus by this group was a series of international workshops. Discussions began at the Myoelectric Controls (MEC) Symposium 2005, an international symposium of upper limb prosthetics. After that, a workshop was held in Trondheim Norway to further discuss the needs for outcome measurement in this population. The results of that workshop were brought to the 12th World Congress of the International Society for Prosthetics and Orthotics in August 2007 for discussion among an international, multidisciplinary group of professionals with background in the areas of research and development of prosthetic limbs as well as adult and pediatric upper limb amputee rehabilitation.
Discussions held at these conferences addressed concerns over the timing of using outcome measures in the rehabilitation process to ensure participation from the patient and the quest for a “gold standard” measure from which to compare other measures. More focused discussions came from the participants being divided into smaller groups according to their area of interest: pediatric, adult, and technical or research. Within the small groups, participants were asked to share details of outcome measures being used in their facilities, identify issues with specific measures being used, and identify gaps or needs in outcome measures related to their area of interest. These results fed into the aims and ideas of the ULPOM.
Although the three interest groups (pediatric, adult, and technical or research) have some specific measurement requirements, they are not mutually exclusive and do have some similar needs. It is important to know what to measure, that the particular measure does indeed measure what it says it should measure, and that it captures the information that is required. This needs then to be adopted by the appropriate profession, so that as many people as possible can communicate the ideas and the knowledge as clearly as possible.
GENERAL REQUIREMENTS FOR A MEASUREMENT TOOL
It was anticipated that one tool could not cover all ICF-related components, and, therefore, there would need to be a kit of tools that covered the different aspects of the range. Although individual tools were not yet identified, it was possible to draw up a list of requirements for what such a toolkit would look like.
It is crucial that unambiguous terminology be used that allows effective and precise cross-disciplinary communication.
Properties Required From a Measurement Tool
Consensus from meetings and workshops suggests that the ultimate measurement toolkit should posses a number of features. It should:
- Be easy to quantify
- Be moderately easy to use
- Relate to activities of daily living
- Be sensitive to change in control configuration
- Be sensitive to change in component design
- Be usable for high-level upper limb amputees
- Be usable for bilateral upper limb amputees
- Be able to track improvement and function over time
- Measure user satisfaction
- Accommodate users' different tolerance to complexity of control
From a practical perspective, the last point may require screening of subjects before enrolling them in device assessment.
Two fundamental types of questions need to be answered by the measure are as follows:
- Does the prosthesis work according to the specifications?
Does it offer something beneficial to the end user?
- Is it strictly able to/suitable for performing a certain function or activity?
- How well does it suit this purpose? (To be answered quantitatively on an objective scale.)
- How does it compare with other prostheses? (To be answered qualitatively and quantitatively.)
To answer this, there is a need to quantify the benefit for the patient at the activity and participation level.
Terminal Device Specifications
It is important to measure the terminal device specifications such as grasp patterns, width of opening, grip force, and effectiveness of grasp for particular activities. There are many methods of measuring these specifications; however, it is important that the same measures be used universally to allow comparison between prostheses and terminal devices.
Value of the Prosthesis
It is difficult for a clinician or a third-party payer to determine a person's prosthetic needs without knowing the value of the prosthetic device. The problem is that the value of a prosthesis or device to the wearer is a subjective measure and cannot be related to time worn or tasks performed with it. The person may use their prosthesis for a single specific task, (not daily or for a long time in any one day), but if that task is important to the person and the prosthesis makes the task possible, then the prosthesis's value to the user is great. Naturally, there is also a psychological component to its use. The usefulness or value to a person also changes over time depending on developmental stages, and experiences with types of prostheses, experience with a prosthetist or therapist, fit of a prosthesis, and external real-life experiences. Currently, many assessments equate function and wearing time to usefulness, which is clearly not an accurate metric.
Control of a Prosthesis
The group wished to measure the both abstract qualities of the device and its use in the field, so that function was seen as accuracy of control of a terminal device, dexterity, grasp, and speed. It was also desired that a measure be sensitive enough to differentiate between control schemes of the same device and show changes in ability to control the prosthesis over time.
Function also was seen to be concerning a person's potential to use and benefit from a particular type of prosthesis or prosthetic component, and the expected level of independence the person could achieve. This may help with anticipating the amount of caregiver support a person may require.
Evidence to Support Clinical Judgment
There is a need to provide evidence to support clinical judgment in recommending particular devices or prosthetic components.
Effectiveness of Therapy/Treatment
An outcome measure must demonstrate the effectiveness of treatment with regards to the amount and quality of therapy received by the patient. It should also be able to show differences in function related to the timing of initiating prosthetic fitting after amputation or the progress being made during therapy.
Quality of Life
It is important to take into account a patient's perspective and preferences as they relate to quality of life and function.
Clinical experience suggests that overuse issues seem to be more prevalent in the adult amputee population, but there is far less in the literature concerning this critical matter, perhaps because it is hard to measure. It is clearly worthwhile to investigate the incidence of overuse syndromes in the adult amputee population for all levels of use, from the non wearer to the full-time wearers of a prosthesis.
REQUIREMENTS SPECIFIC TO THE PAEDIATRIC POPULATION
Satisfaction of Child/Parent
A requirement specific to this population was the need to measure satisfaction of the child and the parents with a prosthesis or prosthetic services. The parents' satisfaction is felt to have a great impact on the child's acceptance of a prosthesis. The child and family's needs also change over time, as the child grows and reaches different developmental stages. This too needs to be measured.
Requirements Specific to the Adult Population
Simultaneous Versus Sequential Control
It would be beneficial to researchers to be able to evaluate the use of simultaneous control of various components versus seamless sequential control. This is hard to achieve at present.
It is desirable to be able to determine the appropriate prosthetic components or control strategy for a particular person based on the person's cognitive or mental status.
REQUIREMENTS SPECIFIC TO THE RESEARCH COMMUNITY
It is important to be able to define a problem in terms of:
- What population the device is to be made for or used by?
- What specific aspects of the device need to be improved, or what functions need to be offered?
These aspects need to be measured clearly amongst the amputee population. Needs, as expressed by end users, should be analyzed and perhaps rephrased because the consumers might not be in a position to express their needs in a sufficiently precise way.
Upper-limb prosthetics development is a translational research. Specific hypotheses should always be formulated before selecting and using a measure, and any measures adopted must answer questions pertinent to the hypotheses.
WORK OF THE UPPER LIMB PROSTHETIC OUTCOME MEASURES GROUP
After the meeting during the 12th World Congress of the International Society for Prosthetics and Orthotics in Vancouver in August 2007, two things occurred. First, a web-based discussion forum was established on the ISPO website, which was replaced eventually by a Google Group discussion forum, initiated and moderated by Shawn Swanson.
At the same time, volunteers from the meeting were requested to form a smaller working group, representing clinicians and researchers in the areas of adult and pediatric amputee rehabilitation as well as research and development of prosthetic components. The volunteers were expected to meet via teleconference and e-mail, which was again organized and moderated by Shawn Swanson. This group was known as the ULPOM.
Its intention was to create a “toolkit” of validated procedures for the entire development cycle and promote it's use within the appropriate professions.7,8 This toolkit was not aimed at prescribing a single method or a single assessment tool for each ICF-related component, or to develop new tests to perform these assessments, but rather to identify a collection of existing tests that could be used in upper limb prosthetics, and to identify any remaining gaps that need to be filled.
The initial activity of the ULPOM working group (from February 2008) was to develop a list of all the assessments of potential use and then critically assess each of them to see if they were appropriate for the ULPOM application. Thirty-four assessments covering areas of, hand function, overall function related to the upper limb, goal setting, quality of life, and client satisfaction were identified. They were sorted and categorized according to the most appropriate ICF-related component, function, activity or participation, and the type of measure: observer-rated or self-rated.
The ULPOM Process
The measures included for analysis are as follows:
- The Assisting Hand Assessment
- Child Amputee Prosthetics Project-Functional Status Inventory (CAPP-FSI)
- CAPP-FSI toddler
- CAPP-FSI preschool
- Prosthetic Upper Limb Functional Index
- Assessment of Capacity for Myoelectric Control
- Arm Motor Ability Test
- Assessment of Motor and Process Skills
- Pediatric Evaluation of Disability Inventory
- Quality of Upper Limb Skills Test
- School Function Assessment
- University of New Brunswick (UNB) Test of Prosthetic Function
- Unilateral Below-Elbow Test
- Activity scale for kids
Hand Function Tests
- Southampton Hand Assessment Procedure (SHAP)
- Sollerman Hand Function Test
- Box and Blocks
- Jebsen Standardized Test of Hand Function
- Purdue Pegboard
- Canadian Occupational Performance Measure
- Goal Attainment Scaling
Satisfaction/Quality of Life
- Assessment of Life Habits (LIFE-H)
- Assessment of LIFE-H—Children Short Form (LIFE-H for Children)
- Child Amputee Prosthetics Project-Prosthesis Satisfaction Inventory
- Child Health Questionnaire
- Disability of the Arm, Shoulder, and Hand Outcome Measure
- Orthotics and Prosthetics User Survey-Upper Limb Functional Status Module
- Pediatric Outcomes Data Collection Instrument
- Trinity Amputation And Prosthetic Experience Scales
- World Health Organization Quality of Life (WHOQOL-BREF)
The thirty-four measures were analyzed according to a checklist developed by Liselotte Hermansson (Appendices A and B) and discussed by the group to determine recommendations. The checklist included information about the administration of the measure, the intended purpose and target group, and the psychometric properties. The group recommendations were one of three things: recommend, consider, or exclude (see Appendix A for definitions).
Following this lengthy analysis, it was determined that several tests were appropriate for a toolkit, and that there are gaps within the range that need addressing. In the case that tests have been validated for other groups, the anticipated effort that would be required to extend its remit to prosthetics was discussed. This data fed into discussions at the State of the Science Conference (SSC) and form part of the overall conclusions of the SSC.
The analysis was completed on all of the measures. Some of the measures were not familiar to members of the ULPOM group, and, in those circumstances, analysis was based on literature found to support their use in upper limb prosthetics. Recommendations were based on the perceived clinical utility when compared with similar measures. Therefore, some measures were excluded because their validity had not been tested for use with people with upper limb loss, whereas others were excluded because they were not deemed to be clinically useful for this population. Some of the assessments included in the original list were intended for neurologically impaired people, and therefore were not appropriate to be used with people with limb loss. Other assessments were very general in nature and had a perceived ceiling effect when applied to an upper limb loss population. If it was deemed by the group that a particular test would require a substantial amount of work to become valid and useful, then that test was excluded. A test was also excluded if there were other tests that provided that same information and would require less work to be listed as a recommended test.
The final recommendations were categorized by ICF- related component and type of measure and were noted to be intended for pediatric or adult populations. Table 1 shows the recommendations from the ULPOM group. There are appropriate measures for adults and pediatrics, but there is a perceived need for more appropriate functional assessments for adults.
The results of the ULPOM analyses of the outcome measures were presented at a workshop at the MEC Symposium held in Fredericton, New Brunswick, August 2008. There was discussion about the inclusion or exclusion of particular tests. The feedback from MEC was brought back to the ULPOM working group, who then looked more closely at the tests in question. As a result, other tests were added to the list for consideration.
Those tests included the following:
- Michigan Hand Questionnaire
- Functional Independence Measure
These tests have been discussed by the group but have not been fully analyzed; therefore, they are not included on the lists in this article. We are aware that some of these tests are being used in some facilities with people with upper limb loss, and further analysis is warranted.
There are a sizable number of the reviewed tests that were perceived to be clinically useful by the group but require further validation studies to be acceptable for use in upper limb prosthetics. This would be the most viable solution to fill a particular need or gap in the assessment process. However, such a task could only be taken on by a facility with a sufficiently large number of appropriate clients or by a group of facilities who could combine data. Table 2 gives details of the actions required for the tests in the consider category.
HIERARCHY, COMPLETENESS AND TRACEABILITY
One more radical aspect of measurement proposed during the workshop process was that measures should have a hierarchical structure. That is, the measurement functions should be integrated into a single chain from research to the home, with each element of a later measure including elements from the preceding measures. If constructed this way, a chain of outcome measures would be hierarchical in structure and as a result traceable in content and implications.
Such a measurement chain would provide tools for assessing all relevant aspects of a device. Complex aspects can then be said to be composed of simpler aspects; hence, traceability can be identified or required two ways.
Each simpler aspect (function, activity, and participative role) of a device that is subject to assessment at one stage must play a role in some aspects that are in focus at the next (i.e., more complex) stage, otherwise it could be argued that the simpler aspect is not necessary and thus does not need to be assessed.
Each simpler element of the complex aspect that is in focus at one stage must be assessable at a preceding or more elementary stage. Otherwise it could be argued that the tool chain is incomplete and, strictly speaking, the device is not ready for the assessment at a more complex level.
This issue relates to the completeness of the measurement chain only and does not imply that all parts and aspects of the chain are required to be used in every development process. It is also appreciated that complete traceability is a theoretically ideal goal that might not be achievable in practical life. This ultimate goal is beyond the aims of the ULPOM group at this time but represents an interesting avenue for subsequent research.
FUTURE OF THE APPROACH
Having established a toolkit and identified the gaps within the exiting tools the next phase for the ULPOM is to:
- Promote the toolkit to other members of the profession, through conferences and professional bodies.
- Encourage the proper use of the tools to perform assessment and to communicate the results to the profession.
- Establish groups and collaboration to perform the required testing and analysis to bring the tests in the “consider” category to a level where they can be approved for use. This also includes performing some of the validation tests on approved tools to confirm their properties.
- Publish, and encourage others to publish, results of tests to make the subject more visible to the profession.
Many of these steps require a level of funding to allow them to take place, but the effort will be worth it as it will enable professionals to have confidence in what they say and what others report to them.
The authors thank those who have contributed their time and insights into the process described here, especially the non-ULPOM members of the Trondheim workshop, Tordis Magne, Kerstin Caine-Winterberger, Stewe Jönsson, Kathy Stubblefield, Laura Miller, Sigbjørn Rønning, Stian Salomonsen, and the other members of the ULPOM working group, Kristin Gulick and MacJulian Lang. They also thank Virginia Wright and Helen Lindner, MSc, Reg OT, PhD student for their input. They also thank their sponsors Norwegian Research Council, the Norwegian University of Science and Technology (NTNU) and Foreign Affairs and International Trade, Canada, National Science and Engineering Canada, Canada Institutes of Health Research, The Canada Chairs Program, and New Brunswick Innovation Foundation.
1. Hill W, Stavdahl Ø, Hermansson LN, et al. Functional outcomes in the WHO-ICF model: establishment of the Upper Limb Prosthetic Outcome Measures Group. J Prosthet Orthot
2. Light CM, Chappell PH, Kyberd PJ, Ellis BS. Critical review of functionality assessment
in natural and prosthetic hands
. Br J Occup Ther
3. Bazzini G, Orlandini D, Moscato D, Nicita D. I Test Di Funzionalità Della Mano In Amputati Di Aart Superiore Con Protesi. Giornale Italiano Di Medician Del Lavoro Ed Ergonomia
, 2007;29:203–209; ISSN1592-7830.
4. Buffart LM, Roebroeck ME, Pesch-Batenburg JM, et al. Assessment
of arm/hand functioning in children with a congenital transverse or longitudinal reduction deficiency of the upper limb. Disabil Rehabil
5. Metcalf C, Adams J, Burridge J, et al. A review of clinical upper limb assessments within the framework of the WHO ICF. Musculoskeletal Care
6. World Health Organization. Towards a Common Language for Functionary, Disability and Health: ICF Beginner's Guide (WHO/EIP/GPE/CAS/01.3)
. WHO: Geneva; 2002.
7. Demers L, Desrosiers J, Ska B, et al. Assembling a toolkit to measure geriatric rehabilitation outcomes. Am J Phys Med Rehabil
8. Child Health Care Quality Toolbox. Fact Sheet
. Rockville, MD: Agency for Healthcare Research and Quality; 2004. AHRQ publication No. 01-0025. Available T: http://www.ahrq.gov/news/chtoolfact.htm
APPENDIX A CHECKLIST FOR EVALUATION OF OUTCOME MEASURES
Scientifically sound, good quality tools can help clinicians improve the rehabilitation of persons with upper limb prostheses by identifying the areas in need of training or adjustments. By using valid and reliable instruments, the clinicians will have tools to measure true improvements in their clients function, activity, or sense of participation. To critically review and evaluate the tests that we use are of outmost importance. As stated in standards for educational and psychological testing,1 appropriate test use and sound interpretation of test scores remain primarily the responsibility of the test user.
“Prior to the adoption and use of a published test, the test user should study and evaluate the materials provided by the test developer. Of particular importance are those that summarize the test's purpose, specify the procedures for test administration, define the intended populations of test takers, and discuss the score interpretations for which validity and reliability are available” (Standard 11.1).1
However, many clinicians find it difficult to evaluate the quality of the instruments and consequently several tests with debatable properties are in use. Hence, there is a need for a “standards” for outcome measures in upper limb prosthetics.
For the Upper Limb Prosthetic Outcome Measures (ULPOM) group to make critical reviews of outcome measures in use in upper limb prosthetics this checklist was constructed. This checklist was based on issues from the above-mentioned Standards for educational and psychological testing2 and other aspects on the use of tests in research and clinic, e.g., Law and MacDermid.3 The checklist was refined through discussion (Figure 3). Guidelines were formulated to help the reviewer, one addressed the psychometric evidence, and the other on the focus and its clinical utility.
What Was Included and Why?
There are three main aspects of a test that need to be covered by a review: the purpose, the clinical utility, and the quality of the measures generated by the test. The process included a search for these aspects in the literature of the tests under review, and these details are included in the checklist.
PURPOSE OF THE TEST
The focus of the test should be described by the originator; the original authors' stated purpose for the test was included. Another way of describing the focus of the test is by placing it under one of the International Classification of Functioning, Disability, and Health (ICF) components: 1) body function/body structure, 2) activity, or 3) participation.4 This has also been recommended or used by others.5,6 After these recommendations, we decided to, in addition to the original author's purpose, make an attempt to classify the tests according to the ICF components.
Another important aspect of the purpose of a test is the target group for the instrument, its scope, or domain.7 A test can be generic, i.e., applicable to any population or it can be diagnosis specific and even, in cases of upper limb amputation, specific for a certain level of amputation. Accordingly, this was included in the checklist.
CLINICAL UTILITY OF THE TEST
Outcome measures for upper limb prosthetics can be either observational based, timed, or self-reported by means of interviews or questionnaires. For this project, description of the test administration, the intended settings, and the structure, i.e., number of items and response options, of the tests were added to the checklist. The ease of administration, i.e., the expected time for completion or scoring, of the test as well as information regarding the access to use of the test, and the cost, are important information for clinicians and were, thus, searched for and added to the checklist.
PSYCHOMETRIC PROPERTIES OF THE TEST
The quality of a test determines the quality of the measure, i.e., the outcome from the test. This is expressed usually by validity and reliability.
Validity can be expressed in many ways. As suggested by Bond and Fox,8 construct validity is a comprehensive concept that includes content validity, face validity, and concurrent validity. Another implementation of the concept validity is to use several lines of validity as accumulation of evidence to provide information regarding the intended interpretation of the tests' scores for a specific purpose or population.2 Hence, the ULPOM group decided to describe the psychometric evidence available from the literature for the test in each specific checklist and leave the interpretation of the evidence to the reader. In the same way as with the evidence of validity, the reported evidences for reliability are presented in the checklists.
The guidelines used for the review can be used when deciding which instrument to choose. For example, what are the reported psychometric qualities of the instrument for our target group? Is it demonstrated that the items (questions or “observational points”) measure what the instrument is intended for, and fits this group of patients (not only have been used for …)? Can you rely on the results generated from the instrument?
Instead of providing a quantitative measure on the quality of the tests, the ULPOM group decided to add a recommendation for use to the checklist. To do so, the members of the group discussed the focus of the tests and based upon their accumulated experience (face-validity), sometimes ascertained by judgments from other clinicians, a decision regarding potential use (clinical or research) of the tests were decided.
The categories were as follows: recommend—valid for upper limb prostheses users; consider—could be/potential valid for upper limb prostheses users but lacks psychometric merits; exclude—not valid for upper limb prostheses users.
Validity is an accumulation of evidence. In the checklist, the ULPOM group reports the published results on validity and reliability for each instrument without rating the level of validity or reliability for the tests. To increase the clinical use of the review, a critical appraisal of each test would have been useful. For example, Law and MacDermid5,9 recommend the rating form used at CANCHILD Centre for Disability Research at Macmaster University, Ontario, Canada.
We used the International Classification of Function (ICF)4 to categorize the intended focus of the tests. However, the mixed focus of the tests made it difficult to select one component only per test. Consequently, in many tests, we had difficulty deciding in which component the test should be categorized. This wide focus of tests has been shown in another study of upper limb assessments.6
Another way of using the ICF components for evaluation of tests is by linking each test item to an ICF domain.10 By this content comparison,11,12 it is possible to see which domains are covered in a specific instrument and, thus, choose the most appropriate instrument for the intended purpose.13
1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Part 11, The responsibilities of Test Users. Standards for Educational and Psychological Testing
. Washington: American Educational Research Association; 2004:111–118.
2. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. I. Test construction, Evaluation, and Documentation. Standards for Educational and Psychological Testing
. Washington: American Educational Research Association; 2004:7–70.
3. Law M, MacDermid J. Evidence-Based Rehabilitation: A Guide to Practice
. Thorofare, NJ: SLACK Inc; 2008.
4. World Health Organization. Towards a common language for Functionary, Disability and Health: ICF Beginner's Guide (WHO/EIP/GPE/CAS/01.3)
. WHO: Geneva; 2002.
5. Law M. Appendix E: Outcome measures rating form. In: Law M, MacDermid J, eds. Evidence-Based Rehabilitation: A Guide to Practice
. 2nd ed. Thorofare, NJ: SLACK Inc; 2008:367–379.
6. Metcalf C, Adams J, Burridge J, Chappell P. A review of clinical upper limb assessments within the framework of the WHO ICF. Musculoskelet Care
7. MacDermid J, Michlovitz S. Incorporating outcomes measures into evidence-based practice. In: Law M, MacDermid J, eds. Evidence-Based Rehabilitation: A Guide to Practice
. Thorofare, NJ: SLACK Inc; 2008:63–94.
8. Bond TG, Fox CM. Applying the Rasch Model: Fundamental Measurement in the Human Sciences
. Mahwah, NJ: Lawrence Erlbaum Associates; 2001.
9. Law M. Appendix F: Outcome measures rating form guidelines. In: Law M, MacDermid J, eds. Evidence-Based Rehabilitation: A Guide to Practice
. 2nd ed. Thorofare, NJ: SLACK Inc; 2008: 381–386.
10. Cieza A, Brockow T, Ewert T, et al. Linking health-status measurements to the International Classification of Functioning, Disability and Health. J Rehabil Med
11. Stucki A, Cieza A, Schuurmans MM, et al. Content comparison of health-related quality of life instruments for obstructive sleep apnea. Sleep Med
12. Tschiesner U, Rogers SN, Harréus U, et al. Content comparison of quality of life questionnaires used in head and neck cancer based on the international classification of functioning, disability and health: a systematic review. Eur Arch Otorhinolaryngol
13. Stucki G, Cieza A. The International Classification of Functioning, Disability and Health (ICF) core sets for rheumatoid arthritis: a way to specify functioning. Ann Rheum Dis
APPENDIX B ULPOM REFERENCE LIST
Penta M, Thonnard J, Tesio L. ABILHAND: a Rasch-built measure of manual ability. Arch Phys Med Rehabil 1998;79:1038–1042.
Penta M, Tesio L, Arnould C, et al. The ABILHAND questionnaire as a measure of manual ability in chronic stroke patients: rasch-based validation and relationship to upper limb impairment. Stroke 2001;32:1627–1634.
Arnould C, Penta M, Renders A, Thonnard J-L. ABILHAND-Kids: a measure of manual ability in children with cerebral palsy. Neurology 2004;63:1045–1052.
ACTIVITIES SCALE FOR KIDS
Nancy Y, Karen Y, Ivan WJ, et al. The role of children in reporting their physical disability. Arch Phys Med Rehabil 1995;76:913–918.
Mary L, Carolyn B, Winnie D. Measuring Occupational Performance in Basic Activities of Daily Living Chapter in Measuring Occupational Performance: Supporting Best Practice in Occupational Therapy. Thorofare, NJ: SLACK Inc.; 2001:125–126.
THE ARM MOTOR ABILITY TASK
Bruno K, Annett K, Herta F, et al. The arm motor ability task: reliability, validity and sensitivity to change of an instrument for assessing disabilities. Arch Phys Med Rehabil 1997;78:615–620.
ASSESSMENT OF CAPACITY FOR MYOELECTRIC CONTROL
Hermansson L, Fisher A, Bernspång B, Eliasson AC. Assessment of capacity for myoelectric control: a new rasch-built measure of prosthetic hand control. J Rehabil Med 2005;37:166–171.
Hermansson L, Bodin L, Eliasson A-C. Intra- and inter-rater reliability of the assessment of capacity for myoelectric control. J Rehabil Med 2006;38:118–123.
Lindner H, Linacre M, Hermansson NL. The assessment of capacity for myoelectric control: evaluation of the construct and the rating scale. J Rehabil Med. In press.
ASSESSMENT OF LIFE HABITS
Noreau L, Fougeyrollas P, Vincent C. The LIFE-H: assessment of the quality of social participation. Technol Disabil 2002;14:113–118.
ASSESSMENT OF LIFE HABITS: CHILDREN SHORT FORM
Noreau L, Lepage C, Boissiere L, et al. Measuring participation in children with disabilities using the assessment of life habits. Dev Med Child Neurol 2007;49:666–671.
ASSESSMENT OF MOTOR AND PROCESS SKILLS
Donnelly C, Carswell A. Individualized outcome measures: a review of the literature. Can J Occup Ther 2002;69:84–94.
ASSISTING HAND ASSESSMENT
Krumlinde-Sundholm L, Eliasson A-C. Development of the assisting hand assessment: a rasch-built measure intended for children with unilateral upper limb impairments. Scand J Occup Ther 2003;10:16–26.
Krumlinde-Sundholm L, Holmefur M, Kottorp A, Eliasson AC. The assisting hand assessment: current evidence of validity, reliability, and responsiveness to change. Dev Med Child Neurol 2007;49:259–264.
Holmefur M, Krumlinde-Sundholm L, Eliasson AC. Interrater and intrarater reliability of the Assisting Hand Assessment. Am J Occup Ther 2007;61:79–84.
BOX AND BLOCKS
Test description. Available at: http://healthsciences.qmuc.ac.uk/labweb/Equipment/Box%20and%20Block%20test.htm.
Mathiowetz V, Volland G, Kashman N, Weber K. Adult norms for the Box and Block Test of manual dexterity. Am J Occup Ther 1985;39:386–391.
CHILD AMPUTEE PROSTHETICS PROJECT-FUNCTIONAL STATUS INVENTORY
Pruitt SD, Varni JW, Setoguchi Y. Functional status in children with limb deficiency: development and initial validation of an outcome measure. Arch Phys Med Rehabil 1996;77:1233–1238.
CHILD AMPUTEE PROSTHETICS PROJECT-FUNCTIONAL STATUS INVENTORY FOR PRESCHOOL CHILDREN
Pruitt SD, Varni JW, Seid M, Setoguchi Y. Functional status in limb deficiency: development of an outcome measure for preschool children. Arch Phys Med Rehabil 1998;79:405–411.
CHILD AMPUTEE PROSTHETICS PROJECT-FUNCTIONAL STATUS INVENTORY FOR TODDLERS
Pruitt SD, Seid M, Varni JW, Setoguchi Y. Toddlers with limb deficiency: conceptual basis and initial application of a functional status outcome measure. Arch Phys Med Rehabil 1999;80:819–824.
CHILD AMPUTEE PROSTHETICS PROJECT PROSTHESIS SATISFACTION INVENTORY
Sheri PD, James VW, Michael S, Yoshio S. Prosthesis satisfaction outcome measurement in pediatric limb deficiency. Arch Phys Med Rehabil 1997;78:750–754.
CHILD HEALTH QUESTIONNAIRE
Mary L, Carolyn B, Winnie D. Measuring occupational performance in basic activities of daily living. In: Measuring Occupational Performance: Supporting Best Practice in Occupational Therapy. Thorofare, NJ: SLACK Inc.; 2001:134–135.
Landgraf JM, Maunsell E, Speechley KN, et al. Canadian-French, German and UK versions of the Child Health Questionnaire: methodology and preliminary item scaling results. Qual Life Res 1998;7:433–445.
Waters E, Wright M, Wake M, et al. Measuring the health and well-being of children and adolescents: a preliminary comparative evaluation of the child health questionnaire. Ambul Child Health 1999;5:131–141.
CANADIAN OCCUPATIONAL PERFORMANCE MEASURE
Donnelly C, Carswell A. Individualized outcome measures: a review of the literature, Can J Occup Ther 2002;69:84–94.
Law M, Polatajko H,Carswell A, McColl M. The Canadian Occupational Performance Measure. 3rd ed. Ottawa, ON: CAOT Publications ACE; 1998.
DISABILITY OF ARM, SHOULDER, AND HAND MEASURE
Hudak P, Amadio P, Bobarier C, UECG. Development of an upper extremity outcome measure: the DASH (disabilities of the arm, shoulder, and hand). Am J Industrial Med 1996;29:602–608.
Beaton D, Katz J, Fossel A, et al. Measuring the whole or the parts? Validity, reliability, and responsiveness of the Disabilities of the Arm, Shoulder and Hand outcome measure in different regions of the upper extremity. J Hand Ther 2001:128–141.
Bullinger M, Schmidt S, Petersen C; DISABKIDS Group. Assessing quality of life of children with chronic health conditions and disabilities: a European approach. Int J Rehabil Res 2002;25:197–206.
Petersen C, Schmidt S, Bullinger M; DISABKIDS Group. Brief report: development and pilot testing of a coping questionnaire for children and adolescents with chronic health conditions. J Pediatr Psychol 2004;29:635–640.
Petersen C, Schmidt S, Power M, Bullinger M; DISABKIDS Group. Development and pilot-testing of a health-related quality of life chronic generic module for children and adolescents with chronic health conditions: a European perspective. Qual Life Res 2005;14:1065–1077.
Schmidt S, Debensason D, Muhlan H, et al; European DISABKIDS Group. The DISABKIDS generic quality of life instrument showed cross-cultural validity. J Clin Epidemiol, 2006;59:587–598.
Simeoni MC, Schmidt S, Muehlan H, et al; The DISABKIDS Group. Field testing of a European quality of life instrument for children and adolescents with chronic conditions: the 37-item DISABKIDS Chronic Generic Module. Qual Life Res 2007;16:881–893.
GOAL ATTAINMENT SCALING
Donnelly C, Carswell A. Individualized outcome measures: a review of the literature. Can J Occup Ther 2002;69:84–94.
JEBSEN TAYLOR TEST OF HAND FUNCTION
Jebsen R, Taylor N, Trieschmann R, et al. An objective and standardized test of hand function. Arch Phys Med Rehabil 1969:311–319.
Lynch K, Bridle M. Validity of the Jebsen-Talyor hand function test in predicting activities of daily living. Occ Ther J Res 1989;9:316–319.
ORTHOTICS AND PROSTHETICS USER SURVEY
Actual assessment copyright 2001 Northwestern University.
Burger H, Franchignoni F, Heinenmann A, et al. Validation of the orthotics and prosthetics user survey upper extremity functional status module in people with unilateral upper limb amputation. J Rehabil Med 2008;40:393–399.
Heinemann AW, Bode RK, O'Reilly CO. Development and measurement properties of the Orthotics and Prostheics Users' Survey (OPUS): a comprehensive se of clinical outcome instruments. Prosthet Orthot Int 2003;27:191–206.
Heinemann AW, Gershon R, Fisher W. Development and application of the orthotics and prosthetics user survey: applications and opportunities for health care quality improvement. J Prosthet Orthot 2006;18:80–85.
PEDIATRIC EVALUATION OF DISABILITY INVENTORY
Laurien BM, Marij RE, Josemiek P-R, et al. Assessment of arm/hand functioning in children with a congenital transverse or longitudinal reduction deficiency of the upper limb. Disabil Rehabil 2006;28:85–95.
Haley SM, Coster W, Ludlow L, et al. Pediatric Evaluation of Disability Inventory (PEDI): Version 1.0. Boston, Mass: New England Medical Centre Hospitals Inc; 1992.
Mary L, Carolyn B, Winnie D. Measuring occupational performance in basic activities of daily living. Measuring Occupational Performance: Supporting Best Practice in Occupational Therapy. Thorofare, NJ: SLACK Inc.; 2001:149–151.
PEDIATRIC QUALITY OF LIFE INVENTORY (PedsQL)
James V, Tasha B, Michael S, Douglas S. The PedsQL 4.0 as a pediatric population health measure: feasibility, reliability and validity. Ambul Pediatr 2003;3:329–341.
James M, Bagley A, Brasington K, et al. Impact of prostheses on function and quality of life for children with unilateral congenital below-the-elbow deficiency. J Bone Joint Surg 2006;88:2356–2365.
PEDIATRIC OUTCOMES DATA COLLECTION INSTRUMENT/POSNA
Lawren D, Matthew L, Anne F, Michael G. The POSNA pediatric musculoskeletal functional health questionnaire: report on reliability, validity and sensitivity to change. J Pediatr Orthop 1998;18:561–571.
Buffart LM, Roebroeck ME, Pesch-Batenburg JM, et al. Assessment of arm/hand functioning in children with a congenital transverse or longitudinal reduction deficiency of the upper limb. Disabil Rehabil 2006;28:85–95.
James MA, Bagley AM, Brasington K, et al. Impact of prostheses on function and quality of life for children with unilateral congenital below-the-elbow deficiency. J Bone Joint Surg Am 2006;88:2356–2365.
Richard H, Elroy S. The Pediatricorthopaedic Society of North America pediatric orthopaedic functional health questionnaire: an analysis of normals. J Pediatr Orthop 2001;21:619–621.
PROSTHETIC UPPER EXTREMITY FUNCTIONAL INDEX
Wright FV, Hubbard S, Jutai J, Naumann S. The Prosthetic Upper Extremity Functional Index: development and reliability testing of a new functional status questionnaire for children who use upper extremity prostheses. J Hand Ther 2001;14:91–104.
Wright F, Hubbard S, Naumann S, Jutai J. Evaluation of the validity of the prosthetic upper extremity functional index for children. Arch Phys Med Rehabil 2003;84:518–527.
Gallus J, Mathiowetz V. Test-retest reliability of the Purdue Pegboard for persons with multiple sclerosis. Am J Occup Ther 2003;57:108–111.
Buddenberg L, Davis C. Test-retest reliability of the Purdue Pegboard Test. Am J Occup Ther 2000;54:555–558.
THE QUALITY OF UPPER EXTREMITY SKILLS TEST
Laurien BM, Marij RE, Josemiek P-B, et al. Assessment of arm/hand functioning in children with a congenital transverse or longitudinal reduction deficiency of the upper limb. Disabil Rehabil 2006;28:85–95.
DeMatteo C, Law M, Russell D, et al. The reliability and validity of the quality of upper extremity skills test. Phys Occup Ther Pediatr 1993;13:1–18.
SCHOOL FUNCTION ASSESSMENT
Coster W, Beeney T, Haltiwinger J, Haley SM. School Function Assessment. San Antonio, TX: The Psychological Corporation/Therapy Skill Builders; 1998.
Law M, Baum C, Dunn W. Occupational performance: measuring the perspectives of others. In: Measuring Occupational Performance: Supporting Best Practice in Occupational Therapy. Thorofare, NJ: SLACK Inc.; 2001:80–81.
SOUTHAMPTON HAND ASSESSMENT PROCEDURE
Light CM, Chappell PH, Kyberd PJ, Ellis BS. Critical review of functionality assessment in natural and prosthetic hands. Br J Occup Ther, 1999;62:7–12.
Light CM, Chappell PH, Kyberd PJ. Establishing a standardized clinical assessment tool of pathologic and prosthetic hand function: normative data, reliability, and validity. Arch Phys Med Rehabil 2002;83:776–783.
Kyberd PJ, Murgia A, Gasson M, et al. Barnhill case studies to demonstrate the range of application of the Southampton hand assessment procedure. Br J Occup Ther. In Press.
SOLLERMAN HAND FUNCTION TEST
Sollerman C. Grip function of the hand: analysis, evaluation and a new test method. Doctoral Thesis. Göteborg, Sweden: University of Gothenburg; 1980.
TRINITY AMPUTATION AND PROSTHESIS EXPERIENCE SCALES
Desmond DM, MacLachlan L. Factor structure of the trinity amputation and prosthesis experience scales (TAPES) with individuals with acquired upper limb amputations. Am J Phys Med Rehabil 2005;84:506–513.
Gallagher P, MacLachlan L. Development and psychometric evaluation of the trinity amputation and prosthesis experience scales (TAPES). Rehabil Psychol 2000;45:130–154.
Actual assessment is freely downloaded. Produced at the Department of Psychology, Trinity College, Dublin. Available at: http://www.tcd.ie/Psychoprosthetics/pages/tapes.html.
UNILATERAL BELOW ELBOW TEST
Bagley AM, Molitor F, Wagner LV, et al. The unilateral below elbow test: a function test for children with unilateral congenital below elbow deficiency. Dev Med Child Neurol 2006;48:569–575.
UNIVERSITY OF NEW BRUNSWICK TEST OF PROSTHETIC FUNCTION
Sanderson ER, Scott RN. UNB Test of Prosthetics Function Manual. Fredericton, NB: Bioengineering Department, University of New Brunswick; 1985.
WORLD HEALTH ORGANIZATION′S QUALITY OF LIFE (WHOQOL–BREF)
Skevington SM, Lotfy M, O'Connell KA. The World Health Organization's WHOQOL-BREF quality of life assessment: psychometric properties and results of the international field trial. A report from the WHOQOL group. Qual Life Res 2004;13:299–310.
The WHOQOL Group. Development of the World Health Organization WHOQOL-BREF quality of life assessment. Psychol Med 1998;28:551–558.