PERSPECTIVES AND HISTORY
After the presentation of Wright's evidence-based review (EBR), a history of the outcomes in upper limb pediatric prosthetics, and the work by the Upper Limb Prosthetic Outcome Measures (ULPOM) group, it was felt that conference questions 1, 2, and 3 (as listed in the Introduction to the State of Science [SSC]) were addressed. To understand better what various stakeholders want from an outcome measure, presentations were given from the following perspectives (Stubblefield, occupational therapy/research; Heckathorne, research; Lake and Uellendahl, prosthetics; Esquenazi, physician and prosthetic device user).
The uniqueness and individual needs of the upper limb population was an overarching theme of all of the presentations. When considering any kind of outcome measure, and especially given this uniqueness, it is imperative to formulate the question to be asked before an outcome measure is selected. There was no one “gold standard” outcome measure identified that covered all related components and would work in all fields of application (i.e., research or patient care). During a review of the research literature, Heckathorne found that many people developing new prosthetic systems also designed a new measure. He noted that many of the measures would be more appropriately termed “assessments” (e.g., Fitt's law test, time to complete test, motion analysis, or virtual reality) rather than outcome measures. During this discussion, the various types of measures were reviewed: discriminative, predictive, and outcome. It was noted that research may fall more toward the discriminative or assessment side of the spectrum. Some of the questions discussed later (e.g., predicting usage prior to fitting) fall toward the predictive side of the spectrum.
Heckathorne suggested that instead of creating new outcome measures (also referred to as “tools” in this text) to investigate a new product or device for the upper limb amputation population and validate it, it may be more effective and efficient to take the measures recommended by the group and break them down into subsections that would be applicable in the research setting. Although the use of subsections of a measure compromises its validity, as a project progresses toward a clinical research study, the tasks and measurements used early in the research and development can then be integrated as all sections of the outcome measure are able to be used. Stubblefield also lamented the applicability or suitability of available measures in the research setting, commenting that often the time to complete a task is measured, but that might not be the best way to measure the quality of the control system and may not address or answer the question being asked. Also, the ability to control a prosthetic device will change in the context (e.g., opening a bag of cookies might be a lot more motivating than opening a bag of laundry). She reinforced the benefit of using the same tool in a research and clinic setting, so that it provides continuous useful information to all involved (therapist, engineer, and prosthetist). Also, as pointed out by Heckathorne, it is important to identify the research subject and the reference: a person with an upper limb amputation compared with a nonamputee, other amputees, or themselves. In some evaluation situations, the subject may be a person without an upper limb amputation simulating an amputation.
From the clinician perspective (Uellendahl, Lake, and Esquenazi), the uniqueness of the upper limb patient population was reviewed. For example, Uellendahl pointed out that few outcome measures address passive prosthetic use (for adults) or the impact of the device appearance. In addition, there is a benefit in knowing what individuals with different limb amputation levels choose to wear over time (from immediately postoperatively to many years later), and the device choice may change as the patient's perspective and needs change. Lake commented that measures that could help predict usage would be beneficial (predictive measures). Both Uellendahl and Lake stressed that the user has the most important perspective, and various goals of the user will determine what the most appropriate device would be to reach those goals. To address this aspect, the Canadian Occupational Performance Measure (COPM) and Goal Attainment Scale (GAS) were discussed. It was suggested that both of these measures would facilitate focusing on the personal factors of prosthetic usage and successful goal attainment but would likely not be used independent of standard item outcome measures. It was also noted that currently there are no adequate training materials or time available for the prosthetist to apply the COPM or GAS outcome measure in a clinical setting.
Esquenazi reviewed many of the reasons that individuals with upper limb loss may not use a prosthesis including, but not limited to, lack of shoulder mobility, lack of occupational therapy training, financial concerns, and an unnatural appearance. He discussed that a large number of individuals may eventually develop overuse injures of the contralateral side whether they use a prosthesis or not. This warrants further investigation as it may influence the recommendations of the physician to prescribe a prosthesis and occupational therapy training in an attempt to address the impact on future health care and to help justify the use of prosthetic devices to users and insurers.
WHAT TOOLS ARE AVAILABLE AND WHEN THEY COULD BE USED
One recommendation of the ULPOM group was to build a “toolkit” of “validated procedures for the entire development cycle to be developed and promoted within the appropriate professions.”1 Work by the ULPOM group was started before the SSC in which they classified identified measures into three categories: accept, consider, reject.1 At the SSC meeting, the terminology was revised to better describe the categories: recommended, to consider, and excluded.
After the specific perspectives presentations in the SSC meeting, the notes from the International Society for Prosthetics and Orthotics (ISPO) 2007 breakout groups were reviewed to address the fourth SSC question. These discussions identified what the various breakout groups at ISPO (adult, pediatric, and technical) wanted to be able obtain from an outcome measure. From these notes and the SSC discussion, a list of example questions across a range of evaluation topics (Table 1) was prepared. As a group, appropriate tools for each question were identified and categorized according to the field of application (research or patient care). The International Classification of Functioning, Disability and Health (ICF)-related component under the evaluation was also identified for each measure.
During this process, the group adopted and modified terminology used by the ULPOM group for evaluating tools as “clinically useful” or “research useful” or both. Specifically, we needed to identify what “research” meant to our SSC group, because it is a broad term that covers measurement along the spectrum from product development through commercialization into the clinic and ending in long-term or longitudinal or community use. To best classify the available tools and how they might be used, the SSC group identified three fields of application with the assumption that all the three would use human subjects, rather than bench-top testing of hardware. Research was expanded into two fields: 1) “development research,” identifying the time when componentry and controls might be early prototypes and used in the laboratory; and 2) “clinical research,” identifying the transition to a clinical environment or an in-depth investigation to test a certain intervention. The third field, “patient care,” relates to the assessments that are conducted in everyday patient interaction or clinical care settings. After identifying the general questions, the tools that were recommended in the EBR and by the ULPOM group were reviewed to see which, if any, would directly answer the identified questions in the field of application category. It is important to note that this is only a general summary, and it would still be necessary to make sure that the measures listed were appropriate to answer the specific question. The results of this discussion can be found in Table 1.
There were some questions that were not easily addressed with the existing toolkit of recommended measures, including the global value of the upper limb prosthesis to the individual, which may be more suitably addressed by qualitative methodologies, and the effect of the prosthesis on mental load. Three questions were determined to only be answerable with longitudinal studies using existing measures: 1) “what provides evidence to support prosthetic clinical recommendations?”; 2) “how can we compare one upper limb prosthetic device to another?” which would depend on the tools demonstrating sufficient sensitivity; and 3) “does the tool measure change over time (longitudinal outcomes)?” which would be dependent on the tools lacking ceiling or floor effects over the period of time in question.
As Table 1 was completed, there were noticeable gaps, especially in the development end of the spectrum. Therefore, the tools listed as “consider” by the ULPOM group were evaluated by the SSC group to see what field of application would be addressed. Tools were organized into three groups: 1) emerging, tools under active development; 2) promising, tools with only minor work needed to confirm if they should be listed as “recommended”; 3) potential, tools that require significant work to develop and validate for use as recommended. The tools in the “promising” and “potential” groups were also discussed to determine whether any showed specific benefit, either for answering questions in areas that are lacking applicable tools (e.g., Jebsen-Taylor, Southampton Hand Assessment Procedure [SHAP], and Box and Blocks for the “research development” stage) or for that are very close to application, with very specific work needed to make the tool appropriate for application (e.g., Assisting Hand Assessment [AHA]-prosthetics is popular in Sweden in pediatric applications but has never been validated, though work is in progress). The results of this discussion can be found in Table 2.
IDENTIFICATION OF RESEARCH PRIORITY QUESTIONS
Throughout the various discussions, a list was kept of commonly identified issues, questions, and missing links. Approximately, 20 points were listed. After much discussion about these needs, seven key research priority questions were identified that collectively covered the specific points:
- How should outcome measures be disseminated to the various stakeholders along the continuum? That is, what are the best methods to enable all of the stakeholders to use outcome measures on a routine basis?
- What are appropriate and recommended measures that can be identified across the continuum from research and development through community integration and across all the ICF-related components?
- How can we leverage multidisciplinary, multicenter, longitudinal, and collective studies to answer the interest questions of the various stakeholders?
- What measures are sensitive enough to evaluate acceptance and rejection of prosthetic devices?
- How does the team approach influence upper limb prosthetic outcomes? (multidisciplinary, experience, specialty groups or specialized training, and complexity of the case [bilateral and/or higher levels of amputation, other comorbidities, etc]).
- What are the contributing factors to overuse injury in upper limb deficiency and is there a difference in injury incidence and severity for those who do or do not use an upper limb prosthesis?
- How are overall clinical upper limb outcomes related to individualized interventions or decisions?
The group was unanimous in their recommendation that research priority question 1 was the most important to emphasize at this state. Many outcome measures exist, but overall, they are not being applied, especially in a clinical setting. New tools are being developed and it will be necessary to disseminate information about these new measures as they become available. Decisions need to be made regarding who is going to administer the various tests and how they should be administered. The time and equipment needed to administer, and score the tests also needs to be considered.
One key limitation to application that was discussed was the lack of available training for the various assessments. For example, the COPM and GAS seemed to be able to address many aspects of care that prosthetists would like to know. However, these are both occupational therapy tools, and there are not applicable training documents available for the prosthetics community. Because of the individuality of goals, it would also be difficult to compare individual's abilities using these measures on their own. It was suggested that a library or menu of individualized goals and achievement levels could be developed to make longitudinal studies more feasible using this tool (e.g., goals related to dressing, feeding, self-care, work, sports, leisure).
Agreement was reached by the group regarding the identification of the Assessment for Capacity of Myoelectric Control (ACMC), a measure that is, currently, aimed at the Occupational Therapy (OT) community, to be one of the measures with the most potential for two of the questions identified in Table 1, although certification is required to use it and training is now done in a labor-intensive small group basis (i.e., one expert trainer with groups of ten clinicians), limiting the opportunities for widespread sharing of this measure. Facilitating web-based training modules may make this and other measures more accessible to the general public. Timely and well-targeted dissemination of the results of this meeting was also suggested as a way to make the various stakeholders aware of the recommendations of the SSC group and thus help further future longitudinal studies. In addition to presentations at national and international meetings, it was suggested that education of clinical professionals (prosthetists, occupational therapists, and physical therapists) earlier in their careers (e.g., as students) may have a significant positive influence on the acceptance of the tools in the future.
Research priority question 2 addresses the gaps in our toolkit. For example, the development component of the research column has very few appropriate tools to address the questions in Table 1. One of the suggested methods to address this research or development area was to evaluate the “activity” tools and see what elements could be used in a research setting. It was also noted that there are few tests for adults in the ICF domain of “function.” Other gaps included the lack of validated functional measures for higher complexity level amputees, bilateral amputees, or body-powered users. Currently, existing measures can at best evaluate one arm at a time or are limited to assessment of only the prosthetic hand. None of the existing measures evaluate the mental load associated with prosthetic use (i.e., how much focus must be given to control the device). It was also concluded that to evaluate the global value of the device, it may be necessary to also incorporate the use of qualitative evaluation methodologies. For some of the other missing links (e.g., satisfaction, etc), lower limb measures may be able to be modified to fit the upper limb case.2
Long-term care and evaluation was the focus of research priority question 3, specifically the need to organize and fund multidisciplinary, multicenter, longitudinal, and collective studies. Multicenter collective studies would provide larger numbers to permit a more thorough and generalizable examination that would provide evidence that meets the evidence needs of multiple stakeholders. In particular, a need was expressed for longitudinal studies comparing various devices over time. This necessitates the use of a strong set of validated outcome measures that cover the various ICF-related components. It was also believed that the use of longitudinal studies may help us to predict the impact of the prosthesis and associated rehabilitation on long-term functional success and device acceptance and the incidence of related overuse issues.
With respect to longitudinal studies, research needs to be done to determine which measures are sensitive enough to evaluate the ultimate acceptance and rejection of prosthetic designs (research priority question 4). These evaluations may include a focus many of the same issues that were identified in research priority question 3 (e.g., long-term acceptance) and also consider the physical load or effort, such as energy expenditure, heat, weight, and socket comfort. Research priority question 4 may also relate to the evaluation of the patient education practices as reflected in options and goals. If this information is given at a time when it is not of priority to the user, i.e., provided too early in the prosthetic rehabilitation program, it may be forgotten or overlooked, reducing its potential impact on the outcome. The study by Biddiss and Chau3 provides suggestions on appropriate areas of investigation related to research priority question 4.
Another common topic of discussion among the group was the need for team approach when fitting and training persons with upper-limb prostheses (research priority question 5). It was noted that there has been little study of how the team approach influences outcomes. Longitudinal studies in this area could evaluate the effect of the multidisciplinary team on outcomes, the effect of clinician experience on the success of fitting, the effect of a specialty designation group or specialty training on the fitting, and the effect of the complexity of the case on the outcome. Many aspects of care could be evaluated as part of the team approach including component selection, patient education, and client satisfaction.
As a follow-up to a recent publication in the Journal of Prosthetics and Orthotics,4 more work needs to be done to investigate the contributing factors to overuse injury in upper limb deficiency (research priority question 6). In addition to evaluating if there is a difference in injury rates and severity for those who do or do not use an upper limb prosthesis, it is important to know if there are other factors that are associated with injury (type of device used, vocational activities, recreational activities, other physical involvement, etc).
The group also recognized the need to be able to relate overall clinical upper limb outcomes to individualized interventions or decisions (research priority question 7). That is, we need better ways to go from outcome measure results to specific recommendations for caring for a particular client, which is a limitation of the existing standard item measures but of great importance. This brought the discussion back to the potential of individualized measures such as the COPM or GAS as indictors of outcome.
SUMMARY AND CONCLUSIONS
One question from the readership of the Academy publications (i.e., prosthetic clinicians) may be “How should I interpret this work to influence my clinical practice and what tools should I start using now?” Even with the significant amount of work done to evaluate various outcome measures, there are few tools that can be recommended for the clinical prosthetist to integrate immediately into a clinical care setting. There are many measures that show promise for application into the clinical prosthetics setting soon, either tools that require slightly more work or the development of educational materials for the prosthetist to make them available for use. Measures at the body function level may be ones that most directly relate to the work done by a clinical prosthetist. For most of these measures, we still need more information on their reliability and validity when used in prosthetics.
At the activity level, the COPM and GAS allow individualized goal setting, ideal for a clinical setting; however, prosthetist-specific training does not currently exist. Another primary concern of the clinical prosthetist is satisfaction and use or acceptance. Surveys are ideal for this application, and it is recommended that prosthetic companies use existing, validated measures (such as the upper limb module of the Orthotics and prosthetics user survey, which is under refinement), perhaps supplemented by an additional page of additional information required for specific business purposes, rather than developing entire new surveys. There may be many other outcome topics (e.g., specific functional assessment) that the prosthetist may wish to address, although many of the identified tools that link with these areas have been traditionally administered by therapists. Although prosthetist should certainly become more involved in the application and work on outcome measures, it is hoped that future work will be done as part of a team approach to permit use of measures that reflect the breadth of the ICF outcome areas and, as discussed in this report, will be facilitated by education initiatives for all interested clinical team members on these use of the tools.
This State of the Science document contains the summary of significant work in outcomes that has been conducted in upper limb prosthetics. The work that was accomplished at the State of the Science meeting could not have been completed without the evidence-based report prepared by Wright or the work that was done independent of Wright by the Upper Limb Prosthetic Outcome Measures Group. From this background work, the SSC group has selected and recommended a list of tools potentially applicable for various questions (Table 1) as well as other measures that in the future may be more fully developed for recommendation across the spectrum from research and development through home use (Table 2). As previously stated, we use the word “potentially” because it is necessary to formulate the question before confirming whether an outcome measure is truly appropriate.
A key outcome of the SSC meeting was the identification of seven specific research priorities. The most important of these is the need to facilitate the implementation of the existing recommended measures into the prosthetics community (including prosthetists, therapists, physicians, and engineers). The group plans to widely disseminate this information so that instead of continuing to discuss why outcomes are not more widely used, there will be resources developed for training and eventually data will be collected from longitudinal and multicenter studies that advance the field.