Kerfeld, Cheryl I. PT, PhD; Dudgeon, Brian J. PhD, OTR; Engel, Joyce M. PhD, OT; Kartin, Deborah PT, PhD
INTRODUCTION AND PURPOSE
Physical function is a commonly used construct in pediatric rehabilitation. It is implicitly described within the World Health Organization's International Classification of Functioning, Disability and Health (ICF)1 activities and participation domains. Evaluation of pediatric physical function explores and measures capacity and performance, levels of independence, and environmental and equipment needs of children in their daily living routines.2
Self-reported outcome measures can play a large role in the evaluation of physical function by obtaining information from children about their abilities to do what they want to do and how they do it in their home and community settings. Children, properly queried, can be the best source of information about their health and treatment effects,3 and self-reporting can also be a means to facilitate children's participation in their own health care.4
A common assumption in many measures of physical function is that the capacity to engage in functional activities is essential to the formation of a sense of well-being and satisfaction in life.5 However, adults with disabilities often report their quality of life as good and that their quality of life is only moderately correlated with their symptoms or impairments.6 Similarly, it should not be assumed that children who use a wheelchair (WC) for mobility have poor health or diminished quality of life.
Therefore, the common challenge in developing self-reported outcome measures is to assure that the constructs being measured are well understood and the foundation of the measures is grounded in the expertise of the respondents, thereby making the questions asked clearly understood and appropriate.7 Cognitive interviewing (CI), a technique used to obtain respondent feedback on potential questions, is considered important in the development of self-reported outcome measures that are psychometrically sound and valid.8 The intents of CI are to be aware of what the respondent understands a particular question to be asking, the processes used by each person to retrieve relevant information from memory, and the decisions used to arrive at an answer.9
Available CI research with individuals who use WCs is very limited. Meyers and Anderson10 administered the 36-Item Short Form Health Survey, a 36-item questionnaire that is widely used to examine functional health and well-being, to a sample of adults who use WCs. Respondents rejected words such as “walk” and “climb.” However, when the word “go” was substituted and the questions changed to reflect function using assistive devices, results suggested that this version could be successfully used to measure performance among adults who had mobility impairments, including those who use WCs.11
Cognitive interviewing was used by Dumas et al2 to assess the content, format, and comprehension of questions and responses developed for use in a parent report of physical function measure for children with cerebral palsy (CP). Respondents commented that certain questions had limited applicability to their children (eg, advanced physical activities for children with severe physical function limitations) and that they were unsure how to factor in (or not) level of assistance when responding to questions. Tucker et al12 examined the psychometric properties of a computer adaptive test, using parent report of physical function in children with CP. They found a lack of questions related to assistive devices and WC use for mobility, though it was believed that the inclusion of these questions would improve the responsiveness of the measure.
The Patient Reported Outcomes Measurement Information System (PROMIS)13,14 is a multicenter project funded under the National Institutes of Health15 to develop a psychometrically sound and clinically meaningful measurement system of self-reported outcomes. Using state-of-the-art qualitative and quantitative methods, PROMIS personnel have developed a set of item banks for children and adults that allow the assessment of key symptoms and health concepts relevant to both community and clinical populations, including people with chronic conditions.13 The PROMIS pediatric physical function related to mobility (PFM) item bank was developed through a stepwise item review and extensive field testing. For this domain, computer adaptive test and short-form formats are available.16 Greater detail about the PROMIS qualitative and quantitative analytic plan is publicly available.13
The pediatric PFM domain satisfies general criteria for unidimensionality, the assumption that the construct being measured can be expressed as a range of ability on a definable continuum characterized by a number of items that can be ordered from low to high or easy to difficult.16 However, a conceptual and practical problem suggests that a unidimensional model might not be a valid representation when disability is present.17 The PROMIS pediatric PFM item bank was developed through focus groups and CI of children with asthma and children without medical conditions or disabilities, and it was noted that the assistive-device items included in the item bank (ie, questions about using a walker or WC) did not apply to a large number of those children.9
The outcome measurement field would benefit from further study and application of the PROMIS pediatric PFM item bank with children having a greater variety of diseases and disabilities, including those at the extreme ends of physical functioning. The purpose of this study was to use CI to assess the content, format, and comprehension of the PROMIS pediatric PFM items and appraise responses from children who use WCs.
This study was approved by the Seattle Children's Hospital Institutional Review Board before initiation of study procedures.
A sample of 14 children, aged 8 to 12 years, who used a WC participated in CI. The children were recruited through professional contacts, consumer organizations for children with disabilities, and flyers posted in the community. Inclusion criteria required that each child had used a WC as his or her primary means of mobility for at least 3 prior consecutive months and had one of the following parent-reported diagnoses: CP, spinal cord injury, spina bifida, Duchenne's muscular dystrophy, or other chronic physical disabilities. Each child also had to read English at the second-grade level, and parents had to report that the child could meaningfully answer the questions on the survey during the study screening. All participants and their parents provided written informed assent and consent before the children's participation in the study.
On the basis of the World Health Organization definition of health, “a state of complete physical, mental, and social well-being and not merely the absence of disease or infirmity,”18 PROMIS defines physical function as “one's ability to carry out various activities, ranging from self-care (activities of daily living) to more challenging and vigorous activities that require increasing degrees of mobility, strength, or endurance.”13 PROMIS further describes the pediatric PFM item banks as focusing on “activities of physical mobility such as getting out of bed or a chair to activities such as running.” These are reported13 to be “consistent with the International Classification of Functioning for Children and Youth (ICF-CY).”19
The PROMIS pediatric PFM items are framed as “capability” (ie, what a child “can do”) rather than “performance” (ie, what a child “does do”).13 It emphasizes walking without accommodations or environmental adaptations as the highest probable level of functioning.16 Table 1 presents the PROMIS pediatric PFM item bank including the item context, stem, and response scale.
This study closely followed the PROMIS qualitative item review CI protocol.20 The 23 PROMIS PFM items were initially reviewed by the lead investigator, a pediatric physical therapist who has worked with children with disabilities for more than 20 years, and an expert panel composed of 2 pediatric physical therapists, 1 occupational therapist, and an educational psychology researcher studying children with disabilities. The lead investigator conducted all of the CI sessions with participants. Cognitive interviewing sessions were conducted in the child's home, at the hospital, or over the telephone, based on child/parent preference. Each child's reading level was assessed through the reading section of the Wide Range Achievement Test 3, a commonly used and reliable assessment of reading level.21
The children first read and responded to all of the 23 PROMIS PFM items in paper and pencil format. Then, the primary investigator, using the PROMIS CI protocol of questions (see Table 2) and spontaneous probes, encouraged the children to verbalize their thoughts when answering each item. The questions and probes were based on previous PROMIS research and addressed 4 main topics: item clarity, time frame, item content, and response format.9
The primary investigator and a research assistant took detailed notes throughout each interview about the child's responses to the probes, questions or concerns about the items that arose during the interviews, and thoughts about the interview format and interaction. When permission was given, the interviews were audio recorded to ensure accuracy of interviewers' notes. All concerns identified by the children were considered potential problem areas. An a priori decision was made to stop data collection when saturation was perceived (ie, no new responses were elicited).
Following each child's CI, field notes were compiled into a single text document, including each item and the child's comments. After completion of all CIs, all comments for a single item were generated. In addition to following the PROMIS qualitative item review CI protocol, we used the Questionnaire Appraisal System (QAS)22 that provided a structured systematic review of the children's comments for each of the items. The goal of using the QAS was to improve the efficiency of the questionnaire-review process by providing an easy-to-use method for categorizing concerns the participants may have with the items.22
Five categories of the modified version of the QAS (clarity, assumptions, knowledge, sensitivity/bias, and response) were used to code each of the item's summarized text documents. Two QAS categories (problems with reading the question and problems with instructions) and the open-ended question subcategory were excluded as they did not apply. Table 3 presents the 5 categories and 20 subsets that were used for coding. When coding disagreements occurred, they were resolved by consensus of the 2 coders and expert panel. Additional item concerns not readily captured by the QAS were also identified and recorded.
Cognitive interviewing sessions were completed with 14 children, and all but 1 gave permission to have their interviews audio recorded. Participant demographics are summarized in Table 4. Each CI session took approximately 90 minutes. The saturation requirement was met. Of the 322 comments that were coded, 4 disagreements required further discussion by the 2 coders and expert panel before achieving consensus.
The item context used the time frame “in the past 7 days.” The children were asked what time frame they thought of when responding to these items. Of those who indicated using the requested time frame (n = 4), they reported either thinking about the last week of school, Monday to Monday, or counted back 7 days. Others reported using a time frame other than the past week or not using a time frame at all (n = 10). Two children read right through “in the past 7 days,” saying that it would be the same if they had been asked “in the last month.” Two children said that it would be better to ask in the past month because of recent special circumstances, 1 child being in the hospital and 1 at a camp, and that answers would be different depending on the time frame.
Although some children (n = 5) felt the response options made sense to them and they would not change the options, a majority (n = 9) felt that response categories overlapped. With a little trouble and with some trouble seemed the same, and they preferred to have 4 response options rather than 5. Most of the children were comfortable with the use of the word “trouble.” Only 1 child felt that the response should use the word “difficulty” rather than “trouble.”
The Questionnaire Appraisal System analysis identified 3 broad themes surrounding the categories of clarity, assumptions, and sensitivity of the PROMIS PFM related to mobility items (Table 5).
During the analysis, clarity problems were identified with items related to communicating the intent of meaning of the question to the respondent. All of the children (N = 14) expressed concerns about some individual items being vague, that there were multiple ways to interpret the question or to decide what was to be included or excluded.
Six children wondered if the kids in the question “I could keep up when I played with other kids” could walk or if they too used a WC, which would affect the way they would respond. Many of the children's answers had nothing to do with their functional ability, but more to do with the limitations placed on them by the environment (eg, terrain). When an item was too general, some said that they would answer with the middle choice “with some trouble.”
Other items on which children asked for clarification included “I could bend over to pick something up.” Answers varied, depending on where the object was located. All of the children in one way or another wondered if they could use assistance to accomplish some of the other items, such as I could move my legs, I can get up from the floor, and I can stand up by myself.
Regarding getting up from the floor and standing, the children said that they could do it if they could use their hands, WC, walker, or when someone holds on to them. The children consistently asked for clarification regarding the questions: I could get in and out of a car, I could get into or out of bed by myself, and I could ride a bike. They wanted more specific information such as type of car, bed height, and characteristics of the bike.
The children expressed concerns about the inappropriate assumptions that some items made about them or about their situation. These included the following items: I could walk across the room, I could walk more than one block, I could run a mile, and I could get up from a regular toilet. Many of these items assume the respondent is ambulatory. The children said that they would answer differently if the wordings were changed. Many said that they get around in different ways, using various assistive devices or WC, depending on the destination and activity. Some of their answers would depend on the environment in which they were performing the activity.
Sensitivity of Items
Finally, the QAS analysis assessed the items for sensitive nature or wording, and for bias. Many of the children were not bothered with answering the questions that had nothing to do with how they moved or played. Three children, however, commented that they were not happy answering “not able to do” on several of the questions. Nevertheless, many were pleased they could express how to change the wording of the questions. For example, instead of “I could walk across the room,” a child changed the wording to “I could wheel across the room.” With that small change many of the children's responses changed from 0, not able to do, to 4, with no trouble. The children were forthcoming about suggestions of what they would like to be asked about their PFM, taking into account what they “can do” with their WC.
Cognitive interviewing provided valuable information about the PROMIS pediatric PFM items for children aged 8 to 12 years who use WCs. The items were easily comprehended by the children in this study. The children, in meaningful and insightful ways, provided feedback on the PROMIS items they thought were not applicable to them and offered suggestions for personally important items that would better capture their perspectives of PFM. These findings are consistent with other studies examining the development of self-reported outcome measures for children9 and populations that use WCs for their PFM.2,23
The children had various responses when asked what the phrase “in the past 7 days” meant to them. The selection of the recall period is an important decision in the development of a self-reported outcome measure. In general, researchers have suggested that shorter recall periods (eg, 24 hours or 1 week at most) can be preferable to longer recall periods. Longer recall data can be heavily biased by current health and any significant events. Choice of an optimal recall period may depend on a number of other factors such as the measure's purpose and intended use; the variability, duration, frequency, and intensity of the concept measured; and the disease or condition's characteristics.24 Interestingly, in our study, some of the children's responses were influenced by the environment in which they were present during the recall period.
Unlike the PROMIS CI study,9 the children in this study expressed difficulty with differentiating between 2 of the 5 response options and suggested that 4 response options would be sufficient. Previous research has shown that 8-year-old children have accurately used the full range of 5- to 7-point Likert scales to rate their health, whereas younger children seem to respond more easily to a 3-point Likert scale.25 Our study's qualitative findings regarding the optimal recall period and number of response options should be explored further, using quantitative methods similar to the PROMIS-recommended item development methodology.
Participants identified items containing vague concepts, unclear assumptions, or that were otherwise sensitive in nature. All of the children expressed concerns about the items being vague, with many ways to interpret and answer a question. The difficulty answering a question was not because of their functional limitations, but rather because of environmental limitations and the uncertainty about how to take these into account when reporting. Their functioning also depended on whether or not they were allowed to use their WC or other kinds of assistance. Despite the acknowledged role of environmental and personal factors to compensate for physical functioning limitations or environmental barriers in characterizing physical functioning, most self-reported outcome measures of PFM do not appear to take these factors into account.26
Tieman et al27 examined the variability of mobility methods of children with CP in different Gross Motor Function Classification System categories. Results indicated that children within the same Gross Motor Function Classification System category exhibited differences in their mobility independence that varied by environment, especially in the school and outdoors/community settings. Certain personal factors of the child such as preferences, personality (eg, sense of adventure, cautiousness), cognition, and health condition may also affect mobility. For instance, some children may want to walk despite the slow speed and high energy cost, whereas others may prefer wheeled mobility.
Of the 100 physical function measures commonly used in rehabilitation research reviewed by Rust and Smith,28 few measures addressed use of assistive technology (AT), and if included, AT use tended to lower the overall score. This suggests that using AT reflects a negative rehabilitation outcome and must be addressed to reconcile different conceptual models of disability and physical functioning measures.
Participants also expressed concerns about the type of questions that were being asked, with particular reference to walking and performing activities using one's legs. Many stated that their answers would differ by changing the word “walk” to “wheel.” This suggests that it is important to measure PFM from multiple perspectives and has implications for the development of valid outcome measures for specific populations.
In the context of health, the ICF defines activities and participation as “the execution of a task or action” and “involvement in a range of life situations,” respectively1 The authors of the ICF indicated that it is impossible to differentiate between the domains that represented individual (activity) and societal (participation) perspectives, given international variations and varying professional approaches.1 The absence of clear definitions for these 2 constructs can create a dilemma for measurement. Different measures may be developed that identify themselves as measures of activity or participation, while actually representing quite distinct definitions of these constructs.
Researchers and clinicians also continue to debate the difference between the 2 ICF activity and participation qualifiers: capacity (child's ability in a standard environment) and performance (what the child does in his or her current environment).1,29,30 The decision to measure capacity, performance, or both should depend on what the researcher or clinician plans to do with the data. Treatment is sometimes intended to improve capacity, and ultimately to improve performance. In other situations, the intent of treatment is to improve the quality of existing functions so that activities can be performed more easily.
One potential advantage of using ICF-defined performance measures is that they provide a direct assessment of home or community functions and assess limitations of direct relevance to children. Since home, school, and play environments are most meaningful to children, the use of performance measures in these settings may be more appropriate.31 Some argue that both capacity and performance should be measured. This strategy may identify specific areas of differences between capacity and performance and thereby may identify environmental factors that limit performance. This ultimately may lead to alternative intervention strategies such as environmental modifications.32
The “capability” definition used for the PROMIS PFM item banks appears to be similar to the ICF activity and participation capacity, rather than performance qualifier. For PROMIS, gathering information about one's inability to walk or climb stairs is a valuable measure for children who use WCs. Since the PROMIS PFM items exist as part of a calibrated item bank, when given to participants using a computer adaptive test, children who use WCs would not be given the walking items.
Although this may capture the physical capacity of a child, it may not provide insight into the child's overall functional abilities when using a WC. It is possible for a child who uses a WC to have better functional mobility than a child who ambulates without assistive devices. This suggests the potential need for a PFM item bank that is specifically for children who use WCs, possibly 1 for manual WCs and 1 for power WCs.
In 2008, PROMIS developed a physical function item bank appropriate for adults who use WCs, mobility aids, and other AT. The expert panel working on the item bank recommended that physical function should include what people with chronic conditions can accomplish by using AT, any device that would assist participants in independently accomplishing physical tasks and chores.33 Subsequent analysis of this item bank indicated that the added items substantially improved the precision of the item bank, particularly for use with individuals with low physical function.33
The selection of a self-reported outcome measure should be concept driven and/or include what is most meaningful and important to the individuals reporting. The children participating in our study requested that items be more specific and include more options for reporting adaptive ways of performing and participating, saying their answers to items depended on the situation and the environmental supports and barriers. The children also suggested other items they would like to be asked about their functional mobility; however, these items likely represent a different construct(s) than the one used to develop the PROMIS pediatric PFM item bank. To develop these suggested items into a new self-reported outcome measure for children who use WCs, these need rigorous review and further development.
Table 6 provides potential items to consider, including an item bank that measures PFM of children who use WCs. For this item bank, PFM of children who use WCs can be defined as the child's ability to move within the environment in a purposeful way for activities of daily living, with modifications and equipment needed. Clearly, these items would need to be carefully trialed and amended as needed using the PROMIS mixed method (qualitative and quantitative) analyses based on a larger and more diverse sample of children who use WCs.
As guidelines for interpreting CI results are not yet standardized in the research literature, the primary investigator and expert panel made judgments regarding the importance of a QAS item problem. Given the small sample size, the results may not be representative of all children who use WCs.
Overall, children as young as 8 years who use WCs are able to respond to items and discuss aspects of their PFM in meaningful ways. They comprehended varying response options and accurately understood a 7-day recall period. The children resoundingly wanted to share what they were able to do, as evidenced in their suggested items regarding PFM. Most current self-reported PFM outcome measures do not capture the activity/participation of children who use WCs. Although the data from the cognitive interviews suggested items for a PFM outcome measure for children who use WCs, further development including qualitative and quantitative analyses using a larger sample is needed.
In the future, integrating pediatric outcome measurement within the context of the ICF framework may clarify the relationship between physical function, activities/participation, and capacity/performance. As pediatric rehabilitation professionals develop and increasingly use sound self-reported outcome measures, they should continue to explore the authentic reflections made by children who are WC users regarding their physical functioning, the influences of environments, and the variability in the use of devices to assist with mobility. Suggested enhancements should result in continued refinement and successive generations of measures in which children who use WCs can become central reporters about their experiences and needs.
The authors gratefully thank the study participants and their families. We also thank Chelsea Vuylsteke, DPT, PROMIS, and the University of Washington Center on Outcomes Research in Rehabilitation for their assistance.
1. World Health Organization. International Classification of Functioning, Disability and Health (ICF). Geneva, Switzerland: World Health Organization; 2001. http://www.who.int/classifications/icf/en/
. Accessed April 11, 2010.
2. Dumas HM, Watson K, Fragala-Pinkham MA, et al. Using cognitive interviewing for test items to assess physical function in children with cerebral palsy. Pediatr Phys Ther. 2008;20(4):356–362.
3. Drotar D. Measuring child health: scientific questions, challenges, and recommendations. Ambul Pediatr. 2004;4(suppl)(4):353–357.
4. Kieckhefer GM, Trahms CM. Supporting development of children with chronic conditions: from compliance toward shared management. Pediatr Nurs. 2000;26(4):354–363.
5. Hays RD, Hahn H, Marshall G. Use of the SF-36 and other health-related quality of life measures to assess persons with disabilities. Arch Phys Med Rehabil. 2002;83(suppl 2)(12):S4–S9.
6. Albrecht GL, Devlieger PJ. The disability paradox: high quality of life against all odds. Soc Sci Med. 1999;48(8):977–988.
7. Cremeens J, Eiser C, Blades M. A qualitative investigation of school-aged children's answers to items from a generic quality of life measure. Child Care Health Dev. 2007;33(1):83–89.
8. Willis GB. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications; 2004.
9. Irwin DE, Varni JW, Yeatts K, DeWal DA. Cognitive interviewing methodology in the development of a pediatric item bank: a Patient Reported Outcomes Measurement Information System (PROMIS) study. Health Qual Life Outcomes. 2009;7:3.
10. Meyers AR, Andresen EM. Enabling our instruments: accommodation, universal design, and access to participation in research. Arch Phys Med Rehabil. 2000;81(suppl 2)(12):S5–S9.
11. Froehlich-Grobe K, Andresen EM, Caburnay C, White GW. Measuring health-related quality of life for persons with mobility impairments: an enabled version of the short-form 36 (SF-36E). Qual Life Res. 2008;17(5):751–770.
12. Tucker CA, Gorton GE, Watson K, et al. Development of a parent-report computer-adaptive test to assess physical functioning in children with cerebral palsy 1: lower-extremity and mobility skills. Dev Med Child Neurol. 2009;51(9):717–724.
13. Patient Reported Outcomes Measurement Information System (PROMIS). PROMIS homepage. http://www.nihpromis.org
. Accessed April 11, 2012.
14. Cella D, Yount S, Roghrock N, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH roadmap cooperative group during its first two years. Med Care. 2007;45(5, suppl 1):S3–S11.
15. Zerhouni E. Medicine: the NIH roadmap. Science. 2003;302:63–64,72.
16. DeWitt EM, Stucky BD, Thissen D, et al. Construction of the eight-item Patient-Reported Outcomes Measurement Information System pediatric physical function scales: built using item response theory. J Clin Epidemiol. 2011;64(7):794–804.
17. Bruce B, Fries JF, Ambrosini D, et al. Better assessment of physical function: item improvement is neglected but essential. Arthritis Res Ther. 2009;11:191–202.
18. World Health Organization. Constitution of the World Health Organization as Adopted by the International Health Conference, New York, June 19–July 22, 1946. Geneva, Switzerland: World Health Organization.
20. Dewalt D, Rothrock N, Yount S, Stone A. Evaluation of item candidates: the PROMIS Qualitative Item Review. Med Care. 2007;45(suppl 1)(5):S12–S21.
21. Wilkinson GS. Wide Range Achievement Test WRAT3 Administration Manual. Wilmington, DE: Wide Range Inc; 1993.
22. Willis GB, Lessler JT. Questionnaire Appraisal System: QAS-99. Rockville, MD: Research Triangle Institute; 1999.
23. Young NL, Williams JI, Yoshida KK, Wright JG. Measurement properties of the Activities Scale for Kids. J Clin Epidemiol. 2000;53(2):125–137.
25. Eiser C, Morse R. Can parents rate their child's health-related quality of life? Results of a systematic review. Qual Life Res. 2001;10(4):347–357.
26. Tomey KM, Sowers MR. Assessment of physical functioning: a conceptual model encompassing environmental factors and individual compensation strategies. Phys Ther. 2009;89(7):705–714.
27. Tieman B, Palisano RJ, Gracely EJ, Rosenbaum PL. Variability in mobility of children with cerebral palsy. Pediatr Phys Ther. 2007;19(3):180–187.
28. Rust KL, Smith SO. Assistive technology in the measurement of rehabilitation outcomes. Am J Phys Med Rehabil. 2005;84(10):780–793.
29. Jette MJ. Toward a common language for function, disability, and health. Phys Ther. 2006;86(5):726–734.
30. Harris F. Conceptual issues in the measurement of participation among wheeled mobility device users. Disabil Rehabil Assistive Technol. 2007;2(3):137–148.
31. Rosenbaum P, Gorter JW. The ‘F-words’ in childhood disability: I swear this is how we should think! Child Care Health Dev. 2011;38(4):457–463.
32. Tieman BL, Palisan RJ, Gracely EJ, Rosenbaum PL. Gross motor capability and performance of mobility in children with cerebral palsy: a comparison across home, school, and outdoors/community settings. Phys Ther. 2004;84(5):419–429.
33. Amtmann D, Cook KF, Johnson KL, Cella D. The PROMIS Initiative: involvement of rehabilitation stakeholders in development and examples of applications in rehabilitation research. Arch Phys Med Rehabil. 2011;92(suppl 1):S12–S19.
assistive technology; child; cognition; disability evaluation; environment; female; humans; interviews as topic; male; psychometrics; self-report; severity of illness index; social environment; wheelchairs
© 2013 Lippincott Williams & Wilkins, Inc.