Journal Logo

CLINICAL SCIENCES

Global Switch Cost as an Index for Concussion Assessment: Reliability and Stability

OKUMURA, MICHELLE S.; COOPER, STEPHANIE L.; FERRARA, MICHAEL S.; TOMPOROWSKI, PHILLIP D.

Author Information
Medicine & Science in Sports & Exercise: June 2013 - Volume 45 - Issue 6 - p 1038-1042
doi: 10.1249/MSS.0b013e318281e1fb
  • Free

Abstract

Decisions concerning an athlete’s return to play (RTP) after a concussion are made on the basis of multiple indices. Over the past decade, neuropsychological testing has increasingly become part of the concussion assessment and in making RTP decisions. Several computer-based neuropsychological test batteries have been developed and marketed for evaluation of concussion effects. The advantage of computerized testing has been purported to be that the tests provide fine-grain analyses of basic cognitive functions such as visual scanning, information processing speed, executive function, and memory. Typically, baseline measures of athlete’s performance on these test batteries are obtained before beginning a sport season and compared with test performance after a concussive event. The athlete’s cognitive function may be assessed at multiple time points to track recovery, and the interpretation of these data can be used as part of the RTP decision. Recently, the merits of this approach have been questioned, particularly in terms of the sensitivity and reliability of the tests used (2,5,25).

The dual-task (DT) testing method has been proposed as a complementary approach to assess the effects of concussion in athletes. Researchers have shown that specific combinations of motor-control tests and cognitive tests lead to changes in performance of one or both tasks. Reduced performance observed during DT conditions is typically explained in terms of limitations of attentional resources. Theorists hypothesize that individuals’ attentional resources are fixed and that concurrent performance of multiple tasks draws on limited resources to the point that reductions in performance are observed (13,28). Numerous studies have examined the challenges that can arise when individuals are asked to perform a cognitive task and a sensorimotor task simultaneously (14). The majority of recent studies used cognitive tests of executive function, which is thought to be particularly sensitive to the effects of concussion (4), and sensorimotor tests that involve balance (6,16,21,26) or gait (3,15). The results obtained from research conducted to assess age-related differences in these DT studies suggest that balance and locomotion, which have traditionally been seen as reflexive, is influenced by higher CNS control systems and draws on attentional resources. Older adults, for example, who are characterized as having fewer attentional resources than younger adults, tend to use strategies in which locomotion and balance control are prioritized over cognitive test performance (23).

The behavioral characteristics of concussed athletes (e.g., difficulties in balance control, mental confusion, and difficulties in short-term or working memory) have lead us to consider the merits of the DT approach to evaluate recovery from brain injury and in making safe RTP decisions. However, although theories of attention have been bolstered by the findings of DT studies, several factors impede the translation of laboratory-based DT methods to assess concussed athletes. Lacking is a “gold standard” measure of DT interference that provides adequate sensitivity and reliability (2). The cognitive tests and motor tests used in previous studies vary widely, and none of the protocols describe measures that can be adapted and used to estimate the magnitude of DT interference caused by concussion (1,7,19,20). In addition, absent from prior research studies are data concerning the stability of an individual’s cognitive test performed over the time ranges that are necessary for tracking recovery from concussion and making RTP decisions. The paucity of test–retest reliability data has also been noted for the existing neuropsychological test batteries often used to assist into RTP decisions (24).

As a prelude to assessing concussed individuals, we have developed a DT method that uses a single test of executive function—the switch task. Executive functions are hypothesized to reflect the operations of three component processes that, together, regulate complex, goal-oriented actions: response inhibition, working memory, and switching (11). The switch task requires participants to perform different tasks that alternate the response sets associated with specific stimuli. For example, in the present study, if a letter followed a number, the participant was required to make discriminations based on an alphabetic decision rule; when a number followed a letter, the discriminations were made via a numeric decision rule. The response time (RT) measured after an alternation provides a switch cost index, which is taken to reflect the changes in the attention-demanding mental processes required to abandon one response set and to reconfigure a different response set (17,22). The switch-task protocol has been used in three DT experiments performed in our laboratory designed to assess interference between balance control and cognitive test performance in nonconcussed young adults. The balance control in each of these experiments was measured via standardized instrumentation, which provided measures of changes in postural control in response to systematic variation of somatosensory input. A visual switch task was used by Broglio et al. (6), whereas auditory switch tasks were used in studies conducted by May et al. (16) and Resch et al. (21). The switch tasks were introduced shortly after balance perturbation and concluded before the end of the balance period. Dual-task interference, indexed by switch-task performance, was observed in all the experiments. However, the switch tasks used in these studies differed in length, and it was unclear if the number of alternation trials might affect the sensitivity of the cognitive test and whether the within-session reliability remained consistent across multiple trials. Also unknown was the stability reliability of switch-task performance under single-task (ST) and DT conditions. The present study was designed to address these issues. Young, nonconcussed adults performed switch tasks that differed in the number of alternation trials under ST and DT conditions. In addition, the stability reliability was measured after 1 wk and approximately 7 months.

METHOD

Participants.

Fifty-nine young adults (Mage = 20.32 ± 1.84 yr, 19 males) volunteered to participate in the study. The participants were recruited via an e-mail and course announcements. Each participant provided written consent approved by the university institutional review board before participating in the study. Based on the responses to the structured medical and health history questionnaires, the participants were free of neurological disease, medication influencing nervous system function, cardiovascular disease, or any other contraindication to exercise.

Procedures.

All the participants completed two laboratory sessions scheduled 1 wk apart. During the first session, the participants completed informed consent, a medical history questionnaire, and a 24-h history questionnaire. Next, the participants were trained to perform an auditory switch task (cognitive task) and a modified Harvard Step Test (motor task). The cognitive task was described to the participant, who sat next to the computer station. The participants were instructed to listen to a series of 60 numbers and discriminate between even and odd numbers with the appropriate mouse key press (left = even, right = odd). Next, a series of 60 letters were presented, and the participant was asked to discriminate between vowels and consonants with the appropriate key press (left = vowel, right = consonant). Each key press on a serial mouse was followed 100 ms later by the presentation of the next auditory stimulus. These within-category discrimination tests comprised the pure switch-task condition. Finally, the participant was told that both letters and numbers were going to be presented. The stimuli consisted of 60 letters or numbers, which were repeated in series lengths of 2, 3, or 4 and then switched from one category to the other. The letters consisted of four vowels (A, E, I, and O) and four randomly selected consonants (B, D, L, and C). The numbers consisted of four even numbers (2, 4, 6, and 8) and four odd numbers (1, 3, 5, and 7). There were 40 nonswitch and 20 switch trials, with an equal number of switches to even–odd and vowel–consonant conditions. The between-category discrimination test comprised the mixed switch-task condition. The participants were asked to respond as quickly and accurately as possible. Computer-generated letter or number stimuli were presented binaurally to a headphone via a commercial software program (8).

After training, the participants performed a series of nine tests, which differed in the type of discrimination required and the number of stimuli presented. Two test types required within-category decisions (pure conditions): number presentation (even/odd discrimination) or letter presentation (consonant/vowel discrimination). One test type required alternating-category decisions (mixed condition) made of numbers and letters. The participants completed separate pure- and mixed-condition tests that consisted of 30 trials, 40 trials, or 60 trials. The mixed-condition tests yielded 8, 12, and 18 switch trials, which required alternation between categories. The order for test length was counterbalanced across the participants. The participants performed the tests while seated and were instructed to perform each test as quickly and as accurately as possible.

During the final phase of session 1, the participants were trained to perform a modified Harvard Step Test. The sound level of a metronome was adjusted, and the participant was instructed to maintain a step frequency of 30 steps per minute. After a brief practice, the participant completed 5 min of stepping on an 8-inch platform.

Session 2 took place 1 wk later at the same time of day for each participant. The session consisted of three phases. First, the participants completed a 24-h history questionnaire and received retraining on the pure- and mixed-condition switch test (60 trials each). Second, the participants completed alternate forms of the nine cognitive tests completed in session 1. Third, each participant was assigned to one of three DT conditions during which they performed the mixed-condition switch test while stair stepping. During the 5-min stepping test, the participants completed five 30-trial tests, three 40-trial tests, or two 60-trial tests. In each condition, the participants began stepping to the metronome, and after 10 s, the researcher signaled the participant to depress a mouse key to begin a switch test. At the end of each switch test, an auditory “stop” stimulus was presented; the participants continued stepping until the researchers signaled to depress the mouse key to begin the next switch test. Start times for each switch test were determined to ensure data were obtained during DT conditions. The metronome volume was lowered while the participants performed the cognitive task. A researcher monitored the metronome and signaled the participant to alter stepping pace if it deviated by more than a step from the 30 steps per minute cadence.

Twenty-one participants (Mage = 21.33 ± 2.03 yr, four males) completed a third session, which took place approximately 7 months (M = 7.78 ± 0.92 months, range = 165 to 279 d) after the first session. The protocol for session 3 was a systematic replication of session 2, performed with alternate forms of the switch task.

Data analyses.

Separate switch cost scores were calculated for tests composed of 30, 40, and 60 trials. Global switch cost scores were derived as recommended by Wasylyshyn et al. (27). The average RT of the two pure, within-category conditions (number and letter discrimination) was subtracted from the average RT of switch and nonswitch trials in the mixed, between-category condition (Table 1). Chronbach α was used to calculate the switch cost score stability reliability of the three tests between sessions 1 and 2 and between sessions 1 and 3. Dual-task interference on switch cost scores and response errors on tests composed of 30, 40, and 60 trials was assessed separately by comparing the participants’ ST and DT performance during session 2. The frequency of the response errors made during the mixed, between-category tests of 30, 40, and 60 trials was converted to percentage scores. The global switch cost scores and error scores are presented in Table 2. For both the global switch cost scores and percentage error scores, an initial 3 (group: 30, 40, 60 trials) × 2 (condition: sitting, stepping) ANOVA was conducted to assess the between-group group performance, and the effect size was estimated via partial eta square (pη2). In addition, separate paired t-tests were used to evaluate the performance of the participants assigned to 30, 40, and 60 trial conditions. Analyses were conducted using SPSS 19 software. The 0.05 rejection level was used in all analyses.

TABLE 1
TABLE 1:
Single-category, mixed-category switch-task RT (ms) (SE) and global switch costs (ms) (SE) for tests of 30, 40, and 60 trial lengths measured at three time points.
TABLE 2
TABLE 2:
Mean global switch cost scores (ms) (SE) and errors (percent) under ST and DT conditions for tests of 30-, 40-, and 60-trial lengths.

RESULTS

Stability reliability.

The global switch cost scores of 58 participants were measured twice with a 7-d intertest interval. One participant’s score from the second session was unavailable for switch-task analyses. The test–retest reliability estimates were 0.64 for the 30-trial test, 0.86 for the 40-trial test, and 0.83 for the 60-trial test. The global switch cost scores of 21 participants who completed sessions 1 and 3 were separated by approximately 7 months. The test–retest reliability for the 30-trial task could not be calculated because of a negative average covariance, and it violated the reliability model assumptions. The remaining tasks yielded the test–retest reliability estimates of 0.32 for the 40-trial test and 0.59 for the 60-trial test.

Dual-task interference.

The ANOVA performed to assess DT competition on the participants’ global switch costs yielded a significant effect for the condition factor, F(1,55) = 44.53, P < 0.0001, pη2 = 0.44. The average switch cost score was 89.96 ms during the ST, sitting condition, and 209.37 ms during the DT, stepping condition. There was no significant difference in performance among the three groups. The global switch costs were 124, 123, and 123 ms for the 30-, 40-, and 60-trial conditions, respectively. The paired t-test analyses for the 19 participants assigned to the 30-trial condition revealed a t = −3.00, P = 0.008; for the 18 participants assigned to the 40-trial condition, a t = −4.19, P = 0.001; and for the 21 participants assigned to the 60-trial condition, a t = −5.18, P < 0.001. The ANOVA performed on the participants’ percentage error scores yielded a significant effect for the condition factor, F(1,56) = 64.69, P < 0.001, pη2 = 0.54. The average percentage error score was 3.10 during the ST, sitting condition, and 6.82 during the DT, stepping condition. There was no significant difference in performance among the groups. The paired t-test analyses for the 19 participants assigned to the 30-trial condition revealed a t = −5.33, P = <0.001; for the 18 participants assigned to the 40-trial condition, a t = −3.78, P = 0.001; and for the 22 participants assigned to the 60-trial condition, a t = −4.96, P < 0.001.

DISCUSSION

The purpose of the present study was twofold: first, to evaluate the reliability of a single index of executive functioning (task switching) across multiple task parameters and at multiple time points; and second, to assess DT interference in task switching that arises when individuals simultaneously perform a motor task and a cognitive task. Analyses of the reliability of global switch cost scores were shown to have acceptable reliability across a period of 1 wk (range = 0.64–0.86) and that the reliability improved with longer test-trial lengths. The 1-wk reliability estimates of the switch-task scores found in the present study are considerably higher than those reported by Broglio et al. (6), who evaluated individual tests that comprise three computer-based test batteries across test–retest periods of 5 and 45 d. Intraclass correlation coefficients over a 5-d test–retest period ranged between 0.39 and 0.61 on the ImPACT, 0.39 to 0.66 on the Concussion Sentinel, and 0.03 to 0.66 on the Concussion Resolution Index. Intraclass correlation coefficients over a 45-d test–retest period ranged between 0.15 and 0.39 on the ImPACT, 0.23 to 0.65 on the Concussion Sentinel, and 0.15 to 0.66 on the Concussion Resolution Index. Although the test–retest reliability estimates of the switch cost scores were, at best, only marginally acceptable for one test configuration across a period of approximately 7 months (range = 0.32–0.59), the results are encouraging when compared with the reliabilities of the computer-based test batteries over a longer test–retest period and given the relatively small sample size evaluated. Further assessment of the switch cost index reliability is indicated to more closely replicate typical concussion assessment time points.

The participants’ switch-task performance was affected systematically when performed simultaneously with a stair-stepping task (modified Harvard Step Test). The switch cost interference was substantial, increasing from 86.38 ms in the ST condition to 209.75 ms in the DT condition, and suggestive that the switch index may provide researchers with a common metric to assess the influence of different types of perturbation on cognitive function. Furthermore, the magnitude of the effect was virtually identical across the participants assigned to three different test-length conditions.

The observed interference could be explained by the capacity theories of attention. Theorists argue that individuals have a limited amount of attentional resources that can be divided among several tasks. As long as the attentional resources exceed the attentional demands of the concurrent tasks, performance of the tasks will be allowed to proceed without degradation. If, however, the attentional demands of multiple tasks exceed the available attentional resources, interference occurs (13,28). Further support for a capacity explanation of DT interference comes from researchers who have reported that individuals experiencing a concussion may show a decline in the capacity of the attentional resources that are available (7,9,10). Additional research in which the level or type of motoric involvement is manipulated will be required to determine the sensitivity of the switch index. As expected, the number of response errors was higher during the DT conditions than ST conditions; however, the overall percentage of errors was relatively low under both the single (3.15%) and dual (6.82%) conditions. Of note was the lack of a speed–accuracy tradeoff between the switch cost scores and response errors, which often occurs when individuals attempt to maintain or reduce RT but at the cost of increasing errors. However, we postulate that the error rate could be much higher when evaluating postconcussion patients in which there is cognitive impairment as Geurts et al. (12) previously observed. In the present study, DT demands led to a general increase in mental processing time and response errors. These could be important measures when tracking concussion recovery.

The results obtained are similar to those of our prior research studies in which balance perturbations were produced by a standardized measurement system (18). The rationale for selecting the stair-stepping protocol in the present study was to determine whether a low-cost alternative methodology, appropriate for field applications, was feasible. Presently, the use of widespread neuropsychological testing for concussion effects is limited by the costs, computer equipment availability, and interpretation of test data. In addition, the DT approach used in the present study was designed to simulate the conditions that challenge an athlete’s decision making during game conditions, that is, simultaneously challenging the cognitive and motor systems of the brain closely mimicking physical activity.

There are some important clinical implications for this study for the use of the switch cost index in concussion assessment. First and foremost is that all assessment tools undergo rigorous psychometric testing before clinical implementation. We have taken a measured and stepwise approach to the development of the DT methodology using a variety of different testing instruments, adjusting the timing of the test, and using different physical conditions (3,8,11). This has allowed us to further refine the development of a dual-tasking concussion assessment tool for clinical use. Furthermore, current concussion testing methodologies assess neurocognitive and motor abilities in isolation. A DT testing methodology potentially allows for a truer assessment on how an individual is recovering from concussion and his or her readiness for play by combining the cognitive and motor assessments simultaneously. It is recognized that the next step in the evaluation of this testing methodology must include the specificity and sensitivity to concussion to further demonstrate its psychometric properties and possible use in clinical practice.

There are several limitations to the present study. The participants were healthy, young, nonconcussed adults; as such, the findings cannot be generalized to the concussed individuals. In addition, there was a broad range in the time between testing at baseline and in the third session. Because of the large variation between test dates, it may be difficult to determine the true stability reliability of the DT test. In addition, the roles of the participants’ motivation and level of fatigue were not assessed to determine their effect on the study outcomes.

The stability reliability of the global switch cost index observed in the present study suggests that it may be a useful measure for clinicians who assess effects of concussion on athletes’ mental function. It should be cautioned, however, that the switch task provides information about only one of the three interrelated components of executive function. It remains to be determined whether tests of working memory and response inhibition yield similar test–retest reliabilities. At this juncture, however, the switch cost index is attractive because it does not appear to be influenced by the effects of practice, which are often seen in the tests of other components of executive function. The magnitude of the DT costs in cognitive function observed in nonconcussed young adults was sizable, and the findings of the present study bode well for the use of the global switch cost index and the DT approach to assess the effects of concussion or other events that influence the CNS.

No funding was provided to conduct this study. None of the authors report potential conflict of interest. The results of the present study do not constitute endorsement by the American College of Sports Medicine.

REFERENCES

1. Azouvi P, Jokic C, Van der Linden D, Marlier M, Bussel B. Working memory and supervisory control after severe closed-head injury: a study of dual task performance and random generation. J Clin Exp Neuropsychol. 1996; 18 (3): 317–37.
2. Barr WB, McCrea M. Diagnosis and assessment of concussion. In: Webbe FM, editor. The Handbook of Sport Neuropsychology. New York: Springer; 2011. pp. 91–111.
3. Beauchet O, Dubost V, Gonthier R, Kressig RW. Dual-task–related gait changes in transitionally frail older adults: the type of the walking-associated task matters. Gerontology. 2005; 51: 48–52.
4. Belanger HG, Vanderploeg RD. The neuropsychological impact of sport-related concussion: a meta-analysis. J Int Neuropsychol Soc. 2005; 11: 345–57.
5. Broglio SP, Ferrara MS, Macciocchi SN, Baumgartner TA, Elliott R. Test–retest reliability of computerized concussion assessment programs. J Athl Train. 2007; 42 (4): 509–14.
6. Broglio SP, Tomporowski PD, Ferrara MS. Balance performance with a cognitive task: a dual-task testing paradigm. Med Sci Sports Exerc. 2005; 37: 689–95.
7. Catena RD, Van Donkelaar P, Chou L. Cognitive task effects on gait stability following concussion. Exp Brain Res. 2007; 176 (1): 23–31.
8. Cedrus. SuperLab. 2.2 ed. San Pedro (CA): Cedrus Corporation.
9. Cicerone KD. Attention deficits and dual task demands after mild traumatic brain injury. Brain Inj. 1996; 10 (2): 79–90.
10. Ebersbach G, Dimitrijevic MR, Poewe W. Influence of concurrent tasks on gait: a dual-task approach. Percept Mot Skills. 1995; 81: 107–13.
11. Etnier JL, Chang Y-K. The effect of physical activity on executive function: a brief commentary on definitions, measurement issues, and the current state of the literature. J Sport Exerc Psychol. 2009; 31: 469–83.
12. Geurts ACH, Ribbers GM, Knoop JA, van Limbeek J. Identification of static and dynamic postural instability following traumatic brain injury. Arch Phys Med Rehabil. 1996; 77: 639–44.
13. Kahneman D. Attention and Effort. Englewood Cliffs (NJ): Prentice-Hall, Inc.; 1973.
14. Li KZH, Lindenberger U. Relations between aging sensory/sensorimotor and cognitive functions. Neurosci Biobehav Rev. 2002; 26: 777–83.
15. Lovden M, Schaefer S, Pohlmeyer AE, Lindenberger U. Walking variability and working-memory load in aging: a dual-process account relating cognitive control to motor control performance. J Gerontol. 2008; 63B (3): P121–P8.
16. May BE, Tomporowski PD, Ferrara MS. Effects of backpack load on balance and decisional processes. Mil Med. 2009; 174 (12): 1308–12.
17. Monsell S, Driver J, editors. Control of Cognitive Processes: Attention and Performance XVIII. Cambridge (MA): The MIT Press; 2000.
18. NEUROCOM. NeuroCom Systems Operator’s Manual. 8.0 ed. Clackamas (OR): Neurocom International, Inc.; 2001. pp. PO-1–10.
19. Parker TM, Osternig LR, Lee H, Van Donkelaar P, Chou L. The effect of divided attention on gait stability following concussion. Clin Biomech. 2005; 20: 389–95.
20. Parker TM, Osternig LR, Van Donkelaar P, Chou L. Gait stability following concussion. Med Sci Sports Exerc. 2006; 38 (6): 1032–40.
21. Resch JE, May B, Tomporowski PD, Ferrara MS. Balance performance with a cognitive task: a continuation of the dual-task testing paradigm. J Athl Train. 2011; 46 (2): 170–5.
22. Rogers DR, Monsell S. Costs of a predictable switch between simple tasks. J Exp Psychol Gen. 1995; 124: 207–31.
23. Schaefer S, Schumacher V. The interplay between cognitive and motor functioning in healthy older adults: findings from dual-task studies and suggestions for intervention. Gerontology. 2011; 57: 239–46.
24. Schatz P. Computerized neuropsychological assessment in sport. In: Webbe FM, editor. The Handbook of Sport Neuropsychology. New York: Springer; 2011. pp. 173–86.
25. Schatz P, Putz BO. Cross-validation of measures used for computer-based assessment of concussion. Appl Neuropsychol. 2006; 13 (3): 151–9.
26. Shumway-Cook AM, Woollacott MH. Attentional demands and postural control: the effect of sensory context. J Gerontol A Biol Sci Med Sci. 2000; 55: M10-6.
27. Wasylyshyn C, Verhaeghen P, Sliwinski MJ. Aging and task switching: a meta-analysis. Psychol Aging. 2011; 26 (1): 15–20.
28. Wickens CD. Multiple resources and mental workload. Hum Factors. 2008; 50 (3): 449–55.
Keywords:

COGNITION; DUAL TASKING; EXECUTIVE FUNCTION; RETURN TO PLAY

©2013The American College of Sports Medicine