Baseline Neuropsychological Testing in Managing Sport-Related Concussion

Does It Modify Risk?

Abstract

Baseline neuropsychological testing is mandated at various levels of play for hundreds of thousands of athletes each year. This paper reviews the risks associated with sport-related concussion, and the clinical validity and reliability data for the most commonly used baseline test, the ImPACT program. There is no evidence to suggest that the use of baseline testing alters any risk from sport-related concussion, nor is there even a good rationale as to how such tests might influence outcome. Given the poor sensitivity and low reliability of these measures, they have an associated high false negative rate (i.e., classifying a player's neurocognitive status is normal, when in fact, it is not). The use of baseline neuropsychological testing, therefore, is not likely to diminish risk, and to the extent that there is a risk associated with "premature" return-to-play, the use of these measures even may increase that risk in some cases.

Introduction

The use of baseline neuropsychological testing in the management of sport-related concussion has gained widespread acceptance, largely in the absence of any evidence suggesting that it modifies risk for athletes. The use of these tests currently is mandated in football and hockey programs (among other sports) at the elementary, high-school, and collegiate levels throughout the United States, and in the National Football League (NFL) and National Hockey League (NHL). In a National Collegiate Athletic Association (NCAA) memorandum dated April 29, 2010, the NCAA Executive Committee outlined a policy that institutions must have a concussion management plan on file, and the memorandum goes on to state that "neuropsychological testing (e.g., computerized standard paper and pencil) has been shown to be effective in the evaluation and management of concussion." There even have been attempts by state legislatures to mandate or strongly encourage the use of such programs. For example, New Jersey Senate resolution no. 74, introduced June 24, 2010, states "Whereas, baseline cognitive testing programs, such as the Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) program, are specifically designed for the management of sports-related concussions…In order to ensure the safety of student-athletes, it is imperative that school districts implement a baseline cognitive testing program[…]."

The topic of sport-related concussion has been highly visible in the media over the last few years, as well, and the issue of concussion management in the NFL even has reached the point of Congressional hearings. What consistently is missing from these debates, however, is a rational, empirically-based discussion of the true risks associated with sport-related concussion, the potential for any given management strategy to modify these risks, and the actual evidence that such risk modification can be achieved. In addition, there has been little attention paid to the clinical validity or psychometric characteristics of baseline tests, despite widespread use of these measures. The purpose of this review is to evaluate critically the use of baseline testing in the context of potential risk modification. In addition, a specific focus on the clinical validity and psychometric data for the ImPACT program was undertaken in this context, as this is by far the most widely-used of the baseline tests, and an exhaustive review of data from all such tests is beyond the scope of this paper.

Risk Identification

The first step in any attempt to modify risk via a management strategy is to identify the risks associated with a particular injury, as well as to identify the incidence of each risk. We recently reviewed the risks that have been identified as real or potential outcomes from sport-related concussion (²⁰). These included injuries that resulted in death or permanent neurological injury, same-season repeat concussion, delayed/atypical recovery, and late-life effects of multiple concussions or repetitive head trauma. To quantify the incidence of these risks, American football was used as the sport with the most available data, as well as being a sport with one of the highest rates of concussion.

It has been estimated that there are approximately 1.8 million annual participants in American football, at all levels of play (²⁵). This is probably a reasonable estimate. The National Federation of State High Schools Association lists a participation number in high-school football of approximately 1,100,000 for the 2009 to 2010 season, and the most recent NCAA data (2007) indicate an annual participation rate of approximately 62,000. The Pop Warner Web site reports more than 400,000 participants, and there are between 1,700 and 2,500 players in the NFL, approximately, depending on the time of year. These numbers obviously are not exhaustive but allow a reasonable estimate of annual participation in American football upon which to base epidemiological estimates regarding the incidence of specific risks.

Death or Permanent Brain Injury, Including Diffuse Cerebral Swelling (or "Second Impact Syndrome")

The National Center for Catastrophic Sport Injury Research at the University of North Carolina (²⁵) maintains a database of sport-related injuries to the brain and spinal cord that resulted in permanent disability (determined by residual impairment at 2 to 3 months postinjury) or death. In reviewing 10 seasons of American football at all levels of play (ending in 2006), there were a total of 50 cases of permanent disability and 38 deaths from cerebral injuries. The great majority of the deaths were from acute subdural hematoma. Based upon an annual participation rate of 1.8 million, this corresponds to 88 instances of permanent disability or death for 18,000,000 player-seasons, or one instance for every 205,000 player-seasons. For a squad size of 100 players, this would translate to one such injury every 2,050 seasons. As almost all of these injuries occur as the result of the immediate consequences of a single traumatic brain injury (typically resulting in intracranial bleeding), the risk of this type of injury obviously cannot be modified by baseline testing.

The very rare phenomenon of diffuse cerebral swelling has been referred to as "second impact syndrome" because of the belief by some that it is caused by incurring a second concussion before the effects of the initial concussion have resolved. This syndrome often is cited by distributors of computerized tests as a rationale for baseline testing. On the ImPACT Web site, for example (⁸), the following statement is posted: "Suffering a second blow to the head while recovering from an initial concussion can have catastrophic consequences as in the case of 'Second Impact Syndrome,' which has led to approximately 30 to 40 deaths over the past decade."

During the 10-yr period reviewed previously in this article, there was only a single death attributed to diffuse cerebral swelling at all levels of play for American football. This constitutes a risk of one event for every 18 million player-seasons. If diffuse cerebral swelling were preventable by baseline testing, it would require 18 million baseline assessments before a single case of diffuse cerebral swelling was encountered.

There is a host of evidence, however, to suggest that the phenomenon of diffuse cerebral swelling is not preventable by any management technique. It is clear, first of all, that a second injury is not necessary for this syndrome to occur. In the only systematic review of the literature on diffuse cerebral swelling as the result of sport-related injury, McCory and Berkovic (¹⁸) identified just 17 cases, only 5 of which involved a repeat injury. Far more cases have been reported in the pediatric neurosurgical literature, and the typical clinical characteristics of these cases involve a minor head injury followed by a period of lucidity before marked worsening of neurological status. This has been described as "malignant brain edema" by Bruce et al. (⁴). Additional series of diffuse cerebral swelling in children following a single minor brain trauma have been reported by Snoek et al. (²³) and Mandera et al. (¹⁵). This phenomenon may be related to a form of familial hemiplegic migraine and due to a calcium channel subunit receptor gene mutation (^11,24).

In addition to the overwhelming evidence to suggest that diffuse cerebral swelling occurs most often from a single injury and may be related to a genetic mutation, the hypothesis that a mechanistic etiology is playing a role seems unlikely, given how commonly players are exposed to additional head injuries soon after incurring a concussion. In a recent study of 635 concussions in high-school and college athletes (¹⁶), it was reported that approximately 40% of players were returned to play while still actively symptomatic. This would suggest that, nationwide, tens of thousands of players are returned to play while still symptomatic every year in American football alone. If diffuse cerebral swelling were attributable to two closely-spaced injuries, it stands to reason that it should be observed much more commonly in this setting.

Same-Season Repeat Concussion

It frequently is suggested, primarily on the basis of animal models, that the brain is in a state of vulnerability following a concussion (^5,12). If this were true, one of the potential risks following a sport-related concussion would be vulnerability to a second concussion. Surprisingly, however, few studies have investigated the risk of same-season repeat concussion. In one early survey involving nearly 18,000 high-school and college football players, almost 15% of players who had experienced an initial concussion reported having incurred a same-season repeat concussion (⁷). The inter-injury intervals were not reported in this paper. In a more closely studied sample enrolled in a prospective controlled study that involved 25 colleges over three seasons, only 6.5% of the injured sample incurred a same-season repeat concussion (⁶). However, 90% of these occurred within 10 d of the initial concussion. Macciocchi et al. reported on a separate sample of 195 concussed players, and in this sample only 2.6% experienced a same-season repeat concussion (¹⁴). For those with a repeat concussion in this study, the average inter-injury interval was 33 d (range 14 to 70 d).

Based on the data from the latter two prospective studies, it does not appear that the rate of same-season repeat concussion is markedly higher than the base rate of concussion in American football (typically reported to be between 3% and 6% of players each season). There does appear to be possibly a period of vulnerability within the first 10 d postinjury, but this needs to be confirmed with additional study. Presuming a base rate of concussion at 5% of player-seasons for American football and a same-season repeat concussion rate of 5% of initial concussions, a same-season repeat concussion would be expected to occur approximately once every 400 player-seasons.

It also remains unclear whether baseline testing could play any role in diminishing the risk of same-season repeat concussion. In a recent study by McCrea et al., same-season repeat concussions actually were observed more frequently in the sample of players who were held out of play until symptom-free than in the sample of players who were returned to play "prematurely" (¹⁷). Of the 24 same-season repeat concussions that were observed in this sample of 635 concussed athletes, 22 occurred in the group for whom a symptom-free waiting period was observed, and only two in the group who returned to play while still symptomatic. While this finding probably should not be taken as evidence that returning to play while symptomatic is advisable, it suggests that any vulnerability to a second concussion is more likely to be based on some factor other than resolution of symptoms. In this study (which had some overlap in subjects with the Guskiewicz et al. 2003 study) (⁶), approximately 80% of the same-season repeat concussions occurred within 10 d of the initial injury.

Overall, therefore, same-season repeat concussions are relatively infrequent (approximately once every 400 player-seasons), do not appear to result in a markedly different outcome than single concussions, and do not appear to be prevented by ensuring that players are asymptomatic before return-to-play. This certainly diminishes the potential for the use of baseline testing in modifying the risk of this injury, which does not seem to be associated with any serious adverse consequences that differ from single concussions.

Delayed/Atypical Recovery

There also is frequent mention made, primarily within the popular media and by purveyors of baseline testing, of long-lasting or "permanent" symptoms or impairments from concussion. There essentially is no support for this in the scientific literature. The natural history of any disorder is best understood via prospective controlled studies. Sport-related concussion is a relatively easy disorder to study in this fashion, as the rate of injury is fairly predictable, allowing for preseason testing of cognition, symptoms, balance, etc., with postinjury assessments that can be made and compared with matched controls (uninjured teammates). Several of these studies have been performed to date, and they are consistent in reporting a rapid resolution of symptoms, balance impairment, and cognitive deficits. A review of the baseline testing literature reveals that concussed and control players often were not statistically different on these measures by 48 h postinjury, and no such study has documented reliable group differences on cognitive variables 7 d postinjury (²³). A meta-analysis of the literature reached the same conclusion (¹).

These group studies do not rule out the occasional case of prolonged recovery, although such cases must be relatively rare, as they are insufficient to produce statistically significant differences in group studies. This particularly is true for cognitive test data, and there obviously is no way that baseline testing could prevent a prolonged or atypical recovery, although in theory, baseline testing could help track recovery (presuming the tests were sufficiently reliable). Since there never has been a prospective controlled study that has identified a subsample of individuals with persistent cognitive impairment, it is unclear whether such a condition even exists. It would be difficult to advocate the routine use of baseline neurocognitive testing simply to track a condition whose existence is uncertain.

Late-Life Consequences of Repeated Concussions/Repetitive Head Trauma

There has been increasing concern that experiencing multiple concussions over a long playing career or simply being exposed to repetitive head trauma over that period of time could lead to diminished cerebral reserve and increased vulnerability to the early clinical expression of neurodegenerative diseases like Alzheimer's disease and Parkinson's disease. It also is conceivable that repetitive trauma can result in distinct neuropathological changes (e.g., chronic traumatic encephalopathy), which could be the mechanism underlying diminished reserve or a separate factor in contributing to late-life neurocognitive decline. This issue has been reviewed at length elsewhere (²⁰), and since such a consequence clearly cannot be modified by baseline testing, it will not be addressed further in this paper.

Summary of Risks

Thus far, the serious risks associated with sport-related concussion appear to be unlikely to be modified by baseline testing. The vast majority of deaths and long-lasting neurological impairments are secondary to acute subdural hematomas, which cannot be prevented by baseline testing. Diffuse cerebral swelling, also termed "second impact syndrome" is extremely rare, does not require two separate injuries, and likely is due to a genetic abnormality. There was only one case of this in a recent review of 10 yr of American football, amounting to a risk of one event for every 18 million player-seasons. Same-season repeat concussion occurs approximately once every 400 player-seasons in American football. The only study exploring possible risks leading to repeat concussions that has been published to date concluded that the risk was not diminished by ensuring that players were symptom-free before return-to-play. It is unclear, therefore, what risks possibly could be modified by the routine use of baseline testing, despite the ubiquity of this practice.

The Science Behind Baseline Testing-Impact as a Model

The issues relating to the clinical validity and utility of baseline neurocognitive testing were addressed in a lengthy review article published in 2005 (²¹) that examined all available data from prospective controlled studies for test batteries in use at that time. That article concluded that no test battery had met the necessary criteria to support a clinical application at that time. It is of interest that little additional research in this area has been published since, although the use of these batteries has grown exponentially. Rather than reiterate all of the material presented in that article, the issues can be best illustrated by a specific focus on the most popular of the baseline testing batteries: the ImPACT test.

The Hypothesis Behind Baseline Testing

The use of baseline testing is predicated on the following beliefs: 1) concussion results in impairment of cognition; 2) it is important to ensure that players are free of the effects of concussion before return-to-play; and 3) measurement of cognition at baseline will allow the reliable detection of cognitive impairment following concussion. Moreover, it often is assumed (and this assumption is reinforced by purveyors of baseline tests) that players will deny experiencing symptoms following concussion, and therefore the use of baseline testing is an important objective measurement of residual impairment.

The issue of whether players need to be free of any residual effects of concussion before returning to play remains purely speculative. The only study of this in the literature to date (¹⁷) was observational in design, but no risk of "premature" return-to-play was identified. Leaving this issue aside for the moment, it should be apparent that, given the costs and time involved in baseline testing, there should be some clearly-defined added benefit of baseline testing over the use of symptom checklists, which are cost-effective and easily/quickly administered.

As delineated in the Randolph et al. (²¹) review on the topic, the following criteria should be met by a baseline battery before it is adopted for routine clinical use:

1. It should be proven to be sensitive to the effects of concussion via a prospective controlled study and ideally be capable of measuring impairment in the concussed group after self-reported symptoms have resolved. Obviously, baseline testing adds nothing to return-to-play decision-making in an athlete who is still reporting subjective symptoms.
2. The test should be reliable, and a clear-cut algorithm for identifying "impairment" should be available, from test-retest data using clinically relevant time intervals. The use of Reliable Change Index (RCI) scores is widespread in neuropsychology and allows for a probability to be derived for a given change score. In other words, for a given test there might be data indicating that a decline of 10 points or more over 1 yr occurs less than 5% of the time due to chance in the normal older population. Therefore, when observing a decline of this magnitude or larger, the clinician can conclude there is a 95% probability that the decline was due to something other than chance. Because RCI is calculated on the basis of test-retest stability coefficients and these coefficients tend to decline as retest intervals increase, it is important to derive retest stability coefficients from clinically-relevant time intervals.

Does ImPACT Meet These Criteria?

The short answer is clearly no. As discussed previously in this article, the most appropriate way of demonstrating test sensitivity is via a prospective controlled study. In this context, such a study requires baseline testing a large sample of athletes, and then retesting concussed athletes in comparison with noninjured teammates, tested concurrently over the same time intervals. To demonstrate that baseline testing provides information beyond that derived from simple symptom checklists, it would be most appropriate to delay the first postconcussion test session until the concussed player is symptom-free. The importance of the prospective controlled design is to account for other factors that could influence postinjury testing (e.g., practice effects, effects of simply participating in a contact sport over the course of a season).

There does not appear to be even one prospective controlled study of the current version of ImPACT (version 2.0) that has ever been published in the peer-reviewed literature, nor is such information available through the ImPACT Web site. There is only one prospective controlled study of an earlier version (1.0) of the battery that was published (¹³). In this study, data were reported for 64 concussed high-school athletes and 24 control subjects. The concussed and control groups were mismatched in terms of gender and sport. The concussed group was 94% male, and the control group was 67% male. The majority of the concussed group consisted of football players (69%), whereas only 8% of the control group comprised football players (92% were swimmers). The authors for some reason chose to report data for only one index score from the ImPACT battery, the memory composite. The control group and the concussed group differed significantly in their baseline performance on the memory composite score, and there were no direct postinjury comparisons between the two groups. The concussed group was reported to fall below their preseason baseline at testing done 36 h, 4 d, and 7 d postinjury. The largest decline in performance was at the 36-h testing, with a mean decline of 8.4 points. It should be noted, however, that the 90% confidence interval for change on this measure was reported to be 12.82 points (⁹). This would suggest that, even at the 36-h test interval, the majority of the injured athletes were within the 90% test-retest confidence interval of measurement reliability.

Since there has been no prospective controlled study of the current version of ImPACT published to date, there also are obviously no data that would allow a determination to be made as to whether ImPACT is capable of detecting impairment in a significant percentage of athletes once symptom-free as measured by a symptom checklist. The authors of the ImPACT attempted to address this "value-added" issue in a study that involved 122 concussed high-school and college athletes compared with 70 nonconcussed athletes who took the ImPACT twice within 1 wk (²⁶). Performance on the ImPACT cognitive measures and on the 22-item symptom scale (PCS) that is included with ImPACT was examined. Concussed players were tested at baseline and within 2 d postinjury. The control test-retest data were used to calculate RCI with 80% confidence intervals, and these were applied to the neurocognitive and PCS scores for the injured group. It should be noted that, like the study cited above, the concussed and control groups in this report were mismatched in terms of both gender and sport.

The authors did not examine ImPACT scores directly for those athletes who were symptom-free. They reported that 64% of the concussed sample exhibited an elevation in their PCS symptom score that was outside of the RCI, with 0% false positives in the control group. They indicated that 83% of the concussed sample had at least one ImPACT test score that was below the RCI, but the false positive rate was 30%. Requiring at least one abnormal ImPACT score in addition to an abnormal symptom score increased the "true positive" rate in the concussed sample from 64% to 81%, but it also increased the "false positive" rate in the controls from 0% to 17%.

In a more recent study that lacked control subjects, ImPACT scores were examined once a sample of 21 concussed athletes became symptom-free (³). The authors reported that 38% of the concussed athletes were classified as impaired on at least one ImPACT variable once symptom-free. Considering the 30% false positive rate in the first study, this suggests that an additional 8% of subjects were identified as true positives via ImPACT testing, once symptoms had resolved. This is very close to the percentage of concussed subjects (7%) identified as impaired on the basis of paper-and-pencil testing in another study, once the false positive rate was controlled for (¹⁶).

The bulk of the evidence, therefore, suggests that ImPACT is not particularly sensitive to the effects of concussion, particularly once subjective symptoms have resolved. These types of investigations, however, cannot substitute for an appropriate prospective controlled study to explore the actual sensitivity of the test in symptom-free athletes, and this type of study has yet to be conducted for ImPACT.

Reliability and Individual Decision-Making

While the sensitivity of a test like ImPACT is explored best through a prospective controlled group study, the intended application of the test is in the context of individual decision-making. Given that there is a certain amount of variability inherent in any psychological test score (or biological measurement in general), it is important to develop an appropriate algorithm for interpreting change.

In the field of neuropsychology, the use of RCI scores is an accepted methodology for decision-making regarding individual change. As noted previously, this requires obtaining test-retest data in the population of interest over time intervals relevant to the intervals involved in clinical decision-making. This is because it cannot be assumed that reliability over long time intervals is as high as it is over short time intervals, and if practice effects exist, they typically are of a higher magnitude over shorter time intervals.

Once data for clinically relevant retest intervals are obtained, RCI can be calculated. Assuming that there are no significant practice effects, the RCI will be centered around the baseline score, and the magnitude of a given RCI will depend upon the test-retest coefficient. The more reliable the test, the smaller the RCI will be. For individual decision-making with psychological tests, it has been recommended that the test-retest coefficient should be above 0.80 (¹⁹). Coefficients significantly lower than this result in RCI scores that are so broad that they require dramatic score changes to exceed the confidence interval.

For the current version of the ImPACT battery, Iverson et al. calculated RCI scores from a sample of 56 adolescents and young adults, with a mean age of 17.6 (range 15 to 22), with an average retest interval of 5.8 d (range 1 to 13) (¹⁰). This appears to be the study upon which the RCI scores are calculated for the "Clinical Report" that is generated by the ImPACT program, although this is not specified. There are two additional recent publications of test-retest data on this version of ImPACT. A study by Broglio et al. (²) involved 118 students drawn from a general university population and tested 45 d apart. Another study by Schatz (²²) reported on 95 collegiate athletes tested approximately 2 yr apart. The correlation coefficients reported in these three separate studies are listed in the Table 1. It is clear that the coefficients obtained from more clinically-relevant time intervals (Broglio and Schatz studies) are considerably lower than those utilized by the ImPACT; therefore, the RCI scores would be correspondingly much higher. In fact, the data from the Broglio and Schatz studies suggest that the ImPACT battery lacks sufficient reliability to be useful in individual decision-making. For example, utilizing the Broglio reliability coefficient for verbal memory, an athlete's score would have to decline by more than 18 points to fall outside of the 90% confidence interval, as opposed to a decline of 11 points using the reliability coefficient reported by Iverson et al.

Table: Test-Retest Reliability Coefficients of ImPACT.

In addition to the lack of reliability illustrated by these published data, it should be pointed out that RCI is calculated for a single score and represents the probability of an "abnormal" finding for that given score. When dealing with multiple scores, the probability of a single "abnormal" finding will continue to increase, depending upon the number of scores reported. If the scores are orthogonal (do not covary), the probability will be additive. That is, if the probability of a false positive is 10% with any given score, it will be 20% in any one of two scores, 30% in any one of three scores, and so on. This was illustrated in the Broglio et al. study, where up to 40% of normal subjects were classified as impaired on at least one ImPACT change score during follow-up testing.

Since the ImPACT system now reports a total of five cognitive scores (the four listed previously, plus "impulse control"), it essentially is impossible to calculate an overall probability of "impairment" from the data provided. This reduces the utility of the tool from the level of a clear-cut statistical algorithm to that of clinical guesswork in most cases.

Conclusion

Despite the widespread use of baseline neuropsychological testing in the management of sport-related concussion, there appears to be essentially no evidence in the medical literature to suggest that this approach has modified any associated risks. In addition, a review of the known risks involved does not reveal any clear rationale for how baseline testing might modify risk. The burgeoning use of these measures appears to be largely driven by concerns around liability and/or a misunderstanding of the etiology and frequency of specific risks associated with sport-related concussion. It should be noted that the use of these tests is distinct from standard clinical neuropsychological assessment, which would be appropriate in the case of lingering cognitive or emotional symptoms following traumatic brain injury. These may be multifactorial in etiology and indicate the need for differential diagnosis and appropriate treatment planning. This fundamentally is distinct from baseline testing and should be carried out by a board-certified clinical neuropsychologist.

It is clear that, to date, the baseline tests used in sport-related concussion management programs lack sufficient clinical validity and reliability for their intended purpose. These data were reviewed in some detail for the test that is by far the most widely used, the ImPACT computerized battery. There is not one single prospective controlled study in the peer-reviewed literature involving the current version of this test to establish sensitivity, and the reliability as reported by independent investigators appears to be far too low to be useful for individual decision-making. In addition, the ImPACT battery reports five separate cognitive measures without any associated algorithm for classifying overall performance as "impaired" or "recovered." This essentially negates the utility of the baseline testing paradigm, which is intended to provide a statistically-based classification of performance.

Finally, given the relatively low sensitivity and poor reliability of this test, ImPACT is likely to have a "false negative" rate that is as high as the "false positive" rate. In this setting, a false negative would be classifying an athlete as recovered, when in fact he or she was still experiencing cognitive impairment secondary to concussion. The true false negative rate can never be determined, as there is no "gold standard" for measuring cerebral integrity postconcussion. The false positive rate, on the other hand, can be established clearly from the percentage of uninjured controls that are classified as impaired. Since the false positive rate appears to be 30% to 40% of subjects for ImPACT (^2,26), it is reasonable to assume that the false negative rate may be comparable.

To the extent that there is any risk of "premature" return-to-play, the determination of recovery based on the use of an instrument with such a high false negative rate may paradoxically increase risk, by returning athletes who might otherwise be withheld from play longer in the absence of such data. Regardless of whether risk can be reduced via the use of a well-validated baseline test under any circumstances, until appropriate clinical validity and psychometric data are established for one of these batteries, team medical personnel may be better advised to rely upon their own clinical judgment, in conjunction with a validated symptom checklist, in making return-to-play decisions. The use of a baseline neurocognitive test with poor sensitivity and inadequate reliability in this context may lead to an unfounded sense of security that recovery has taken place, and this is true particularly for clinicians who lack adequate training in psychometrics to understand the limitations of such an instrument.