Secondary Logo

Stability and Validity of the Veterans Health Administration's Traumatic Brain Injury Clinical Reminder Screen

Vanderploeg, Rodney D. PhD; Belanger, Heather G. PhD

Journal of Head Trauma Rehabilitation: September/October 2015 - Volume 30 - Issue 5 - p E29–E39
doi: 10.1097/HTR.0000000000000095
Original Articles
Free

Objective: Determine the test-retest stability and validity of the Veterans Health Administration's Traumatic Brain Injury Clinical Reminder (VHA-TBI-CR) screen.

Design: Criterion-standard.

Setting: Veterans Health Administration.

Participants: Ninety-five Operations Enduring Freedom and Iraqi Freedom veterans who had previously completed the VHA-TBI-CR screen were recruited from the Tampa Veterans Affairs Medical Center. Participants were primarily male (89.5%), with a mean age of 41.6 (SD = 11.14) years; 86% completed at least some college and over half were employed (56.8%). The majority of the participants were enlisted (83.2%).

Main Outcome Measures: Phi coefficients of stability were calculated comparing the original clinically administered VHA-TBI-CR screen, with a paper-and-pencil version of the screen completed as part of the current study. Validity analyses were conducted by comparing the results from both screens to the study's TBI Identification Clinical Interview findings regarding a history of deployment-related traumatic brain injury (TBI). Sensitivity, specificity, positive predictive value, and negative predictive value of the screen were computed. Kappas were also calculated to examine agreement between screens and the study's “criterion standard” TBI diagnosis.

Interventions: None.

Results: The VHA-TBI-CR screen had poor test-retest stability (ϕ = 0.34). The clinically administered VHA-TBI-CR screen had only moderate sensitivity (0.61) but relatively good specificity (0.88). Historical drift was present in the data. Clinical VHA-TBI-CR screens completed within the past 24 months were less accurate (κ = 0.27) than screens completed between 25 and 38 months earlier (κ = 0.63) or those completed between 38 and 57 months earlier (κ = 0.53).

Conclusion: The VHA-TBI-CR screen has poor test-retest stability and poor validity when compared with a semistructured TBI identification interview. In this sample, sensitivity was unacceptably low such that the screen failed to identify 39% of veterans who were determined to have had a deployment-related TBI.

Department of Mental Health and Behavioral Sciences, James A. Haley Veterans Hospital, Tampa, Florida; Departments of Psychology and Psychiatry and Behavioral Neurosciences, University of South Florida, Tampa, Florida; Defense and Veterans Brain Injury Center, Silver Spring, Maryland; and HSR&D/RR&D Center of Excellence Maximizing Rehabilitation Outcomes, James A. Haley Veterans Hospital, Tampa, Florida).

Corresponding Author: Rodney D. Vanderploeg, PhD, Psychology Service (116B), James A. Haley Veterans Hospital, Tampa, FL 33612 (Rodney.Vanderploeg@va.gov).

The research reported here was supported by the Department of Veterans Affairs, Veterans Health Administration (VHA), through SDR HSR&D #08-411–-TBI Screening Instruments and Processes, and the Defense and Veterans Brain Injury Center (DVBIC). Further support was provided by the James A. Haley Veterans Hospital. The views expressed herein are those of the authors and do not necessarily reflect the views or the official policy of the Department of Veterans Affairs or US government. The authors thank Joseph Gutmann, MD, and Keith White, MD, who served as the research assistants conducting the semistructured clinical interviews. The authors also thank Shirley Groer, PhD, who was originally coprincipal investigator, and Valerie Larson, who served as the study coordinator for the SDR HSR&D #08-411 grant that supported this study.

No work resembling the enclosed article has been published or is being submitted for publication elsewhere. All authors have certified in writing that they have read the final version of the manuscript and have made a substantial contribution to the research and final manuscript in accordance with the guidelines of the Uniform Requirements. All authors declare that the answer to the questions on your competing interest form are all “No” and, therefore, have nothing to declare.

The authors declare no conflicts of interest.

IN APRIL 2007, the Veterans Health Administration (VHA) implemented a series of screening instruments for various deployment-related conditions in the Operation Enduring Freedom/Iraqi Freedom/New Dawn (OEF/OIF/OND) cohort. One of these screens was for traumatic brain injury (TBI) and is referred to as the VHA TBI Clinical Reminder (VHA-TBI-CR).1 The VHA-TBI-CR consists of 4 questions: (1) During any of your OIF/OEF deployment(s), did you experience any of the following events? (blast or explosion; vehicular accident/crash; fragment wound or bullet wound above the shoulders; fall; blow to head; other injury to head); (2) Did you have any of these symptoms immediately afterward? (losing consciousness/“knocked out,” being dazed, confused or “seeing stars,” not remembering the event, concussion, head injury); (3) Did any of the following problems begin or get worse afterward? (memory problems or lapses, balance problems or dizziness, sensitivity to bright light, irritability, headaches, sleep problems); (4) In the past week, have you had any of the symptoms from section 3? A positive response to all 4 questions constitutes a positive screen, that is, the person likely experienced a TBI (first 2 questions) and continues to experience postconcussive-like symptoms (questions 3 and 4). The VHA-TBI-CR is to be administered by the first medical provider to see the veteran within the VHA healthcare system following military deployment. Typically, it is administered by a primary care provider.

Following a positive VHA-TBI-CR, veterans are referred for a more extensive comprehensive TBI evaluation completed by TBI clinicians throughout the Polytrauma System of Care.1 This evaluation consists of further assessment of blast exposures and injurious events, targeted review of symptoms, and a physical examination conducted by a licensed medical practitioner with expertise in TBI. The comprehensive TBI evaluation is designed to determine whether the patient sustained a TBI and then institute an appropriate plan of care regardless of whether or not a TBI was sustained.

When implemented, the TBI screen had unknown reliability and validity.1 Initial studies of the psychometric properties of the VA's TBI screen have been mixed. The screen was found to have adequate internal consistency (α = .77).2 Test-retest stability has ranged from poor3 to good.2 Overall estimates of the military-related TBI screening sensitivity have ranged from 60%4 to 85%5,6 to 94%,2 and estimates of specificity have ranged from 13%5 to 59%2 to 82%6 to the mid-90%.4 Sensitivity improved when only items 1 and 2 of the screen were considered.4

The purpose of the current study was to examine the temporal stability and validity of the VHA-TBI-CR. Sensitivity/specificity analyses were conducted by comparing TBI screening findings with TBI diagnostic results from the VA TBI Identification Clinical Interview.7 A secondary objective was to examine the extent to which the concurrence between the initial clinical VHA-TBI-CR and the criterion VA TBI Identification Clinical Interview was stable over time.

Back to Top | Article Outline

METHODS

Participants

Potential participants were identified by review of the VHA's Patent Care Services database for veterans who completed the VHA-TBI-CR at the Tampa VA Medical Center. Eligibility criteria included (a) Active Duty/Veteran, (b) age 18 years or more, (c) deployed in OEF/OIF, (d) completed a clinical VHA-TBI-CR, (e) English speaking and conversant, (f) able to consent, and (g) not participating in another study. Selection was set for approximately 20% positive clinical VHA-TBI-CR and 80% negative to be consistent with the general positive rate in the overall VHA population.8 Invitation letters were sent to eligible veterans explaining the study and inviting participation. Veterans responding to the letter were scheduled for in-person interviews. Ninety-five participants were recruited from the Tampa VAMC. Veterans were paid $40.00 for completion of the data collection and interview. The protocol was approved by the Institutional Review Board of the University of South Florida.

Back to Top | Article Outline

Measures

VHA-TBI-CR Screen

A paper-and-pencil version of the VHA-TBI-CR was utilized for rescreening as part of the current study. This screen was identical to the one used by the VHA except that it was done via paper-and-pencil as opposed to verbally with a clinician. As with the clinical VHA-TBI-CR, it included 4 questions regarding (a) exposure to a potential TBI event, (b) loss or alteration of consciousness, (c) 6 postconcussive symptoms at the time of the event, and (d) the same 6 postconcussive symptoms currently–memory problems or lapses, balance problems or dizziness, sensitivity to bright light, irritability, headaches, and sleep problems. Endorsement of all 4 questions constituted a positive screen. Failure to endorse any 1 of the 4 questions constituted a negative screen.

Back to Top | Article Outline

VA TBI Identification Clinical Interview

The VA TBI Identification Clinical Interview is a semistructured interview developed for this study.7 It was developed through consultation with TBI subject matter experts and built on the strengths of existing semistructured interviews in the literature. It provides a standardized method for determining whether or not an individual experienced a TBI in the past (within a certain prespecified time frame) as defined by World Health Organization9 and American Congress of Rehabilitation Medicine10 criteria for mild TBI but not limited to mild severity. The VA TBI Identification Clinical Interview was developed specifically to address concerns expressed in the literature regarding (1) nonshared terminology, (2) memory for remote events, (3) limits of self-report, particularly in relationship to over- or underreporting, and (4) linkage of neurological symptoms to TBI versus other potential causes. It uses a patient-narrated story approach and consists of 3 parts.

Part A is a series of open-ended questions (and follow-up probes) to (a) facilitate the patient's freely told, spontaneous description of a potentially injurious event, (b) help determine whether the event resulted in a TBI, (c) determine the cause of the physical trauma (eg, motor vehicle accident, fall, explosive blast), (d) identify any immediate new-onset symptoms or problems, and (e) determine the course of any new symptoms. Part B is a checklist for recording the patient's spontaneously reported information from part A's semistructured interview. Part C is a series of questions and a recording form for confirming the patient's responses acquired during part A and recorded on the part B form. This includes confirming the negative if the interviewee did not report an event with alteration of consciousness and/or did not report experiencing acute onset or worsening of new symptoms. For example, if participants were able to describe in detail events that occurred immediately before, during, and after the injury/trauma event with no memory gaps and with indications that they continued to function in a normal cognitive manner without disorientation or mental confusion, then TBI was ruled out for those participants. This “negative TBI finding” typically was also confirmed by asking them to describe what others had observed about their behavior during and immediately after the event. They were specifically asked: “It sounds like there were no holes or gaps in your memory from that day and that you were functioning in a pretty normal way under the circumstances at the scene and afterward–is that correct?”

At the conclusion of the VA TBI Identification Clinical Interview (after parts A, B, and C were completed), participants were asked:

How many events have you experienced with significant physical force to your head that resulted in a loss of consciousness, or memory gaps for part of the event or what happened immediately afterward, or not being able to function normally afterward (that is, being confused, not making sense, not knowing what was going on).

Participants' response to this question reflects their perception of concussions experienced, regardless of whether the earlier portions of the interview “confirmed” any deployment-related TBI.

Back to Top | Article Outline

Procedures

When veterans arrived for the scheduled data collection interviews, they were rescreened for eligibility. They then completed a series of paper-and-pencil forms that included (a) an informed consent form, (b) a 9-item demographic questionnaire that included information about military service and number of deployments, (c) the paper-and-pencil VHA-TBI-CR, (d) the Neurobehavioral Symptom Inventory, and (e) the Posttraumatic Stress Disorder (PTSD)-Checklist civilian version (PCL-C). They were then interviewed using the semistructured VA TBI Identification Clinical Interview. Results from the Neurobehavioral Symptom Inventory and PCL-C were not used in the current analyses.

Two research-trained physicians completed in-person interviews using the criterion standard VA TBI Identification Clinical Interview. Results from these interviews were recorded on the TBI Identification Clinical Interview form and a determination made regarding whether or not the person had sustained a TBI during OEF/OIF military deployment, that is, did he or she experience an acute physical force to the head resulting in confusion or disorientation, loss of consciousness, posttraumatic amnesia, and/or other transient neurologic abnormalities. Interviews were audio recorded.

Using the audio recordings of the interview, interrater reliability was conducted using 18 interviews from interviewer 1 and 11 interviews from interviewer 2. The alternative interviewer listened to the audio recording and independently and blindly completed the VA TBI Identification Clinical Interview form and made a determination regarding whether or not the person had sustained a deployment-related TBI. Interrater reliability agreement was measured using Cohen κ. The κ score was 0.85, with a standard error of 0.10. Of the 29 cases, 2 were not in agreement. In those 2 cases, the original interviewer's decision was used in the analyses reported later.

Back to Top | Article Outline

Data analysis

Results of the original clinically administered VHA-TBI-CR were obtained from the VHA Patent Care Services database. A paper-and-pencil version of the VHA-TBI-CR was also completed as part of the current study, and stability analyses were conducted examining positive screening rates and ϕ stability coefficients across the 2 assessments.

Validity analyses were conducted by comparing the results from both the original clinical and the current study's paper-and-pencil VHA-TBI-CR screens to the study's criterion standard TBI Identification Clinical Interview findings regarding a history of deployment-related TBI. Sensitivity, specificity, positive predictive value, and negative predictive value (NPV) of the VHA-TBI-CR screens were computed. Kappas were also calculated to examine agreement between VHA-TBI-CR screens and the study's criterion standard TBI diagnosis. A κ of 1 indicates perfect agreement while a value of 0 indicates chance-level agreement. Kappa values below 0.40 are generally interpreted as poor, 0.40 to 0.75 as fair to good, and more than 0.75 as excellent.11 Finally, the VHA-TBI-CR is positive only when patients report both acute and current symptoms (ie, Questions 3 and 4 of the VHA-TBI-CR), whereas only the first 2 questions constitute the actual definition of TBI (ie, Question 1: a physical trauma event, and Question 2: acute alterations of consciousness). As such, we also examined validity of the original clinical VHA-TBI-CR on the basis of only the first 2 questions. Prior validation studies4,5 found that sensitivity rates of the VHA-TBI-CR improved when only Questions 1 and 2 were considered.

Back to Top | Article Outline

RESULTS

Participant characteristics

Demographic and military characteristics of the sample are presented in Table 1. Participants were primarily male (89.5%) with diverse racial backgrounds. The mean age of the participants was 41.6 (SD = 11.14) years, and 86% completed at least some college. More than half were employed (56.8%), and of those, 74% worked 30 hours per week or more. The majority of the participants were enlisted (83.2%), only 16.8% were officers. All branches of military service were represented.

TABLE 1

TABLE 1

Back to Top | Article Outline

Positive VHA-TBI-CR screening rates and rates of confirmed TBI

Table 2 shows that the VHA-TBI-CR completed as part of participants' VHA clinical care had, by study design, approximately a 20% positive rate, with 20 of the 95 participants (21.1%) screening positive for TBI. On the current study's paper-and-pencil VHA-TBI-CR rescreen, twice as many screened positive (n = 40, 42.1%) an average of 32 months after their clinically administered VHA-TBI-CR. On the research study's VA TBI Identification Clinical Interview, 45 participants (47.4%) reported sustaining 1 or more deployment-related mild TBIs. However, the interview confirmed TBIs in only 18 of these 45 participants, as 27 participants were able to describe injury events in detail with indication of immediate postevent normal mental processing and had no indication of postevent confusion, disorientation, or memory gaps. Self-reported TBIs by participants and interview-based TBI confirmation when the trauma events were questioned in detail were clearly divergent. In those with a TBI confirmed via interview, the time elapsed since the most recent TBI averaged 73.2 months (SD = 29.4). In the remaining participants reporting some deployment-related trauma event that was determined not to be a TBI (n = 31), the time elapsed averaged 76.8 months (SD = 18.3).

TABLE 2

TABLE 2

Back to Top | Article Outline

TBI Clinical Reminder stability

Table 3 shows that the overall stability of the VHA-TBI-CR screen was low (ϕ = 0.34). Only 68% responded in a consistent manner across the 2 screens. This may not be surprising in that the average time between rescreens was almost 3 years. Therefore, the sample was divided in thirds on the basis of time interval between the 2 administrations of the VHA-TBI-CR. The third of the sample with the shortest duration between VHA-TBI-CRs had the highest level of test-retest consistency (ϕ = 0.55); however, at best it was only fair. With greater durations between screenings, the stability of the VHA-TBI-CR declined to unacceptably low levels (ϕ = 0.25 and 0.21, respectively). Across all 3 intervals between readministration of the VHA-TBI-CR, there was an increase in the rate of positive rescreens compared with the original clinical screen, with increasing positive rates associated with longer intervals between screens (1.6, 2.1, and 2.4 times higher rates across the 3 increasingly longer interscreening intervals, respectively; see Table 3).

TABLE 3

TABLE 3

Examining the discrepant findings across the 2 VHA-TBI-CR screens found that 5 participants went from an initial positive clinical VHA-TBI-CR screen to a subsequent negative research rescreen. On the VHA-TBI-CR rescreen, these 5 participants all continued to report a deployment-related trauma event but resolution of symptoms by the time of the rescreen. In contrast, 25 participants went from an initial negative VHA-TBI-CR screen to a subsequent positive rescreen. Eighteen of the 25 denied even experiencing a deployment-related trauma event on the initial clinical VHA-TBI-CR screen. On the initial VHA-TBI-CR, 5 of the 25 reported a trauma event but without any alteration of consciousness (ie, no TBI). The remaining 2 of the 25 reported a trauma event with alteration of consciousness but a resolution of symptoms by the time of their initial clinical VHA-TBI-CR screen; however, on rescreening, these 2 participants reported being symptomatic again. The original clinical VHA-TBI-CR was determined to be consistent with findings on the VA TBI Identification Clinical Interview criterion standard for 2 of the 5 cases (40%) that went from a positive screen to a negative screen and consistent with the findings on the TBI criterion standard in 21/25 cases (84%) that went from a negative screen to a positive screen. Overall, the initial clinical VHA-TBI-CR was accurate in 77% of these cases, while the research rescreen was accurate in only 23% of these cases.

Back to Top | Article Outline

TBI Clinical Reminder validity

Validity in the current study was operationally defined as the diagnostic concordance between the VHA-TBI-CR screen and the criterion standard VA TBI Identification Clinical Interview, as well as an evaluation of sensitivity, specificity, positive predictive value, and NPV of the VHA-TBI-CR screen. The stability analyses raise the question of whether participants' original clinical VHA-TBI-CR or the current study's VHA-TBI-CR rescreen is diagnostically more accurate or valid when compared with the VA TBI Identification Clinical Interview findings. The final 2 columns in Table 3 demonstrate that more recent screens are less accurate with unacceptably low κ coefficients. This was true for clinical screens completed within the past 2 years (κ = 0.27), as well as for the current study's rescreens (κ = 0.25 for the entire sample). The same temporal pattern was found using just the first 2 questions of the clinical VHA-TBI-CR screen (ie, event with alteration of consciousness). That is, for screens within the past 2 years, κ was 0.27; for screens completed more than 2 years ago, κ was 0.52.

Table 4 shows the correspondence between clinical and subsequent research rescreen with the VHA-TBI-CR and the criterion standard semistructured interview, while Table 5 shows the psychometric accuracy statistics. As can be seen, the original clinical VHA-TBI-CR had only moderate sensitivity (0.61) but relatively good specificity (0.88). However, on the rescreen done as part of the current study, twice as many participants now screened positive, resulting in a modest increase in sensitivity (0.72) but a corresponding drop in specificity (0.65). Of concern is that in this study, a clinical VHA-TBI-CR screen failed to identify a TBI 39% of the time (ie, 7 false negative out of the 18 determined to have a TBI) but had a relatively acceptable false-positive rate of 11.7%. For the 7 individuals with false-negative screens, we examined their endorsement pattern from the 4 items of the VHA-TBI-CR. Four of the 7 reported some deployment trauma event, but none responded “yes” to item 2 (ie, alteration or loss of consciousness). Because item 2 was “no,” items 3 and 4 about symptoms would not be asked on the VHA-TBI-CR. However, on VA TBI Identification Interview all 7 were determined to have sustained a deployment-related TBI, only 1 of whom reported being currently asymptomatic.

TABLE 4

TABLE 4

TABLE 5

TABLE 5

Sensitivity and specificity rates of the original clinical VHA-TBI-CR were similar for the first 2 items versus all 4-item screens (see Tables 4 and 5). In contrast, the 2-item version of the research rescreen was slightly more sensitive than the 4-item version. Even the research rescreen failed to identify 28% of those confirmed to have sustained a TBI (5/18). Three of the 5 false-negative research rescreen participants were also false negative on the original clinical 4-item screen. The other 2 participants were true positives on the original clinical 4-item screen and false negatives on the research rescreen because they reported that their symptoms had resolved over 2.5 years between screens.

Back to Top | Article Outline

DISCUSSION

When the VHA-TBI-CR was implemented across VA in 2007, its reliability and validity were unknown. The current study is one of several examining its reliability and validity. Results of the current study are disappointing. The validity of the VHA-TBI-CR is in question, given its low agreement with a semistructured VA TBI Identification Clinical Interview used as the diagnostic criterion standard in this study. This is similar to a prior finding of poor agreement with clinical comprehensive evaluation findings.5 Furthermore, the VHA-TBI-CR was found to be unstable across time, with only 68% of the participants responding in a consistent manner (ϕ = 0.34), though rescreens were administered via paper-and-pencil rather than interview. When rescreening durations were closer together in time (ie, within the past 2 years), consistency was higher (ϕ = 0.55) but still far lower than desired. With intervals greater than 2 years between rescreening, the VHA-TBI-CR was very unstable (ϕ < 0.25). The sensitivity of the VHA-TBI-CR to TBI was also disappointingly low in this study (0.61), far lower than previous studies, while the specificity was reasonable (0.88) and higher than previous studies. On the readministration of the VHA-TBI-CR done as part of the current study, twice as many participants screened positive, resulting in a modest increase in sensitivity (0.72) but a corresponding drop in specificity (0.65). Of concern is that in this study, a clinical VHA-TBI-CR failed to identify a TBI 39% of the time, although the false-positive rate (11.7%) was relatively acceptable.

As with prior research,4,5 using just the first 2 screening items improved sensitivity but only in the research rescreening. There was very little difference in sensitivity across the 2 versus 4-item clinical screens. This suggests that inaccuracies in clinical VHA-TBI-CR screening, particularly false negatives, were not due to symptom resolution in participants who sustained a deployment-related TBI, as the 2-item screen does not ask about symptoms and solely reflects the mild TBI diagnosis.

Back to Top | Article Outline

Stability considerations

A larger proportion of participants in the current study screened positive for TBI on rescreening with the VHA-TBI-CR. The original clinical VHA-TBI-CR was positive in 21% of the participants, while the rescreen was positive in 42% of the participants. These findings are similar to those reported by Polusny and colleagues12 in a National Guard sample. They found that when assessed in theater toward the end of their deployment, 9.2% of the participants reported a deployment-related concussion or mild TBI. However, when reassessed 1 year following deployment, 22% of participants reported having sustained a concussion. Their postdeployment positive screening rate is comparable with the rate of positive clinical TBI screens completed across the VHA healthcare system.8 Polusny and colleagues thought it unlikely that the increase in positive rates on rescreening were due to additional concussions sustained in theater during the 1 month between assessment and departure from theater.

As with the study by Polusney et al,12 in the current study, it is unlikely that the participants sustained additional deployment-related concussions in the interval between rescreening, although some may have had subsequent redeployments. Three current findings are inconsistent with additional concussions occurring between screens accounting for the increased rate of positive TBI screens. First, if the increase in positive VHA-TBI-CR screens from 21% to 42% was due to additional deployment-related concussions occurring between the 2 screens, then the second VHA-TBI-CR should be more accurate when compared with the current VA TBI Identification Clinical Interview findings. The opposite was found. Using the semistructured interview as the criterion standard, the first VHA-TBI-CR screen was more accurate (κ = 0.47) than the second (κ = 0.25). Second, even within the same research study visit, 42% had a positive screen at the beginning of the session, but 47.4% reported having sustained 1 or more deployment-related mild TBIs during their clinical interview, an increase of 5.4% TBI affirmation within 30 minutes. Another way to investigate this increased reporting within session is to examine those participants who responded “no” to item 2 on the research VHA-TBI-CR rescreen (ie, no alteration of loss of consciousness) but then reported having sustained a deployment-related TBI later in the same session. Nine of 50 participants responding “no” to screening question 2 met this criterion, an increase of 18% in affirmation rate during a 30-minute interview when clearly no deployment-related TBI occurred. Clearly, self-reporting was unstable in these participants but generally increasing across repeated questioning regarding potential TBI experiences and symptoms. Third, if the original clinical VHA-TBI-CR screen was completed between 38 and 57 months earlier, it was more accurate (κ = 0.53) than if the VHA-TBI-CR had been completed within the past 24 months (κ = 0.27). That is, clinical VHA-TBI-CR screens done further in the past were more accurate than more recent screens. Psychometrically, these findings are counterintuitive. Evaluations conducted closer in time should be more consistent than evaluations completed more distant in time. Therefore, other factors must be driving the increased rate of positive TBI screens over time. It is possible that the paper-and-pencil nature of the research VHA-TBI-CR and the context of the evaluation (ie, research vs clinical) resulted in a higher affirmation rate than the interview-based clinical VHA-TBI-CR. However, similar findings of doubling of positive screening rates over time by Polusney et al, with both screenings in a research context, makes it unlikely that this explanation accounts for the current doubling of positive TBI screening rates.

Back to Top | Article Outline

Validity considerations

The instability of the VHA-TBI-CR raises significant concerns regarding its validity. Reliability is necessary (but not sufficient) for validity. Given the seeming unreliability of the screen, the poor validity of the screen is not surprising. Far more participants go from a negative VHA-TBI-CR screen to a positive rescreen, than vice versa. Examination of cases with divergent findings across the 2 screens found the original VHA-TBI-CR to be accurate in 77% of these cases compared with 23% accuracy of the opposite findings in the research rescreen. Furthermore, within the overall sample, VHA-TBI-CR screens completed in the distant past (>24 months earlier) are more consistent with the criterion standard diagnosis (and, therefore, presumably more “valid”) than VHA-TBI-CR screens completed in the more recent past (within the last 24 months). Therefore, it should not be surprising that the overall agreement between the VHA-TBI-CR screens and the criterion standard diagnosis are unacceptably low, particularly for those completed within the past 24 months (within the last 24 months clinical VHA-TBI-CR screen κ = 0.27; current research VHA-TBI-CR rescreen κ = 0.25).

Of concern is the lower sensitivity rate of the TBI clinical screen in the current study (0.61) than that in prior studies (0.942 and 0.87-0.905). In the current study, the clinical TBI screen failed to identify a TBI 39% of the time. Ideally, screening tools are more than 90% sensitive to the condition of interest. There are likely 4 primary reasons for sensitivity differences across VHA-TBI-CR studies: (a) population sampling issues, (b) research methodology issues related to administration of the TBI screen (ie, method variance), (c) the criterion standard used to confirm a history of TBI, and (d) different rates of positive VHA-TBI-CR screens across the study samples.

The data in the study by Donnelly et al2 were research data collected by trained research assistants in a research setting. These research findings were never compared to the clinical VHA-TBI-CR screens completed as part of the veterans' VHA healthcare. Half of their sample was recruited from referrals of OEF/OIF veterans suspected of having a TBI who were then referred for neuropsychological evaluations; the other half was recruited from an OEF/OIF registry. Their overall rate of positive screens was 64% on the first research VHA-TBI-CR screen and 61.4% on the VHA-TBI-CR rescreen (K. Donnelly, PhD, written communication, 2014), far higher than the actual positive screening rate (about 20%) across the VA healthcare system.8 As a result, veterans with negative VHA-TBI-CR screens are underrepresented in their sample, increasing the likelihood of artificially increasing sensitivity rates. Their use of a semistructured TBI confirmation interview similar to that used in the current study likely increased the accuracy of TBI confirmation.

In contrast to our prior study,5 which evaluated the VHA-TBI-CR's validity on the basis of clinicians' Comprehensive TBI Evaluation findings (ie, an unstructured clinical examination), there were many fewer false positives in this study when using a semistructured interview as the criterion standard. The differences in specificity between the current study (0.88) and our prior study (0.15) are likely due to differences in setting and TBI criterion standard used. In a clinical setting, in which there is perhaps a less standardized approach to diagnosis, it should not be surprising that rates of TBI confirmation following a positive TBI screen range from 19% to 93% across VA facilities completing the Comprehensive TBI Evaluation.13 Given this large variability, the accuracy of these TBI diagnostic determinations is suspect with respect to a true reflection of the screen's sensitivity and specificity to TBI. In the current study, the agreement between the VHA-TBI-CR and semistructured interview criterion standard (κ = 0.47) was substantially higher than that in the prior study using a clinical evaluation criterion standard (κ from 0.03 to 0.08).5 As argued elsewhere,1,5,14 the use of this screen within VA may be problematic for these and other reasons.

In the current study, we were interested in determining the sensitivity and specificity of the TBI clinical screen using a careful semistructured clinical interview as the criterion standard for determination of whether a deployment-related TBI had occurred. A standardized approach to diagnosis of TBI allows for more accurate appraisal of the screen's validity with respect to actual TBI history. The current methodology with its use of a semistructured interview likely reflects a better estimate of the actual incidence of (mild) TBI within this sample. Although these findings indicated that the screen failed to identify 39% of veterans who sustained a deployment-related TBI, the current study's negative predictive power (0.91) is actually quite high because the overall TBI prevalence rates in this population are around 17% to 19% (calculated at 19% in the current study). That is, we can be confident in 91% of the cases with a negative screen that no TBI occurred per the criterion standard interview. This is similar to the favorable NPV also found in the clinical setting when using similar base rates.5 In contrast, positive predictive power is low (0.55). When the TBI screen is positive, the person likely sustained a deployment-related TBI only 55% of the time.

An important issue in studying the VHA-TBI-CR's validity is that the VHA-TBI-CR and the criterion standard interview had different goals. Namely, the VHA-TBI-CR was designed to identify patients with a history of TBI who are currently symptomatic. The criterion standard in this study was an interview whose purpose was to identify the presence or absence of historical TBI (regardless of current symptomatology). As such, validity analyses of the VHA-TBI-CR may result in deceptively high false-negative rates because of lack of current symptom endorsement on the VHA-TBI-CR (ie, a negative VHA-TBI-CR finding, which is later evaluated on interview to be positive for TBI). However, current findings indicate that this was not the reason for false negatives in the current study. Clinical screenings based on only the first 2 screening items had the same false-positive participants as did screenings based on the 4-item screen. Of 7 individuals with false-negative screens, 4 reported some deployment trauma event, but none responded “yes” to item 2 (ie, alteration or loss of consciousness).

Underappreciated is that a study's sensitivity and specificity calculations are affected by the study's positive screening rate. The higher the positive screening rate, the higher will be the study's sensitivity. This is because sensitivity equals true positives divided by true positives plus false negatives. With higher positive screening rates, the numerator increases more than the denominator. The current study's findings were compared with prior VHA-TBI-CR validity studies that had a similar methodology with an external semistructured TBI identification clinical interview as the criterion standard. Using the rates of TBI confirmation in these studies for both positive and negative VHA-TBI-CR screens, and recalculating sensitivity and specificity based on a VHA's overall positive VHA-TBI-CR screening rate of 20%, findings were very similar across studies. In the study by Donnelly and colleagues,2 adjusted sensitivity would be 70%, with 91% specificity. In the study by Fortier and colleagues,6 adjusted sensitivity would be 65%, with 93% specificity. The current study's adjusted sensitivity would be 60%, with 89% specificity. Across studies, as the VHA-TBI-CR is used across the VHA healthcare system, sensitivity ranges from 60% to 70% and specificity ranges from 89% to 93%. These sensitivity values are clearly unacceptable in that in general screening, tools should be more than 90% sensitive. The specificity values are acceptable with resulting high NPVs.

Back to Top | Article Outline

Historical drift: stability and validity issues

The historical drift of screening responses within the same participant deserves discussion. Using the VHA-TBI-CR, twice as many participants screened positive on rescreening (from 21.1% to 42.1% positive), and the original clinical screen was more accurate than the subsequent rescreen. Some of this divergence may be due to method variance (ie, using paper-and-pencil on the screen vs person-to-person interview). However, it is unlikely that this method variance completely accounts for this discrepancy. There was historical drift in the accuracy of the clinically administered VHA-TBI-CR screen over the course of the past 4½ years. Clinical VHA-TBI-CR screens completed further in the past were more accurate than more recent screens.

Reasons for these historical drift findings are not clear, although they replicate findings reported by Polusny and colleagues,12 in which positive screening rates increased from 9.2% to 22% over a 1-year reassessment interval. It is certainly possible that initial denial or minimization of problems over time gives way to gradual recognition and acknowledgement. That explanation might explain the findings by Polusny et al, that is, low military theater rates more than doubling a year later postdeployment. However, in the current study, only 2 participants were less than 33 months since injury when initially screened. Any postdeployment “honeymoon period” and initial denial or minimization of problems would likely have ended long before. In addition, in the current study, 25 participants went from an initial negative VHA-TBI-CR screen to a subsequent positive rescreen. Eighteen of these 25 denied even experiencing a deployment-related trauma event on the initial clinical VHA-TBI-CR screen. Therefore, any initial postdeployment denial would apply to both symptoms and physically traumatic events themselves, an unlikely scenario. Importantly, the original screen was consistent with the criterion standard TBI determination in 84% of these cases.

Simply bringing participants' attention to their deployment experiences and current symptoms may increase affirmation. However, this explanation does not account for the doubling of TBI positive rates between rescreening with VHA-TBI-CR in which the shortest rescreening interval was 8.7 months. It is far more likely that societal, media, and medical factors increased participants' attention to and concern regarding their deployment experiences and subsequent symptoms in the interval between rescreenings. Over the past 5 to 10 years, there has been a great deal of media attention to, and likely false causal relationships asserted, between military concussions, multiple concussions, blast injuries, and football concussions and supposedly related cognitive impairment, suicide, chronic traumatic encephalopathy, and eventual dementia.

The initial VHA-TBI-CR screen, which was conducted as part of VA clinical care, was embedded in a series of screens, including screens for depression, PTSD, infections, and so forth, as part of a postdeployment screen package.1 As such, it could be that the higher rate of positive screens conducted later is due to method variance. That is, the current study used an isolated paper-and-pencil approach to screening and conducted the VHA-TBI-CR in isolation. This could, in part, account for greater endorsement because of the human tendency to give greater meaning to questions presented in isolation. Certainly, social psychological research has revealed that simultaneous versus isolated presentation can significantly impact responses (eg, distinction bias).15

Other factors also could potentially play a role in the doubling of positive TBI rates on rescreening. Over time, residual or developing symptoms may influence retrospective recall of deployment traumas and concussion history.16 Events may be remembered as causing disorientation for reasons other than a TBI, and symptoms due to comorbid conditions and psychosocial factors may be misattributed to a disorienting but nonconcussive trauma event. Similarly, comorbid deployment-related mental health (eg, PTSD or depression) or physical conditions (eg, chronic pain conditions) with similar symptoms may be misattributed to a trauma episode with a presumed concussion/mild TBI. Alternatively, participants with chronic symptoms over time may begin to seek compensation for what they perceived as deployment-related problems, and TBI or concussion is a potentially compensable condition. Additional research will be needed to delineate the factors accounting for these increased positive rates on TBI rescreening.

Back to Top | Article Outline

Study limitations

The current sample is relatively small and may not be representative of the VA population of returning OEF/OIF/OND population at large. However, the use of a 21:79 positive to negative screening ratio is comparable to the VA national average of 20:80. Furthermore, it likely provides a better estimate of the actual incidence of mild TBI by using an independent, semistructured interview, relative to studies conducted in strictly clinical5 or research2 settings. Although the outcome from a semistructured clinical interview is considered to be the best standard for an identification and diagnosis of a remote history of TBI when no acute care records are available, it is imperfect. No biomarkers for a remote mild TBI currently exist, and direct observational information from the time of injury is frequently unavailable.

Back to Top | Article Outline

CONCLUSION

Limitations notwithstanding, this study suggests that the VHA-TBI-CR has poor validity from the perspective of comparing it to a semistructured interview that was designed to avoid over- and underdiagnosis.7 The poor validity is likely in part due to its poor reliability. That is, the screen does not produce the same results in the same patients across time, particularly as the time between screens increases and as the screen occurs later in historical time. Screening is intended to cast a wide net with a high-sensitivity rate, typically more than 90%. However, this study found that the original clinical VHA-TBI-CR missed a TBI 39% of the time, whereas the VHA-TBI-CR rescreen, with a doubled positive screening rate, still missed a TBI 28% of the time. The screen does not appear to perform as intended. As argued elsewhere,1,5,14 the use of this screen within VA may be problematic for these and other reasons.

Back to Top | Article Outline

REFERENCES

1. Belanger HG, Uomoto JM, Vanderploeg RD. The Veterans Health Administration's (VHA's) Polytrauma System of Care for mild traumatic brain injury: costs, benefits, and controversies. J Head Trauma Rehabil. 2009;24:4–13.
2. Donnelly KT, Donnelly JP, Dunnam M, et al. Reliability, sensitivity, and specificity of the VA traumatic brain injury screening tool. J Head Trauma Rehabil. 2011;26:439–453.
3. Van Dyke SA, Axelrod BN, Schutte C. Test-retest reliability of the Traumatic Brain Injury Screening Instrument. Mil Med. 2010;175:947–949.
4. Terrio HP, Nelson LA, Betthauser LM, Harwood JE, Brenner LA. Postdeployment traumatic brain injury screening questions: sensitivity, specificity, and predictive values in returning soldiers. Rehabil Psychol. 2011;56:26–31.
5. Belanger HG, Vanderploeg RD, Soble JR, Richardson M, Groer S. Validity of the Veterans Health Administration's traumatic brain injury screen. Arch Phys Med and Rehabil. 2012;93(7):1234–1239. doi:10.1016/j.apmr.2012.03.003.
6. Fortier CB, Amick MM, Kenna A, Milberg WP, McGlinchey RE. Correspondence of the Boston Assessment of Traumatic Brain Injury-Lifetime (BAT-L) Clinical Interview and the VA TBI Screen [published online ahead of print December 12, 2013]. J Head Trauma Rehabil.
7. Vanderploeg RD, Groer S, Belanger HG. Initial developmental process of a VA semi-structured clinical interview for TBI identification. J Rehabil Res Dev. 2012;49(4):545–556.
8. Lew HL, Otis JD, Tun C, Kerns RD, Clark ME, Cifu DX. Prevalence of chronic pain, posttraumatic stress disorder, and persistent postconcussive symptoms in OIF/OEF veterans: polytrauma clinical triad. J Rehabil Res Dev. 2009;46(6):697–702.
9. Carroll LJ, Cassidy JD, Holm L, Kraus J, Coronado VG, WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury. Methodological issues and research recommendations for mild traumatic brain injury: the WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury. J Rehabil Med. 2004;36(43 suppl):113–125.
10. The American Congress of Rehabilitation Medicine, Mild Traumatic Brain Injury Committee of the Head Injury Interdisciplinary Special Interest Group of the American Congress of Rehabilitation Medicine. Definition of mild traumatic brain injury. J Head Trauma Rehabil. 1993;8:86–87.
11. Fleiss JL. Statistical Methods for Rates and Proportions. 2nd ed. New York, NY: Wiley; 1981.
12. Polusny MA, Kehle SM, Nelson NW, Erbes CR, Arbisi PA, Thuras P. Longitudinal effects of mild traumatic brain injury and posttraumatic stress disorder comorbidity on postdeployment outcomes in national guard soldiers deployed to Iraq. Arch Gen Psychiatry. 2011;68(1):79–89. doi:10.1001/archgenpsychiatry.2010.172.
13. Cifu DX. Co-morbidities 101: putting it in perspective. Paper presented at: VHA Polytrauma Conference; 2010; Orlando, FL.
14. Vanderploeg RD, Belanger HG. Screening for a remote history of mild TBI: when a good idea is bad. J Head Trauma Rehabil. 2013;28(3):211–218.
15. Hsee CK, Zhang J. Distinction bias: misprediction and mischoice due to joint evaluation. J Pers Soc Psychol. 2004;86(5):680–695. doi:10.1037/0022-3514.86.5.680.
16. Wessely S, Unwin C, Hotopf M, et al. Stability of recall of military hazards over time: evidence from the Persian Gulf War of 1991. Br J Psychiatry. 2003;183(4):314–322.
Keywords:

concussion; traumatic brain injury; screening; Veterans Health Administration

Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.