Donnelly, Kerry T. PhD; Donnelly, James P. PhD; Dunnam, Mina PhD; Warner, Gary C. PhD; Kittleson, C. J. PsyD; Constance, Janet E. PhD; Bradshaw, Charles B. PhD; Alt, Michelle MA
In recent years, the Department of Defense (DoD) and the Department of Veterans Affairs (VA) have placed special emphasis on efforts to identify and treat service members and veterans of Operation Enduring Freedom (OEF) and Operation Iraqi Freedom (OIF) who may have sustained deployment-related traumatic brain injury (TBI). Apparent urgent clinical need and congressional mandate for mass TBI screening of our troops and veterans resulted in the rapid development and implementation of screening tools that had face validity but were not subjected to systematic psychometric scrutiny. The purpose of this article is to provide item analyses, estimates of temporal reliability and internal consistency, and examination of the sensitivity and specificity of the veteran traumatic brain injury-screening tool (VATBIST), also known as the VA TBI clinical reminder.
Over the course of the wars in Afghanistan and Iraq, TBI has consistently been identified as one of the most common injuries sustained by service members.1–3 Injuries from explosive devices, extended deployments, and increased survivability have led to an unprecedented number of troops suffering from TBI and posttraumatic stress disorder (PTSD).3,4 The Rand report on the Invisible Wounds of War, summarizing nearly 2000 surveys of returning OEF/OIF service members, estimated that 19.5% of OEF/OIF service members (approximately 320,000 as of 2008) may have sustained TBI during deployment, 31% may have sustained TBI or have a mental health condition associated with deployment (ie, PTSD or depression), and 7.3% may have both sustained TBI and have a mental health condition associated with deployment.
Similarly, Hoge et al5 studied over 2000 Army infantry soldiers returning from Iraq and found that 15% reported a brain injury during deployment that involved loss or alteration of consciousness. The soldiers with mild TBI (mTBI, also known as concussion) were significantly more likely to report high combat and blast exposure than were the 17% of soldiers who reported other types of injuries. Soldiers with mTBI reported more physical and mental health problems than soldiers with other types of injuries. Further, mTBI marked by loss of consciousness carried a higher risk of health problems than did injuries with only altered mental status.
Wilk and colleagues6 surveyed 3952 soldiers 3 to 6 months after return from deployment to Iraq and found that nearly 15% met criteria for concussion sustained in theater. Of the concussed group, slightly over one-third reported loss of consciousness from the injury, and nearly three-quarters of that subgroup attributed their injury to a blast mechanism. Among soldiers who lost consciousness from their injuries, blast was related to the presence of postconcussive symptoms 3 to 6 months postdeployment significantly more often than were TBIs due to other causes. This difference in symptoms based on mechanism of injury was not apparent for soldiers with milder injuries, who reported only alteration of consciousness. Taken together, it appears that about 15% to 20% of OEF/OIF service members are reporting deployment-related TBI. As such, the need for routine, reliable, and valid detection of TBI seems essential to the well being of our military forces and veterans.
DEPARTMENT OF DEFENSE TBI SCREENING
Regular screening for problems associated with deployment-related TBI was first implemented by the DoD. Operation Enduring Freedom and Operation Iraqi Freedom patients admitted to Walter Reed Army Medical Center (WRAMC) for head injuries of all types were evaluated for associated symptoms by the Defense and Veterans Brain Injury Center (DVBIC).7 The vast majority of these cases involved moderate to severe head injuries. Such injuries tend to be self-evident. Warden7 suggested that routine TBI screening of returning service members should be conducted more widely regardless of obvious injury status. Deficits associated with mTBI are usually more subtle and difficult to observe than those related to moderate or severe injuries. The Centers for Disease Control and Prevention8 estimated that 80% to 90% of all TBIs fall in the mild range of injury. Without comprehensive screening for TBI, many of these mild—but potentially significant—injuries would not be detected.
Schwab and colleagues9 developed an instrument for screening active duty troops for TBI. The brief traumatic brain injury screen (BTBIS) is a self-report measure that includes items concerning personal identifying information, deployment history, helmet characteristics, exposure to a possible TBI event, and alteration of consciousness and current postconcussive symptoms. A preliminary psychometric study of the BTBIS based on 596 recently returned OEF/OIF soldiers revealed that responses among those who screened positive for TBI on the BTBIS were generally consistent with information given in a follow-up interview, either immediately after completion of the BTBIS or within 2 weeks. The BTBIS had low to moderate correlations with 2 criterion TBI questionnaires, with TBI reported more frequently on the BTBIS than on the more detailed surveys. Although the authors were encouraged by the potential of the BTBIS as a mass-screening tool, they acknowledged that more complete analysis of the instrument's reliability and validity was needed.
Researchers at Fort Carson, Colorado evaluated nearly 4000 returning OIF soldiers with another self-administered TBI screening tool, one based on the BTBIS.10 Similar to its predecessor, the warrior administered retrospective casualty assessment tool (WARCAT) gathers data on possible TBI-related war injuries, alteration of consciousness, and presence of postconcussive symptoms at the time of injury. The WARCAT extends the inquiry to include postconcussive symptoms evident postdeployment as well. On the basis of responses to the WARCAT followed up with clinical diagnostic interviews, about 23% of the soldiers in the study were confirmed to have deployment-related TBI. Of that group, only about one-third reported experiencing 3 or more postconcussive symptoms proximal to the injury, and less than 8% endorsed that symptom threshold postdeployment. Nonetheless, as expected, postconcussive symptom reports were considerably higher among soldiers diagnosed with TBI than for those without such injuries, both at the time of injury and after returning home.
Although it seems intuitive that advances in screening and diagnosis of TBI made by the DoD would seamlessly migrate to VA practice, administrative and logistical differences, as well as related but distinct missions, resulted in the development of complementary but parallel methods of care.
VA TBI SCREENING
In late 2006, VA convened an interdisciplinary task force to develop a TBI screening tool and evaluation protocol. The task force reviewed the literature on screening for TBI, examined the efforts of individual military medical treatment centers and VA medical centers that had implemented TBI screening locally, consulted with DVBIC and reviewed information on the natural history of TBI.11 The task force subsequently created a TBI screening instrument and follow-up protocol for system-wide VA use, which was implemented in April 2007. The VA TBI Screening Tool (VATBIST: see Appendix B) is a brief clinician-administered screen that collects information similar to the WARCAT. Four questions pertain to exposure to a potential TBI-causing event, loss or alteration of consciousness, postconcussive symptoms at the time of injury and current postconcussive symptoms. As of January 2010, approximately 392,000 OEF/OIF veterans have been screened with the VATBIST nationwide, with approximately 20% yielding positive screens (N. A. Sayer at National Statistics on VA TBI Screening, e-mail communication, June 10, 2010). Although several projects are underway to evaluate the integrity of the VATBIST, to our knowledge no reliability or validity data have been published on the measure to date.
COMPLICATIONS OF EVALUATING TBI SCREENING METHODS
Standard measures of TBI (eg, The Glasgow Coma Scale, American Academy of Neurology Diagnostic Criteria) provide severity ratings, rather than definitive determination of the presence or absence of TBI. Mild TBI in particular is something of an amorphous concept, and there is significant debate on its definition.12 Furthermore, so called postconcussive symptoms are not specific to TBI; many are common in the general population or may be primarily due to other disorders, such as depression, PTSD, or other affective disturbances.
To determine the reliability and validity of a TBI screening tool there needs to be an accepted criterion against which the screening tool can be compared. Indirect measurement of mTBI is complicated by the lack of such a true criterion measure, or “gold standard.” There are no medical tests, imaging studies, or biological markers that definitively identify mTBI at this time. Observation by a medical provider of the injury in progress qualifies as an acceptable criterion, but this is very rarely possible in the context of deployment.
The Rand report quoted Jablensky's13 assertion that diagnostic interviews are commonly considered to provide the most accurate means of identifying cases, but there is also little consensus to date as to what constitutes a suitable diagnostic interview for TBI. The interview used in this study, based on review of the literature and combined clinical experience of the authors, is described in detail below.
1. Provide an item analysis of the VATBIST.
2. Determine the temporal stability and internal consistency of the VATBIST.
3. Estimate the sensitivity and specificity of the VATBIST relative to a diagnostic interview.
4. Describe the sensitivity and specificity of the VATBIST for veterans with probable PTSD.
This examination of the psychometric properties of the VA TBI screening tool was conducted as part of a larger, multisite prospective cohort study of OEF/OIF veterans. The current project provides item analyses, estimates of temporal reliability and internal consistency, and examination of concurrent and predictive validity by means of sensitivity/specificity analyses. Ratings derived from a “gold standard” structured diagnostic interview for TBI constitute the criterion for the validity studies. Each participating site was granted approval from its institutional review board (IRB) and Research and Development Committee. Veterans' responses are protected by a National Institute of Mental Health Certificate of Confidentiality.
Five hundred participants were recruited from 5 VA medical centers (Buffalo, Albany, Canandaigua, Syracuse and Bath) and a VA outpatient clinic (Rochester) in Veterans Integrated Service Network 2 (VISN 2), which provides services for veterans in upstate New York. All were recruited for participation in a larger study of returning veterans. This sample size was powered for the most complex multivariate analysis of the parent project and is considered to be more than adequate for the current study. Flahault et al14 table of power estimates indicates that a sample of 474 is sufficient to detect an expected sensitivity of 0.90, with a lower bound 95% confidence interval value of 0.85. This estimate is well matched to the current sample size and test sensitivity.
The study settings reflect a mix of urban, suburban, and rural locations. All OEF/OIF veterans who intended to remain in upstate New York for the 18-month duration of the parent study were eligible for inclusion, except for those who lacked capacity to give informed consent (none were excluded based on this criterion). No vulnerable participants were included. The sample included veterans who were recruited from a VISN 2 OEF/OIF registry, using IRB approved procedures (n = 248, 49.6%) described later, and/or were referred clinically for neuropsychological or polytrauma assessment (n = 252, 50.4%). It is possible that some of the veterans referred for clinical evaluation were not included in the registry at the time of the initial contact. Participants were compensated for their time and travel.
The Standards for the Reporting of Diagnostic Accuracy Studies (STARD) Checklist was followed, and all items in the checklist were completed (http://www.stard-statement.org/). In Appendix A, a flow diagram (as recommended in the STARD statement) illustrates participation in each screening question.
VA TBI screening tool (VATBIST)
This brief measure (see Appendix B) was implemented across the VA system in April 2007. A clinical reminder within the computerized patient record system prompts all VA health care providers to complete the questionnaire once with each OEF/OIF veteran. When the screening has been completed, the clinical reminder is discontinued. Those previously diagnosed with deployment-related TBI, as documented in computerized patient record system, are excluded from the otherwise mandatory clinical screening. For the purpose of this study, the VATBIST was administered by research staff, whether or not the participant had already been screened clinically. The measure includes 4 questions regarding exposure to a potential TBI event, loss or alteration of consciousness, and postconcussive symptoms at the time of the event and at the time of evaluation. Affirmative endorsement of at least 1 element in each of the 4 questions yields a positive screen. That is, a positive screen requires report of: (1) Exposure to an event involving application of force to the head that could result in brain injury (5 items); (2) An alteration in mental status or injury to the head after that event (5 items); (3) Presence of new or worsened symptoms after the event (6 items); and (4) Continued presence of 1 or more of those symptoms in the week prior to screening (6 items).
If no element of a given question is endorsed, the screen is discontinued and is considered to be negative.
The VATBIST was created by a workgroup consisting of interdisciplinary VA TBI providers, primary care providers, and representatives from the DVBIC and from VA Central Office. The goal was to develop an instrument to screen OEF/OIF returnees presenting to the VA for health care in an effort to identify those who may have sustained deployment-related TBI and were currently experiencing symptoms that could be related to the TBI. A deliberate choice was made to cast a wide net, yet to identify only those who may have had a TBI and were symptomatic at the time of screening—those who would benefit most from the resources of a health care system.15 The VA measure was based largely on the brief traumatic brain injury screen (BTBIS) developed by the DVBIC/DoD and described by Schwab et al.9
Structured interview for TBI diagnosis
A 22-item interview was developed to establish the nature, probability, and severity of deployment-related TBI among OEF/OIF veterans and to provide the criterion for the sensitivity and specificity analyses (see Appendix C). The interview was created by the study team and follows Cifu et al16 TBI diagnostic criteria, which include confirmation of: (1) a possible TBI event, (2) alteration of consciousness, and (3) postconcussion symptoms. Diagnostic items were added based on the clinical experience of the authors to include detailed questions regarding circumstances and events leading up to, during, and following the injury, confirmation of information that was personally recalled versus that which was recounted to the veteran later, and careful documentation of associated symptoms and of any treatment received. The interview yielded ratings for the probability and severity of each TBI event, based on the summary judgments of the interviewing psychologists in consideration of the totality of the responses. Any injury rated as “very likely” or “almost certainly” to have resulted in a TBI was counted as a positive diagnosis. A companion decision tool facilitated standard ratings across examiners. For those participants who reported experiencing multiple TBI events, a separate set of interview questions and ratings was completed for each event. The content of the diagnostic interview was reviewed via e-mail and was approved by the National Director of Physical Medicine and Rehabilitation Services for Veterans Health Administration and by a VA Health Services Research and Development appointed steering committee of nationally recognized TBI experts.
PTSD checklist (military version)
The PCL-M is a widely used 17-item checklist developed at the National Center for PTSD. Referenced to a self-identified military-related traumatic event, the PCL-M measures symptoms of DSM-IV diagnostic criteria B (reexperiencing of the trauma), C (avoidance of trauma-related stimuli), and D (increased arousal). Although diagnostic cutoff scores for the PCL-M in the OEF/OIF cohort as low as 30 have been suggested,17 a standard (and more conservative) cutoff score of 50 is typically used to identify probable PTSD in the general veteran population.
Data for this study were extracted from a comprehensive test battery completed as part of the parent project.
All OEF/OIF veterans in the VA Registry living in VISN 2 at the beginning of the study (August 2008: approximately 7000) were sent introductory letters describing the parent project. Opt-out return postcards were included for those veterans who preferred not to be contacted by telephone for more information, screening, or enrollment. Approximately, 8% (569) of the postcards were returned, and those veterans were removed from our list. Veterans who did not return the postcards were considered to be available for telephone recruitment and TBI screening and were contacted by research assistants. A waiver of informed consent was obtained for the telephone-screening phase of the study. Eligible OEF/OIF veterans referred to VISN 2 neuropsychology and polytrauma clinics were also invited to participate. Referred veterans were informed of the study and recruited over the phone, at the time that their clinic appointment was scheduled. Efforts were focused on oversampling women and minorities by identification on the registry and in consultation with the local Minority Affairs Coordinator. Enrollment of the first 500 veterans to consent to the study was completed in January 2010.
TBI and PTSD screening and diagnosis
The first administration of the VATBIST was completed over the phone by 1 of 5 master's level research assistants who were trained and supervised by licensed psychologists. Initial TBI screening was typically completed in less than 5 minutes. The measure was administered again, in person, approximately 2 weeks later, in the context of the first parent study visit. The second administration was for the purpose of assessing test reliability of the VATBIST and is not the VA standard of care. Following completion of informed consent procedures, the VATBIST and comprehensive structured interviews were administered by 1 of 6 licensed psychologists who were unaware of the results of the initial screening. Results of the initial screen were retained in the offices of the research assistants and were not viewed by the psychologists prior to readministration of the measure. The second administration of the VATBIST and the structured interviews were completed within 30 minutes for most participants. The PCL-M was administered, along with several other self-report forms for the parent study immediately following the TBI diagnostic interview.
Overview of data analysis
Prior to analysis, we conducted multiple data integrity checks at the site level and in the combined database. Questionable and missing data values were identified, checked against the original article forms, and corrected. This resulted in complete VATBIST and clinical interview results for the first administration for all 500 veterans. Ninety-seven percent (n = 487) were retained for the second administration of the VATBIST data for test-retest reliability assessment. Item analyses included examination of frequencies for each item, followed by estimation of reliability from the first to second administration of the VATBIST. Sensitivity, specificity, false positive and negative rates, and diagnostic odds ratios (DOR) were computed, with 95% confidence intervals for the estimates. Glas and colleagues18 reviewed various commonly reported indicators of test performance and recommended the DOR as a superior single indicator of discriminatory performance. These analyses were based on the criterion of TBI classification (presence or absence based on probability rating) by the neuropsychologist at each site, according to the standardized interview. Computations were performed with PASW Statistics, Version 18.0 and DAGStat.19
Follow-up analyses of reliability and diagnostic accuracy of the VATBIST by PTSD status were also completed. For these analyses, we examined both standard and OEF/OIF-specific cutoff scores on the PTSD Checklist.
Demographic characteristics of the sample are presented in Table 1. Participants self-identified their race/ethnicity according to their own standards. This information was used solely to describe the sample and to determine its similarity to the OEF/OIF population. The sample, whereas ethnically homogeneous, was consistent with overall OEF/OIF sex division.20 The mean age of the participants was 32 (range = 20–60), and 73% completed at least some college. Half were employed full time, and over half (54%) were service-connected for disability. The time elapsed since the most recent TBI averaged 41 months, with a range of 0 to 96 months.
Descriptive statistics for VATBIST
Table 2 lists the frequency of endorsement for each of the 4 VATBIST questions and their constituent components. Recall that if no element of a given question is endorsed, the screen is discontinued and is considered to be negative. It is noteworthy that all 500 participants endorsed at least one experience that might have caused a TBI; there were no completely negative responses to question 1. The most frequently occurring experience was blast or explosion exposure, with a positive response rate of 85% (n = 421). In contrast, less than 6% (n = 28) endorsed a bullet or fragment impact above the shoulder. The mean number of potential TBI events experienced was 2.2/5 (SD = 1.3). The most frequently reported immediate symptom following the event was “being dazed, confused, or ‘seeing stars',” with 70% (n = 314) of the veterans screened providing affirmative responses. The mean number of subitems endorsed on question 2 was 3.47/5 (SD = 2.24).
Questions 3 and 4 of the VATBIST assess the same symptoms, but with reference to different time frames. Question 3 assesses symptoms proximal to the event, whereas question 4 refers to recent symptoms (within the past week). As many of these injuries occurred in the fairly distant past, it is reasonable to assume that responses to questions 3 and 4 would differ significantly. Nonetheless, these questions both produced high rates of positive responses, ranging from a low on current “sensitivity to bright light” reported by 176 veterans (54% of those asked) in question 4 to a high of 287 (or 85.2%) for “sleep problems” around the time of the event in question 3. The mean number of problems reported in questions 3 and 4 was similar: 3.47/6 initial problems (SD = 2.24) on question 3 and 3.30/6 symptoms in the past week (SD = 2.12). Complaints of “memory problems” and “irritability” were endorsed more frequently for current status than at the time of the injury event, whereas “balance problems” declined significantly. The increase of subjective memory impairment and irritability over time may be more reflective of primary affective disturbances than of residual TBI.
Descriptive statistics for the structured interview for TBI diagnosis
The structured interview for TBI diagnosis includes 22 items detailing events and symptoms. The number of deployment-related injury events and severity of injury are 2 of the key items in the interview and are summarized in Table 3. Most of the sample had no TBI (n = 281, 56.2%) or one event (n = 168, 33.6%), with a total of 219 (43.8%) given a positive diagnosis based on the interview. The severity ratings ranged from “no TBI” to loss of consciousness for more than 1 day. Most TBIs diagnosed were mild (96.8%), with loss of consciousness (if any) of less than an hour. Episodes marked by any loss of consciousness were reported by 115 (23%) veterans. The most frequently reported cause of injury (n = 138, 35.6%) was detonation of an improvised explosive device (IED).
Reliability of the VATBIST
As the items of interest are dichotomous, internal consistency was estimated with the KR-20 coefficient (equivalent to Cronbach's alpha). The estimate was 0.78 for the 309 participants with complete data for all 4 questions at the first visit and 0.77 for the 304 veterans with complete data who responded to the VATBIST in a subsequent screening interview approximately 2 weeks following the initial screen. Recall that the VATBIST is discontinued after a negative response to all components of any single question. This accounts for the reduced sample sizes of 309 and 304.
Subgroup internal consistency analyses were conducted based on PTSD status. The mean PCL-M score for the sample was 45.45 (SD = 16.63) with a range of 17 to 84. Because there is some question as to the most valid cutoff for a probable PTSD diagnosis based on the PCL-M for the OEF/OIF cohort, our criterion was set at both 30 and 50. For the lower criterion of a PCL-M score of 30, the KR-20 coefficients were 0.76 (n = 280) and 0.75 (n = 273) for the first and second administrations, respectively. When the PCL-M criterion was set at 50, KR-20 coefficients were 0.77 (n = 166) for the first administration and 0.76 (n = 166) for the second. The smaller sample sizes for these analyses reflect the impact of selecting out veterans below the criterion, along with missing data. KR-20 coefficients for the full sample and for the PTSD subgroups were remarkably similar, and all suggested robust internal consistency.
Responses to the VATBIST were examined for consistency of reports across the 2 assessment points with 2 indicators: percent that were consistent and Phi coefficients (equivalent to the Pearson Correlation Coefficient). Consistency in reports was examined by summing the number of participants who gave the same answer at both time points and those who did not. Frequencies, percentages and the Phi coefficients for each item are shown in Table 4. The overall Phi coefficient for the complete VATBIST, representing test-retest reliability, was high (.80, P < .001). The percentages of consistent responses were moderate to high, ranging from 72 to 96. The least consistent items were complaints of memory and balance problems proximal to the injury event (72% consistent from Time 1 to Time 2).
Sensitivity and specificity of the VATBIST
The diagnostic accuracy of the VATBIST was assessed by estimating sensitivity, specificity, false positive and negative rates, and diagnostic odds ratios (DOR) for the overall screening tool, as well as for each question and subitem. The results of these analyses are presented in Table 5. The VATBIST as a whole was sensitive to the presence of a confirmed TBI 94% of the time, and was specific in 59% of the cases. These estimates are associated with a false positive rate of 41% (the complement of the specificity estimate) and a false negative rate of 6% (the complement of the sensitivity estimate). As the goal of the VATBIST was to cast a wide net in identifying veterans with potential TBI, this low false negative rate is reassuring. Nonetheless, we were curious as to why the sensitivity rate was not even higher. Inspection of the false negative cases revealed that all sustained their TBIs outside of deployment, either in predeployment training activities or postdeployment activities. As such, the VATBIST was perfectly sensitive for deployment related injuries.
The lower bound estimate of the DOR was 12.63, meaning that veterans with a positive screen on the VATBIST were at least 12.6 times as likely to have a confirmed TBI compared to those with a negative screen. Examination of the estimates on the 4 individual questions shows that the first 2, events and immediate symptoms, were strongly associated with the final TBI diagnosis, but with different patterns of sensitivity and specificity. Question 1 produced nearly perfect sensitivity at 0.99, but with a low specificity (0.18) and high-false positive rate (0.82). In contrast, the subitems in question 2 were less sensitive than those in question 1 (0.63 vs 0.99) but were much more specific (0.94 vs 0.18). The overall diagnostic odds ratios for questions 1 and 2 were significant predictors of a TBI diagnosis. The DOR for question 1 had a lower bound estimate of 4.79. The lower bound estimate for question 2 was 11.62, meaning that a positive response to at least one of the subitems in question 2 would result in a positive diagnosis at nearly a 12 to 1 rate, relative to negative responses to the question.
As in the examination of the potential impact of PTSD on the reliability of the VATBIST, the diagnostic accuracy estimates were also broken down by PTSD status, again using PCL-M total score cutoffs of 30 and 50. Figure 1 presents the results graphically. It is clear that the presence of probable PTSD, using either criterion, differentially affects the validity of the VATBIST. The DORs when PTSD is absent, using either cutoff, are the strongest of the 4 at 24.5 (cutoff = 50) and 24 (cutoff = 30). Although the sensitivity of the screen drops from 0.91 to 0.78 (with an associated 17% increase in false negatives) when the criterion is set at the lower cutoff of 30, the difference is offset by the better specificity of the lower criterion (87% vs 71%).
In summary, the VATBIST appears to have high-internal consistency, moderate to high test-reliability, high sensitivity and moderate specificity. When veterans with probable PTSD were excluded from analysis, the lower bound estimate of the diagnostic odds ratio for the VATBIST was extremely high (24). In the “real world” context, including all veterans in our sample, the DOR remained impressive, with a lower bound DOR estimate of 12.6, suggesting that veterans who screen positive for TBI based on this tool are over 12 times more likely to have actually sustained such an injury than are veterans who screen negative. Veteran traumatic brain injury screening tool serves as a valid and reliable screening tool for veterans and can inform primary care providers of when to refer for further workup for cases of suspected TBI.
When the VATBIST was implemented across VA in 2007, little was known about its reliability or validity. To our knowledge, this is the first large-scale study of the psychometric properties of the VATBIST. Among our cohort of OEF/OIF veterans, the tool was found to be strongly internally consistent, reliable over a 2-week interval, highly sensitive to interview-confirmed TBI and moderately specific overall. The diagnostic odds ratio of the tool as a whole was impressive at 12.6 and increased markedly to at least 24 when participants with probable PTSD were removed from the sample.
Reliability and validity
Ninety-one percent of our sample responded consistently to the VATBIST as a whole after a test-retest interval of approximately 2 weeks, with a Phi coefficient of 0.80. Internal consistency values were similar, with a minimum value of 0.77, suggesting excellent reliability by multiple indicators. These overall reliability coefficients were strikingly similar to those obtained for probable PTSD subgroups, regardless of PCL-M cutoff (50 or 30). The presence of significant PTSD symptoms did not appear to compromise the reliability of the VATBIST.
The first 2 questions of the VATBIST (regarding exposure to a potential TBI event and immediate symptoms such as alteration of consciousness) appeared to be the most salient. A positive response to any possible TBI-related event in question 1 resulted in a high DOR (15.6). With 0.99 sensitivity and endorsement of at least one of the subitems by the entire cohort, it was clear that a positive response to question 1 was a necessary but insufficient predictor of TBI. Question 2 was arguably the “best” item, as a positive response to any of the subitems yielded a DOR of more than 25. Not surprisingly, endorsement of loss of consciousness was the most specific subitem (0.94), followed closely by concussion and head injury (both 0.92). Alteration of consciousness or “seeing stars” was very frequently endorsed (70%) but was far less specific to confirmed TBI (0.49) than was loss of consciousness. Reported alteration of consciousness may also reflect the emotionally traumatic nature of these events, and its ubiquity underscores the complexity of TBI and stress symptoms.
Endorsement of any element in question 3 (pertaining to postconcussive symptoms around the time of the injury event) resulted in a DOR of over 6, which was significant but less compelling than the odds ratios associated with questions 1 and 2. The subitems of question 3 had generally high sensitivity but low specificity. In question 4 (current postconcussive symptoms) residual balance problems was the element most specfic to a TBI diagnosis (0.73). Overall, however, the DOR of any one subitem of question 4 was low (1.2). As many of the symptoms included in questions 3 and 4 are common in the general population,21 and given that the average time since injury in this sample was over 3 years, it is not surprising that these items provided less diagnostic accuracy than did questions 1 and 2. It seems reasonable that one might likely recall involvement in a TBI-related event and whether or not there was an alteration of consciousness (by direct recall or reports from others), whereas memories of specific symptoms afterward might be less clear (especially for mild injuries). As symptoms associated with mTBI usually resolve spontaneously in the days or weeks following the injury,22 the symptoms reported on question 4 were not necessarily directly related to the injury identified in question 1 and may have reflected affective problems, such as PTSD, depression, or generalized anxiety.
The presence of probable PTSD had no discernible effect on the reliability and little effect on the sensitivity of the VATBIST, regardless of the PCL-M cut score. This was not the case, however, with regard to specificity and diagnostic odds ratios. Specificity was reduced by more than half (from .71 to .32) for participants with PCL-M scores of 50 or higher, relative to those with scores below 50. The corresponding DORs were 11.2 and 24.5, respectively. When the PCL-M criterion was 30, however, specificity was just 0.47 for the PTSD-positive group, compared to a formidable 0.87 for veterans scoring below 30 on the PTSD measure. Nonetheless, the DORs were high for both groups (18.5 and 24, respectively). These findings underscore the importance of screening for PTSD symptoms along with those related to TBI. Although the symptoms of the 2 disorders often overlap, a clear understanding of the relative presence and severity of each will facilitate better care for our veteran patients.
Similar to Schwab's examination of the DoD's Brief TBI Screen, we found that positive VATBIST screens were much more common than confirmed TBIs. This is not a criticism of the screening tool. The measure was designed to err on the side of false positives to avoid missing any significant brain injuries. Positive screens are followed up across VA with comprehensive evaluations to confirm the diagnosis and prescribe treatment according to a standard algorithm.16 Several studies are underway to examine these follow-up procedures (eg, VA Health Services Research and Development IAC 08–101, SDR-08–377, SDR-08–411).
Responses on the VATBIST were compared with the results of a structured diagnostic interview designed to confirm the probability and severity of TBIs in our OEF/OIF sample. We found that nearly half of our veterans (44%) likely sustained at least one TBI. Previous studies3,5–7 found the incidence of TBI in OEF/OIF veterans to be around 20%. As about half of our study group enrolled following clinical referral, this oversampling of TBI-positive veterans is not unexpected. About 10% of our group was confirmed as having sustained more than one TBI. Although a number of veteran participants initially estimated that they had sustained multiple TBIs, careful questioning in the diagnostic interview showed that very few (3%) had likely experienced 3 or more TBIs. Further, the vast majority of these injuries were mild. Twenty-three percent reported a loss of consciousness, but less than 2% noted a loss of consciousness lasting an hour or more. Our most seriously injured participant, who was unconscious for more than 24 hours, was injured in a fall after his return from deployment. The relative lack of moderate or severe deployment-related TBIs in our sample could be related to the nature of our study sites. In the VA polytrauma system of care, many veterans with moderate or severe TBIs are followed by 1 of 4 regional polytrauma rehabilitation centers, none of which were included in our investigation. Veterans with less serious injuries might be more likely to return to VA medical centers closer to their homes.
Limitations of the study
While we attempted to oversample women and minorities for this study, our participants were mostly white (85%). Rutherford and Allegria20 reported that about 66% of OEF/OIF services members overall are white. As such, generalizability of our findings is limited with regard to racial and ethnic diversity. The percentage of women in our study, however, was similar to the full OEF/OIF cohort (10% vs 11% nationally). As our sample was geographically restricted to our local VISN, it might not accurately represent veterans from other parts of the country. About half of our participants were enrolled following referral for neuropsychological assessment, and this may have biased the sample toward a group with a higher likelihood of being TBI and/or PTSD positive.
Although the outcome of a clinical interview is considered to be the best “gold standard” criterion currently available for TBI diagnosis of returning veterans, it is imperfect. No clear biomarkers exist for mTBI, and information gleaned from direct observation at the time of injury, as might be used in cases of civilian TBI, is unavailable for returning warriors. In addition, the psychologists who conducted the diagnostic interviews were unaware of the results of the first research administration of the VATBIST but not the second. It could be argued that knowledge of the screening results at the second time point biased the outcome of the diagnostic interview, although careful clinical due diligence was implemented to avoid such contamination.
Finally, it needs to be emphasized that the research assistants who conducted the initial screens were trained and supervised by licensed psychologists, and the psychologists themselves completed the follow-up screens. This is not the standard of care for TBI screening across the VA. Most staff completing the screens have little to no training or supervision for using the measure, and the screening is often completed along with a number of other surveys in the context of a busy medical appointment. Our previous work23 showed that insufficient training and supervision of clinical staff, as well as “screening fatigue,” likely contributed to poor results with a primary care-based dementia screening initiative. As such, our findings may represent a “best possible” outcome that does not generalize to all clinicians.
Recommendations for future research
We suggest that future research include further psychometric as well as clinically oriented studies. Perhaps the first and most obvious study would be a replication with another veteran sample to add to the evidence base regarding reliability and diagnostic accuracy of the VATBIST. Sample characteristics including injury severity and demographic variables would be of primary importance in planning such a replication. The moderate specificity of the VATBIST raises the question of costs associated with false positive screens, including the cost of follow-up assessment of positively screened veterans. It could be argued that awareness of comorbid PTSD may be more informative than questions 3 and 4 of the VATBIST, particularly with regard to enhancing specificity. Further research might involve modification or supplementation of the screen, as well as consideration of alternative gold standards for diagnosis. Of course, the primary purpose of any clinically oriented screening procedure is to identify cases as soon as possible to increase effectiveness and efficiency of service delivery, as well as to improve long-term clinical outcomes. Therefore, health care utilization and outcome studies are necessary to justify continued screening.
Despite its rapid implementation prior to psychometric scrutiny, the VATBIST proved to be a reliable screening tool with good to excellent diagnostic accuracy in this sample of 500 OEF/OIF veterans. The presence of significant PTSD symptoms reduced the specificity of the measure and highlights the need for careful clinical follow-up of positive screens.
© 2011 Lippincott Williams & Wilkins, Inc.