Secondary Logo

Share this article on:

Digit Symbol Substitution Test: The Case for Sensitivity Over Specificity in Neuropsychological Testing

Jaeger, Judith, PhD, MPA

Journal of Clinical Psychopharmacology: October 2018 - Volume 38 - Issue 5 - p 513–519
doi: 10.1097/JCP.0000000000000941
Review Articles

Purpose/Background Development of the Digit Symbol Substitution Test (DSST) was initiated over a century ago as an experimental tool to understand human associative learning. Its clinical utility, owing to its brevity and high discriminant validity, was first recognized in the 1940s, and now the DSST is among the most commonly used tests in clinical neuropsychology.

Methods Specific studies and articles were reviewed to illustrate what the test measures, to evaluate its sensitivity to change, and to discuss its use in clinical practice.

Results The DSST is a valid and sensitive measure of cognitive dysfunction impacted by many domains. Performance on the DSST correlates with real-world functional outcomes (eg, the ability to accomplish everyday tasks) and recovery from functional disability in a range of psychiatric conditions including schizophrenia and major depressive disorder. Importantly, the DSST has been demonstrated to be sensitive to changes in cognitive functioning in patients with major depressive disorder and offers promise as a clinical decision-making tool for monitoring treatment effects in this and other disorders affecting cognition.

Implications/Conclusions The DSST is sensitive to the presence of cognitive dysfunction as well as to change in cognitive function across a wide range of clinical populations but has low specificity to determine exactly which cognitive domain has been affected. However, the DSST offers a practical and effective method to monitor cognitive functions over time in clinical practice.

From CognitionMetrics, LLC, Wilmington, DE; and Albert Einstein College of Medicine, Bronx, NY.

Received August 3, 2017; accepted after revision June 27, 2018.

Reprints: Judith Jaeger, PhD, MPA, CognitionMetrics, LLC, 4023 Kennett Pike, #253, Wilmington, DE 19807 (e-mail: JJaeger@cognitionmetrics.com).

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

Back to Top | Article Outline

DESCRIPTION AND HISTORY

The Digit Symbol Substitution Test (DSST) was initially developed as an experimental tool over a century ago by researchers seeking to understand human associative learning.1,2 The DSST is a paper-and-pencil cognitive test presented on a single sheet of paper that requires a subject to match symbols to numbers according to a key located on the top of the page. The subject copies the symbol into spaces below a row of numbers. The number of correct symbols within the allowed time, usually 90 to 120 seconds, constitutes the score ( Fig. 1).3 The DSST is perhaps the most commonly used test in all of neuropsychology, owing to several inherent properties: brevity, reliability, and the minimal impact of language, culture, and education on test performance. Yet, the question of “what it measures” still has no definitive answer, a consequence of the historical context from which it emerged.

FIGURE 1

FIGURE 1

The intent of this article is to provide an overview of the DSST, addressing what cognitive operations are required to perform it and what impaired DSST performance may indicate. The aim is to consider its utility in monitoring cognitive functions over time in clinical practice. With this in mind, selected studies have been highlighted to help illustrate the test's sensitivity to change, as well as how performance on the DSST correlates with real-world functioning, so as to provide practitioners with insight into its use for monitoring treatment effects in patients having a range of disorders, with particular attention to its utility for management of major depressive disorder (MDD).

The utility of the DSST as a clinical tool in neuropsychology first became evident when it was shown to reliably distinguish patients with brain damage from healthy patients during the screening of soldiers in World War II. The use of the DSST widened after becoming incorporated in the Wechsler-Bellevue Intelligence Scale (WBIS), developed in 19392; the WBIS relied heavily on early versions of the DSST dating to at least 1900.1 Sometimes termed “coding” or “symbol coding,” the test paradigm has survived with minor alterations to the most recent version found in the Wechsler Adult Intelligence Scale (WAIS), which is currently in its fourth edition (WAIS IV).4,5 As a subtest of the WAIS, the DSST has undergone repeated and rigorous psychometric validation such as test-retest reliability and discriminant validity in a range of patient samples; however, modest differences exist between these versions.

The original WAIS, developed in 1955, has 90 stimulus items (number of blanks into which the symbols are copied), all of which were among the 67 stimulus items in its predecessor, the WBIS.2,6 The DSST was not substantially different in the subsequent version of the WAIS, the WAIS-Revised. However, in WAIS-III, the test duration was extended from 90 to 120 seconds, the response form filled a full page, and the size and spacing of the stimuli were increased so that hooked left-handers were no longer required to move their hands to see the key items.

Symbol coding paradigms comparable to the DSST (with different symbols) are included as subtests in the Brief Assessment of Cognition in Schizophrenia and the Repeatable Battery for the Assessment of Neuropsychological Status.7,8 The Symbol Digit Modalities Test uses essentially the same method, except in reverse; instead of drawing symbols that match digits, the subject is required to write the matching digit in the blank.9 This version of the coding paradigm has the advantage that it can be easily altered to allow oral responses in disorders that affect motor functioning (eg, multiple sclerosis).10

Computerized tests offer another option for monitoring change over time in a clinical setting. To be useful, a measure must be validated for sensitivity to change and reliability in the population in which the clinical application is being contemplated. Several commercial efforts have been undertaken along these lines. For example, the Cambridge Neuropsychological Test Automated Battery developed by Cambridge Cognition is being used to track memory decline in elders with a view toward early detection of dementia.11,12 Similarly, Cognigram, a version of the Cogstate Brief Battery formatted for use in the clinic, consists of a 10-minute battery of computerized tasks requiring attention, processing speed, memory, and executive functioning that have been validated for sensitivity to change in a range of conditions including MDD.13–16 Both of these tests are commercially available and approved for clinical use. THINC-IT is a freely available computerized test that also includes a questionnaire to capture subjective cognitive complaints; it is designed for monitoring change in MDD and is becoming increasingly used, although it has not yet achieved regulatory approval.17 An advantage of the DSST over computerized tools is its simplicity, speed (2 minutes, in contrast to 10–15 minutes for computerized options), and ready accessibility. No computer or software subscription is needed; only the test form, a writing implement, and a timer are required. Like the computerized version, the validity and reliability of the paper DSST depend on the test being administered in the manner it was validated. When scoring the DSST, speed and accuracy are both taken into account, allowing the examiner to use norms or refer to research findings for comparison.

As illustrated below, the DSST is clinically useful because it is sensitive to cognitive deficit in a wide range of brain diseases and conditions. However, without further testing of the patient, the DSST is not informative about the nature of the deficit. The widespread clinical use of the DSST has produced large amounts of data that have helped shed some light on what the DSST measures and the significance of deficits and of change over time in its performance.

Back to Top | Article Outline

What Does the DSST Measure?

The DSST measures a range of cognitive operations. Good performance on the DSST requires intact motor speed, attention, and visuoperceptual functions, including scanning and the ability to write or draw (ie, basic manual dexterity). Performance might also be affected by associative learning. For example, if pairings are rapidly learned following the first few trials, performance speed will improve because the subject will not need to refer to the key to check the accuracy of each pairing. The decision to consciously engage in this learning strategy to improve performance speed calls for the executive functions of planning and strategizing. Working memory, another executive function, is likely required to hold in mind the task rules and for the continual updating of required symbol-digit pairs.

According to Lezak's18 widely used neuropsychological textbook, the DSST is a measure of “Complex Attention,” yet the author indicates that the cognitive aging literature also points to a strong influence of motor speed on performance. For example, in elderly populations, copying speed alone explains 72% of the variance in performance difference between young and elderly groups.18 This finding supports the observation that the DSST is a polyfactorial test and that impairment in any one of the required domains (ie, motor speed associated with aging) results in performance decline. Darby and Walsh19 emphasized that performance on the DSST is the final common pathway for expression of quite diverse types of impairment and described (quoting Kinsbourne) DSST responses as “the end product of the integration of visual perceptual, oculomotor, fine manual motor, and mental functions.”

Maxwell20 argued that the DSST should be classified as a verbal rather than a performance subtest in light of a slightly higher correlation with the verbal intelligence quotient than the performance intelligence quotient in the normative sample of Wechsler. Laux and Lane21 reviewed the correlations of the DSST with other measures in the literature and also suggested that DSST performance involves verbal ability.

Beres and Baron22 measured the effect of active training on DSST performance. The authors showed that both elderly and young subjects improved not only within each day but also between days. Although the older women performed at lower levels than did the younger ones at all points during training, both age groups of women improved fairly consistently over the 20 administrations given on the same day (likely because they were learning the actual symbol pairs). Further, both groups improved from day to day, even though on each day they were given different forms of the test. This observation suggests that improvement with exposure occurs in part for reasons other than associative learning of the pairs. These findings indicate that the executive function of strategizing or consciously exerting effort to learn the pairings is among the activities that contribute to performance on this test. It is noteworthy that both groups improved with practice, and the difference between the groups remained fairly constant. That the DSST is sensitive to age effects is well known.23 The Beres and Baron22 findings suggest that age-related effects are attributable to age differences in motor speed and not in higher cognitive functions. The DSST has been a useful tool to demonstrate age differences in the effects of drugs on cognition, for example, owing to reduced clearance resulting in higher plasma concentrations in older than in younger subjects.24

Finally, the possibility that executive functioning contributes to performance on the DSST is supported by findings of Thornton and Carmody.25 The authors used a Symbol Digit test to examine the relationship between quantitative electroencephalography (QEEG) variables and cognitive performance.25 The authors showed that test performance was associated with QEEG indices, including relative power and coherence at several frequency bands, particularly within the frontal region of the brain.25 This correlation of specific QEEG indices with activation in the frontal lobes supports the theory that executive functioning plays a part in DSST performance.

Back to Top | Article Outline

Sensitivity and Sensitivity to Change of the DSST

David Wechsler2 first described the DSST's sensitivity to brain damage in 1936. Since then, the DSST was found to be the single most sensitive test, among WAIS subtests, to determine the presence of brain damage and discriminate between patient groups (including healthy comparison groups).26 For example, Glosser et al27 noted the sensitivity of the DSST in alcoholic Korsakoff patients, patients with right hemisphere damage, and patients with chronic alcoholism. While psychomotor retardation contributed to impaired performance in these patient populations, the investigators, using other cognitive measures, determined that visuoperceptive processing impairments were also involved in DSST performance.

The sensitivity of the DSST is owed, in part, to its property as a polyfactoriaI measure.28 As alluded to above, impaired performance on a polyfactorial measure may arise for different reasons in differently impaired populations. For example, age-related impaired performance may be attributed, in part, to a decline in motor speed, often associated with aging (as noted earlier in this article). In alcoholic Korsakoff patients, patients with right hemisphere damage, and patients with chronic alcoholism, performance deficits arise from impairments in visuoperceptive processing.27 In schizophrenic patients, processing speed is known to be a critically impaired cognitive ability. In fact, deficits in processing speed as measured by DSST performance may mediate a wide range of other cognitive deficits and may further reflect a final common pathway of impairment in a range of simple and complex cognitive processes that are likely to include working memory and executive functions.29

The ability of the DSST to detect deterioration in processing speed has also been useful as a complementary cognitive assessment in the screening of patients with diabetes mellitus type 2.30 In both middle-aged and older persons, diabetes mellitus type 2 has been found to be associated with an increased risk of mild cognitive impairment and dementia.30 Therefore, brief cognitive tests such as the DSST may prove to be a practical and effective method for use in clinical practice to detect decrements in cognitive processing over time. Interestingly, reports of the predictive nature of lower DSST scores on the development of clinical and subclinical disorders of cognition and mobility have recently been published.31 With additional confirmatory studies, the authors postulate that slower processing speeds as detected by the DSST may serve as a type of biomarker for disorders of cognition, mobility, and possibly even mood. Use of the DSST for studying the pathophysiology of psychomotor slowing with age may also provide insights into the pathology of age-related brain disorders.31 Clearly, the utility of the DSST in the evaluation and management of patients across various therapeutic indications is evident.

The DSST's sensitivity to change, both acutely and chronically, is one of its hallmark features. For this reason, the DSST has been used as a standard tool in clinical pharmacology studies from as early as the 1970s. A review of such studies may offer a means for benchmarking the magnitude of change in standardized effect size units or as a function of the DSST's correlation with other biological measures. The clinical meaningfulness of changes of different magnitudes may then be considered from this vantage point.

Roth et al32 evaluated the sedative effects of antihistamines on a battery of tests including the DSST. The authors demonstrated that the DSST could statistically significantly distinguish the cognitive effect of the H1 antagonist diphenhydramine from placebo with an effect size of 0.28 (calculated from the published statistics).

Thapar et al33 compared the cognitive and sedating effects of several anesthetic agents at doses commonly used in ambulatory surgery. The dependent variable was performance on a modified DSST, timed for 1 minute. As a standard control, alcohol was administered to the subjects to a blood alcohol concentration (BAC) of 0.11%, a dose associated with significant intoxication. Digit Symbol Substitution Test performance was markedly reduced during alcohol intoxication with scores approximately 10 points lower than that of placebo. Although learning or practice effects may occur because the same test is administered multiple times, the randomized design of the study overcomes any resulting confounding factors. Different magnitudes of cognitive effect, as measured with the DSST, can be used to make comparative inferences about the relative effects of various doses of agents, such as alcohol, on cognitive performance (Fig. 2).33

FIGURE 2

FIGURE 2

Greenblatt et al34 compared the sensitivity to change of the DSST with that of the electroencephalogram (EEG) to the effects of a single dose of 0.375 mg triazolam versus placebo. Benzodiazepines (eg, triazolam) are widely prescribed drugs for the treatment of anxiety, insomnia, seizures, alcohol withdrawal, and many other disorders. Compared with placebo, triazolam significantly impaired performance on the DSST (P < 0.001) and increased beta amplitude on the EEG (P < 0.002).34 Both DSST and EEG changes closely tracked changes in plasma triazolam concentrations over time.34

The DSST is often used as a pharmacodynamic marker to assess pharmacokinetic-pharmacodynamic relationships. For example, Greenblatt et al35 used the DSST to evaluate age and sex effects on the pharmacokinetics and pharmacodynamics of 0.25-mg triazolam in 61 healthy men and women, aged 20 to 75 years. This double-blind crossover study demonstrated a statistically significant effect of age on predose DSST performance, with lower baseline scores in the elderly subjects further reinforcing the effect of age on DSST performance. The percent decrement in DSST scores with age was also significantly greater in men than in women. In a similar study, Greenblatt et al36 evaluated the effect of dose and sex on the pharmacokinetics and pharmacodynamics of the sedative zolpidem in healthy, nonelderly male and female volunteers and included the DSST among the pharmacodynamic measures. In this placebo-controlled, 4-way crossover study, evaluation of the pharmacodynamic effect area under the curve indicated a significant influence of sex, independent of dose, on the DSST such that effects on women were seen to be greater than those on men.36

The DSST is sensitive enough to distinguish changes over time between different sleep groups in sleep studies. Sleep experiments have used the DSST as a tool to study the cognitive effects of chronic sleep deprivation. Figure 3 shows the mean performance on a computerized DSST (tested every 2 hours) in a sample of 35 subjects randomized to receive 4, 6, or 8 hours of sleep each night for 14 days.37,38 The 8-hour group showed some improvement over the first few days (possibly a learning effect), whereas the 4-hour group differed from the 8-hour group by more than 12 points by the end of the first week and continued to decline over the second week. In the 6-hour group, there was no change from baseline over time. Tucker et al39 also measured the effect of sleep deprivation on DSST performance and yielded similar results. Another example of the DSST's sensitivity to chronic change is illustrated in abstinence testing of chronic alcoholics. Only the DSST, among a battery of neuropsychological tests, was able to measure the benefit from long-term abstinence (nearly 2 years) over short-term abstinence (6 weeks) in chronic alcoholism, suggesting its greater sensitivity to change compared with other measures.28

FIGURE 3

FIGURE 3

The wide use of the DSST across so many applications permits unique opportunities for comparison. For example, it is reasonable to argue that using widely understood perturbations, such as sleep deprivation, alcohol, or common pharmacologic agents as benchmarks, offers a reasonable approach to appreciating the clinical meaningfulness of a change in cognition on the DSST. Recent interest in office-based clinical monitoring of cognitive change in patients having MDD and the sensitivity of the DSST to cognitive change in this population suggests true clinical value may be realized by understanding the clinical meaningfulness of change on the DSST.

Back to Top | Article Outline

Clinical Meaningfulness in Patients With MDD

Many studies have examined neuropsychological performance, using a variety of neuropsychological tests, in patients with MDD, and overall, impairments of moderate size have been observed.40 Further, these impairments appeared to have been present from the first depressive episode.40 A recent meta-analysis on executive dysfunction estimated the magnitude of impairment relative to control subjects on a wide array of neuropsychological tests and concluded that “MDD is reliably associated with impaired performance on neuropsychological measures of executive functioning.”41 In that report, data from 1904 subjects evaluated using the DSST in a total of 22 studies demonstrated an impairment of moderate magnitude among patients with MDD with a mean effect size (Cohen’s d) relative to control subjects of 0.55 (confidence interval, 0.34–0.75; P < 0.001). The effect size of 0.55 was in line with the effect sizes of other measures examined. In elderly depressed patients relative to elderly control subjects, Nebes et al42 reported DSST decrements of about double this magnitude and further found that performance was worse in patients with late compared with early onset of disease (P < 0.04).

Vortioxetine, an antidepressant indicated to treat MDD, was evaluated in 2 placebo-controlled clinical trials, and the DSST was used to evaluate the efficacy of 10 and 20 mg of vortioxetine for improving cognitive function in patients with MDD.43,44 In the first study, the effect sizes of treatment with 10 or 20 mg of vortioxetine on DSST scores were, respectively, 0.51 and 0.52.43 The second study also showed positive treatment effects on the DSST, although with a smaller effect size of 0.254 for 10 to 20 mg vortioxetine.44 In addition, a meta-analysis showed significant effects of vortioxetine versus placebo in improving cognitive function assessed with the DSST in patients with MDD and that these improvements were greater than those observed with other antidepressants.45

In an earlier report of MDD patients studied 6 months after discharge from hospital for an MDD episode, the DSST was strongly associated with the magnitude of functional recovery in work, school, and housing, yielding an odds ratio of nearly 20.46 Hence, there is evidence that the impairment and change measured by the DSST reflect meaningful differences relevant to the clinical management of MDD patients. Further, there is growing awareness of the prevalence of cognitive symptoms and their impact on life functioning in MDD patients, leading to an increase in the recognition of such symptoms as clinically important targets for therapeutic improvement.47 Although the importance of cognition for functioning and quality of life may be regarded as self-evident, the meaningfulness of a given magnitude of impairment or change on a brief measure such as the DSST is often questioned. A more refined, data-driven method for establishing the meaningfulness of a particular magnitude of change would be of enormous interest to clinicians seeking to use this test for clinical decision making.

As discussed previously, the DSST's sensitivity to change is one of its main strengths. The magnitude of change on DSST performance, as well as the effect on DSST of an illness state relative to a state of health, can be represented using a standard effect size statistic. The standard effect size is a metric that allows one to compare results of different studies, for example, those examining pretreatment versus posttreatment or treatment (drug) versus placebo, or those examining the magnitude of deficit (relative to normal) in various patient groups.

To appreciate the clinical meaningfulness of a statistically significant cognitive effect of a new pharmacologic agent, it is useful to review the effects of a range of known compounds for which the clinical meaningfulness of effect in healthy volunteers is fairly well understood. Figure 4 illustrates one way to consider the clinical meaningfulness of effect sizes through direct comparison with familiar benchmarks.32,39,43,44,48–53 The figure shows the magnitude of impairment seen with a well-understood exposure to ethanol (ie, BAC of 0.088%) and compares it with that of 2 mg of lorazepam, a sedative for the treatment of anxiety disorders or symptoms, or 150 mg of diphenhydramine, an antihistamine with sedative effects, in healthy volunteers. Additional comparisons include effect sizes and direction of change after caffeine intake, caffeine deprivation, and 1 full night of sleep deprivation. Although benchmarks from acute adverse effects in healthy volunteers are not clinically equivalent to a sustained 8-week benefit of vortioxetine in patients with MDD, these examples can provide some context to understand the meaningfulness of effects. Such benchmarking offers an evidentiary basis for concluding that the observed improvements of DSST scores are of a magnitude likely to be clinically meaningful. Indeed, it may be possible to think of an improvement of this magnitude as comparable to removing the transient effect of a similar magnitude resulting from the BAC and lorazepam dose noted above.

FIGURE 4

FIGURE 4

Back to Top | Article Outline

CONCLUDING COMMENTS

The utility of the DSST for monitoring patients in the clinic derives from several properties, many of which are shared by other neuropsychological tests, including its brevity, its sensitivity to change, and minimal impact from education and culture. The DSST offers high sensitivity to detect impairment, but low specificity to determine exactly which cognitive domain has suffered the impairment. A legitimate question may be raised as to whether a single brief cognitive assessment tool can itself be sufficient to draw clinically meaningful conclusions under any circumstances. Neuropsychologists have become accustomed to the use of extensive batteries of tests, and redundancy of measurement may improve the reliability of any conclusion drawn about cognitive dysfunction. However, where the objective of testing is not diagnosis, but rather to identify impairment regardless of its nature and origin or to detect change within a patient, a brief and simple test such as the DSST may offer important benefits to clinical practice.

The DSST is sensitive to both the presence of cognitive dysfunction and change in cognitive function, across a wide range of clinical populations. In psychiatric patients, the DSST is sensitive to impairments and improvement in processing speed, executive functioning, and working memory. Performance on the DSST correlates with real-world functional outcomes (eg, the ability to accomplish everyday tasks) and recovery from functional disability. In addition, the DSST has been demonstrated to be sensitive to change in cognitive functioning in patients with MDD and may offer an effective means to detect clinically relevant treatment effects. In the context of clinical decision making in a physician's office, this simple cognitive test can, in combination with a psychiatric clinical evaluation, be useful at the level of the individual patient for medication choice and dosing.

Back to Top | Article Outline

ACKNOWLEDGMENTS

Medical writing assistance was provided by James Netterwald, PhD, and Martina Schwarzkopf, PhD, of inVentiv Medical Communications and supported by Takeda Pharmaceuticals USA, Inc., and H. Lundbeck A/S.

Back to Top | Article Outline

AUTHOR DISLOSURE INFORMATION

The author declares no conflicts of interest related to this article. The author is employed full time and is the owner and president of CognitionMetrics, LLC. CognitionMetrics, LLC, has, since inception, provided paid consulting services to the following pharmaceutical companies: Axsome Therapeutics, Biogen, Boehringer Ingelheim, Eisai, Forum Pharmaceuticals, iProteos, Ironwood, Janssen, Lundbeck, Pfizer, Sunovion, Takeda, and Teva Pharmaceuticals.

Back to Top | Article Outline

REFERENCES

1. Boake C. From the Binet-Simon to the Wechsler-Bellevue: tracing the history of intelligence testing. J Clin Exp Neuropsychol. 2002;24:383–405.
2. Wechsler D. The Measurement of Adult Intelligence. Baltimore, MD: The Williams & Wilkins Company; 1939.
3. Patel T, Kurdi MS. A comparative study between oral melatonin and oral midazolam on preoperative anxiety, cognitive, and psychomotor functions. J Anaesthesiol Clin Pharmacol. 2015;31:37–43.
4. Wechsler D. Wechsler Adult Intelligence Scale—Fourth Edition. Bloomington, MN: PsychCorp, an imprint of Pearson Clinical Assessment; 2008.
5. Sudarshan NJ, Bowden SC, Saklofske DH, et al. Age-related invariance of abilities measured with the Wechsler Adult Intelligence Scale-IV. Psychol Assess. 2016;28:1489–1501.
6. Wechsler D. Wechsler Adult Intelligence Scale Manual. New York, NY: Psychological Corporation; 1955.
7. Keefe RS, Harvey PD, Goldberg TE, et al. Norms and standardization of the Brief Assessment of Cognition in Schizophrenia (BACS). Schizophr Res. 2008;102:108–115.
8. Randolph C, Tierney MC, Mohr E, et al. The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS): preliminary clinical validity. J Clin Exp Neuropsychol. 1998;20:310–319.
9. Sheridan LK, Fitzgerald HE, Adams KM, et al. Normative Symbol Digit Modalities Test performance in a community-based sample. Arch Clin Neuropsychol. 2006;21:23–28.
10. Benedict RH, DeLuca J, Phillips G, et al. Validity of the Symbol Digit Modalities Test as a cognition performance outcome measure for multiple sclerosis. Mult Scler. 2017;23:721–733.
11. Sahakian BJ, Morris RG, Evenden JL, et al. A comparative study of visuospatial memory and learning in Alzheimer-type dementia and Parkinson's disease. Brain. 1988;111(Pt 3):695–718.
12. Robbins TW, James M, Owen AM, et al. Cambridge Neuropsychological Test Automated Battery (CANTAB): a factor analytic study of a large sample of normal elderly volunteers. Dementia. 1994;5:266–281.
13. Davis MT, DellaGioia N, Matuskey D, et al. Preliminary evidence concerning the pattern and magnitude of cognitive dysfunction in major depressive disorder using cogstate measures. J Affect Disord. 2017;218:82–85.
14. Sicard V, Moore RD, Ellemberg D. Sensitivity of the Cogstate Test Battery for detecting prolonged cognitive alterations stemming from sport-related concussions. Clin J Sport Med. 2017.
15. Cook NA, Kim JU, Pasha Y, et al. A pilot evaluation of a computer-based psychometric test battery designed to detect impairment in patients with cirrhosis. Int J Gen Med. 2017;10:281–289.
16. Farnsworth JL 2nd, Dargo L, Ragan BG, et al. Reliability of computerized neurocognitive tests for concussion assessment: a meta-analysis. J Athl Train. 2017;52:826–833.
17. McIntyre RS, Best MW, Bowie CR, et al. The THINC-Integrated Tool (THINC-it) screening assessment for cognitive dysfunction: validation in patients with major depressive disorder. J Clin Psychiatry. 2017;78:873–881.
18. Lezak MD. Neuropsychological Assessment. 3rd ed. New York: Oxford University Press; 1995.
19. Darby D, Walsh K. Neuropsychology. 5th ed. Melbourne: Churchill Livingstone; 2005.
20. Maxwell AE. Obtaining factor scores on the Wechsler Adult Intelligence Scale. J Ment Sci. 1960;106:1060–1062.
21. Laux LF, Lane DM. Information processing components of substitution test performance. Dermatol Int. 1985;9:111–136.
22. Beres CA, Baron A. Improved digit symbol substitution by older women as a result of extended practice. J Gerontol. 1981;36:591–597.
23. Hoyer WJ, Stawski RS, Wasylyshyn C, et al. Adult age and digit symbol substitution performance: a meta-analysis. Psychol Aging. 2004;19:211–214.
24. Greenblatt DJ, Harmatz JS, Shapiro L, et al. Sensitivity to triazolam in the elderly. N Engl J Med. 1991;324:1691–1698.
25. Thornton KE, Carmody DP. Symbol Digit and the quantitative EEG. Neurotherapy. 2012;16:210–222.
26. Russell EW. WAIS factor analysis with brain-damaged subjects using criterion measures. J Consult Clin Psychol. 1972;39:133–139.
27. Glosser G, Butters N, Kaplan E. Visuoperceptual processes in brain damaged patients on the digit symbol substitution test. Int J Neurosci. 1977;7:59–66.
28. Ryan C, Butters N, Didario B, et al. The relationship between abstinence and recovery of function in male alcoholics. Clin Neuropsychol. 1980;2:125–134.
29. Amaresha AC, Danivas V, Shivakumar V, et al. Clinical correlates of parametric digit-symbol substitution test in schizophrenia. Asian J Psychiatr. 2014;10:45–50.
30. Dong Y, Kua ZJ, Khoo EY, et al. The utility of brief cognitive tests for patients with type 2 diabetes mellitus: a systematic review. J Am Med Dir Assoc. 2016;17:889–895.
31. Rosano C, Perera S, Inzitari M, et al. Digit Symbol Substitution test and future clinical and subclinical disorders of cognition, mobility and mood in older adults. Age Ageing. 2016;45:688–695.
32. Roth T, Roehrs T, Koshorek G, et al. Sedative effects of antihistamines. J Allergy Clin Immunol. 1987;80:94–98.
33. Thapar P, Zacny JP, Thompson W, et al. Using alcohol as a standard to assess the degree of impairment induced by sedative and analgesic drugs used in ambulatory surgery. Anesthesiology. 1995;82:53–59.
34. Greenblatt DJ, Gan L, Harmatz JS, et al. Pharmocokinetics and pharmacodynamics of single-dose triazolam: electroencephalography compared with the Digit-Symbol Substitution Test. Br J Clin Pharmacol. 2005;60:244–248.
35. Greenblatt DJ, Harmatz JS, von Moltke LL, et al. Age and gender effects on the pharmacokinetics and pharmacodynamics of triazolam, a cytochrome P450 3A substrate. Clin Pharmacol Ther. 2004;76:467–479.
36. Greenblatt DJ, Harmatz JS, Singh NN, et al. Gender differences in pharmacokinetics and pharmacodynamics of zolpidem following sublingual administration. J Clin Pharmacol. 2014;54:282–290.
37. Van Dongen HP, Maislin G, Mullington JM, et al. The cumulative cost of additional wakefulness: dose-response effects on neurobehavioral functions and sleep physiology from chronic sleep restriction and total sleep deprivation. Sleep. 2003;26:117–126.
38. Banks S, Dinges DF. Behavioral and physiological consequences of sleep restriction. J Clin Sleep Med. 2007;3:519–528.
39. Tucker AM, Whitney P, Belenky G, et al. Effects of sleep deprivation on dissociated components of executive functioning. Sleep. 2010;33:47–57.
40. Lee RS, Hermens DF, Porter MA, et al. A meta-analysis of cognitive deficits in first-episode major depressive disorder. J Affect Disord. 2012;140:113–124.
41. Snyder HR. Major depressive disorder is associated with broad impairments on neuropsychological measures of executive function: a meta-analysis and review. Psychol Bull. 2013;139:81–132.
42. Nebes RD, Pollock BG, Houck PR, et al. Persistence of cognitive impairment in geriatric patients following antidepressant treatment: a randomized, double-blind clinical trial with nortriptyline and paroxetine. J Psychiatr Res. 2003;37:99–108.
43. McIntyre RS, Lophaven S, Olsen CK. A randomized, double-blind, placebo-controlled study of vortioxetine on cognitive function in depressed adults. Int J Neuropsychopharmacol. 2014;17:1557–1567.
44. Mahableshwarkar AR, Zajecka J, Jacobson W, et al. A randomized, placebo-controlled, active-reference, double-blind, flexible-dose study of the efficacy of vortioxetine on cognitive function in major depressive disorder. Neuropsychopharmacology. 2015;40:2025–2037.
45. Baune BT, Brignone M, Larsen KG. A network meta-analysis comparing effects of various antidepressant classes on the Digit Symbol Substitution Test (DSST) as a measure of cognitive dysfunction in patients with major depressive disorder. Int J Neuropsychopharmacol. 2018;21:97–107.
46. Jaeger J, Berns S, Uzelac S, et al. Neurocognitive deficits and disability in major depressive disorder. Psychiatry Res. 2006;145:39–48.
47. Mattingly G, Anderson RH, Mattingly SG, et al. The impact of cognitive challenges in major depression: the role of the primary care physician. Postgrad Med. 2016;128:665–671.
48. Katona C, Hansen T, Olsen CK. A randomized, double-blind, placebo-controlled, duloxetine-referenced, fixed-dose study comparing the efficacy and safety of Lu AA21004 in elderly patients with major depressive disorder. Int Clin Psychopharmacol. 2012;27:215–223.
49. Mahableshwarkar AR, Jacobsen PL, Chen Y, et al. A randomized, double-blind, duloxetine-referenced study comparing efficacy and tolerability of 2 fixed doses of vortioxetine in the acute treatment of adults with MDD. Psychopharmacology (Berl). 2015;232:2061–2070.
50. Kassis O, Katz N, Ravid S, et al. Double-blind placebo and active (caffeine) controlled study to examine the effects of the herbal nutritional supplement beverage “Wake up” on vigilance and function after lunch. Isr Med Assoc J. 2013;15:419–423.
51. Phillips-Bute BG, Lane JD. Caffeine withdrawal symptoms following brief caffeine deprivation. Physiol Behav. 1997;63:35–39.
52. Mattila MJ, Vanakoski J, Kalska H, et al. Effects of alcohol, zolpidem, and some other sedatives and hypnotics on human performance and memory. Pharmacol Biochem Behav. 1998;59:917–923.
53. Pompéia S, Pradella-Hallinan M, Manzano GM, et al. Effects of lorazepam on visual perceptual abilities. Hum Psychopharmacol. 2008;23:183–192.
Keywords:

cognitive functioning; Digit Symbol Substitution Test (DSST); major depressive disorder (MDD); neuropsychological testing

Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved.