DESCRIPTION AND HISTORY
The Digit Symbol Substitution Test (DSST) was initially developed as an experimental tool over a century ago by researchers seeking to understand human associative learning.1,2 The DSST is a paper-and-pencil cognitive test presented on a single sheet of paper that requires a subject to match symbols to numbers according to a key located on the top of the page. The subject copies the symbol into spaces below a row of numbers. The number of correct symbols within the allowed time, usually 90 to 120 seconds, constitutes the score (Fig. 1).3 The DSST is perhaps the most commonly used test in all of neuropsychology, owing to several inherent properties: brevity, reliability, and the minimal impact of language, culture, and education on test performance. Yet, the question of “what it measures” still has no definitive answer, a consequence of the historical context from which it emerged.
The intent of this article is to provide an overview of the DSST, addressing what cognitive operations are required to perform it and what impaired DSST performance may indicate. The aim is to consider its utility in monitoring cognitive functions over time in clinical practice. With this in mind, selected studies have been highlighted to help illustrate the test's sensitivity to change, as well as how performance on the DSST correlates with real-world functioning, so as to provide practitioners with insight into its use for monitoring treatment effects in patients having a range of disorders, with particular attention to its utility for management of major depressive disorder (MDD).
The utility of the DSST as a clinical tool in neuropsychology first became evident when it was shown to reliably distinguish patients with brain damage from healthy patients during the screening of soldiers in World War II. The use of the DSST widened after becoming incorporated in the Wechsler-Bellevue Intelligence Scale (WBIS), developed in 19392; the WBIS relied heavily on early versions of the DSST dating to at least 1900.1 Sometimes termed “coding” or “symbol coding,” the test paradigm has survived with minor alterations to the most recent version found in the Wechsler Adult Intelligence Scale (WAIS), which is currently in its fourth edition (WAIS IV).4,5 As a subtest of the WAIS, the DSST has undergone repeated and rigorous psychometric validation such as test-retest reliability and discriminant validity in a range of patient samples; however, modest differences exist between these versions.
The original WAIS, developed in 1955, has 90 stimulus items (number of blanks into which the symbols are copied), all of which were among the 67 stimulus items in its predecessor, the WBIS.2,6 The DSST was not substantially different in the subsequent version of the WAIS, the WAIS-Revised. However, in WAIS-III, the test duration was extended from 90 to 120 seconds, the response form filled a full page, and the size and spacing of the stimuli were increased so that hooked left-handers were no longer required to move their hands to see the key items.
Symbol coding paradigms comparable to the DSST (with different symbols) are included as subtests in the Brief Assessment of Cognition in Schizophrenia and the Repeatable Battery for the Assessment of Neuropsychological Status.7,8 The Symbol Digit Modalities Test uses essentially the same method, except in reverse; instead of drawing symbols that match digits, the subject is required to write the matching digit in the blank.9 This version of the coding paradigm has the advantage that it can be easily altered to allow oral responses in disorders that affect motor functioning (eg, multiple sclerosis).10
Computerized tests offer another option for monitoring change over time in a clinical setting. To be useful, a measure must be validated for sensitivity to change and reliability in the population in which the clinical application is being contemplated. Several commercial efforts have been undertaken along these lines. For example, the Cambridge Neuropsychological Test Automated Battery developed by Cambridge Cognition is being used to track memory decline in elders with a view toward early detection of dementia.11,12 Similarly, Cognigram, a version of the Cogstate Brief Battery formatted for use in the clinic, consists of a 10-minute battery of computerized tasks requiring attention, processing speed, memory, and executive functioning that have been validated for sensitivity to change in a range of conditions including MDD.13–16 Both of these tests are commercially available and approved for clinical use. THINC-IT is a freely available computerized test that also includes a questionnaire to capture subjective cognitive complaints; it is designed for monitoring change in MDD and is becoming increasingly used, although it has not yet achieved regulatory approval.17 An advantage of the DSST over computerized tools is its simplicity, speed (2 minutes, in contrast to 10–15 minutes for computerized options), and ready accessibility. No computer or software subscription is needed; only the test form, a writing implement, and a timer are required. Like the computerized version, the validity and reliability of the paper DSST depend on the test being administered in the manner it was validated. When scoring the DSST, speed and accuracy are both taken into account, allowing the examiner to use norms or refer to research findings for comparison.
As illustrated below, the DSST is clinically useful because it is sensitive to cognitive deficit in a wide range of brain diseases and conditions. However, without further testing of the patient, the DSST is not informative about the nature of the deficit. The widespread clinical use of the DSST has produced large amounts of data that have helped shed some light on what the DSST measures and the significance of deficits and of change over time in its performance.
What Does the DSST Measure?
The DSST measures a range of cognitive operations. Good performance on the DSST requires intact motor speed, attention, and visuoperceptual functions, including scanning and the ability to write or draw (ie, basic manual dexterity). Performance might also be affected by associative learning. For example, if pairings are rapidly learned following the first few trials, performance speed will improve because the subject will not need to refer to the key to check the accuracy of each pairing. The decision to consciously engage in this learning strategy to improve performance speed calls for the executive functions of planning and strategizing. Working memory, another executive function, is likely required to hold in mind the task rules and for the continual updating of required symbol-digit pairs.
According to Lezak's18 widely used neuropsychological textbook, the DSST is a measure of “Complex Attention,” yet the author indicates that the cognitive aging literature also points to a strong influence of motor speed on performance. For example, in elderly populations, copying speed alone explains 72% of the variance in performance difference between young and elderly groups.18 This finding supports the observation that the DSST is a polyfactorial test and that impairment in any one of the required domains (ie, motor speed associated with aging) results in performance decline. Darby and Walsh19 emphasized that performance on the DSST is the final common pathway for expression of quite diverse types of impairment and described (quoting Kinsbourne) DSST responses as “the end product of the integration of visual perceptual, oculomotor, fine manual motor, and mental functions.”
Maxwell20 argued that the DSST should be classified as a verbal rather than a performance subtest in light of a slightly higher correlation with the verbal intelligence quotient than the performance intelligence quotient in the normative sample of Wechsler. Laux and Lane21 reviewed the correlations of the DSST with other measures in the literature and also suggested that DSST performance involves verbal ability.
Beres and Baron22 measured the effect of active training on DSST performance. The authors showed that both elderly and young subjects improved not only within each day but also between days. Although the older women performed at lower levels than did the younger ones at all points during training, both age groups of women improved fairly consistently over the 20 administrations given on the same day (likely because they were learning the actual symbol pairs). Further, both groups improved from day to day, even though on each day they were given different forms of the test. This observation suggests that improvement with exposure occurs in part for reasons other than associative learning of the pairs. These findings indicate that the executive function of strategizing or consciously exerting effort to learn the pairings is among the activities that contribute to performance on this test. It is noteworthy that both groups improved with practice, and the difference between the groups remained fairly constant. That the DSST is sensitive to age effects is well known.23 The Beres and Baron22 findings suggest that age-related effects are attributable to age differences in motor speed and not in higher cognitive functions. The DSST has been a useful tool to demonstrate age differences in the effects of drugs on cognition, for example, owing to reduced clearance resulting in higher plasma concentrations in older than in younger subjects.24
Finally, the possibility that executive functioning contributes to performance on the DSST is supported by findings of Thornton and Carmody.25 The authors used a Symbol Digit test to examine the relationship between quantitative electroencephalography (QEEG) variables and cognitive performance.25 The authors showed that test performance was associated with QEEG indices, including relative power and coherence at several frequency bands, particularly within the frontal region of the brain.25 This correlation of specific QEEG indices with activation in the frontal lobes supports the theory that executive functioning plays a part in DSST performance.
Sensitivity and Sensitivity to Change of the DSST
David Wechsler2 first described the DSST's sensitivity to brain damage in 1936. Since then, the DSST was found to be the single most sensitive test, among WAIS subtests, to determine the presence of brain damage and discriminate between patient groups (including healthy comparison groups).26 For example, Glosser et al27 noted the sensitivity of the DSST in alcoholic Korsakoff patients, patients with right hemisphere damage, and patients with chronic alcoholism. While psychomotor retardation contributed to impaired performance in these patient populations, the investigators, using other cognitive measures, determined that visuoperceptive processing impairments were also involved in DSST performance.
The sensitivity of the DSST is owed, in part, to its property as a polyfactoriaI measure.28 As alluded to above, impaired performance on a polyfactorial measure may arise for different reasons in differently impaired populations. For example, age-related impaired performance may be attributed, in part, to a decline in motor speed, often associated with aging (as noted earlier in this article). In alcoholic Korsakoff patients, patients with right hemisphere damage, and patients with chronic alcoholism, performance deficits arise from impairments in visuoperceptive processing.27 In schizophrenic patients, processing speed is known to be a critically impaired cognitive ability. In fact, deficits in processing speed as measured by DSST performance may mediate a wide range of other cognitive deficits and may further reflect a final common pathway of impairment in a range of simple and complex cognitive processes that are likely to include working memory and executive functions.29
The ability of the DSST to detect deterioration in processing speed has also been useful as a complementary cognitive assessment in the screening of patients with diabetes mellitus type 2.30 In both middle-aged and older persons, diabetes mellitus type 2 has been found to be associated with an increased risk of mild cognitive impairment and dementia.30 Therefore, brief cognitive tests such as the DSST may prove to be a practical and effective method for use in clinical practice to detect decrements in cognitive processing over time. Interestingly, reports of the predictive nature of lower DSST scores on the development of clinical and subclinical disorders of cognition and mobility have recently been published.31 With additional confirmatory studies, the authors postulate that slower processing speeds as detected by the DSST may serve as a type of biomarker for disorders of cognition, mobility, and possibly even mood. Use of the DSST for studying the pathophysiology of psychomotor slowing with age may also provide insights into the pathology of age-related brain disorders.31 Clearly, the utility of the DSST in the evaluation and management of patients across various therapeutic indications is evident.
The DSST's sensitivity to change, both acutely and chronically, is one of its hallmark features. For this reason, the DSST has been used as a standard tool in clinical pharmacology studies from as early as the 1970s. A review of such studies may offer a means for benchmarking the magnitude of change in standardized effect size units or as a function of the DSST's correlation with other biological measures. The clinical meaningfulness of changes of different magnitudes may then be considered from this vantage point.
Roth et al32 evaluated the sedative effects of antihistamines on a battery of tests including the DSST. The authors demonstrated that the DSST could statistically significantly distinguish the cognitive effect of the H1 antagonist diphenhydramine from placebo with an effect size of 0.28 (calculated from the published statistics).
Thapar et al33 compared the cognitive and sedating effects of several anesthetic agents at doses commonly used in ambulatory surgery. The dependent variable was performance on a modified DSST, timed for 1 minute. As a standard control, alcohol was administered to the subjects to a blood alcohol concentration (BAC) of 0.11%, a dose associated with significant intoxication. Digit Symbol Substitution Test performance was markedly reduced during alcohol intoxication with scores approximately 10 points lower than that of placebo. Although learning or practice effects may occur because the same test is administered multiple times, the randomized design of the study overcomes any resulting confounding factors. Different magnitudes of cognitive effect, as measured with the DSST, can be used to make comparative inferences about the relative effects of various doses of agents, such as alcohol, on cognitive performance (Fig. 2).33
Greenblatt et al34 compared the sensitivity to change of the DSST with that of the electroencephalogram (EEG) to the effects of a single dose of 0.375 mg triazolam versus placebo. Benzodiazepines (eg, triazolam) are widely prescribed drugs for the treatment of anxiety, insomnia, seizures, alcohol withdrawal, and many other disorders. Compared with placebo, triazolam significantly impaired performance on the DSST (P < 0.001) and increased beta amplitude on the EEG (P < 0.002).34 Both DSST and EEG changes closely tracked changes in plasma triazolam concentrations over time.34
The DSST is often used as a pharmacodynamic marker to assess pharmacokinetic-pharmacodynamic relationships. For example, Greenblatt et al35 used the DSST to evaluate age and sex effects on the pharmacokinetics and pharmacodynamics of 0.25-mg triazolam in 61 healthy men and women, aged 20 to 75 years. This double-blind crossover study demonstrated a statistically significant effect of age on predose DSST performance, with lower baseline scores in the elderly subjects further reinforcing the effect of age on DSST performance. The percent decrement in DSST scores with age was also significantly greater in men than in women. In a similar study, Greenblatt et al36 evaluated the effect of dose and sex on the pharmacokinetics and pharmacodynamics of the sedative zolpidem in healthy, nonelderly male and female volunteers and included the DSST among the pharmacodynamic measures. In this placebo-controlled, 4-way crossover study, evaluation of the pharmacodynamic effect area under the curve indicated a significant influence of sex, independent of dose, on the DSST such that effects on women were seen to be greater than those on men.36
The DSST is sensitive enough to distinguish changes over time between different sleep groups in sleep studies. Sleep experiments have used the DSST as a tool to study the cognitive effects of chronic sleep deprivation. Figure 3 shows the mean performance on a computerized DSST (tested every 2 hours) in a sample of 35 subjects randomized to receive 4, 6, or 8 hours of sleep each night for 14 days.37,38 The 8-hour group showed some improvement over the first few days (possibly a learning effect), whereas the 4-hour group differed from the 8-hour group by more than 12 points by the end of the first week and continued to decline over the second week. In the 6-hour group, there was no change from baseline over time. Tucker et al39 also measured the effect of sleep deprivation on DSST performance and yielded similar results. Another example of the DSST's sensitivity to chronic change is illustrated in abstinence testing of chronic alcoholics. Only the DSST, among a battery of neuropsychological tests, was able to measure the benefit from long-term abstinence (nearly 2 years) over short-term abstinence (6 weeks) in chronic alcoholism, suggesting its greater sensitivity to change compared with other measures.28
The wide use of the DSST across so many applications permits unique opportunities for comparison. For example, it is reasonable to argue that using widely understood perturbations, such as sleep deprivation, alcohol, or common pharmacologic agents as benchmarks, offers a reasonable approach to appreciating the clinical meaningfulness of a change in cognition on the DSST. Recent interest in office-based clinical monitoring of cognitive change in patients having MDD and the sensitivity of the DSST to cognitive change in this population suggests true clinical value may be realized by understanding the clinical meaningfulness of change on the DSST.
Clinical Meaningfulness in Patients With MDD
Many studies have examined neuropsychological performance, using a variety of neuropsychological tests, in patients with MDD, and overall, impairments of moderate size have been observed.40 Further, these impairments appeared to have been present from the first depressive episode.40 A recent meta-analysis on executive dysfunction estimated the magnitude of impairment relative to control subjects on a wide array of neuropsychological tests and concluded that “MDD is reliably associated with impaired performance on neuropsychological measures of executive functioning.”41 In that report, data from 1904 subjects evaluated using the DSST in a total of 22 studies demonstrated an impairment of moderate magnitude among patients with MDD with a mean effect size (Cohen’s d) relative to control subjects of 0.55 (confidence interval, 0.34–0.75; P < 0.001). The effect size of 0.55 was in line with the effect sizes of other measures examined. In elderly depressed patients relative to elderly control subjects, Nebes et al42 reported DSST decrements of about double this magnitude and further found that performance was worse in patients with late compared with early onset of disease (P < 0.04).
Vortioxetine, an antidepressant indicated to treat MDD, was evaluated in 2 placebo-controlled clinical trials, and the DSST was used to evaluate the efficacy of 10 and 20 mg of vortioxetine for improving cognitive function in patients with MDD.43,44 In the first study, the effect sizes of treatment with 10 or 20 mg of vortioxetine on DSST scores were, respectively, 0.51 and 0.52.43 The second study also showed positive treatment effects on the DSST, although with a smaller effect size of 0.254 for 10 to 20 mg vortioxetine.44 In addition, a meta-analysis showed significant effects of vortioxetine versus placebo in improving cognitive function assessed with the DSST in patients with MDD and that these improvements were greater than those observed with other antidepressants.45
In an earlier report of MDD patients studied 6 months after discharge from hospital for an MDD episode, the DSST was strongly associated with the magnitude of functional recovery in work, school, and housing, yielding an odds ratio of nearly 20.46 Hence, there is evidence that the impairment and change measured by the DSST reflect meaningful differences relevant to the clinical management of MDD patients. Further, there is growing awareness of the prevalence of cognitive symptoms and their impact on life functioning in MDD patients, leading to an increase in the recognition of such symptoms as clinically important targets for therapeutic improvement.47 Although the importance of cognition for functioning and quality of life may be regarded as self-evident, the meaningfulness of a given magnitude of impairment or change on a brief measure such as the DSST is often questioned. A more refined, data-driven method for establishing the meaningfulness of a particular magnitude of change would be of enormous interest to clinicians seeking to use this test for clinical decision making.
As discussed previously, the DSST's sensitivity to change is one of its main strengths. The magnitude of change on DSST performance, as well as the effect on DSST of an illness state relative to a state of health, can be represented using a standard effect size statistic. The standard effect size is a metric that allows one to compare results of different studies, for example, those examining pretreatment versus posttreatment or treatment (drug) versus placebo, or those examining the magnitude of deficit (relative to normal) in various patient groups.
To appreciate the clinical meaningfulness of a statistically significant cognitive effect of a new pharmacologic agent, it is useful to review the effects of a range of known compounds for which the clinical meaningfulness of effect in healthy volunteers is fairly well understood. Figure 4 illustrates one way to consider the clinical meaningfulness of effect sizes through direct comparison with familiar benchmarks.32,39,43,44,48–53 The figure shows the magnitude of impairment seen with a well-understood exposure to ethanol (ie, BAC of 0.088%) and compares it with that of 2 mg of lorazepam, a sedative for the treatment of anxiety disorders or symptoms, or 150 mg of diphenhydramine, an antihistamine with sedative effects, in healthy volunteers. Additional comparisons include effect sizes and direction of change after caffeine intake, caffeine deprivation, and 1 full night of sleep deprivation. Although benchmarks from acute adverse effects in healthy volunteers are not clinically equivalent to a sustained 8-week benefit of vortioxetine in patients with MDD, these examples can provide some context to understand the meaningfulness of effects. Such benchmarking offers an evidentiary basis for concluding that the observed improvements of DSST scores are of a magnitude likely to be clinically meaningful. Indeed, it may be possible to think of an improvement of this magnitude as comparable to removing the transient effect of a similar magnitude resulting from the BAC and lorazepam dose noted above.
The utility of the DSST for monitoring patients in the clinic derives from several properties, many of which are shared by other neuropsychological tests, including its brevity, its sensitivity to change, and minimal impact from education and culture. The DSST offers high sensitivity to detect impairment, but low specificity to determine exactly which cognitive domain has suffered the impairment. A legitimate question may be raised as to whether a single brief cognitive assessment tool can itself be sufficient to draw clinically meaningful conclusions under any circumstances. Neuropsychologists have become accustomed to the use of extensive batteries of tests, and redundancy of measurement may improve the reliability of any conclusion drawn about cognitive dysfunction. However, where the objective of testing is not diagnosis, but rather to identify impairment regardless of its nature and origin or to detect change within a patient, a brief and simple test such as the DSST may offer important benefits to clinical practice.
The DSST is sensitive to both the presence of cognitive dysfunction and change in cognitive function, across a wide range of clinical populations. In psychiatric patients, the DSST is sensitive to impairments and improvement in processing speed, executive functioning, and working memory. Performance on the DSST correlates with real-world functional outcomes (eg, the ability to accomplish everyday tasks) and recovery from functional disability. In addition, the DSST has been demonstrated to be sensitive to change in cognitive functioning in patients with MDD and may offer an effective means to detect clinically relevant treatment effects. In the context of clinical decision making in a physician's office, this simple cognitive test can, in combination with a psychiatric clinical evaluation, be useful at the level of the individual patient for medication choice and dosing.
Medical writing assistance was provided by James Netterwald, PhD, and Martina Schwarzkopf, PhD, of inVentiv Medical Communications and supported by Takeda Pharmaceuticals USA, Inc., and H. Lundbeck A/S.
AUTHOR DISLOSURE INFORMATION
The author declares no conflicts of interest related to this article. The author is employed full time and is the owner and president of CognitionMetrics, LLC. CognitionMetrics, LLC, has, since inception, provided paid consulting services to the following pharmaceutical companies: Axsome Therapeutics, Biogen, Boehringer Ingelheim, Eisai, Forum Pharmaceuticals, iProteos, Ironwood, Janssen, Lundbeck, Pfizer, Sunovion, Takeda, and Teva Pharmaceuticals.
1. Boake C. From the Binet-Simon to the Wechsler-Bellevue: tracing the history of intelligence testing. J Clin Exp Neuropsychol
2. Wechsler D. The Measurement of Adult Intelligence
. Baltimore, MD: The Williams & Wilkins Company; 1939.
3. Patel T, Kurdi MS. A comparative study between oral melatonin and oral midazolam on preoperative anxiety, cognitive, and psychomotor functions. J Anaesthesiol Clin Pharmacol
4. Wechsler D. Wechsler Adult Intelligence Scale—Fourth Edition
. Bloomington, MN: PsychCorp, an imprint of Pearson Clinical Assessment; 2008.
5. Sudarshan NJ, Bowden SC, Saklofske DH, et al. Age-related invariance of abilities measured with the Wechsler Adult Intelligence Scale-IV. Psychol Assess
6. Wechsler D. Wechsler Adult Intelligence Scale Manual
. New York, NY: Psychological Corporation; 1955.
7. Keefe RS, Harvey PD, Goldberg TE, et al. Norms and standardization of the Brief Assessment of Cognition in Schizophrenia (BACS). Schizophr Res
8. Randolph C, Tierney MC, Mohr E, et al. The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS): preliminary clinical validity. J Clin Exp Neuropsychol
9. Sheridan LK, Fitzgerald HE, Adams KM, et al. Normative Symbol Digit Modalities Test performance in a community-based sample. Arch Clin Neuropsychol
10. Benedict RH, DeLuca J, Phillips G, et al. Validity of the Symbol Digit Modalities Test as a cognition performance outcome measure for multiple sclerosis. Mult Scler
11. Sahakian BJ, Morris RG, Evenden JL, et al. A comparative study of visuospatial memory and learning in Alzheimer-type dementia and Parkinson's disease. Brain
. 1988;111(Pt 3):695–718.
12. Robbins TW, James M, Owen AM, et al. Cambridge Neuropsychological Test Automated Battery (CANTAB): a factor analytic study of a large sample of normal elderly volunteers. Dementia
13. Davis MT, DellaGioia N, Matuskey D, et al. Preliminary evidence concerning the pattern and magnitude of cognitive dysfunction in major depressive disorder using cogstate measures. J Affect Disord
14. Sicard V, Moore RD, Ellemberg D. Sensitivity of the Cogstate Test Battery for detecting prolonged cognitive alterations stemming from sport-related concussions. Clin J Sport Med
15. Cook NA, Kim JU, Pasha Y, et al. A pilot evaluation of a computer-based psychometric test battery designed to detect impairment in patients with cirrhosis. Int J Gen Med
16. Farnsworth JL 2nd, Dargo L, Ragan BG, et al. Reliability of computerized neurocognitive tests for concussion assessment: a meta-analysis. J Athl Train
17. McIntyre RS, Best MW, Bowie CR, et al. The THINC-Integrated Tool (THINC-it) screening assessment for cognitive dysfunction: validation in patients with major depressive disorder. J Clin Psychiatry
18. Lezak MD. Neuropsychological Assessment
. 3rd ed. New York: Oxford University Press; 1995.
19. Darby D, Walsh K. Neuropsychology
. 5th ed. Melbourne: Churchill Livingstone; 2005.
20. Maxwell AE. Obtaining factor scores on the Wechsler Adult Intelligence Scale. J Ment Sci
21. Laux LF, Lane DM. Information processing components of substitution test performance. Dermatol Int
22. Beres CA, Baron A. Improved digit symbol substitution by older women as a result of extended practice. J Gerontol
23. Hoyer WJ, Stawski RS, Wasylyshyn C, et al. Adult age and digit symbol substitution performance: a meta-analysis. Psychol Aging
24. Greenblatt DJ, Harmatz JS, Shapiro L, et al. Sensitivity to triazolam in the elderly. N Engl J Med
25. Thornton KE, Carmody DP. Symbol Digit and the quantitative EEG. Neurotherapy
26. Russell EW. WAIS factor analysis with brain-damaged subjects using criterion measures. J Consult Clin Psychol
27. Glosser G, Butters N, Kaplan E. Visuoperceptual processes in brain damaged patients on the digit symbol substitution test. Int J Neurosci
28. Ryan C, Butters N, Didario B, et al. The relationship between abstinence and recovery of function in male alcoholics. Clin Neuropsychol
29. Amaresha AC, Danivas V, Shivakumar V, et al. Clinical correlates of parametric digit-symbol substitution test in schizophrenia. Asian J Psychiatr
30. Dong Y, Kua ZJ, Khoo EY, et al. The utility of brief cognitive tests for patients with type 2 diabetes mellitus: a systematic review. J Am Med Dir Assoc
31. Rosano C, Perera S, Inzitari M, et al. Digit Symbol Substitution test and future clinical and subclinical disorders of cognition, mobility and mood in older adults. Age Ageing
32. Roth T, Roehrs T, Koshorek G, et al. Sedative effects of antihistamines. J Allergy Clin Immunol
33. Thapar P, Zacny JP, Thompson W, et al. Using alcohol as a standard to assess the degree of impairment induced by sedative and analgesic drugs used in ambulatory surgery. Anesthesiology
34. Greenblatt DJ, Gan L, Harmatz JS, et al. Pharmocokinetics and pharmacodynamics of single-dose triazolam: electroencephalography compared with the Digit-Symbol Substitution Test. Br J Clin Pharmacol
35. Greenblatt DJ, Harmatz JS, von Moltke LL, et al. Age and gender effects on the pharmacokinetics and pharmacodynamics of triazolam, a cytochrome P450 3A substrate. Clin Pharmacol Ther
36. Greenblatt DJ, Harmatz JS, Singh NN, et al. Gender differences in pharmacokinetics and pharmacodynamics of zolpidem following sublingual administration. J Clin Pharmacol
37. Van Dongen HP, Maislin G, Mullington JM, et al. The cumulative cost of additional wakefulness: dose-response effects on neurobehavioral functions and sleep physiology from chronic sleep restriction and total sleep deprivation. Sleep
38. Banks S, Dinges DF. Behavioral and physiological consequences of sleep restriction. J Clin Sleep Med
39. Tucker AM, Whitney P, Belenky G, et al. Effects of sleep deprivation on dissociated components of executive functioning. Sleep
40. Lee RS, Hermens DF, Porter MA, et al. A meta-analysis of cognitive deficits in first-episode major depressive disorder. J Affect Disord
41. Snyder HR. Major depressive disorder is associated with broad impairments on neuropsychological measures of executive function: a meta-analysis and review. Psychol Bull
42. Nebes RD, Pollock BG, Houck PR, et al. Persistence of cognitive impairment in geriatric patients following antidepressant treatment: a randomized, double-blind clinical trial with nortriptyline and paroxetine. J Psychiatr Res
43. McIntyre RS, Lophaven S, Olsen CK. A randomized, double-blind, placebo-controlled study of vortioxetine on cognitive function in depressed adults. Int J Neuropsychopharmacol
44. Mahableshwarkar AR, Zajecka J, Jacobson W, et al. A randomized, placebo-controlled, active-reference, double-blind, flexible-dose study of the efficacy of vortioxetine on cognitive function in major depressive disorder. Neuropsychopharmacology
45. Baune BT, Brignone M, Larsen KG. A network meta-analysis comparing effects of various antidepressant classes on the Digit Symbol Substitution Test (DSST) as a measure of cognitive dysfunction in patients with major depressive disorder. Int J Neuropsychopharmacol
46. Jaeger J, Berns S, Uzelac S, et al. Neurocognitive deficits and disability in major depressive disorder. Psychiatry Res
47. Mattingly G, Anderson RH, Mattingly SG, et al. The impact of cognitive challenges in major depression: the role of the primary care physician. Postgrad Med
48. Katona C, Hansen T, Olsen CK. A randomized, double-blind, placebo-controlled, duloxetine-referenced, fixed-dose study comparing the efficacy and safety of Lu AA21004 in elderly patients with major depressive disorder. Int Clin Psychopharmacol
49. Mahableshwarkar AR, Jacobsen PL, Chen Y, et al. A randomized, double-blind, duloxetine-referenced study comparing efficacy and tolerability of 2 fixed doses of vortioxetine in the acute treatment of adults with MDD. Psychopharmacology (Berl)
50. Kassis O, Katz N, Ravid S, et al. Double-blind placebo and active (caffeine) controlled study to examine the effects of the herbal nutritional supplement beverage “Wake up” on vigilance and function after lunch. Isr Med Assoc J
51. Phillips-Bute BG, Lane JD. Caffeine withdrawal symptoms following brief caffeine deprivation. Physiol Behav
52. Mattila MJ, Vanakoski J, Kalska H, et al. Effects of alcohol, zolpidem, and some other sedatives and hypnotics on human performance and memory. Pharmacol Biochem Behav
53. Pompéia S, Pradella-Hallinan M, Manzano GM, et al. Effects of lorazepam on visual perceptual abilities. Hum Psychopharmacol