Lies and Statistics: Study Advocates Caution in Interpreting Tests of Human Neural Responses

Basu, I.; Anderson, WS

doi: 10.1227/01.neu.0000445335.57950.55
Science Times

    New techniques for recording single neuron spiking behavior in human subjects during pathological activity (seizures) or physiological activity (cognitive or motor testing paradigms) are being rapidly introduced into the community. The interpretation of changes in firing rates for neurons often rely on statistical tests of differences in rates taken between different experimental conditions, or changes from the background firing rate occurring during testing. As Steinmetz and Thorp point out in this new paper, care must be taken in how these tests are implemented because the results may be spurious.1 Specifically, the problem arises when a pretest upon neural activity is applied which first seeks changes in neural firing compared to background based on testing stimulus category, and then using this preselected subset of neurons, changes in firing behavior between neurons are sought between these filtered groups of neurons by stimulus category.

    To demonstrate that this technique of pre-enriching the responsive neural sample, by first selecting neurons responding to stimulus category at rates statistically different than background, the authors used first a Monte Carlo style simulation technique. The responses of 50000 neurons to 2 separate categories were simulated assuming normal distributions for the changes in firing rates centered at zero. This study looked at 2 levels of pre-test selectivity for statistical changes in firing rate compared to background for 2 stimulus categories (α = 0.05 and α = 0.001). The authors found that just by using the tighter requirement on response to enrich the testing sample (which now comes from the tail of the Gaussian distribution), they were able to spuriously induce a statistical signal for response as a function of category because of the larger statistical fluctuations present in this enriched sample. This appears to be a widely used technique, with the authors reporting that 37% of sampled articles in the neuroscience literature involving neuronal recordings used it at some level.

    The authors developed 3 approaches to avoid this type of statistical error: 1) Splitting the response data to perform the enrichment test first, and then test by response category on the second data set with the identified neurons; 2) Performing an estimate of the conditional point-process intensity function within a generalized linear model framework with the stimulus categories used as model predictors; and 3) Splitting the hypotheses being tested, by performing separate statistical tests to demonstrate neurons that fire differently than background, and neurons that fire differently from each other based on stimulus category. The authors formally developed the last technique by creating non-parametric tests for estimating significant changes from background and across categories. In order to test changes from background, the authors performed a likelihood ratio test in which they test the null hypothesis that the observed responses to each category arose from background firing and generalized this change from background test (CBT) by using Monte Carlo methods to account for low firing rates which are poorly approximated by normal distributions. In order to check the effect of stimulus category on responses, the authors designed a bootstrapped F-ratio test (FRT), and showed that higher average firing rates and larger changes of the firing rates from background increased the chances of detecting changes amongst the categories.

    To demonstrate its use, the CBT was applied to test the statistical significance of neuronal firing rates obtained during epilepsy recordings from amygdala, hippocampus, anterior cingulate, and ventromedial prefrontal cortices. The recordings were performed with subjects viewing 5 categories of images (or stimulus categories). The difference in firing rate from background was tested with the CBT and an unpaired t-test, while the effect of categories was tested using the FRT. The authors found groups of neurons that responded to categories only, changed firing rate from background only, and a group that exhibited changes in background along with responses to one or more categories.

    The authors illustrated through these examples that changes in neuronal firing rates may be more subtle than those often observed in animal studies, particularly those in sensory areas. Such changes may be only detected when pre-selection of the responses for large changes from baseline are not applied, and imply that care needs to be taken in designing these types of physiological testing with sparse human recordings.

    Back to Top | Article Outline


    1. Steinmetz PN, Thorp C. Testing for effects of different stimuli on neuronal firing relative to background activity. J Neural Eng. 2013;10(5):056019.
    Copyright © by the Congress of Neurological Surgeons