ARTICLE IN BRIEF
A re-analysis of data from a 2011 The Lancet study finding EEG-detected awareness in patients in a persistent vegetative state raises questions about the methodology and findings of the original paper.
In research, methodology is critical. It drives findings and gives voice to implications. This point fueled debate in a series of published papers and open letters by two groups of scientists that study disorders of consciousness.
A team of scientists at Weill Cornell Medical College questioned the results of the findings from a study, originally published in the online edition of The Lancet Nov. 9, 2011, by scientists at the University of Western Ontario. Based on the analysis of EEG recordings from 16 patients in a persistent vegetative state (PVS) and a dozen normal controls, the Canadian scientists suggested that as many as 20 percent of PVS patients might be able to follow mental imagery commands and that an electroencephalogram (EEG) could be used to test whether someone who can't move and doesn't seem to respond to the environment actually has higher cognitive function that has gone unnoticed. [See Neurology Today's “A More Portable EEG Used to Assess Consciousness in Patients in Vegetative State,” http://bit.ly/rpLSK0.]
At question is the actual method used to test the Canadian scientists' hypothesis that some patients in PVS might be able to actually understand directions and follow commands. The Canadian team used a complicated design where patients and normal volunteers were hooked up to an EEG and asked to carry out two different mental imagery tasks: “When you hear a beep, imagine wiggling your toes or when you hear a beep, imagine opening and closing your hand.”
Once the data were collected, they applied a modern computerized method — a support vector machine — to identify shared features of consciousness and then determine which patients were able to follow commands in their mind. [See “The Study's Use of the Support Vector Machine.”]
In The Lancet study in question, Damian Cruse, PhD, Adrian Owen, PhD, and their colleagues at the University of Western Ontario reported that three of the 16 patients showed an EEG pattern suggestive of awareness. But they also found that the method showed that only 75 percent of the normal volunteers could carry out the task.
QUESTIONS ABOUT THE LANCET STUDY
To the Cornell team, led by Nicholas Schiff, MD, there were several crucial aspects of this important study that deserved scrutiny. How does one know what signals to look for and then how do you correlate them with the task itself?
“We can only rely on the performance of the classifier as the reporter of a positive result. This can bring false positive findings as classification algorithms can be very sensitive and overfit noise [that is, they can describe random error or noise] in EEG data,” said Dr. Schiff, professor of neurology and neuroscience. “It would be necessary to show the actual differences in the signals classified and to validate the statistical model to mitigate this kind of error.”
[Overfitting generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations.]
The Cornell scientists suspected that the support vector machine was crunching too many numbers — 30,000 features to be exact — and that the high-powered algorithm could not tell the difference between EEG signals that were performing two different mental imagery tasks or merely represented reflexive muscle activity. (EEG electrodes rest on the skin and are highly reactive to muscle activity.) They also thought that the design of the study — which was conducted in blocks where the subject was asked to carry out either of the two different mental imagery tasks after hearing a beep — might lead to errors of overfitting, especially when so many features were under consideration.
“We discussed our concerns with Dr. Owen and his colleagues,” said Dr. Schiff. Surprised that some of the intermediate signal analysis steps were not in the published paper, Dr. Schiff and colleagues wrote a short letter to the journal editors, which was published, and had already begun analysis using raw data shared with them by their Canadian colleagues. The New York group wanted to test the model themselves.
The Cornell team carried out tests of the statistical assumptions of the Canadian group's analysis, and also ran new analyses of the same data that were not sensitive to these assumptions, including variants of the Canadian group's approach, and separate analyses using more traditional methods.
They concluded that “the methods that the Canadian team used to ask whether patients in a persistent vegetative state could follow mental imagery commands is just not right,” said Andrew Goldfine, MD, assistant professor of neurology and neuroscience at Cornell. And therefore, they said in a paper published in the Jan. 26 issue of The Lancet, the approach does not answer the age-old question: Is there awareness in a person who appears to be in a persistent vegetative state?
“Once we began looking closer to the raw data, there was no signal related to task performance at all in any of the patient subjects,” added Dr. Schiff. But exactly where did the Canadian team go wrong?
The Canadian group compared EEG results between blocks of tests, treating each block and each trial within each block as independent of the next. They were asking the computer program to differentiate between the EEG activity while patients (and controls) were thinking about squeezing their right hand or wiggling their toes on the same side of the body.
The Cornell team looked at the way that the Canadian group determined that their classifier's performance was better than chance. Were the trials within a block really independent and should the block sequences be factored into the analysis? “The patient data did not meet the assumptions of the statistical model they used, and therefore we concluded that their method is invalid,” said Dr. Goldfine.
To understand the implications of these statistical concerns, the Cornell group, who had been doing the same kinds of mental imagery tests on their patients, took a different approach. They compared the EEG data taken during rest to the recordings during the time when the patient was supposed to be performing a mental task. They looked for pattern differences between rest and presumed activity. Can the patients do the task over and over again and have the same EEG pattern? The answer they got was no; they could not.
Jonathan Victor, MD, PhD, the Fred Plum professor of neurology and neuroscience, supervised the calculations. “Only after we found that the assumptions underlying their statistical model were probably wrong did we re-analyze the data with a different statistical model,” explained Dr. Victor. They used a permutation test and a different cross-validation strategy and showed that the classifier used by the Canadian group was performing at chance in the patient population.
Finally, after conducting these two tests, they used another approach to look for command following — which is not even a classifier — just to see whether there was a response following the beep, as per the instruction. They found a response in all of the normal volunteer data and in none of the patients, Dr. Victor said.
It turned out that the machine-learning algorithm had a hard time knowing the difference between muscle activity and actual brain wave activity generated from thinking, Dr. Goldfine added. The machine analyzed the data in blocks of information from each of the beeping and thinking tests. “Their statistical model makes the assumption that all trials are independent but we found that trials within a block are more similar than trials in other blocks,” explained Dr. Goldfine.
Dr. Cruse and his colleagues stand by their original findings. In an e-mail response to queries from Neurology Today, he said that the content of The Lancet letter by the Cornell team “comes down to a difference of opinion regarding the best statistical model to use to analyze EEG data. There are obviously many different ways of analyzing EEG data, and strong opinions about the pros and cons in all cases.”
Dr. Cruse added, “For two of the three key patients, we have corroborative evidence using independent approaches (e.g. fMRI) that they were aware at the time of the EEG testing. So, even if you ignore the EEG evidence, the fact remains that a subset of our patients were aware, supporting previously published data in this area (including data from the Cornell group).”
Even as the Canadian scientists continue to defend their data, a study published in November in PLoS One suggests that they were listening to the criticism. Dr. Owen and his colleagues used more standard statistics in this later study, which involved a single patient. They used signal-processing steps that the Cornell team used to test their model. They conducted the same study design on a patient who had been in a persistent vegetative state for a decade. The analysis suggested that the patient's EEG response looked similar to a healthy adult doing the same imaging task.
With only one patient, this was a proof-of-concept study. But, said Cornell's Dr. Goldfine, “the model looked good. This is a win for science. The McDonnell Foundation funded this collaboration to validate methods to use in disorders of consciousness.”
This public controversy comes on the threshold of the second phase of the study, where the consortium scientists will select shared methods to begin a large-scale study to identify features that might predict future outcome.
At the heart of the issue, said Cornell's Dr. Schiff, every patient comes to the table, or in this case the bedside, with such variability that it would seem virtually impossible to identify factors that could predict inner awareness. At present, he added, “we've done very little to fill in the biology of these variations. Any tests that are developed will be no substitute for a thorough clinical assessment.”
EXPERTS WEIGH IN
John Whyte, MD, PhD, director of the Moss Rehabilitation Research Institute in Elkins Park, PA, said “the discussion between these two groups points to a number of important issues. First, although recent technological advances in brain electrophysiology and functional imaging allow us to have insight into brain processing in the absence of overt movement, they also require a sequence of complex analytic steps. These steps involve a number of important assumptions, and it is critical for any investigator seeking to interpret such studies to understand and assess the appropriateness of these assumptions and analyses.
“Methods that rely on overt behavior, although they may ultimately turn out to be less sensitive, have the advantage of being relatively straightforward to analyze. But by either method, consciousness has to be inferred from a pattern of results; it can never be measured directly. And that means that the steps that link a particular pattern of data to a conclusion about a patient's state of consciousness are always interpretations rather than ‘results,’ and it's very important that these links are vetted by other scientists before we make too much of the conclusions. Moreover, one can never prove unconsciousness; one can only fail to find evidence of consciousness.”
Another challenge in this kind of research is patient variability. Indeed, one of the defining features of the minimally conscious state is fluctuation and inconsistency in behavior, Dr. Whyte explained. “Accordingly, even with a very good behavioral assessment tool, absence of evidence of consciousness on a single occasion means very little and a question that we haven't really answered rigorously is how long and how hard do we need to search for consciousness before we feel comfortable giving up the search? It may be that assessment methods that rely on brain physiology are less variable than behavioral methods and, thus, could give us a more clear answer with fewer assessments. But this has not been demonstrated, and on the face of it, doesn't seem terribly likely.”
James Bernat, MD, the Louis and Ruth Frank professor of neuroscience and professor of neurology and medicine at the Geisel School of Medicine at Dartmouth University, added that the finding is particularly important because “we assume that the peer review would guarantee that results are valid.” The fMRI data with similar types of study designs have shown that in rare cases one can see a signal of awareness even when the clinical assessment led to a diagnosis of persistent vegetative state. [See “Former Israeli Prime Minister Ariel Sharon Found to Have Brain Activity While in Vegetative State.”] Most centers do not have fMRI machines so finding a tool that is readily available is a goal. “If we could get the same information out of an EEG it would certainly be more valuable,” he said. And that is the goal for many centers involved in the study of disorders of consciousness, he added.
THE STUDY'S USE OF THE SUPPORT VECTOR MACHINE
The support vector machine seeks patterns in information in much the same way that e-mail programs try to filter spam out from relevant email messages.
In the original Lancet study, the support vector machine was asked to classify an EEG segment as a hand or a foot trial, and the features consisted of how much activity is in the EEG at each of 25 channels, 4 frequencies, and 300 time windows, for a total of 30,000 (25 × 4 × 300) features.
One set of trials is used as the training set, and a second set of trials is used to determine whether the classifier is valid — that is, whether it generalizes. In making this statistical evaluation, a crucial aspect is the extent to which the trials are independent, both within the training and test sets, and between them.
FORMER ISRAELI PRIME MINISTER ARIEL SHARON FOUND TO HAVE BRAIN ACTIVITY WHILE IN VEGETATIVE STATE
Seven years after Israel's former Prime Minister Ariel Sharon suffered a severe stroke that led to a presumed persistent vegetative state, a team of scientists decided to test whether he has any islands of awareness. There is growing evidence that some areas of the brain in the rare patient may still be processing information even though the body shows no sign of it.
Sharon's doctors said that they did not see, in the last few years, behavioral signs that he understands the world around him, but his family and the hospital nurses who tend to him on a daily basis said that they felt he is more aware than his body belies.
In late January, Sharon was transported to Soroka University Medical Center in Beersheba, where scientists performed a number of complex behavioral studies while he was inside a magnetic resonance imaging machine.
Martin M. Monti, PhD, an expert on fMRI and consciousness, traveled to Israel to help the team carry out the exam. Dr. Monti, an assistant professor of psychology and neurosurgery at the University of California, Los Angeles, has spent his career developing tests to understand whether patients like Sharon are conscious and how much of their brainpower is actually still available.
The big question that drove the present experiment is this: Can a picture of someone's brain — its structure and its function — provide clues to whether that person is conscious?
Dr. Monti was asked to help design the paradigm that was used to assess consciousness in Sharon during his time spent inside the Philips 3-tesla MRI. Not knowing much about the exact medical condition of the former Prime Minister, Dr. Monti used the standard fare that he has been working on for several years to see whether the brain is able to show signs of understanding stimuli in the world beyond the body.
Once Sharon was in the scanner, brain activity was recorded while he was shown pictures of familiar faces and places, listened to a tape of his son's voice and another segment of his son's words mixed up in a type of linguistic gibberish, and responded to someone stroking his hand and asking him to focus on one of two images of a picture of a face superimposed on a house. Each task is testing a specific brain region and assessing lower and higher order processes.
When Dr. Monti first met Sharon, who is on a ventilator, the scientist used the Coma Recovery Scale to assess consciousness. He said there was very little response, and he was not expecting much from the fMRI.
When the pictures of familiar faces and a house were shown in front of Sharon's eyes in the scanner, the fMRI uncovered some brain activity. “There was an appropriate visual response in an area of the brain that responds when looking at a face (the fusiform gyrus) and a different area that becomes active when looking at houses, the parahippocampal gyrus.
“When his hand was stroked, the brain's parietal lobe showed a strong response. And the brain also showed activity when hearing the son's voice in the well-formed sentences compared to the incomprehensible utterances.”
The next series of fMRI tests involved the types of mental imagery experiments that are now being done to assess consciousness. The scientists asked Sharon to move his hand, even though he had not shown that he was able to voluntarily carry out the task. He was asked to imagine playing tennis or imagine walking around his house. Each time, the fMRI recorded brain activity that suggested the possibility of some level of awareness.
But is this evidence of conscious awareness? “We can say that the brain is responding appropriately but it is unclear whether Sharon is actually conscious,” said Dr. Monti, who carried out the study with neuroscientist Alon Friedman, MD, PhD, and his colleagues at Ben-Gurion University.
When Sharon came out of the scanner his eyes were open. Dr. Monti performed an item from the Coma Recovery Scale that called for putting a mirror in front of his eyes and moving the mirror to see whether his eyes moved with it. He was told in Hebrew to follow the mirror with his eyes, which had to follow 45 degrees of arc. Although not quite for 45 degrees, it seemed that he had tracked the mirror. Dr. Monti called a doctor over to watch Sharon's eyes. Again, there was some tracking. On the last try, the eye movements were not so clear, he said.
Many in the room took the findings as evidence of brain activity but stopped short on the interpretation that Sharon was aware of his surroundings. Still, for family, any sign of awareness is critically important. Sharon's son Gilad told the team that he never had doubt that his father showed signs of awareness.
LISTEN UP, TUNE IN: In the December 2011 of The Lancet, researchers reported on a study finding that one in five patients in a vegetative state showed evidence on an EEG recording that they understood commands — squeezing their hand and wiggling toes — repeatedly during a 20-minute testing period. Listen here as study co-author Damian Cruse, PhD, a post-doctoral fellow at the University of Western Ontario, describes the potential for using EEG to assess patients in a vegetative state: http://bit.ly/YKg7jc.
FOR FURTHER READING:
• Goldfine AM, Bardin JC, Victor JD, et al. Reanalysis of “Bedside detection of awareness in the vegetative state: A cohort study.” Lancet
• Cruse D, Chennu S, Owen AM, et al. Reanalysis of “Bedside detection of awareness in the vegetative state: a cohort study” — Authors' reply. Lancet
• Cruse D, Chennu S, Young GB, et al. Detecting awareness in the vegetative state: Electroencephalographic evidence for attempted movements to command. PLoS One
2012; 7(11): e49933.
• Cruse D, Chennu S, Owen AM, et al. Bedside detection of awareness in the vegetative state: a cohort study. Lancet 2011;378(9809):2088–2094.
©2013 American Academy of Neurology
• Overgaard M, Overgaard R. Comment: Measurements of consciousness in the vegetative state. Lancet
Neurology Today Quick Links