Subscribe to eTOC

Novel Brain-Computer Interface Turns Mental Writing Into Text

Article In Brief

A novel brain-computer interface may help to turn mental writing into text. This advancement may offer a new means of communications for patients with neurologic conditions like brainstem stroke, cervical spinal cord injury, and amyotrophic lateral sclerosis.

Figure

Two implanted electrode arrays record the brain activity produced by thinking about writing letters. This information is then collected and processed in real-time by a computer, which converts that data into words on a screen.

A brain-computer interface (BCI) can turn mental “handwriting” into text in real-time, potentially opening up new avenues of communication for individuals with neurologic conditions such as cervical spinal cord injury, brainstem stroke, and amyotrophic lateral sclerosis (ALS), according to a groundbreaking new study published in Nature in May.

A team of investigators led by Krishna Shenoy, PhD, a Howard Hughes Medical Institute (HHMI) investigator, the Hong Seh and Vivian W. M. Lim professor of engineering and the director of the Neural Prosthetic Systems Laboratory at Stanford University, developed a machine-learning algorithm that can recognize the patterns produced in the brain when an individual thinks about writing individual letters.

They tested the new algorithm with a participant enrolled in a clinical trial called BrainGate2 (www.braingate.org. The trial is testing the safety of BCIs that relay information directly from a participant's brain to a computer via an implanted chip connected to implanted electrodes, an external decoder device. The device is connected to another external system, such as robotic arm, a wheelchair, or in this case, a computer, and uses the brain's signals to manipulate them.

Figure

“It would be nice, particularly for people with a degenerative condition like ALS, if you didnt have to have this external electrode implant. And there are companies like Elon Musks Neuralink and others who are making devices that are implantable and wireless, and much less invasive, so that certainly seems possible.”—DR. BRIAN LITT

The study participant, a 65-year-old man with tetraplegia who had been paralyzed for years after a spinal cord injury, was able to type 92 characters per minute, about the same speed at which someone his age might type on a smartphone and more than double the previous record for any other BCI.

“As he was thinking about writing each letter, we were able to detect the neuronal ensemble activity, the patterns of which were reproducibly related to each of the 26 letters in the English alphabet and a few characters of punctuation,” said one of the study's co-authors and director of the BrainGate clinical trials, Leigh Hochberg, MD, PhD, FAAN. Dr. Hochberg is also director of the Center for Neurotechnology and Neurorecovery and a neurointensivist at Massachusetts General Hospital, professor of engineering at Brown University, and director of the VA RR&D Center for Neurorestoration and Neurotechnology at the Providence VA Medical Center. “The speed and accuracy of this decoding was so exciting to see,” said one of the study's co-authors and director of the BrainGate clinical trials, Leigh Hochberg, MD, PhD, FAAN, who is also director of the Center for Neurotechnology and Neurorecovery and a neurointensivist at Massachusetts General Hospital, professor of engineering at Brown University, and director of the VA RR&D Center for Neurorestoration and Neurotechnology at the Providence VA Medical Center.

The algorithm, developed by Frank Willett, PhD, a research scientist with HHMI and Stanford, was retrained and calibrated on a daily basis over a series of five days, and investigators found that the participant could accurately type dictated sentences like “You wish to purchase something?” and “I interrupted, unable to keep silent,” along with answering questions such as “How much spice do you like in your food?” (“Lots and lots”) and “What has taken you the longest to get good or decent at?” (“I worked for years to perfect my photography”).

Expert Commentary

Previous BCI technologies have allowed participants with implanted sensors to use their thoughts associated with attempted arm movements to move cursors on a screen, a point-and-click method that allowed them to type about 40 characters per minute.

“But it turns out that if you think about writing the letter A, it's a lot quicker to decode that neural activity than it is to translate thinking about moving a cursor five inches up and three inches to the right,” said Brian Litt, MD, professor of neurology and professor of bioengineering, director of the Penn Epilepsy Center, and director of the Center for neuroengineering and therapeutics (CNT) at the University of Pennsylvania Perelman School of Medicine.

Figure

“The speed and accuracy of this decoding was so exciting to see.”—DR. LEIGH HOCHBERG

“It's a much more efficient way of doing language. If you train an algorithm to recognize the cluster of neuronal activity that represents writing the letter A, then boom, you've got the letter A. So people can generate written language with their mind by just thinking about it in a way that's much faster than anything else we've had previously. If you have no way to communicate and it takes you a minute to write a six-letter word, people will wait because that's the best they have, but 90 characters a minute is so much faster.”

This particular study participant is able to speak, but investigators next plan to work with individuals who have lost the ability to speak due to a condition like ALS or stroke. And independent experts offer a key cautionary note: These findings must be reproduced.

“If it turns out that this particular individual is a little bit faster than everyone else with this BCI, that would still be okay, but if he's ten times as fast, it would diminish the practical implications of the study,” said Jonathan Wolpaw, MD, director of the National Center for Adaptive Neurotechnologies.

That's a hurdle the authors acknowledged in their paper. “More work is needed to demonstrate high performance in additional people, expand the character set (for example, capital letters), enable text editing and deletion, and maintain robustness to changes in neural activity without interrupting the user for decoder retraining,” they wrote.

“To make this available outside of a clinical trial, you would need a commercial-grade entity that's robust and able to track those neuronal signals long term,” noted Karunesh Ganguly, MD, PhD, an associate professor of neurology and a founding member of the Center for Neural Engineering and Prostheses at the University of California, San Francisco.

“How stable is the mapping across days, weeks, and months? The paper does suggest that it might be stable, but will it be sufficiently so that you don't have to remap on a daily basis?”

Also, of importance is the issue of living indefinitely with what is, at least at the moment, a connector percutaneously attached to the electrode system in your brain. Having wires or connectors through the skin, however, has been common in implanted device development, including the original cardiac pacemakers and cochlear implants.

It's also something that individuals in the BrainGate trials have been doing, noted Dr. Hochberg, including paralyzed individuals who are able to control the movement of robotic arms by thinking about it. “Our participants have always used the investigational BrainGate system in their place of residence,” he says. “We believe it is very important to develop and test this intracortical brain-computer interface technology in the place where it ultimately has to work, in someone's own home.”

Figure

“If it turns out that this particular individual is a little bit faster than everyone else with this BCI, that would still be okay, but if hes ten times as fast, that would be a problem.”—DR. JONATHAN WOLPAW

Dr. Wolpaw suggested that if a BCI like this were to be widely adopted, it would ideally be in a form that could be internally implanted. “It should be possible to implant the electronics through a burrhole and close the skin; that's what you would want long term,” he said.

Dr. Litt agreed. “It would be nice, particularly for people with a degenerative condition like ALS, if you didn't have to have this external electrode implant,” he says. “And there are companies like Elon Musk's Neuralink and others who are making devices that are implantable and wireless, and much less invasive, so that certainly seems possible.”

Many of these issues are solvable engineering challenges, Dr. Ganguly said. “The major importance of this study is that it tells us that this technology would allow sufficient rates of communication where it would be clinically meaningful to patients, suggesting that if it could be implemented in a scalable way, there are a significant number of patients who would benefit fairly immediately.”

How soon might that be? “It depends on how quickly someone goes after it,” said Dr. Litt. “I don't think there needs to be a long latency to larger clinical trials.”

Dr. Hochberg encourages neurologists working with people who have cervical spinal cord injury, brain stem stroke, or ALS to check for ongoing BCI trials in their area and refer interested patients to the trials if interested.

“With multiple academic institutions and companies now developing different versions of these systems, I am very hopeful that in the next few years implanted BCIs will emerge from clinical trials and find their way to becoming a system that's more widely available,” he said.

“And the same intracortical recordings that can be harnessed for computer cursor control or text communication can also be used for the control of other assistive devices. The flexibility to use those signals to pick up an object or open a door, much like someone who is able-bodied may use their hands to do the same tasks, is going to be made possible by recording powerful and information-rich signals from motor-related areas of the brain.”

“Hold on to your hat,” Dr. Litt agreed. “There's going to be a lot coming.

Link Up for More Information

• Willett FR, Avansino DT, Hochberg LR, et al. High-performance brain-to-test communication via handwriting https://www.nature.com/articles/s41586-021-03506-2. Nature 2021;593(7858):249–254.