“Mindwriting” Technique Helps Paralyzed Patient Use Brain Activity To Write

How do you help someone who cannot move to communicate? This is a challenge that neuroscientists have repeatedly attempted to solve with several ingenious innovations.

One study showed that tracking eye movements could allow paralyzed users to write. Others recorded speech patterns and turned them into text. Some even directly sampled information from the brain itself, either using non-invasive EEG devices or by inserting electrodes into users’ heads that allowed them to control a cursor’s movement, like using a smart TV’s remote control to press on-screen keys. All these devices have achieved some levels of success but face two huge hurdles: they are slow and they often occupy a user’s eyes, reducing their adaptability.

Doubled speed

A study from researchers at Stanford University has produced what may be the most successful solution yet. It works by reading the mind of a paralyzed user who thinks about handwriting letters and converting that brain activity into on-screen text. The method could, one day, allow people with paralysis to rapidly type, just by thinking about writing, say authors.

Jaimie Henderson, MD, a professor of neurosurgery at Stanford who co-led the research, said in a university press release, “This approach allowed a person with paralysis to compose sentences at speeds nearly comparable to those of able-bodied adults of the same age typing on a smartphone.”

The study, published in Nature, doubled the writing speed achieved by other invasive and non-invasive brain communication techniques, hitting 90 characters per minute.

When an accident or spinal cord injury removes a person’s ability to move their body, the brain activity that accompanies those movements is unaffected. State-of-the-art neuroprosthetics use this principle to allow amputees to control mechanical hands with their thoughts. But whether fine motor skills, such as handwriting, would be equally well-preserved was unclear.

To test this, the team worked with an anonymous participant, named T5 in the study, who was 65-years-old and paralyzed from the neck down after a spinal cord injury over a decade previously. T5 was part of another trial, called BrainGate2, in which he had a pair of brain computer interface (BCI) chips implanted into his motor cortex, the part of the brain that controls movement, by Henderson.

The magic of mindwriting

In the current study, T5 was asked to think about handwriting, imagining that he was holding a pen to a piece of paper. While his hands stayed motionless, his brain activity fired up. By practicing “mindwriting” the 26 characters of the English alphabet, alongside a tilde (~) and a (>) to represent full stops and spaces respectively, T5’s recorded brain activity was used to train an algorithm. This decoded T5’s brain activity, assessing the aspects of the activity that most clearly defined mindwriting an “a” from a “p”, for example. T5 spent hours copying pre-written sentences without capitalization, such as “i dumped the tools in the hut” using mindwriting. After T5 mindwrote a letter, it would appear on screen slightly less than a second later.

After three days of training, T5 had produced 242 sentences, all used as further training data for the algorithm, which was calibrated at the start of each of five subsequent test days. On those days, T5 was presented with previously unseen sentences to copy. In these tests, T5 averaged 90 characters (roughly 18 words) per minute. While he had a character error rate of 5.4%, when a language model was added (effectively a BCI version of your smartphone’s autocorrect feature), this rate reduced to 0.89%. These figures match the best performances of state-of-the-art speech recognition systems.

"While handwriting can approach 20 words per minute, we tend to speak around 125 words per minute, and this is another exciting direction that complements handwriting. If combined, these systems could together offer even more options for patients to communicate effectively,” commented co-author and Stanford professor of electrical engineering Krishna Shenoy, the study's co-leader.

When T5 was given the opportunity to mindwrite answers to open-ended questions that appeared on screen, he could average 73.8 characters per minute, with an autocorrected error rate of just 2.25%*. This rate nearly trebled performance of other invasive BCI systems.

With these results secured, the authors set themselves one final task – to work out why handwriting was so much faster than controlled cursor movement. The speed limit of cursor control systems is due to the accuracy of the algorithms used. Imagine you set your computer mouse to five times its normal sensitivity and then tried to type with an on-screen keyboard – eventually accuracy breaks down, limiting usable speed. What the researchers concluded was that the neural activity generated by mindwriting varies more in duration than the activity generated by moving a cursor along a straight line. In contrast to what we might have expected, recording complex, fine movement might actually be easier than reading simple ones. The researchers even generated a “BCI-friendly” alphabet that minimized the similarities between discrete characters.

The study represents another huge lead for BCI technology. In Nature, a paired News & Views article called it a “milestone that broadens the horizon” of this research area. The technology has advanced even in the relatively short amount of time (in scientific terms) between T5’s injury and his enrolment in the current study. One question in the study asked T5 to write what advice he would give to his younger self. After he thought about writing his answer, the algorithm analyzed and released the response – “be patient, it will get better”.

Reference:

Willett FR, Avansino DT, Hochberg, LR, Henderson, JM and Shenoy, KV. High-performance brain-to-text communication via handwriting. Nature;18. doi: 10.1038/s41586-021-03506-2

*Correction: This piece previously stated that a patient using the mindwriting software could average 73.8 words per minute. The software can instead average 73.8 characters per minute.