When Walter was asked whether he wanted coffee, his virtual avatar gave an unexpected answer: instead of nodding, it waved goodbye. The amusing slip points to a serious problem. When technology speaks for a person, the wrong gesture can change the meaning of an entire conversation.
Walter, who has amyotrophic lateral sclerosis, or ALS, knows how much is lost when communication becomes difficult. His speech grew slurred over roughly six months. Eventually he could communicate only through email, text-to-speech programs and telephone relay services. ALS is a progressive neurological disease that damages the nerve cells responsible for voluntary muscle movement. It gradually affects a person's ability to speak, swallow, move and breathe, but it does not necessarily impair thinking.
"You try to keep communicating somehow, but you have to sacrifice everything except the bare essentials," Walter says. "There's simply no room left for emotions and gestures."
Looking for a better way to communicate, Walter began researching brain-computer interfaces. At a webinar hosted by the ALS Association, he learned about the work of Edward Chang, a neurosurgeon at the University of California, San Francisco. Walter contacted the research team himself, undeterred by the need for an implant. His illness is shortening his life, and he saw taking part as a chance to help advance a technology that will benefit future patients.
The team began working with Walter in the summer of 2025. The system was gradually trained to recognize his attempts to speak and move, first separately and then together. "Every step and milestone felt exciting and, at the same time, unbelievable," he recalls. But the decisive moment came when the avatar successfully decoded his speech and gestures simultaneously.
What a great milestone!
— Walter
How it works
The brain-computer interface developed by the researchers decodes attempted speech and gestures at the same time, using a single implant. Those signals are translated into words and movements performed by a personalized virtual avatar. The study, published in the journal Nature Neuroscience, offers a glimpse of a technology that could restore something more than words alone: the expressive dimensions of human communication.
The process begins with electrocorticography, the recording of electrical activity from electrodes placed on the surface of the brain. The implant covers a broad area of the sensorimotor cortex, which is involved in controlling speech and movement. Different parts of this cortex are linked to different movements of the body, giving machine-learning models clues for distinguishing attempted speech from attempted gestures.
The researchers, however, ran into an unexpected complication. "We were surprised that models trained on speech and gestures separately performed worse when participants tried to do both at once," says Samantha Brosler, a researcher at the University of California and one of the study's co-first authors.
The problem lay in overlapping neural activity. Some areas of the brain respond both when a person tries to speak and when they try to gesture, and the pattern of that activity can change when the two happen together. Jennifer Collinger, a neuroprosthetics researcher at the University of Pittsburgh who was not involved in the work, calls this context-dependent neural activity. The neural activity associated with a particular movement, such as a wave of the hand, she explains, can differ depending on whether the person is speaking at that moment.
To solve the problem, the team led by Edward Chang trained two parallel models, using recordings in which participants tried to speak and gesture both separately and together. When a participant tried to say "hello" while waving, for example, the resulting neural activity was used to train one model to recognize the phrase and the other to recognize the movement. This combined training improved the quality of decoding.
Encouragingly, the models were able to recognize some combinations of speech and gestures they had not encountered during training. That matters, because teaching the system every possible combination of human words and movements would be impractical.

What comes next
Three people with paralysis took part in the study, although the main experiments with simultaneous speech and gestures were conducted with two of them. The system of one participant with ALS was tested on ten predefined phrases, such as "hello," "thank you" and "nice to meet you," and ten gestures, including a wave, a nod, a shrug and a handshake. The avatar could combine them, making it possible, for instance, to say "hello" while waving. The scientists tested all one hundred possible pairings of phrases and gestures. In real-time testing, speech decoding reached 70 percent accuracy and gesture decoding 66 percent.
Jennifer Collinger finds the results encouraging but stresses that they are experimental. The system did not generate spontaneous conversation; it recognized predefined phrases, displaying them as text, and triggered the avatar's animations when it detected gestures.
That raises questions about user control. Walter's accidental goodbye was funny, but in another situation the wrong gesture could misrepresent a person's intentions. User feedback, Collinger says, is essential to ensure that the technology conveys exactly what the person wants to say.
In future versions of the system, users might be able to confirm or correct the decoded output, perhaps through eye tracking, says Samantha Brosler. For now the system requires a wired connection to external computers. Everyday use will demand wireless data transmission, portability and reduced latency. According to Brosler, that means solving several engineering challenges: shrinking the components, achieving reliable recording of neural signals, and lowering power consumption and heat output.
Walter has his own priorities. "It's a two-way learning process, for both the avatar and the patient," he explains. The ideal solution, in his view, should decode the user's own neural signals, and wireless mobility and less invasive technologies could make such systems more appealing.
He recalls one of the successful exchanges during the experiment, when the system flawlessly decoded his attempts to speak and gesture at the same time: "I was so moved by the avatar's response to my 'thoughts' that I couldn't help laughing out of sheer joy." Despite the limitations, Walter calls his avatar a "digital twin" whose movements reflect his intentions and personality.
For Jennifer Collinger, Walter's experience shows why success cannot be measured by accuracy alone. "Restoring the ability to communicate isn't measured only in words per minute," she says. Independence, relationships, participation in society and self-expression matter too.
The experiment has not yet given people back the ability to hold a natural, unconstrained conversation. But it suggests that the future of brain-computer interfaces may depend on understanding communication as something more than the transmission of words, and on giving people greater control over everything that words cannot express.




Letters