
An Avatar Speaks for the Paralyzed
Walter, who suffers from amyotrophic lateral sclerosis, a neurological disease that gradually paralyzes the muscles, had almost entirely lost the ability to communicate. He volunteered to take part in a study led by the neurosurgeon Edward Chang of the University of California, San Francisco, agreeing to have an implant placed in his brain.
The system reads electrical activity on the brain’s surface that arises when a person tries to speak and move. These signals are translated into words and gestures, which are then reproduced by a virtual avatar. The main difficulty facing the scientists was overlapping neural activity, when the brain responds to speech and to gestures in similar ways. The team’s solution was to train the models on simultaneous actions.
In the course of the experiment, the system recognized ten phrases and ten gestures. The avatar was able, for example, to say “hello” and wave its hand. Speech was decoded with an accuracy of 70%, and gestures with an accuracy of 66%. For now the technology remains experimental and still requires a wired connection. Despite the system’s errors, Walter calls the avatar his “digital twin” and recalls how he laughed with joy the first time it accurately conveyed both his thought and his movement.
