Silent speech with ultrasound
6 hours ago
- #silent-speech
- #speech-decoding
- #ultrasound-tongue-imaging
- Researchers developed a model to predict speech from ultrasound tongue images during silent speech, achieving a 15.6% word error rate on open-vocabulary speech.
- The system uses a 50-hour dataset collected from audible speech, as tongue movements are similar between silent and vocalized speech, enabling quality-check with audio transcription.
- It employs ResNet-18 2+1d for video encoding and Whisper decoder for text prediction, with training focused on aligning video embeddings to Whisper's audio embeddings.
- The model generalizes to new speakers with American accents but struggles with non-American accents, and performance improves with more data without signs of plateauing.
- Hardware challenges include miniaturizing the ultrasound probe and replacing gel with hydrogel, aiming for a wearable or adhesive patch for practical daily use.