25 days ago
- Researchers developed a model to predict speech from ultrasound tongue images during silent speech, achieving a 15.6% word error rate on open-vocabulary speech.
- The system uses a 50-hour dataset collected from audible speech, as tongue movements are similar between silent and vocalized speech, enabling quality-check with audio transcription.
- It employs ResNet-18 2+1d for video encoding and Whisper decoder for text prediction, with training focused on aligning video embeddings to Whisper's audio embeddings.
- The model generalizes to new speakers with American accents but struggles with non-American accents, and performance improves with more data without signs of plateauing.