
Speech Recognition Is Not a Solved Problem — Pavan Kumar Reddy
Pavan Kumar Reddy leads audio research at Mistral AI. He joins Tim Scarfe for a deep technical tour of Voxtral — and explains why the frontier of deployed voice is still a cascade of specialised models rather than one end-to-end system. IN PARTNERSHIP WITH MISTRAL AI: --- This episode was produced in partnership with Mistral AI. Mistral AI: https://mistral.ai/ --- The conversation opens on architecture. Voxtral Chat feeds a 3B Ministral text trunk with continuous embeddings from an audio encoder, passed to the decoder as direct token input rather than through cross-attention as in Whisper, so the model can answer questions about emotion, timing and who spoke when without an intermediate transcript to lose them. The real-time model becomes a dual-stream decoder that consumes audio and emits text at once, at a target delay down to 160ms, with slower streams in parallel for anything that can wait for more context. On generation, Pavan explains why Voxtral TTS predicts continuous latents rather than discrete codec tokens, traces the lineage from SoundStream through EnCodec to Mimi's split of semantic and acoustic codebooks, and places FSQ and flow matching in it. Tim presses on the priors underneath: why a mel spectrogram instead of raw waveform, what noise augmentation buys, and when acoustic overfitting becomes somebody's fine-tuning problem. Then the failure modes.
Publisher
Machine Learning Street Talk (MLST)
Link
podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/Speech-Recognition-Is-Not-a-Solved-Problem--Pavan-Muddireddy-e3orlk9
Published
September 14, 2026
Duration
1h 42m
Format
Podcast
Category
Issues · Tech News
Source
RSS feed
Machine Learning Street Talk (MLST)
@mlstreettalk
#speechrecognition #solvedproblem #pavankumarreddy
