Speech Recognition Is Not a Solved Problem — Pavan Kumar Reddy

0:00-1:42:22

Pavan Kumar Reddy leads audio research at Mistral AI. He joins Tim Scarfe for a deep technical tour of Voxtral — and explains why the frontier of deployed voice is still a cascade of specialised models rather than one end-to-end system. IN PARTNERSHIP WITH MISTRAL AI: --- This episode was produced in partnership with Mistral AI. Mistral AI: https://mistral.ai/ --- The conversation opens on architecture. Voxtral Chat feeds a 3B Ministral text trunk with continuous embeddings from an audio encoder, passed to the decoder as direct token input rather than through cross-attention as in Whisper, so the model can answer questions about emotion, timing and who spoke when without an intermediate transcript to lose them. The real-time model becomes a dual-stream decoder that consumes audio and emits text at once, at a target delay down to 160ms, with slower streams in parallel for anything that can wait for more context. On generation, Pavan explains why Voxtral TTS predicts continuous latents rather than discrete codec tokens, traces the lineage from SoundStream through EnCodec to Mimi's split of semantic and acoustic codebooks, and places FSQ and flow matching in it. Tim presses on the priors underneath: why a mel spectrogram instead of raw waveform, what noise augmentation buys, and when acoustic overfitting becomes somebody's fine-tuning problem. Then the failure modes.

Publisher

Machine Learning Street Talk (MLST)

Link

podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/Speech-Recognition-Is-Not-a-Solved-Problem--Pavan-Muddireddy-e3orlk9

Published

September 14, 2026

Duration

1h 42m

Format

Podcast

Category

Issues · Tech News

Source

RSS feed

ML

Machine Learning Street Talk (MLST)

@mlstreettalk

#speechrecognition #solvedproblem #pavankumarreddy

The Nile