
How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen
Tsung-Hsien (Shawn) Wen, CTO of PolyAI, tells Tim Scarfe why voice agents are harder than text agents. Voice adds time, and a good conversation depends on adapting to the person on the line, not just on reasoning to the best answer. Shawn describes an audio-native model (Dialog-RSN-1) that first predicts a turn-taking signal, then replies in text with citations, and writes the transcript last so enterprises can audit it. Along the way: training on real, noisy calls with synthetic noise added, and why over-cleaned audio made the new model worse. Latency, and what a voice agent should do while it thinks. Why a voice with a hint of regional accent beats a generic one. Why public benchmarks fall short for voice, why enterprises want to own their agent harness, and whether behaviour belongs in the harness or in the weights. The last stretch is about working with agents: cognitive debt, the shift from producing content to checking it, Wispr Flow, building tools that agents can use, and whether slop is in the eye of the reader.
Publisher
Machine Learning Street Talk
Link
podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/How-a-Voice-Agent-Learns-the-Rhythm-of-Conversation--Shawn-Wen-e3pndhc
Published
October 1, 2026
Duration
1h 10m
Format
Podcast
Category
Issues · Tech News
Source
RSS feed
Host
Machine Learning Street Talk (MLST)
Machine Learning Street Talk
@mlstreettalk
#voiceagentlearns #conversation #shawnwen
