How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen

0:00-1:10:09

Tsung-Hsien (Shawn) Wen, CTO of PolyAI, tells Tim Scarfe why voice agents are harder than text agents. Voice adds time, and a good conversation depends on adapting to the person on the line, not just on reasoning to the best answer. Shawn describes an audio-native model (Dialog-RSN-1) that first predicts a turn-taking signal, then replies in text with citations, and writes the transcript last so enterprises can audit it. Along the way: training on real, noisy calls with synthetic noise added, and why over-cleaned audio made the new model worse. Latency, and what a voice agent should do while it thinks. Why a voice with a hint of regional accent beats a generic one. Why public benchmarks fall short for voice, why enterprises want to own their agent harness, and whether behaviour belongs in the harness or in the weights. The last stretch is about working with agents: cognitive debt, the shift from producing content to checking it, Wispr Flow, building tools that agents can use, and whether slop is in the eye of the reader.

Publisher

Machine Learning Street Talk

Link

podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/How-a-Voice-Agent-Learns-the-Rhythm-of-Conversation--Shawn-Wen-e3pndhc

Published

October 1, 2026

Duration

1h 10m

Format

Podcast

Category

Issues · Tech News

Source

RSS feed

Host

Machine Learning Street Talk (MLST)

ML

Machine Learning Street Talk

@mlstreettalk

#voiceagentlearns #conversation #shawnwen

The Nile