
How to Engineer AI Inference Systems with Philip Kiely
Episode 766
In this episode, Philip Kiely, head of AI education at Baseten, joins us to unpack the fast-evolving discipline of inference engineering. We explore why inference has become the stickiest and most critical workload in AI, how it blends GPU programming, applied research, and large-scale distributed systems, and where the line sits between inference and model serving. Philip shares how research-to-production can move in hours, not months, and why understanding “the knobs” of inference—batching, quantization, speculation, and KV cache reuse—lets teams design better products and SLAs. We trace the inference maturity journey from closed APIs to dedicated deployments and in-house platforms, discuss GPU lifecycles, and survey today’s runtime landscape, including vLLM, SGLang, and TensorRT LLM. Finally, we look ahead to agents and multimodality, making the case for specialized, workload-specific runtimes when performance and efficiency matter most.
Publisher
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
Link
twimlai.com/podcast/twimlai/how-engineer-ai-inference-systems
Published
April 30, 2026
Duration
55 min
Format
Podcast
Category
Issues · Tech News
Source
RSS feed
Host
Sam Charrington
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
@twimlai
#engineeraiinference #philipkiely
