How to Engineer AI Inference Systems with Philip Kiely

Episode 766

0:00-54:51

In this episode, Philip Kiely, head of AI education at Baseten, joins us to unpack the fast-evolving discipline of inference engineering. We explore why inference has become the stickiest and most critical workload in AI, how it blends GPU programming, applied research, and large-scale distributed systems, and where the line sits between inference and model serving. Philip shares how research-to-production can move in hours, not months, and why understanding “the knobs” of inference—batching, quantization, speculation, and KV cache reuse—lets teams design better products and SLAs. We trace the inference maturity journey from closed APIs to dedicated deployments and in-house platforms, discuss GPU lifecycles, and survey today’s runtime landscape, including vLLM, SGLang, and TensorRT LLM. Finally, we look ahead to agents and multimodality, making the case for specialized, workload-specific runtimes when performance and efficiency matter most.

Publisher

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

Link

twimlai.com/podcast/twimlai/how-engineer-ai-inference-systems

Published

April 30, 2026

Duration

55 min

Format

Podcast

Category

Issues · Tech News

Source

RSS feed

Host

Sam Charrington

TT

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

@twimlai

#engineeraiinference #philipkiely

The Nile