Recurrence and Attention for Long-Context Transformers with Jacob Buckman

Episode 750

0:00-57:23

Today, we're joined by Jacob Buckman, co-founder and CEO of Manifest AI to discuss achieving long context in transformers. We discuss the bottlenecks of scaling context length and recent techniques to overcome them, including windowed attention, grouped query attention, and latent space attention. We explore the idea of weight-state balance and the weight-state FLOP ratio as a way of reasoning about the optimality of compute architectures, and we dig into the Power Retention architecture, which blends the parallelization of attention with the linear scaling of recurrence and promises speedups of >10x during training and >100x during inference. We review Manifest AI’s recent open source projects as well: Vidrial—a custom CUDA framework for building highly optimized GPU kernels in Python, and PowerCoder—a 3B-parameter coding model fine-tuned from StarCoder to use power retention. Our chat also covers the use of metrics like in-context learning curves and negative log likelihood to measure context utility, the implications of scaling laws, and the future of long context lengths in AI applications.

Publisher

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

Link

twimlai.com/podcast/twimlai/recurrence-and-attention-for-long-context-transformers/

Published

October 7, 2025

Duration

57 min

Format

Podcast

Category

Issues · Tech News

Source

RSS feed

Host

Sam Charrington

TT

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

@twimlai

#recurrence #contexttransformers #jacobbuckman

The Nile