The Benchmark With No Instructions — ARC-AGI-3 (winning team!)

0:00-1:24:34

Tim Scarfe travels to Zurich to sit down with the Tufa Labs ARC-AGI-3 team — founder Benjamin Crouzier, with Jeroen Cottaar, Dries Smit, Stefano Viel and Michal Tesnar — to work out what their leaderboard-topping system does and what the benchmark is really testing. The cut opens on the games: a walkthrough of the Locksmith game, where you read the rules of an unfamiliar world straight from raw frames. ARC-AGI-3 makes ARC interactive and agentic, so the model has to *discover* the goal rather than transduce a static grid. It stays easy for humans and breaks LLMs, and it runs through everything that follows. Dries traces his StochasticGoose preview win — brute force that only searched actions which changed the frame — and why it collapsed once the organisers added action-efficiency scoring and unseen games. Induction and transduction run through the middle of the conversation — how much of an answer is really priors leaking back the moment a model recognises a maze. The abstraction mountain, and Tim's case that LLMs reach the right answer through fractured, entangled representations — performance, not competence. Whether transformers plan at all or just fake it well enough. Why the score really measures action efficiency, not games solved, and why agents lock onto the wrong goal and cannot climb back out.

Publisher

Machine Learning Street Talk (MLST)

Link

podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/The-Benchmark-With-No-Instructions--ARC-AGI-3-winning-team-e3lh6i9

Published

July 1, 2026

Duration

1h 25m

Format

Podcast

Category

Issues · Tech News

Source

RSS feed

ML

Machine Learning Street Talk (MLST)

@mlstreettalk

#benchmark #noinstructions #winningteam

The Nile