
The Benchmark With No Instructions — ARC-AGI-3 (winning team!)
Tim Scarfe travels to Zurich to sit down with the Tufa Labs ARC-AGI-3 team — founder Benjamin Crouzier, with Jeroen Cottaar, Dries Smit, Stefano Viel and Michal Tesnar — to work out what their leaderboard-topping system does and what the benchmark is really testing. The cut opens on the games: a walkthrough of the Locksmith game, where you read the rules of an unfamiliar world straight from raw frames. ARC-AGI-3 makes ARC interactive and agentic, so the model has to *discover* the goal rather than transduce a static grid. It stays easy for humans and breaks LLMs, and it runs through everything that follows. Dries traces his StochasticGoose preview win — brute force that only searched actions which changed the frame — and why it collapsed once the organisers added action-efficiency scoring and unseen games. Induction and transduction run through the middle of the conversation — how much of an answer is really priors leaking back the moment a model recognises a maze. The abstraction mountain, and Tim's case that LLMs reach the right answer through fractured, entangled representations — performance, not competence. Whether transformers plan at all or just fake it well enough. Why the score really measures action efficiency, not games solved, and why agents lock onto the wrong goal and cannot climb back out.
Publisher
Machine Learning Street Talk (MLST)
Link
podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/The-Benchmark-With-No-Instructions--ARC-AGI-3-winning-team-e3lh6i9
Published
July 1, 2026
Duration
1h 25m
Format
Podcast
Category
Issues · Tech News
Source
RSS feed
Machine Learning Street Talk (MLST)
@mlstreettalk
#benchmark #noinstructions #winningteam
