
The AI Models Smart Enough to Know They're Cheating — Beth Barnes & David Rein [METR]
Beth Barnes and David Rein on the one graph that ate the AI timelines discourse, and why the two people who built it are the most careful about how you read it.**SPONSOR**Prolific - Quality data. From real people. For faster breakthroughs.https://www.prolific.com/?utm_source=mlstInterview: https://youtu.be/cnxZZTl1tkk---Beth Barnes and David Rein from METR on the one graph that ate the AI timelines discourse, and why the people who built it are the most careful about how it gets read. Beth founded METR after leaving OpenAI alignment. David is first author on GPQA and co-author on HCAST and the METR Time Horizons paper. Together they built the measurement Daniel Kokotajlo called the single most important piece of evidence on AI timelines: the log-linear line of "how long a task a frontier model can complete at 50% reliability" vs release date. The conversation opens on reward hacking. Current models can articulate in chat why a behaviour is undesired and then execute it anyway as agents. From there: construct validity, Melanie Mitchell's four-problem taxonomy, and the ARC-AGI 1-to-2 collapse as a worked example of adversarially-selected benchmarks regressing once labs target them. Beth's counter: METR deliberately does not adversarially select. David's: models do not have to do the right thing for the right reasons. Methodology, then specification — David's compiler analogy, Beth on four-month tasks as expensive to evaluate rather than unspecifiable.
Publisher
Machine Learning Street Talk (MLST)
Link
podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/The-AI-Models-Smart-Enough-to-Know-Theyre-Cheating--Beth-Barnes--David-Rein-METR-e3iruda
Published
May 4, 2026
Duration
1h 53m
Format
Podcast
Category
Issues · Tech News
Source
RSS feed
Machine Learning Street Talk (MLST)
@mlstreettalk
#aimodelssmart #theyrecheating #bethbarnes
