The alpha, twice a week
That was 3 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
MTS · 3 min · Nov 6, 2026
Models still struggle to diagnose a failed training run
Qasim Wani describes a GDPX task where agents examined a thousand rollouts to explain a reward regression.
The episode’s best minutes
Full episode- 1:36Recursive self-improvement means inventing a new lab6 min
- 10:20High-quality data is the bottleneck for recursive improvement4 min
- 18:53Models still struggle to diagnose a failed training run3 min
- One more moment in the full episode
“almost every single time all the models gave the complete different answer and an incorrect answer”
Takeaways
- The task was to diagnose why an RL run's reward regressed.
- Models reviewed about 1,000 rollouts with 20 million tokens of context.
- The best score was about 24%, despite the task being manageable for an engineer.
About this segment
Qasim Wani describes a GDPX task where agents examined a thousand rollouts to explain a reward regression. Models received 20 million tokens and five hours, yet their answers were usually wrong.
Qasim Wani is co-founder of Extrapolate and former xAI researcher.
Theo Jaffee, Host · Co-founder & host, MTS
One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.
Sophia Dew, Host · Host, MTS
From “Frontier AI Models Fail at Automated Research | Qasim Wani”, Nov 6, 2026. Watch the full episode
Answers “Are AI agents ready for real work?” See all 99 answers
Thanks. We'll look at and fix what's off.
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 18:58
get to rsi like within labs there are already teams that are doing a lot of handoff of like particular like they're just pressing yes right like they're just using claude to press yes so how do you also wait what teams like like AI researchers yeah you can just assume yeah you can assume that they're like using Claude to make to hand off decisions yeah so like what do you make of that sort of thing where it's like yeah maybe we don't have recursive self-improvement in the sense that you and I conceptualize it but we do have the models making a huge amount of the decisions like micro decisions about um where certain directions go or like micro engineering decisions you know things like that i think i think that's great i think we need that um there's still a lot of human involvement in a lot of these processes and um but at the same time
- 19:50
like certain functions have been like almost completely automated like models kicking off a training run and like monitoring for like a big run and these types of pre-training runs like a lot lot of that. has been automated, but things like auditing for data quality, that is not a solved problem. In fact, on GDPX, the public task that we have shown is how good is an agent in detecting for potential reward hacking behavior on about a thousand rollouts. And here, just for context, the task was we had some model post-training RL run that we have done, and we give an agent, and during this run the reward actually regressed it went from like 0.14 to like something super super low and uh this is like a fairly normal thing that happens at a lab and you're like i want to i want to know what happened and why it happened we give the agent this task astra fable
- 20:45
like all the models and we give them 20 million tokens worth of context which is about a thousand rollouts and we ask it to spend five hours to figure out what's wrong and almost every single time all the models gave the complete different answer and an incorrect answer and they only were able to get the highest score was like 24 percent and we made the task fairly simple where like a competent engineer could just like look through the logs and figure out oh this is like a weird thing that happened with the harness but the point that you know I'm bringing up is certain parts have been automated but certain other parts that are super critical like just looking at the data we're nowhere close to fully automating it yeah yeah well i think we're coming up on time but this has been super interesting thanks so much for joining us thanks for having me