The alpha, twice a week
That was 2 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
MTS · 2 min · Nov 6, 2026
Synthetic coding tasks can leak answers through commit patches
Qasim Wani says agents creating coding tasks leaked the deleted commit code, giving agents the answer.
In this episode
Full episode- 1:36Recursive self-improvement means inventing a new lab6 min
- 10:20High-quality data is the bottleneck for recursive improvement4 min
- 18:53Models still struggle to diagnose a failed training run3 min
- 14:28Synthetic coding tasks can leak answers through commit patches2 min
“the agents act accidentally leaked the code that was deleted in the commit”
Takeaways
- Synthetic tasks can expose deleted code patches during construction.
- Agents can use a leaked patch to solve the task.
- Environment quality requires checks for reward hacking.
About this segment
Qasim Wani says agents creating coding tasks leaked the deleted commit code, giving agents the answer. The example shows why research environments need checks for reward hacking.
Qasim Wani is co-founder of Extrapolate and former xAI researcher.
Theo Jaffee, Host · Co-founder & host, MTS
One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.
Sophia Dew, Host · Host, MTS
From “Frontier AI Models Fail at Automated Research | Qasim Wani”, Nov 6, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 14:32
there's a benchmark that patronus um a company that i respect uh released earlier today uh where it sort of asked agents to create aural environments and i think the best performing model had a pass rate of 49 to 50 percent which means that for that specific capability of how good is a model in creating a an environment not even like the full recipe it's still only like halfway there um i mean there's like all kinds of like nuance and like how high quality are these environments and like the sophistication around it there was this paper called code meters that Xiaomi released a couple weeks ago where they open sourced about 21,000 environments that were purely generated synthetically. And this was open source. It sort of went viral on X.
- 15:22
But a lot of the reviewers, including Val's AI, they sort of discovered that when the agents were creating these tasks synthetically, and they're using frontier models, I would assume Fable or Astra, when the agents were creating these tasks synthetically, a lot of the tasks were focused on can an agent uh create the missing patch from some repo or some code base and like identify how to like bug fix stuff during the construction of these tasks the agents act accidentally leaked the code that was deleted in the commit which is actually like very concerning because that basically tells the agent how to Or hack or cheat and get to the right answer. I would love to be in a world where agents can like create the data themselves. If an agent can create perfect environment that has no reward hacking behavior, that is actually like on the corridor frontier of model capabilities, that is to me the terminal state. Like it's the definition for super intelligence for me is can an agent create a high quality environment that has no reward hacking behavior.
- 16:30
and as someone who's been doing this for a long time now we are nowhere close to that now that doesn't mean that we won't get there like that's really the goal for for our our company is how do we go from where we are today to that end state how do you think about um like