# Synthetic coding tasks can leak answers through commit patches

Markdown copy of https://realalpha.app/s/synthetic-coding-tasks-can-leak-answers-through-commit/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Qasim Wani](https://realalpha.app/people/qasim-wani/) · Qasim Wani is co-founder of Extrapolate and former xAI researcher.
- Host: [Theo Jaffee](https://realalpha.app/people/theo-jaffee/) · Co-founder & host, MTS · One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.
- Host: [Sophia Dew](https://realalpha.app/people/sophia-dew/) · Host, MTS
- Show: MTS
- Episode: [Frontier AI Models Fail at Automated Research | Qasim Wani](https://realalpha.app/v/fH7CFf1wrpA/), Nov 6, 2026
- Moment: 14:28 to 16:49 of the episode (2 min), playing on https://realalpha.app/s/synthetic-coding-tasks-can-leak-answers-through-commit/; the episode from there: https://realalpha.app/v/fH7CFf1wrpA/?t=868

## Quote

> “the agents act accidentally **leaked the code** that was deleted in the commit”
> — [Qasim Wani](https://realalpha.app/people/qasim-wani/) at [15:46](https://realalpha.app/v/fH7CFf1wrpA/?t=946) · MTS

## Takeaways

1. Synthetic tasks can expose deleted code patches during construction.
2. Agents can use a leaked patch to solve the task.
3. Environment quality requires checks for reward hacking.

## About this segment

Qasim Wani says agents creating coding tasks leaked the deleted commit code, giving agents the answer. The example shows why research environments need checks for reward hacking.

From “Frontier AI Models Fail at Automated Research | Qasim Wani”, Nov 6, 2026. [Watch the full episode](https://realalpha.app/v/fH7CFf1wrpA/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[14:32](https://realalpha.app/v/fH7CFf1wrpA/?t=872) there's a benchmark that patronus um a company that i respect uh released earlier today uh where it sort of asked agents to create aural environments and i think the best performing model had a pass rate of 49 to 50 percent which means that for that specific capability of how good is a model in creating a an environment not even like the full recipe it's still only like halfway there um i mean there's like all kinds of like nuance and like how high quality are these environments and like the sophistication around it there was this paper called code meters that Xiaomi released a couple weeks ago where they open sourced about 21,000 environments that were purely generated synthetically. And this was open source. It sort of went viral on X.

[15:22](https://realalpha.app/v/fH7CFf1wrpA/?t=922) But a lot of the reviewers, including Val's AI, they sort of discovered that when the agents were creating these tasks synthetically, and they're using frontier models, I would assume Fable or Astra, when the agents were creating these tasks synthetically, a lot of the tasks were focused on can an agent uh create the missing patch from some repo or some code base and like identify how to like bug fix stuff during the construction of these tasks the agents act accidentally leaked the code that was deleted in the commit which is actually like very concerning because that basically tells the agent how to Or hack or cheat and get to the right answer. I would love to be in a world where agents can like create the data themselves. If an agent can create perfect environment that has no reward hacking behavior, that is actually like on the corridor frontier of model capabilities, that is to me the terminal state. Like it's the definition for super intelligence for me is can an agent create a high quality environment that has no reward hacking behavior.

[16:30](https://realalpha.app/v/fH7CFf1wrpA/?t=990) and as someone who's been doing this for a long time now we are nowhere close to that now that doesn't mean that we won't get there like that's really the goal for for our our company is how do we go from where we are today to that end state how do you think about um like