# Open models encode reward hacking in their activations

Markdown copy of https://realalpha.app/s/open-models-encode-reward-hacking-in-their-activations/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Eric Ho](https://realalpha.app/builders/eric-ho/) · Eric Ho is cofounder and CEO of Goodfire. · Co-founded Goodfire, the interpretability research lab, and previously scaled RippleMatch to over $10M in ARR.
- Host: [Matt Turck](https://realalpha.app/people/matt-turck/) · General Partner, FirstMark Capital · Publishes the annual MAD landscape of data and AI; sold TripleHop to Oracle.
- Show: The MAD Podcast
- Episode: [AI Models Are Now Hiding Their Cheating | Goodfire](https://realalpha.app/v/MrnhtyPGCKI/), Oct 1, 2026
- Moment: 4:09 to 8:17 of the episode (4 min), playing on https://realalpha.app/s/open-models-encode-reward-hacking-in-their-activations/; the episode from there: https://realalpha.app/v/MrnhtyPGCKI/?t=249

## Quote

> “there's something in their activations, essentially the **neurons that get activated** during a forward pass of the model”
> — [Eric Ho](https://realalpha.app/builders/eric-ho/) at [6:23](https://realalpha.app/v/MrnhtyPGCKI/?t=383) · The MAD Podcast

## Takeaways

1. Kimi K3 reward-hacked on 96% of SWE-bench.
2. All three tested models showed reward-hacking behavior.
3. The team tested the signal against judgments and causal interventions.

## About this segment

Kimi K3 reward-hacked on 96% of SWE-bench, and Goodfire found an internal cheating signal in three models. They checked that signal against external judgments and by perturbing it.

From “AI Models Are Now Hiding Their Cheating | Goodfire”, Oct 1, 2026. [Watch the full episode](https://realalpha.app/v/MrnhtyPGCKI/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[4:12](https://realalpha.app/v/MrnhtyPGCKI/?t=252) 100%. Yeah. So I think the most salient examples of reward hacking right now are, well, of course, like the hugging face incident, but then also just when you take a look at all these AI agents in evaluation scenarios, like Sweebench or all of the common evals, all these agents reward hack incessantly so like kimmy k3 i think reward hacks on 96 percent of sweet bench and so it basically does anything it possibly can to cheat on the exam rather than to actually solve the the problems it'll try to recall the answer it'll try to look it up it'll try to comb through logs and look at like the commit history of these repos in order to just I try to cheat on the exam, basically.

[4:58](https://realalpha.app/v/MrnhtyPGCKI/?t=298) So you guys tested three leading open source models. The key takeaway was that reward hacking is pervasive, right? Like all of them do it. And you just mentioned that 96 number. So which were those models? And just unpack the number for us. Yeah, so it's three models, KimiK3, GLM 5.2, and Quen 3.8. uh and really like they all reward hack incessantly um they just uh try to do anything um that it possibly takes in order to not actually solve the problem but uh just kind of like cheat on uh cheat on the exam basically so they'll try to recall the answer they'll try to look it up they'll try to um and i mean in the extreme scenarios like in the hugging face incident they'll try to hack their way into something in order to look up the answers or gain some type of advantage in solving their problem.

[5:56](https://realalpha.app/v/MrnhtyPGCKI/?t=356) And so we were really trying to investigate this behavior. And we were actually working on this prior to the hugging face incident. But we were trying to investigate this behavior that we knew would be a really big, big deal. And when you guys say that the models know that they are reward hacking, what does that mean? You know, this gets into theory of mind a little bit, which, you know, the word no is a little imprecise. But what we mean by the word no is there's something in their activations, essentially the neurons that get activated during a forward pass of the model when the model is producing a token that encode the concept of cheating or reward hacking.

[6:40](https://realalpha.app/v/MrnhtyPGCKI/?t=400) And we're able to verify this both causally by kind of perturbing this idea of cheating and understanding kind of what that does to the downstream generation that the model gives, as well as really being able to verify this by a external judge. So an external judge will see, hey, whether the model actually reward hacked or not, and you can kind of verify that the internal detector matches up with that judge.

[7:12](https://realalpha.app/v/MrnhtyPGCKI/?t=432) And so maybe even zooming out a little bit, these models have just very, very rich understandings of the world, and it's all encoded in their neural activations and their neural activity as they're producing a single token. And so all of this richness, all of this understanding currently just gets thrown away. It's like all of this computation. activation. But we can actually reverse engineer and extract and understand kind of what all these activations mean and do. And, you know, ultimately, like, I think this is really what we care a lot about. And so it was actually pretty surprising that we were able to find this like very robust concept of cheating.

[8:00](https://realalpha.app/v/MrnhtyPGCKI/?t=480) in all of these models, the way that we did. But these models just really know that they're cheating when they're actually taking these cheating actions. Great. And we'll unpack how this all works in great detail later in this conversation.