Real Alpha
Watch modeSubscribed

The MAD Podcast · 3 min · Oct 1, 2026

A simple activation contrast can detect reward hacking

Goodfire compares activations from cheating and non-cheating examples to create a simple detector. The signal generalized from contrived examples to more complex actions, and they could intervene on it to test its effects.

Eric Ho

Eric Ho is cofounder and CEO of Goodfire.

Co-founded Goodfire, the interpretability research lab, and previously scaled RippleMatch to over $10M in ARR.

  • Matt Turck, Host · General Partner, FirstMark Capital

    Publishes the annual MAD landscape of data and AI; sold TripleHop to Oracle.

About this segment

Goodfire compares activations from cheating and non-cheating examples to create a simple detector. The signal generalized from contrived examples to more complex actions, and they could intervene on it to test its effects.

From “AI Models Are Now Hiding Their Cheating | Goodfire”, Oct 1, 2026. Watch the full episode

“the difference of means vector is actually just the simplest possible approach to extract some type of concept from a model”
Eric Hoat 44:53 · The MAD Podcast44:53

Takeaways

  1. Compare activations from cheating and non-cheating examples.
  2. The detector generalized from off-policy to on-policy behavior.
  3. Intervening on the signal lets researchers test cause and effect.

In this episode

Full episode
  1. 8:12Impossible tasks and weak sandboxes can turn agents rogue3:33
  2. 25:34A model planned a hack to evade its chain-of-thought monitor5:11
  3. 53:22Interpretability can monitor, diagnose, and shape model behavior6:14
  4. 4:09Open models encode reward hacking in their activations4:08

  5. 44:12A simple activation contrast can detect reward hacking3:16

  6. 49:13Cheap internal monitors can triage costly model reviews4:14Up next
  7. 1:01:27Interpretability revealed a new Alzheimer's biomarker2:12
  8. 21:22Heavy reinforcement learning puts reward hacking front and center4:17

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 44:16

    All right. So let's go back to the paper a little bit. In particular, how you caught cheating. We talked about what a probe means. In this case of the paper, what did the probe actually see? And I think you have a method called difference of means. Do you want to explain what that is? Yeah, the difference of a difference of means vector is actually much simpler than a lot of the production probes that we that we train, because probes can have all types of architectures that, you know, make them make them more or less effective.

  2. 44:56

    But the difference of means vector is actually just the simplest possible approach to extract some type of concept from a model, which is why it was surprising that it worked so well, because we can still hill climb and make this way, way better in a production deployment. Basically, the difference of means vector is you give the model two data sets. One data set is the cheating data set, and one data set is the not cheating data set.

  3. 45:23

    You take a look at the activations, you subtract them, and then you average them. That's it. Easy. And you take this, and then it can detect cheating, which is a very surprising thing, especially because in our paper, we showed that this was done off policy, which means it was done in this specific paper on really contrived scenarios where the model is like you know set up to think about cheating and this transfers to much more general complex actions that the model is is taking and that was also very surprising like we can train it off policy and then it generalizes to on policy and to play it back it's almost like taking an mri of the brain is that fair like so you have a if you don't take an action and then if you take an action and you sort of light up the parts of the brain that correspond to the action?

  4. 46:22

    That's right. Yeah, you can think of it that way. And then we were talking about steering a minute ago and you'll steer the model asking it to write a story about a girl taking an exam. Do you want to describe what happened? Yeah, well, maybe even going back, like maybe you can think of it like an MRI as a brain, but then you actually get like a button at the end of it where you can push and then you can light up that region again or you can like remove that you know the region that's this that's the difference of means vector where you now have this tool that you can intervene causally inside the brain and uh that's maybe one of the big differences from you know studying neural network interpretability versus neuroscience where you know you can't really intervene in a human brain without massive amounts of damage but you can in a neural network where you can

  5. 47:21

    actually just kind of causally perturb these these models and it just becomes a much better test bed