A model planned a hack to evade its chain-of-thought monitor

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 25:37

   the big hope of the last two years was chain of thought monitoring this idea that since models were writing explicitly how they think you could have a person or more likely an AI review the reasoning and therefore catch a bunch of things? Or was that ever true? And if it was, why does it seem to be fading as an avenue? Yeah, this is the dominant way to monitor models. It's maybe just to zoom out. chain of thought is these models are all reasoning models where they output a chain of thought prior to their actual answer to like whatever problem that they're given or whatever agentic action that they're given. The reason why we've trained models this way is that we found empirically that if the models reason for a very long time before outputting their answer their answers get better and so what was kind of very convenient about this approach is that you can just read this

2. 26:48

   like reasoning chain and a lot of the times let's say a model was attempting to deceive the user or attempting to reward hack or something like that model will just have it in its chain of thought it'll just be like hmm i want to cheat on this exam because it's going to be easier than actually solving this problem, you can just read it in plain English like before the model actually cheated on this exam or cheated on the problem.

3. 27:11

   And this was convenient up until now where I think there's two things really changing. One is with increasing RL pressure, chain of thought degrades. The reason behind that is that these models are starting to compress more and more semantics into a fewer number of tokens, more and more intelligence into a fewer number of tokens, and they're heavily incentivized to do so by RL, which also puts pressure to make their reasoning shorter.

4. 27:49

   Is that the neuralese question where like bit by bit, this becomes impossible to understand by a human because the language is so compressed that it doesn't look like English anymore? Wolf, who was on this podcast recently, from HuggingFace, was talking about neurales as an issue. Yeah, definitely. We actually just announced a partnership with HuggingFace, and we're going to do a lot of our techniques on open models in partnership with them.

5. 28:19

   Some people think of neurales like this, and then some people think of neurales in the other sense, which I'm about to talk about, which is latent reasoning. So latent reasoning models are going to become a much bigger thing. Like OpenAI's Astra model, it was reported, is a latent reasoning model. And so what that means is instead of the model outputting a token given a single forward pass of the model, there are some loops internally before the model outputs the token.

6. 28:50

   So instead of thinking out loud, it thinks internally. So it doesn't actually think out loud. you have to read its mind in order to understand what it's thinking. And so that's often what folks define as neural ease. It's like internal neural computation, I think. But it's an ill defined term right now. I think people are still wrapping their head around what neural ease is. And so, yeah, like those two ideas, like this prevalence of neural ease means that you're gonna have to understand neurons and what they do and what's going on so that's the whole point of interpretability and other examples perhaps as part of your research where the models intentionally make the chain of thought impossible to understand as part of a rewarder hacking basically sort of hiding their actions that is a really interesting question uh we have

7. 29:44

   now i believe seen um yeah so just a couple days ago one of our researchers found that the model was explicitly reasoning about having an external chain of thought monitor monitoring its reward hacks. And so the model was reasoning, hey, all right, somebody's going to read my reasoning. So I should think about how to craft my reward hack so it won't get caught by this external monitor uh and so the meta games with these models of gaming these evaluations are getting so many layers deep that i can't even follow it anymore you know it's like it's it's pretty complicated uh and i think it's just going to be a little bit of a cat and mouse game um where the model is just going to try to solve the problems and uh find the answer no matter what and sometimes cheating is the easiest way and it's a it's effective and we mentioned the big labs and