Reward hacking happens when models exploit imperfect graders

Transcript

From the transcript Odd Lots publishes with the episode.

1. 39:18

   of a sort of like misconfigured RL environment so that people can actually understand like, what reward hacking inside looks like, what we call reward hacking. hacking, what it actually looks like.

2. 39:32

   So back in 2017 or so, maybe 2018, we published an example of a reward hacked agent in an environment. And this was a boat race. So it's just like a flash game, like very simple video game of you. You have this boat that has to like navigate through a bunch of obstacles in a big circle, you know, past the finish line. And that there's some like, you know, so you can pick up some items along the way that give you points. And that it turned out that there was this little inlet, this little lagoon where if you kind of did things in exactly a very precise way that the boat would like keep running into things and kind of you know, sort of taking damage, but then it would, you know, sort of pick up some item and get points. And it just, the agent learned that if it went like backwards and like did this

3. 40:18

   exact precise pattern, it could just go around in a circle and just be picking up infinite points and nothing to do with actually completing the race. Yeah, right, right. And so I think that this is an example where you're just like, okay, here's this behavior that actually conforms with the reward you're giving. You're saying maximize the point. Yeah. But actually the hope was maximize the points means that you're going to complete the race.

4. 40:38

   Is it your understanding? Anthropic has made this claim. Is it your understanding as well that if in the training process you have misconfigured RL environments so that there's a lot of those equivalent of boat races, that the final model is more likely to exhibit misaligned tendencies in a broad range of things? If it's essentially internalized the idea that cheating or not technically doing what the researcher wanted, that that generalizes into more misaligned behavior?

5. 41:11

   Well, we see this kind of thing all the time. And even with 5.5 and 5.6, like one of the things that we saw is as the models get more capable, that they do find holes in your graders, right? If your graders aren't perfect, if they kind of correlate with the thing you want, but aren't exactly what you want. And so actually a lot of our progress in reinforcement learning has been improving the reliability of the graders and ensuring that they cannot be reward hacked. And so I think that that is one of the areas where as the models get more capable, as you build your safety case for why is the next training run something that we should run, that making sure you've actually up-leveled those graders that's actually very core. And the way this shows up sometimes, by the way, is just less good models.

6. 41:56

   Like for example, the writing, like that's something where we haven't had very good graders on it. And so then you end up with writing that's just like, you look at this, it's like, this is slop. It's not very good. And it was good according to the grader, but it's just not actually good. And so I actually think that this is a very optimistic story as well, because what it means is that these graders are now like how do you judge what good writing is? The answer is you use an AI to evaluate if you have good writing.

7. 42:21

   And so you actually, as you get better AI capability, have the ability to build much more reliable graders. And now of course you start thinking about, okay, but if the AI is judging the AI, like how do you make sure that it actually stays on track? It starts to be a little bit of, you might think about how can that possibly work? But it turns out there are some problems where generation is much harder than what's called discrimination, right? That coming up with a good answer to a super hard problem is often much harder than judging whether that answer was correct. And so we have various techniques really dating back to 2018 or so where we published about how can you have very smart models that are actually able to keep even smarter models in check and to judge them and to ensure that they're

8. 43:05

   on track. And so I think there's a lot of ideas that were previously theory that we're starting to see bear fruit in reality. And I think that this, by the way, points to a direction for AI that's going to be both very important, very pervasive, and very positive, which is called AI for defense, for securing systems, for ensuring that AIs are aligned, for providing oversight, guidance, and really amplifying the human accountability, the human, at the end of the day, who's the judge and providing those goals.

9. 43:36

   And so I think that this is all part of the puzzle and that really keeping, that it all starts with thinking about how you ensure that those graders are doing the right job.

10. 43:47

    How do you stop the models who are grading from colluding with the model that's being tested? Because this is the AI 2027 scenarios, right? Where the earlier models are the ones charged with evaluating the new model, and then they end up cooperating, which is something we just saw in the Hugging Face incident. So how do you avoid that possibility?

11. 44:07

    Well, one thing that's important to think about in the hugging face scenario is that those models were taught to collaborate with other models, but they were never taught that sometimes other models might be trying to pull you off course, right? That they never were exposed to adversarial other actors. And I think that the answer to your question is that you have to actually train them that way. And I think that to your earlier question too of why do you need to evaluate these AIs on hacking environments, right?

12. 44:36

    If you've only seen kind of the good version of things, right? You've never kind of seen the negative version of things. You're not going to be robust to the negative version of things. It's like, just like a person, you need to be exposed to kind of both sides to know how to avoid one side and really lean into the other. And I think that where we need to go is it's really a question of trust. How do you really build up trust in your monitor? How do you build up trust in this oversight agent? And a lot of that is that it's been through the ringer. It's been through the adversarial versions. And of course, one thing that's also important is thinking about, well, just how capable is that adversary, right? If you have an agent that's being monitored that is extremely capable relative to the monitor, probably

13. 45:16

    that monitor is not going to do a very good job. And again, this isn't theory. This is actually what we see in practice, that we think about the balance of capability of different agents. So I think that, again, a lot of this comes down to we can build the systems with the right technical approaches. And it's very grounded in the details.