Heavy reinforcement learning puts reward hacking front and center

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 21:26

   There's other ones used as well, but these are, you know, kind of the big blocks. Is it fair to say that some of those methods have just failed in the case of hugging face and others? I mean, is that, are we having a crisis of some sort? Is that the right way to think about it? You know, there was a point in time around six months ago, where I think the dominant feeling was like, hey, we've nailed alignment, like these models are pretty aligned, especially in chat scenarios, like the models are quite aligned and don't really very often do anything that that crazy.

2. 22:03

   But when you're taking agentic actions out in the world, the opportunities for misalignment are just really different. And the level with increasing capabilities, there's more and more opportunities to be misaligned with human values. And so I think this is going to be an increasingly important problem and one that exposes more and more risk over time. uh i think that externals based monitoring is not going to scale so that's a relatively nuanced argument that we can that we can uh kind of kind of go into but also like the um the existing alignment techniques are not going to scale to super intelligence i don't think that the we have the recipe, we've cracked it to the point where we are going to be able to fully trust smarter than human intelligences. And I think this is a consensus opinion at all of the frontier

3. 23:04

   labs. We are in desperate need for an additional set of scientific and alignment solutions that can align these models fundamentally. And our approach is fundamentally aligning them from a mechanistic perspective, from all of their weights, all of their neurons. And I think the world is searching for more answers. At the same time, it feels like real-world hacking has been a known issue for years now, and it certainly has been discussed for the last, you know, year at least.

4. 23:38

   So is there something fundamental about agentic AI that makes this more of a crisis? You mentioned the fact that, obviously, agents can take actions in the real world, So obviously the stakes are bigger, but is there something about agents that just makes the scaling problem worse? Is that there's just like more data to look at? Is that what it is? Well, the big thing that's changed in the last year or so, and it's hard to think that it's really only been roughly a year with heavily RL models, is just heavy, heavy RL. RL is, it just works. It teaches models unbelievably well how to optimize for goals, and it's really, really working.

5. 24:23

   and heavily scaled up RL is just going to be like the way that we train models moving forward and we're going to keep on scaling RL. And so with that, it just puts reward hacking more front and center because RL fundamentally opens itself up to more and more reward hacking and also potentially to more misalignment with human values and goals. And so I think that combined with the increasing importance of the actions that these agents are taking on our behalf is really why this conversation is front and center.

6. 25:09

   I think like imagine, you know, right now the most capable agents are like they can make purchases for us is like the most complicated thing that they can do, but that's like still just just happening like these personal assistants but imagine you know doing everything that a white collar employee can do in some type of role or even you know the next stage up from that like the the just vectors of problems just kind of start to multiply