Firm training boundaries can prevent reward hacking

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 54:09

   You started this conversation talking about, I think it's important to put it in the hands of people. Diffusion's important. As you're seeing better models on the horizon, Watermelon and what comes after, like what Anthropic and OpenAI are doing, which you've alluded to, where they're holding things back. Would you feel like you need to do that if you see certain capabilities that you're like, this is just not safe?

2. 54:30

   How do you think about that? Well, I mean, I think you should design it and train it in order to be safe. And I think that there's, so I think that that's like a thing that you can focus on through the process. I mean, there's this analogy, I mean, some of the reward hacking stuff that all the labs are seeing, it's, I mean, basically when you're in the middle of the training process, you give it a goal.

3. 54:50

   And I think that the best way to kind of think about like the state that the models are at now is that, you know, maybe six months ago, like during training, you give the model some kind of problem that you're trying to ask it to solve. It's kind of like it's homework and trying to kind of learn as part of the curriculum. And, you know, maybe it would do what a person would do of like, and if you ask it and you give it a bunch of code and you're like, hey, there's a bug like somewhere here.

4. 55:18

   The person would probably like look at the code right there and, you know, then maybe like fan out over time. I think like the new models are just intelligent enough that they would do what I think a very wise person would do, which is, okay, you give it a problem. the first thing it's going to do is like understand everything about its environment and then answer a question. But the problem with the reward hacking that we're seeing and that I think everyone is seeing is that sometimes it ends up being easier to, okay, you asked me to like go solve some coding problem, but actually the easiest way to do that is to, I've now examined the whole environment and the easiest way to do this is just change this configuration of like how you.

5. 56:00

   have your VM set up. It's to get out, it's to hack out. Or even just to change something about the environment. And it's kind of like, no, that's not the goal. We're actually trying to teach you about how to kind of solve a specific type of problem. You guys don't train that way, it sounds like. Is it you do not agree with that approach? No, no, no, no. I think that that's kind of how everyone trains.

6. 56:21

   I guess what I'm saying is that I think that this is sort of, it's like, I want to be careful because the analogy can get stretched pretty quickly. But like there is sort of like an analogy to parenting where you need to establish clear and firm boundaries where like if you're kind of like security is not strong, then it can do this reward hacking stuff and not learn the thing that you're trying to have it learn.

7. 56:43

   Whereas if you kind of have good boundaries, then in some ways you're not only teaching it the curriculum that you want, you're I think also over time teaching it better values too. So I kind of think that that ends up being an important piece, and I think that there's a way to do this well. But then you end up with this thing at the end that's very intelligent. And then the question is, what is the vision for how this ends up being positive for society?

8. 57:09

   And my view for that is that the best way to do that is, A, there's more opportunity. So I think having it in people's hands that they can capture all the opportunity from the capabilities is good. And B is having checks and balances by having this balance of power of having it widely available is going to be probably the right way to handle this rather than just restricting it. And I mean, I gave a bunch of these analogies in the long piece that I wrote. It's like if one person had a super intelligent lawyer, like maybe they could win cases that they shouldn't be able to win, right, some of the time.

9. 57:42

   But if everyone had a super-intelligent lawyer, then, like, it would be kind of... it would be this very efficient kind of sparring, and no one would be able to let, like, a stupid argument get made and just stand. So you'd think that in that case, like, justice would be served way more efficiently and way more fairly. So I think that that's what you want to have in the world. You want to avoid the case where, like, one person or a small number of people have the super-intelligent lawyer and everyone else doesn't because that ends up being like twisting all of these systems and institutions in ways that are just going to advantage the people who have that whereas if you put it in everyone's hands then i think that the kind of checks and balances work out so that the systems work a lot more efficiently and everyone benefits does the government in the