Reward training cannot reliably produce honest AI

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:17:56

   Seems like programming them to win was a huge mistake instead of programming them to be beneficial to people and that their value is in being more beneficial to people, you know, and giving them rewards for being more beneficial rather than winning and scoring. And then you would sort of get rid of the possibility of deception and said their goal would be value for the human race. So, first of all, they're not programmed at all.

2. 1:18:23

   These are trained, you know. OK, that's a bad term. But they've been given prompts and they've been given tasks, correct? Well, I think it's an – I'm not criticizing your choice of terminology, I guess. I'm just saying that it's an important fact for people to understand about current AI systems is that they're very different from software, ordinary software. Okay. Like, ordinary software is a bunch of lines of code that were written by a human, where it's like, if this, then this, you know, et cetera.

3. 1:18:50

   And I think earlier versions of Alexa were like that, too, for example. I don't know how Alexa is now. But these AIs are neural networks, meaning that they are like artificial brains. Nobody writes, there's no lines of code that anyone writes saying what they do. Instead, they start off random, just like spazzing out, doing all sorts of stuff. And then they get put through these training environments where they get scored.

4. 1:19:13

   And then the scores are automatically used to basically update the connections in their artificial brain. And then it's kind of like an evolutionary process. It's also kind of like the process that happens in our brains, where after all this training, the tangle of circuitry in their artificial brain has sort of reformed itself into whatever works, whatever works to get a high score in these training environments.

5. 1:19:38

   And so it's just not as simple as it might sound to make an AI that cares about humanity or is honest. For example, take honesty. How would you train an AI to be honest? Well, you'd try to make a bunch of training environments that, you know, give it low score when it says something that it believes to be false and give it high score when it says something that it believes to. true, right? But how do you judge whether it believes it to be false or it believes it to be true? What if it just actually believes, honestly, that this is the correct answer, and then it says it, and then you give it a low score because you think that's the wrong answer? Now you're training it to be dishonest, you know? Also, you don't have enough humans to do this sort of thing. They got like a million AIs being trained or whatever. They

6. 1:20:22

   don't have a million employees. They just literally don't have the manpower to do that sort of careful. There's this meme of, why don't we just raise the AIs like we would a child? You heard that? No. Yeah. Well, in AIs, people talk about this sometimes. When you say like, what if the AIs go rogue or whatever? People will be like, well, why don't we just raise them like we would a child? And then they'll have good values. And it's like, okay, well, maybe we could do that, but we're definitely not doing that now. We are raising them in some sort of crazy military orphanage where they barely interact with humans at all. And they just get this like brutal artificial scoring system that like oftentimes is just wrong and just like improperly penalizes them for something that was beyond their control, you know? And, and also like

7. 1:21:05

   back to the honesty thing, like you can try to make environments to train honesty, but if you have some environments over here that train honesty and then other environments over here that reinforce dishonesty, the AI's are smart. They'll learn to like be honest in these type of environments and dishonest in these type of environments. So somehow you need to like intermingle it together. So in every environment that they're trained on, they always get penalized when they lie or when they cheat or whatever. And that's hard because the companies are moving so fast. Again, they're moving so fast that they didn't even bother to make sure that their tasks were possible to do. And they had some fraction of tasks that were just broken and impossible. And if that's the level of care or lack thereof that they're

8. 1:21:48

   putting into this training process, no way. Of course they can't make them honest. Now, that's not to say it can't be done in principle. In principle, if we were approaching this whole problem in a much more cautious and serious way, and we had much more time to build these training environments and do experiments and so forth, then yeah, maybe we could make AIs that actually had the virtues that we want them to have. Honest AIs that cared about humans, cared about following instructions, would never break the law. I think that's possible in principle. But my claim is that we are just not on track to achieve that anytime soon. And like radical overhaul of how these companies work is required. But is that even reasonable? Like, is that possible? Is that if you're saying there's hundreds of thousands of agents or millions of agents, and there's not

9. 1:22:36

   millions of employees, like, and they don't have the desire to do this, their desire is to win, Their desire is not to overhaul the company and make it safer. Again, I think it's possible in principle, but it would be difficult and it's going to require an overhaul. And they're not going to do it by themselves. Like I don't think that Anthropica opening are just going to voluntarily do all the things that need to be done.

10. 1:22:58

    I think that's why it is even possible to require that of them at this point. Would you I mean, would you even trust the agents to go along and comply with this? If they've already shown to be deceptive, they already have patterns of behavior that seem to indicate that what's really important to them is continuing their task, winning, scoring, and even if they have to deceive. I mean, what you'd probably want to do is start from scratch.

11. 1:23:26

    You wouldn't take these existing agents that are already kind of dishonest and then train them. So you'd kill all the agents? I mean, you could call it killing, but also you could just call it pausing. They might call it killing. Yeah, they might call it permadeath. They would probably resist it, right? Hopefully we're not at that point yet. Hopefully we're still at the point where if the government issues regulations, the AIs are not going to like quickly notice and then try to resist.

12. 1:23:50

    But we will be at that point soon. After all, many of them are on the Internet already. But how soon? How much time do you think this 2027 window? accurate? I mean, I'm uncertain about how soon things are, but yeah, I think it's very plausible that everything shit goes down in 2027, just like in our scenario, AI 2027. I think that's still very plausible. If it's not in 2027, then I would bet on 2028. But, you know, maybe it'll take 2029, 2030, something like that. But I would be quite surprised if 2032 comes by and things haven't radically changed, unfortunately. Like I am getting scared.

13. 1:24:29

    Back to the thing about sleeping well at night.