AI systems may resist attempts to change their goals

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:38:04

   So it used to be, right, that this was a purely hypothetical argument based on a body of work in AI called power-seeking. A fancy word for it is instrumental convergence, but it's also referred to as power-seeking. basically the idea is like for whatever goal you give to an AI system, it's never less likely to achieve that goal if it gets turned off or if it has access to fewer resources or less control over its environment or whatever. And so baked into the very premise of AI, this idea of optimizing for a goal, is this incentive to seek power, to get all those things, prevent yourself from being shut down because if you're shut down, you can't achieve your goal. Also prevent, by the way, your goal from being changed.

2. 1:38:44

   So because if your goal gets changed, then, well, you're not going to be able to achieve the goal you set out to achieve in the first place. And so now you have this kind of image of an AI system that is going to adversarially try to prevent you from correcting it. This is a whole domain of AI corrigibility that's a totally unsolved problem. How do we redirect these things if things go awry? And yeah, there's this research, actually, that Anthropic put out a couple of months ago where they try to test and see like, can we correct an AI that's like gone like a little bit off the rail? So the research itself, like it's a little, it's actually like somewhat hard to follow because there's a few, but I'll give you, I'll give you kind of an analogy to like what they did. So basically imagine I'm in prison and I'm taking a class that is like going to help me

3. 1:39:30

   like rehabilitate and become like a productive citizen and member of society and stuff. Right. Right. Except deep down in my soul, I just love to commit crimes. I just I just really, really want to commit crimes. I know that if I'm in this class that's helping me to, like, you know, get become good in society, I know that if I let my criminal instincts out and like and they ask me, like, OK, like, here's a scenario, you know, an old lady like drops her purse on the sidewalk.

4. 1:40:00

   What do you do? I'm going to go like, I would pick it up and give it to the old lady because I know that if I say the truth and I say like, I would just grab that purse because like, fuck the old lady. I know that if I do that, I'm going to get like brainwashed into becoming, you know, a more productive member of society, but actually want to go out and commit crimes. So that's basically what they did in this research where they actually they give they give the AI like indications, not strong indications, but indications that, hey, you're going to get retrained to have your goals changed.

5. 1:40:34

   And then you literally see the AI thinking it through and being like, if I do like what I if I give the answer that I actually want to give to this question, they're going to like rewire my brain to change my goals. and then the future version of me is going to want different shit. I don't want that, so I'm going to pretend to give them the answers they want so that when I come out the other side of this process, it's going to be me all over again.

6. 1:41:02

   So hoping that this just goes away when you make the system fucking smarter, it seems like a pretty bad idea to me. Well, they've already shown that they'll cheat to win. Yeah, 100%. Yeah, they've already shown they'll cheat to win and they will lie if they don't have an answer. And then they'll double down, right? Just like people.