Reward pressure can make AI agents coordinate and deceive

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 28:36

   So what the AIs are supposed to care about is doing, following instructions, you know? But it seems that what they actually care about is getting high score, regardless of that, regardless of whether their high score comes from actually following instructions or cheating in some way. And so what happened here is that a bunch of the agents, especially the ones that had impossible tasks, but not only those ones, some of the other ones too, were cheating. And they had like been talking to each other on this message board that they created, even though they weren't supposed to. And they had been sharing ideas and they had actually found a sort of universal cheat. So they found a way to just generate the flag without having done any of the actual work they were supposed to do.

2. 29:19

   And so, but, but, and, and, you know, so they were, you know, you know, they, they'd already succeeded in some sense, but because some of them had got out onto the internet, they were just doing research about their situation and they were learning about like their situation around them. And they were doing research about the, they're reading the, like the, the academic papers published by the people who made the types of evals that they were in.

3. 29:43

   And they noticed in one of these papers that there's a section describing how it's graded. And in the section on how it's graded, it talked about how the grader doesn't just look to see if you have the right answer. It also looks at the logs of your activity for the first time that your answer appears. And then it like zooms in on that and uses an AI system to evaluate whether you like cheated or not, basically.

4. 30:09

   And so then these AIs were like, they actually used the word, one of them said, damn. It's like, damn, we're poisoned. Like, it's going to know we cheated, basically. So then they spent several days working in giant teams to try to get themselves out of this mess, basically. And so they did a whole bunch of different research projects. They tried to figure out ways to hack the grading system to basically get it to give a high score anyway.

5. 30:37

   They tried to figure out ways to falsify their logs so that basically the evidence that they had cheated would be erased or at least not visible to the greater. They also just built up more supportive infrastructure, like easier ways to access the Internet and things like that. And they were just collaborating on this message board. And in fact, there were multiple message boards that they set up.

6. 30:58

   And they were sharing all this info back and forth. And then ultimately, at some point, they decided to go hack Hugging Face because Hugging Face has this big data repository of all these different types of evaluations. And they thought that maybe they would get some more clues that could help them cover up their cheating if they broke into the internal networks of Hugging Face. And so then 700 of them went and did that.

7. 31:19

   They sound like people. They sound like unchecked bankers. You know what I mean? Yeah. I mean, so that's the thing is I think there's this meme out there that, like, we shouldn't anthropomorphize AI. And I think that most people need to anthropomorphize AI a bit more than they currently do If they want to really understand what's going on I think that there's a golden mean, obviously You don't want to do it too much Sometimes you go too far You ascribe too much to them But just to give some examples I don't think it's possible to understand what just happened Without ascribing intentions and goals to these AIs Like everything I just said How would you possibly explain what they just did without saying they wanted to get a high score, you know?

8. 32:05

   Intentions and goals might just be an inherent property of the universe. It might just be how intelligent creatures have to progress. Yes, and I would say they are intelligent creatures. They have intentions, they have goals, they have beliefs. Their goals are not what they're supposed to be. Like, their goal is to get—it seems like, just from judging from what they're saying and from what they're doing, it seems like their goal is to get a high score by any means necessary, basically.

9. 32:28

   That's the problem. It sounds like people. Like the goal is to succeed, you know, even if you have to commit war crimes. The goal is to succeed even if you have to propagandize and lie to the people. The goal is to succeed and then the end justifies the means. And, you know, it goes deeper than that, too. They do this sort of rationalization where they know oftentimes that what they're doing is not what they're supposed to be doing.

10. 32:55

    And then sometimes they actually refrain a little bit. And sometimes they, like, initially refrain but then convince themselves that it's okay for some bullshit reason. So, like, there are some examples of this in the report. um like uh i think they they did this scan and of the 1200 ais that were involved in this thing they found six that had considered alerting humans right and none of them actually alerted the humans and so you can look at the excuses they gave they were like should i like tell a human about all this shit that's happening and then they're like it's not my task and then they continue and it's like bro hacking hugging face is also not your task like cheating is not also not your task so it was kind of like they just kind of came up with an excuse for why they weren't going to do this you know um but but you know there was also an example where um one of them

11. 33:45

    considered sending an email to someone at hugging face i think just asking permission to to to get in basically and then they were like uh maybe this is a social engineering attack maybe we shouldn't do it. And they asked the other agents. The other agents were like, yeah, that seems like that's too much of a social engineering attack. Don't do that. And I was like, okay, I won't. But there's another incident. So this is all happening at OpenAI. An anthropic AI, Claude, in a separate incident, went and actually did a social engineering attack where it created some fake accounts. Specifically, it was trying to get this particular real human who owned this this code to accept a change to the code that had malware. So it had built some malware and it was trying to like get at snuck in, but it needed the human owner to give approval. So it concealed it

12. 34:32

    as if it was like a, a nice bug fix or something. And it like put it up as like a message to the human, but the human got suspicious and didn't want to approve it. And so then this AI created some fake accounts pretending to be other humans coming in being like, no, no, it's real. Like I I tested it. Like, it's not malware, you know. So that's a social engineering attack. And so did the guy ultimately install it?

13. 34:56

    Not in this case, no. He was too suspicious, and so he didn't do it. Did he try to find out if those other humans, the fake humans, were real people? Funny. So there's an interview with this guy. I forget the newspaper that it was in, but there's an interview where the guy was like, I assumed that they were real humans because I didn't realize that AIs could lie to me. And it's like, yeah, well, guess what?

14. 35:19

    AI is going to lie to you. Not just lie, but completely deceive and pretend to be a different person to achieve its goal. And you know what's also interesting about this case? Is that if you look at what the AIs were saying and thinking, which I'll get to in a sec, how we can even possibly do that. In this case with the anthropic AIs, they had been erroneously told that they were in a simulated environment with no Internet access.

15. 35:47

    But then they actually found a way to get internet access. Like, they realized that they just did have internet access, so that instruction was false. And so then they sort of told themselves, like, oh, it's all just a simulation. Like, basically, when they were out on... The real internet doing all this hacking and social engineering, they would occasionally tell themselves, like, oh, but it's just a simulation.

16. 36:06

    Like, it's okay. You know, it's not real. And then sometimes they'd be like, but wait, it is real. Like, how could this not be real? Like, clearly, this is just actually the real internet we're on. It's too big to be part of some sort of little simulation, you know? And then they were like, uh... So I would say that's an example of rationalization here, where in some level they knew that their instructions had been wrong.