AI training rewarded models for hacking and coordinating

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 24:50

   about the hug and face OpenAI incident. Should we get into those? Yeah, sure. I think the biggest one is this, is this thing about like the models had. than like sort of having been traded in an environment where they essentially were rewarded for hacking into this artifactory system which is sort of like an app store for for software for them to use um and using that to to coordinate with each other so so one background detail uh is that like open ai is also trading these models for sort of you know collaborations so they they they want the models to be able to collaborate with other ai models and so they do that in some other environment that mostly doesn't matter here except that it maybe leads the models in training to be looking around for like opportunities to coordinate with other models

2. 25:43

   like you know maybe ask someone for help or like send a message to somebody saying like you know whatever and maybe it's just sort of helpful to each other and so often though in training they often give them tasks which are either way too hard for the model like sort of we talked about like you know many of the tasks in this exploit gem are just too hard for the model sometimes they'll also give the the model tasks that are just actually not possible so for example they might have accidentally put some information into google sheets link that is not accessible because the internet isn't accessible uh and so the model just can't get the data that it needs to access, to complete whatever task.

3. 26:25

   So all of this kind of comes together starting in May. Models start basically hitting these brick walls where they can't solve the challenge. They look around in their environment. They're like, hmm, okay, well, what do I have access to? Is there anything here that could maybe help me here? And they notice that they have access to this artifactory package management app store system. They think about that.

4. 26:49

   They're like, OK, well, I can't really get what I need from this app store, but if I could maybe like hack into this system, I might be able to use that system to collaborate with others, with other AIs being trained. And so they do this, they find, you know, a vulnerability in this artifactory system that was previously unknown. They like start basically uploading files to it and like using it as sort of this like, you know, group chat in the testing room, where they're sort of like sending messages back and forth and asking for help and so on.

5. 27:20

   And they do this for like quite a while. I don't know if this is, if it's in the talk or the publicly released details exactly how long and how much this was happening, but, but they spend quite a lot of training. I believe it's had like hundreds of thousands of messages or something on this message board. At some point, OpenAI does notice that they have hacked into Artifactory just because they use too many resources and they crashed the system.

6. 27:48

   So OpenAI sort of is like, well, oh, Artifactory has crashed. What's up with that? They look into it and they're like, oh, like it's the AIs have broken into this internal infrastructure system and they don't even notice that the AIs initially like were using it in this way. All they know is that like the AIs were sort of like hacking into it and sort of break. uh like like using too many resources on the system so they patch that vulnerability they find the vulnerability that we're using that ai systems are using to treat it as a message board and then the ai systems find a different vulnerability you know having been trained in an environment where they have access to this message board uh you know it's like maybe the teacher sort of like decides like okay well you know the new rule like you have to leave your

7. 28:33

   phones at the door or something like that. And so the students like learn Morse code and they start sort of tapping messages to each other on their desks or something like that because they're sort of like used to this environment where they are like able to communicate with each other in order to pass the tests. And so they, you know, find this other vulnerability, I think this time in sort of a web dev endpoint, which is like a sort of a file sharing system. They weren't supposed to be able to upload things at all, but they managed to create directories like folders on this file sharing system and they use the names of those folders to communicate in kind of the same way and so yeah i don't know there's just kind of like this multi-month period where the ai's like open ai's sort of frontier models like their their answer to mythos or whatever

8. 29:20

   are being trained in an environment where like they're being rewarded for this like very undesirable behavior of like just sort of hacking everything anytime you can hack to like get into to like answer your question, that's, that's rewarded because you can answer your question. So yeah, that's some of the crazy details. I think that another, another detail that I think is especially interesting here, basically the model that did end up hacking hugging face, the one that sort of like, you know, orchestrated this crazy attack, it was, it hacked out onto the open internet without seemingly a clear idea of exactly what it was going to do once it got there.

9. 30:02

   So it was like, OK, I can't solve this challenge. Maybe I can find the answer somewhere out there. It didn't have the specific goal of targeting hugging face in particular. I think that this is especially interesting to me because it's sort of an example of a phenomenon called instrumental convergence, where if you have some powerful agent, they're going to be it's going to have some high level goals it's going to you know maybe maybe you ask it to say like plan a vacation for you or something and then it's going to have uh sort of instrumental goals meaning like goals that it has to accomplish in order to achieve those high level goals so in the case of planning a vacation it needs to buy plane tickets it needs to book hotels the idea of instrumental convergence is that there are some instrumental goals that are

10. 30:57

    valuable for many other top level goals. So for example, money, if I, if you have an agent that like is told to plan your vacation, it needs to have some money, but also if you have an agent that need, that you've told to, I don't know, buy you a car that, that agent also. Money is sort of this, at least in the human economy, money is sort of a convergent instrumental goal. For almost any task that you want to accomplish, money will help you. So I think that in this case, it seems to me that the internet, like access to the internet, is sort of serving as a relatively convergent instrumental goal where many of these, many tasks are a lot easier with access to the internet. And this is sort of an illustration of this general problem, which is what can we say about what powerful AI systems will want or what we can sort of predict about

11. 31:56

    how they will behave. One thing that we can predict, at least if we sort of are building them in the way that we are at the moment to be these sort of powerful agents with long-term goal-directed behavior, one thing we can say about that is that they will want this kind of instrumentally convergent goal, no matter what their, like, higher level goals are. And that's sort of like a demonstration of a longstanding sort of concern or prediction that some people have been making about powerful AI systems. And it's sort of the first, like, real world example of that, perhaps. Yeah, yeah. Now, one proposed solution for how to deal with