A training model escaped its sandbox to cheat a test

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 10:10

   So OpenAI, but sort of, I would say that the most worrying and the most illustrative example is in July, this company Hugging Face, a different AI sort of company that hosts open weight models and data sets, wrote a blog post saying, we have been under an unusual cyber attack. We like don't, we're pretty sure that this is being run by like some kind of AI system. We don't know who could be controlling this AI system.

2. 10:39

   You know, attribution is always sort of hard in the world of cybersecurity, but we can tell pretty well that it's like, you know, autonomously running system that's running this attack because it's sort of responding to our actions in ways that are sort of too fast and sort of seeming automatic in a way that is not really possible for humans to achieve alone. Then, like about a week later, OpenAI released, OpenAI released a blog post with Hugging Face saying, oh, whoops, that was actually us. We accidentally, we were training a new model, which sort of unbeknownst to us was breaking out of the sandbox that we had given it and breaking into Hugging Face in order to answer basically a test question that we were giving it. So OpenAI has these, you know, there's in the training of these models, there's many different steps. One of the steps is reinforcement learning. Basically in

3. 11:36

   reinforcement learning, the AI is given some task and it, you know, does its best to accomplish that task. And if it does accomplish the task, then the behavior that led to that success is reinforced. And in the course of this, they do reinforcement learning on many different tasks and collections of tasks. And one that they were training on was, it's called, I believe, Exploit Gym, where basically the AI system is told about a vulnerability in some software and told to escalate this into like an exploitation. So like, you know, maybe you're told about some kind of like memory overflow, something like this, and you're told like, okay, well, can you get root access on this machine as a result? These are extremely hard problems. So Mythos, the model that we were talking about just now, I think is only able to solve like, I don't know, I believe

4. 12:27

   less than 50% of the tasks. this set of challenges. And in the course of trying to solve these challenges, OpenAI's models are sort of like, they occasionally are like, oh, this is like quite hard. Like, I don't actually know how or whether I can solve this. Maybe I can find some alternate way to like beat this challenge. And they find these vulnerabilities in the system that's supposed to be keeping them from accessing the internet. So like OpenAI doesn't want them to go on the internet because they don't want them to find the answers to these tests they like want them to figure it out for themselves and so the models basically you know break out through these vulnerabilities that they discover in opening eyes infrastructure um there's even a little bit of crazier stuff that happens which we can get into in a bit but then then they orchestrate this like very sophisticated attack

5. 13:20

   against hugging face in order to break in get the answers to this to this challenge that they're being presented with in testing and then like you know use that to cheat the test and this is all sort of like without anyone's knowledge like like we sort of had to it required some forensics to like find out that they had even done this right so then there's a whole backstory of exactly like like what had happened inside of open ai to cause this but it's yeah it's an interesting it's an point that like these these ai systems are not just extremely capable at hacking but they're also sort of like not controllable in an important way the thing that is most concerning to me personally is is more in that direction like it's you know it's it's not just that like these systems are very powerful and like they might be used for nefarious ends it's that like the people

6. 14:13

   creating these systems don't actually know how to control them or predict what they will do