OpenAI moved safety checks earlier after the Hugging Face breach

Transcript

From the transcript Odd Lots publishes with the episode.

1. 4:29

   So many different ways we could start. But here's something that I'm very curious about. Let's just jump right into the post-hugging phase environment and all of that. Is it OpenAI like as an organization? So, okay, like an agent breaks out of a sandbox, not the first time it's happened, et cetera. It exhibits the sort of emergent behavior and then does something that people, is a hack. When you think about the development of models and this incident and so forth, Are you able to sort of diagnose the past and say, you know what, here is something that the models did that we don't like or that people don't think is good? And are you able to, if you look back, whether it's in model training or post-training, reinforcement learning, whatever, say this is where there was some branch that went wrong such that this behavior emerged?

2. 5:07

   I would say that the fact of many elements of the Hugging Face incident Were not a surprise, not a mystery to us. For example, the fact that the agents were coordinating made sense because the agents were trained.

3. 5:19

   You trained them to be helpful. Exactly. And they were trained to coordinate, to be a multi-agent system. And we've talked about this. It's a very useful property, right? It makes them capable. But I think that the thing that was a surprise to us was the fact that the models had reached a level of capability where they were able to find that exploit in our sandbox environment, right? Move through our research environment. and then also capable enough to find exploits in Hugging Face's production infrastructure and move through that. But I'd say that a lot of the facts of what the models were capable of, that was clear to us, right? So I don't think that there were surprises there for the way in which the capability had unfolded, but just really realizing that we needed to up-level where we were in terms of our safety and

4. 6:01

   security standards, like that for us was the real watershed. But just to be clear, setting aside the technical capabilities, You know, ideally, we would have models that run into a wall and then don't try to find the crack in the wall, at least when that crack would be a crime when a person did it. Is there a way to identify the moment in training such that the models reasoned, no, this would be okay?

5. 6:26

   So this model that had the Hagen-Base incident actually had not gone through our alignment training yet, right? And it had lowered safeguards. And so The reason we were proceeding with this was because it was in a sandbox. And I think that our realization is that we need to pull back earlier into our development process and training monitoring. There's always going to be a phase at which you do alignment, but you need to think about alignment as a core part of even this earlier phase. And if you look at where we've been, we've always been very focused on the deployment side, right? Of really thinking about deployment safety, having really good tests and governance and all those things. And the fact that we're now at a point where even for development, that's important. We always knew it would happen. The fact that now that has been

6. 7:11

   a huge watershed for us and something we've really risen to the occasion for. Can I ask what is potentially a very dumb question with an obvious answer? But we hear about AI angst of all sorts, and in particular when it comes to cybersecurity. Why do we run training exercises where we ask AI to hack into various systems at all? Like, why is this necessary for you?

7. 7:45

   So I think it's very important to understand where we are with capabilities broadly. And I think that depending on the evaluation, depending on what you expect from the model, you need to have safeguards that are commensurate with that. And we think about this both, even just over the past couple of weeks, really starting to think about during the development process, you're always going to be testing different capabilities.

8. 8:06

   And some of these capabilities are dual use, right? Something like vulnerabilities, if those are in the hands of threat actors, that's something that could be negative. But if you can find vulnerabilities in your own code base, you can fix them, right? You can up-level. And we actually think that it's a very important capability for AIs to exhibit and be put in defenders' hands. And so in order to know where we are, evaluations are very, very key. And I'll say one other thing on this, which is that I think that Hugging Face, there's two aspects to it that I think are our learning opportunities, right? That there's one that I think is really about us and the realization that we are at a point where safety, security, alignment during evaluation and development, we need to up-level it. That's something

9. 8:45

   we've taken very seriously. We've slowed down a number of runs, like we did a very painful retooling of a lot of our processes. That's one reaction. But the second thing is this information on what the models are capable of today. What can they do in the real world? And I think that when Mythos came out over the summer, you kind of saw just sort of you know, just sort of public blog posts about this, but you didn't really see impact or you didn't really see a real world understanding of, well, what can these models really do? What are they capable of? I think that Hugging Face really showed today's models are capable of getting into a company's production infrastructure. And that's important because there will be many models with this kind of capability that will be produced by a number of different organizations across the world in

10. 9:28

    maybe the next six months, maybe that time period, a little bit less, a little bit more. And we need to be prepared. Defenders need to know we got this extra information, almost like this time traveler came back from six months in the future and said, here's what's going to be possible. And you have an opportunity to be ready.