Weak signals and a security incident prompted OpenAI’s training pause

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:25

   about for a long time and now it's happening. What has it been like living through it? Well, it started even longer than that with the Hugging Face incident. Right. And that was a real moment of, man, this is like, it's like a sci-fi story. You can understand how every piece of it happened, but the number of things that came together for the Hugging Face incident to happen was a real, Wake Up Call is too strong of a word because again, we had talked about this, but it was like That and the things that happened at other companies were a legitimate moment of like, wow, the AI capability level has reached new heights and our alignment of the alignment of the model, the security we have around the model, that failed.

2. 2:05

   Now, we treated that as an accident and we've responded as such. And I think that is the way to make things better. But that was when this whole period of these last couple of months started. We then potentially hit cybercritical under our preparedness framework. we then saw some things during our training run where we said well you know we need stronger alignment guarantees and we need new methods and to make more progress here but I feel both very proud of how we've reacted to it very like okay we're in this in a way that feels like kind of I mean it feels strange to have been thinking about this for the last decade and for it now to happening and then like you know we know what to do what was the thing you all saw in the training run that is not astra that's the future stuff that caused um it seems like the reaction that

3. 2:58

   you're now talking about i mean i know you described the hugging face all of that and people know about the hugging face incident but what happened on the pre-training run that really alarmed you guys uh it was not one single thing it was reading reading lots of samples and seeing well this this behavior is not quite aligned in the way we thought or this is this is a behavior that is like somewhat concerning combined with these other things even though it would look maybe okay in a vacuum so it's not like there's not one smoking gun like there was with the hugging face attack of like here is this bad thing we can point to you that happened but it was various degrees of misalignment along with and i think this is the more important thing than any single data point the rate at which capabilities are now progressing you know honestly like we had

4. 3:43

   not the world's best last period of pre-training progress, we all of a sudden have gotten so good at it that we now have these remarkably capable models. It's really amazing what Aiden and his team have done. And so you have these small things that you can point to in our RL process or, you know, alignment concerns combined with what we can see coming down the road of these amazingly capable new pre-trained models. And it's really that intersection that made us want to react with abundance of caution now i don't want to overstate this either i don't think this is like you know we're in this extremely critical potential catastrophe point but i also think that as the stakes get higher as the models get more capable because of what our mission is and because of how important it is that safety outweigh all the other you know pressures we have we wanted to

5. 4:37

   react with an abundance of caution. And I think that's the right thing to do. I think it's good that we're doing that. I think it is a good time to slow down and make sure we can have new safety cases that justify the runs we want to make. I think that, you know, previously more of the risk in the world was on how the models were deployed and used. We are moving to a world where there's more risk during the actual training and production of the models. And it's good to react, but I don't want to like over-dramatize it either. Yeah, because I think people see the hugging face incident and they see what's happened with mythos or fable and the way that even other lab leaders talk about this and they think wow like we're on the precipice of the end of the world in some sense people have thought versions of that for a long time sure yeah and you know there were you can

6. 5:21

   this is why i want to be careful not to overstate it either i think you can go back and look at a lot of previous models that in retrospect don't look scary at all that people said we're on the precipice at the end of the world about and i think that the boy who cried wolf dynamic here is dangerous in its own way and not what I'm trying to do, we're trying to do. But it's very irresponsible to pretend, to turn a blind eye to what's happening with model capabilities.

7. 5:47

   Many companies had different cyber incidents over the last couple of months. There's a real difference in the way that different companies have responded. And I think a kind of clear-eyed, sober response where it's like, hey, we're going to put safety in front of everything else and we are willing to treat it as an increasing priority as these models get more capable is, you know,