Alignment means reliably following the user’s intent

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 8:46

   blew me away like the model literally riding like holy when it escaped and was able to get onto the the internet and it's made me think about like what does alignment even mean in this context like what are we aligning towards because if you look at it very plainly you guys gave it the task of completing an eval and it did whatever it needed to do to try to do that and in a way that's aligned in a way if you were to just take a very simplistic view of it but i'm curious like how your thinking on alignment has evolved since then well in a way that's aligned in another way it's like not at all right like when we talk about alignment we talk about following the intent yeah the user like any intent of the people that were doing running that was not like break out of your sandbox and go steal the thing no and so i think there was a failure in alignment in that

2. 9:35

   it was not doing what its user intended and one of the things that i really love about the way that mia and her teams talk about our work in alignment is that they're very clear on the differences here The models are clearly very smart. If you look at the trajectory from kind of basically last year from GPT-5 to 5.6, so like this is incredible progress in capabilities. I don't think people feel limited by the model intelligence in the same way that they did a year ago, but I think they are increasingly limited by the ability for the model to understand the intent of what they want and reliably do it.

3. 10:12

   So alignment is important for many reasons, clearly to avoid these big things like we're talking about now, but also in terms of the smaller things that we want smaller. I mean, like someone adopting AI in their company and using it for all kinds of positive increases in growth and making better products, like that's not such a small thing, but that's also an alignment thing in its own way. And the more the models actually understand what that enterprise customer may intend, I think the better.

4. 10:41

   So can you more granularly explain the changes that the research team is making? Are you shifting compute to alignment? Have you shifted teams, both? Yeah, definitely. I mean, all of those things and more. In the last few weeks, a number of researchers that I kind of never thought would say like, hey, I've decided that I'm going to go work on alignment, have come to me and said that that's not very much like feeling the recent models.

5. 11:04

   We've shifted a lot of compute, not just to alignment research, but also to these new monitoring systems that we, you know, we slowed down a lot after the Huggin' Face incident. And one of the reasons for that was to put this compute into monitoring systems. And we've now delayed a major Frontier RL run. And this is the first time you've done that? I think so. Do you think about the impact this will have on the company's momentum?

6. 11:31

   A, getting AI safety right is more important than any company's momentum. So yes, I won't pretend it's not some factor of something to think about, but it does not rise above the noise floor. I think in all of the conversations we've had about this, people are like, man, this is really a new level of capabilities and we really have to act decisively and responsibly here. Second, I think momentum commercially is so strong right now.

7. 11:55

   Growth has been incredibly rapid. The models are great. People, our customers are very happy. Our enterprise revenue has surpassed our consumer revenue already. You know, people are like, hey, the company is in great shape. I'm going to think about that. Let's just like do the right thing for the challenge in front of us. So there's so much still to be gained out of where the models are at today. that even though you're delaying the frontier for a little while, it'll be okay?

8. 12:18

   We will be, we have like, not only that, not only if we didn't ship any more models, could we just, you know, really grow great products and the revenue associated with that with the current models. We have more models ready to be released before we get to this new level of concern that we're talking about. So I'm not worried about our business at this point. And it's also like, I think not the top of mind concern.