AI capability has a scaling curve, but alignment does not

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 43:33

   we've made approximately zero progress towards understanding super alignment, like the genuine win challenge of controlling or directing or sort of steering an entity that is actually smarter than us in all ways, that is to us what a child is to a toddler or increase that gap however you want. An adult is to an aunt maybe. Yeah, I think we actually have exactly zero progress. We've made meaningful progress in that direction, simply because there are all these techniques that are framed as alignment or control techniques that work to varying degrees and fail to work demonstrably on today's AI systems, despite the fact that, by the way, these techniques are gating literally at this point, like hundreds of billions, almost trillions of dollars of capex.

2. 44:24

   So like you not being able to deploy your frontier model is costing you on a per week basis tens or hundreds of millions of dollars, right? So your incentive to develop alignment and control techniques that prevent it from doing exactly the hugging face thing are nominally insane. Like you really should be making progress in this direction. So and yet we have not made that progress. There's competitive industry competition is a huge part of the reason for that.

3. 44:51

   I'm also quite cynical about sort of like the extent to which Sam Altman buys into these threat models. But that's a whole separate thing. Even if there was the willingness to do it and the money backing it, I don't think we've seen from Anthropic the kind of progress that we'd need to see to be like, yep, we're on course to control even the sorts of agentic systems that we're about to build, let alone genuine superintelligence.

4. 45:18

   it's a different theoretical foundation you have to start from when you literally cannot understand the system that you are building. Yeah, I know. And I mean, it's accelerating. By Anthropics' own account, we are getting closer to so-called recursive self-improvement. The models are making a bigger contribution to each subsequent generation of model development. And as a result, as people may have noticed, models seem to be coming out faster. And like the idea that, well, we haven't gotten this alignment thing so far, but once things start moving unbelievably fast and new models are coming just all the time. And so the things we learned from the last models are irrelevant or of limited use faster than ever, then it'll work. I mean, no, right? That's crazy. And so we've been having a series of conversations with folks in the

5. 46:13

   in the frontier labs and by the way the fact that they talk to us itself is quite telling i mean so there are people from all the frontier labs um who who talk to us um a lot because they know that we're we work with you know elements of the u.s government that are interested in the stuff and that must understand it um but um but also a lot because sometimes you know it's it's good to have a shoulder to cry on and you know i i mean i um so you're sharing the these things with with permission, these particular anecdotes, but there are, within OpenAI, at the time of the hugging face thing, an awful lot of people saw it coming. They were looking around at like the security situation and the cultural situation at OpenAI, and they were not at all happy with it.

6. 46:58

   I mean, they were like, look, this is probably going to lead to some kind of incident. And their concerns were ignored by leadership at the most senior levels, which is one of the reasons why i'm i'm very skeptical of of sam um i don't feel like i have a good understanding of his motivations but i have just heard way too many anecdotes and every interaction i've had with him has been has left me feeling pretty nonplussed about his like his follow through like it it just it doesn't seem to add up um anyway so so there's an awareness that things are not proceeding in the way they need to to have a good outcome even for fairly typical prosaic ai that's being built right now. The alignment techniques are not catching up. The key is that they're not riding the same scaling curves, right? So we know how to make systems smarter. We dump more computing power

7. 47:49

   and more data into them. And yes, we add a bunch of algorithmic improvements and we get the AIs to help with that too. But fundamentally, it's more or less number go up, right? Like especially on the, what's called the pre-training side, the kind of earliest, actually it's the text autocomplete training side. There's really just this like one number called compute efficiency that everybody stares at. And you're basically just trying to make that number go up. And so it's a very automatable process. And so you have this great curve of intelligence that you can ride robustly. No one knows how to make a curve of control or a curve of alignment. So the result of that is that you're desperately trying to bolt on these like safety techniques that are very ad ad hoc. And if any one of like half a dozen people at OpenAI don't do their job on safety,

8. 48:35

   like tomorrow, things start going off the rails really quickly. Whereas the converse is true for capabilities. You could actually have a whole bunch of people who say, well, we've got to stop, we do things differently. And the machine will continue to hum along because of incentives and because of culture. And so sort of culturally, institutionally, I think this is like an underappreciated issue with, with all the labs actually. But opening eye, I think is kind of like the poster boy for this because we've had so many cases where, you know, Sam famously committed to 20% of their compute budget being spent on, uh, what was called the time super alignment.

9. 49:11

   So figuring out this, this, the answer to this question, how do you control a system that is vastly smarter than us? And he just didn't do it. Right. And then he lost his head of safety, uh, to, uh, to, uh, anthropic over that. And he's got, I mean, he's just churned through multiple safety, multiple policy teams. Can you imagine if like Los Alamos at the height of the Manhattan project was churning through literally like three different sets of safety and policy, not leads teams that just churn out? Like it would be a matter of such catastrophic national security concern that the lab leadership would have to be changed on the basis of that.

10. 49:47

    Anyway, that's that's my TED talk. I would. That was a good TED talk. And I'm going to add a little