Why Sam Altman is against a single superintelligence

like a full 3D video game like unbelievable anything you can think of in 10 minutes that reminds me of the early internet including just like these people they're never going to be mind blown by that because that's always what they expect out of the world like any any ideas they can come up with anything they want to create anything that they think might be useful anything that they just need to make their lives better like the computer is supposed to make it there was a crazy moment at OpenAI where we had a sev and codex went down. And everyone had forgotten how to code. I mean, not all the way, but it was like a, whoa, what if you couldn't?

I remember when I was a student and I would meet the really old guys and they would once in a while pull out their gigantic Xerox Alto disc or they would pull out their line by line written of assembly or they would pull out their punch cards. And I felt like it is incredible to me that you all bootstrapped us to this point doing it this way. Yes, no, I agree. and keeping programs in your head, right? You know, programs running the first time after, you know, hundreds, thousands of lines were written. No, there's a few lost arts like that.

How quickly are people making progress with CyberBlue? I don't want to declare victory at all on the PHP thing, because I still don't think we know.

And I assume some bad things are going to happen. Relative to my worst fears, if I can, like, think back 10 days ago, the first day of that went much better than I expected. And we... Heard a bunch of stories from people that were using our models to, I mean, not only build patches and build cyber defense, but like individual companies who were just like, I need to just do a quick scan and appointment. Like, I'm going to take my thing down for an hour. I'm going to use this thing and I want to put it back up because my application is offline. Now I feel like I really jinxed it and I shouldn't have said that.

But the first day did go a lot better than I thought. And I think people have been able to use that. the the thing that i've heard a lot of cyber experts talk about is they just no longer are confident that we have the right assumptions baked into our threat model and they're like look if if everything is what we think and hardware exploits are as hard as we think they are then yes we can see how this is going to work but if ai is going to discover really deep novel bugs in hardware that require physics we didn't understand, but all of a sudden means some assumption doesn't hold, then I think people aren't sure what happens.

One thing that always strikes me is that when I talk to very knowledgeable people inside OpenAI and similar companies, there is a pessimism that sandboxing is possible, basically. And that would not have been my mental model going in. It would have been this, yeah, you have to build a really tight sandbox, and we haven't kind of hardened them historically enough compared to the current capabilities. But ultimately, you can have a maximum security prison for the model to work in and keep it inside there. It seems like you guys have kind of a pessimism that sandboxing models is possible. My instinct, well, first of all, I think we knew much better than we're doing now, and we should.

I think we clearly that is the main thing to focus on right now is a better security model if you imagine a theoretical AI that is like a million times smarter than the smartest security researcher would you like really be willing to trust that sandbox especially if the model can like discover new science that people don't already don't yet understand you can use the models to construct the sandboxes I think you can well so this gets let me come back to that I think sandboxing works to a point um then i think monitoring works to a further point where we can look at what the models are doing not only in their chain of thought but we can do activation monitoring and try to really understand like are the models are we happy the models are not deceiving us and then eventually i think you need to believe with huge confidence that the models are truly aligned and are going to value and try faithfully to do to follow their actual the intent of the people operating them and that they're going to value sort of people being in control of the future forever and if you had a model that you could fully trust the alignment that you were absolutely certain of the alignment of then yes i think you can have that to help you build better and better sandboxes

but like if you really let yourself project forward into the like what does it mean to have something a million times smarter than we are i i i don't know like i think that just gets strange because the alignment becomes hard because you need to really really trust the alignment to trust that the sandbox is staying ahead of capabilities can you not give models

And so capital allocation at OpenAI is sort of compute allocation. And it's hard to a priori know how much it will cost to go do something. It's actually also hard to just like control the, like you can't quite prompt an agent where it's like, go do a really good job of this and don't spend more than $10,000.

You can. Oh, okay. Yeah, if you say do the best job you can with $10,000 of compute, like, yes, it will understand that. I mean, there's another kind of hard question for society about what does it mean when you have two companies competing and one says you can spend $10,000 and one says you can spend $100,000 on the same problem. Like, it's clear what's going to happen there, but there's like a question about how society thinks about compute allocation and wealth and things like that. But we have our own version of this inside of OpenAI. We allocate, you know, a lot of compute to internal productivity, and then people get very wide latitude to make their own decisions unless they're really being abusive.

But do you just encourage people to, like, do useful stuff, and if they spend $10 million in compute, it's like, oh, interesting. Correct. We try never to reward people for using a lot of compute. Sure. Like, no token leaderboards. It's actually slightly bad to be high on the token leaderboard, certainly not a promotion. but it's like it's a skeptical default and if you can show that you're doing something important you will have nothing but appreciation. But at a certain point this will need to change because you're describing the increasing orders of magnitude and right now you have a researcher who does some very interesting math for $10 million and you're like a little pricey but Cool math and then a hundred million dollars and you know when they're doing the billion-dollar math you're like look I'm really happy that you're making so much math progress, but I really need you to cool the jets with the math Um, we did let it get out of control for like one week on the math intentionally So because you have to be like well, you don't know how You don't know how far it's gonna go until you try. Yeah

we talked about this a little bit earlier but i i want to like emphasize it again i i'll just call it swarm since that seems like the term right now i i do think the properties of swarms match a lot of the properties of the resilience of society or the robustness of society that that i kind of

i think we're just missing from the like single rollout single-brained version of the world um like for all of the deep flaws of society it is unbelievably robust and this would not happen without like a lot of people keeping each other in check and collaborating and kind of these values of pluralism and democracy and i had been sort of sad that that was gonna trend away in ai and now i like i'm excited about this as a director You're saying basically it shouldn't lead in a centralized direction because you will have many agents and agents pushing against each other in a dynamic equilibrium and, you know, offense, defense is a good example of this.

I would much rather intuitively have a trillion AIs of like slightly superhuman intelligence join society than one AI of one trillion IQ join society. Yeah. And you think we're headed in that direction of the multiple? All of these things are things we have to push for. Like, all of these things are like, we have this incredible technology. It can be pointed and deployed in many ways. I think there are a lot of people who may have a different opinion about how it should be used and what is the best version of the world. But this is the one that I think is right. Let's talk about starting a business.

Thanks to coding models, your weekend side hustle can quickly become a serious undertaking. And we've been busy here at Stripe making that process as simple as possible. Stripe Atlas helps you with everything you need to start a business, like company formation, tax IDs, founder equity, and all the other bits that usually require someone in a suit. So if you've got a company in the making, take a look at stripe.com. How's the compute shortage going? i mean there are all these ways where i think things at various times along the curve are going to get somewhat out of hand and there will be moment like with any new thing of this importance there are going to be moments of irrational exuberance there are going to be moments of bust there can be moments of constraint and glut and up and down but i think the trend is toward a, we always just want more compute.

Like, the demand for sufficiently capable models at a sufficiently low price, I think that trend only goes one way. And we're doing a lot of things to try to be prepared for this. We made a chip that is an incredible performance at, like, you know, units of intelligence, performance units of intelligence per watt. We are learning how to build our own data centers. We have a robotics program that we're going to point at making lots of data centers. People talk a lot about... The real RSI. Yeah, people talk a lot about RSI as like AI that does AI research. There's very little, relatively little talk about like self-replicating data centers.

You know, this idea that you can use AI to control robots and run fabs and build turbines or solar panels and, you know, build data centers. But I think we are, I think we have that like within grasp. And so we'll try to make a lot of really cheap compute. But aren't data centers already not labor bottlenecked? They're permitting bottlenecked and component bottlenecked. We can make more components of the robots. Eventually. Yeah. All of this is how we solve this over a long period of time. My observation to compute stuff is they say the solution

batteries concrete sorry the market is responding there in in high prices yeah i'm seeing where are you seeing any interesting market responses in terms of high prices pulling in innovation components for some components for robots and not like the humanoid like you know kind of like fake demos but like the industrial robots that can just make stuff i see there's new actuator makers and stuff yeah honestly i mean maybe you could say if open ice sells our chip maybe that That would be an example.

Okay, yeah, I want to get to your chip because that was going to be my example. I think the solution to high prices and short, you know, limited demand, or sorry, limited supply, excuse me, is you make your own chip. Presumably you're using a lot of AI to make the chip. I would have thought that's hard because I would have thought that chip design is one of those places limited by the public tokens you can get where there aren't actually that many blog posts on the public internet on how to build a, you know, leading edge two nanometer chip? Well, in terms of like how to do the lithography, yeah, I think there's like one company on earth that seems to know how to do that.

In terms of how to do the chip design, like to build the file that you eventually send it tape out, that is well understood. And that is a great problem for AI. And given that the world is going to be like limited in its silicon capacity, we all, we have like a shared interest. One of the things that I'm proud of about OpenAI, and maybe it sounds obvious, but it was maybe not as obvious at the time, is we have a culture where people, even if they've done something the old-fashioned way for a long time, are super willing to try using AI to do it better. What are other examples of that? AI research, software engineering, but also increasingly like procurement or things that

for AI use in enterprises generally is how we don't have hard budgets for individual engineers.

You don't get cut off at a certain point. It's like, sorry, you've spent all your arcade tokens. Maybe we should, but we don't. And again, I think we all believe in the scale up of AI. But if you take that seriously, it means the dollars of compute or AI per person will go up. And so I feel like we're pretty soon to now actually just running into this question of how you actually decide how much spending power to give individual people inside the organization. There was a time when a lot of companies had this like a thing in their Google Calendar which showed the like approximate cost of each meeting in terms of like everybody's comp that had to be there and I think that was always still an underestimate of it and there was an observation that like Paul Graham made it at one point you on many other companies too, which is that there was like, there are people at companies that have incredible power to save like, you know, $17 on the snack budget of the office every week.

And yeah, you can have these meetings for like unbelievable amounts of money that are adding no value. kind of like no equivalent of the ability to stop that i think that most use okay so i'm like really talking my book here and i'm very biased but i genuinely believe it that use of ai is like not a company's biggest spend problem relative to other things that happen and even if it's like somewhat inefficient it is not where i would focus well but hang on that's not scaling law pilled enough, like, I'm not, okay, two years from now. You want to go send all of Stripe's revenue, all of Stripe's profit on OpenAI services and then complain about this?

I'd be delighted to have a conversation. But we're on a path to that. Great. Yeah. We now spend, you know, 1% a shorter magnitude of Stripe's OPEX is tokens. That is obviously... That feels too low. Yeah. So we're working on fixing it. And it'll get to 5% and 10% and 20%. And then you have a real question. Do you have an intuition on what the right number is? Like what it should be today. What it should be today is maybe tricky, but steady state, isn't it going to be, depending on the kind of company, isn't it going to be tens of percentage points of their OPEX? I think for many companies, the answer to that is clearly yes.

And I think there will come a time when the right thing to do is to really optimize that. And also, like, another thing I'm proud of about OpenAI is we've tried to really drive the cost per unit of intelligence down and down and down. I think we've been, like, a good spokesperson and good champion of this. But still, I think the problem, even at current costs,

Personal agents are the most exciting thing happening in AI right now. There's many exciting things happening in AI, but one of the most exciting things, You guys launched D.A.T.S. a few weeks back at the stage. And there's a bunch of other people also launching in the field. What will be true of the winning personal agent? It's like intelligence, context, tools. It's a limited number of things. And someone is going. I think no one has yet figured out the thing that people...

People like these, but this is not something that people are like, I cannot imagine my life without this thing in a way that when people talk about coding agents or even ChinaGPT, they do say. And someone, I think someone soon is going to figure out what that looks like. And you talked about one of the reasons why this doesn't quite work yet. But I think this is like a small number of ideas away from a super mainstream thing. Oh, interesting. You think we have not had the Macintosh moment for personal agents? I think we have not. Do you have any sense of what the shape of it will be? I think it'll look a lot like the current, you know, Dots or News or GrokBot or whatever, but it'll just work.

Like, it'll just work. It'll be reliable. You'll, like, never have—you'll just, like, give it more complex tasks and it'll do all of those things. Computer use dynamic is very interesting to me because not everyone is embracing the new agentic future with open arms, but it's not totally within their control because you can just have an agent using a computer. Where do you think that dynamic goes, this sort of cat and mouse? Like rough instinct is the world will, when people are going to fight this in both directions, but eventually converge on something like you're allowed to tell an AI to use your computer on your behalf, including software.

People always talk about that, But isn't it a bit like the semantic web? It's like a really nice conceptual idea. Yeah, I was going to say, I don't think this is so black and white. I don't think I have a super high degree of confidence of how this is going to play out. As a consumer, I feel somewhat annoyed if I can't have an AI automate tasks on my own computer. I want it to feel like it's my agent doing what I ask, but then I don't have to go do something like this. Yeah, I note a very interesting dynamic right now,

And I remember every startup was like, or many startups, like most of the YC batch was like, I'm going to build an iPhone app. This is like new distribution. This is awesome. I can do a new thing I couldn't do before. And many new startups got created. And all of the big companies felt like they had to make one, but they kind of didn't want to because I'm like, oh, I'm going to like hand my, you know, relationship with my consumer over to Apple. and it was great for most people.

It was just a new UI. Yeah. Okay, I'm going to go more rapid fire. People complain about bench maxing. Is that a real concern? Are OpenAI models bench maxed? Are you actually Rick Rubin-ing it where you just feel like the vibe of this model is right and you don't care what Sweebench gives it? Maybe there are two things we should really care about models being good at. One is scientific discovery and doing things that people cannot do. And the other is being enjoyable to work with. and i don't think we have great benchmarks for being enjoyable to work with i'm actually not sure we have great benchmarks for like the smart I'm not a scientist either, but we endeavor not to bench-max, and I keep thinking that bench-maxing is over as a phenomenon, and people keep doing it.

So are you better than other labs at not getting sucked in by the siren call of bench-maxing? I believe we are better at that. I don't think we've always been better at that in history. I believe we're better at that right now. It feels like the industry is still figuring out the being nice to work with, where there was the glazing era. And more recently, there was the stroppy, ornery era of models that were anti-glazing and as a result tried to disagree with you and nitpick at you on every minor thing. The current ones are pretty good, actually. It's gotten, I think the latest models are much better than we've had in a long time.

But it also feels like there's probably higher peaks to Ascent. Emmett Scheer said to me years ago that what he wanted was a really thoughtful and challenging but still respectful senior co-worker. And I have still not heard a better like brief encapsulation than that. Yes. After working with him for many years, what's your grand unified theory of Greg Brockman? I think Greg Brockman is like one of a kind in the best sense. He's singular on the Greg Brockman scale. You just kind of have to be there to understand what that is. But I think it's like a combination of like, yeah, intensity, like monster productivity, extreme dedication, and like focus on 16th admission.

It's very cool. You hate tokens as a metric for anything, really. Pricing or just kind of, it's not the standard unit of AI. And yet we found ourselves there. You know, we did talk. And you might say outcomes, but I don't think it can be outcomes. Because outcomes is too different across different use cases. It's what's the right standardized unit. We almost made a very, what would have been a bad call to say we're not going to price our next model in tokens at all. And you're just going to give it a prompt and we're going to give you a price back. And we're going to do it. And it's going to be like AWS.

You just do something and we send you a bill. But the customer response on that is so negative. They're like, I need to know what I'm paying for somehow. Yes. And I asked a few people, would you rather pay more for the same output but have it priced in some number of consistent tokens or something like even if we like inflated the tokens and i'd much rather inflate you inflate the tokens and pay more because i know there's like a real thing there um so i think you are right that people don't just want us to like give them a price because they like they feel like over the long term they're going to get screwed somehow um in the same way that maybe people would like rather pay for cpu hours rather than just like getting their thing done and even if it was less expensive you know the way like nanometer the chip is totally blank yeah the nanometers has very little relation to reality these days. Will we get something like that in tokens where you guys should start saying here is an intelligence equivalent token like, I can't believe it's not butter. Of like the smallest feature.

I, it's like made up, I hope not. I mean, it offends me that we like use the word nanometers for something that has nothing to do with nanometers. I know, it's very silly. You're not like that. I would rather not make up a fake unit. tokens are a kind of fake unit. I mean, they're real in some sense, but they're not a good unit. They're fake from a business value point of view. Isn't the only defense of tokens that if you're wasting tokens, we should fix that pretty soon? Like, we shouldn't be streaming. Claude Shannon would not approve of us just streaming noise rather than signal. And so there's a floor of the information theoretic most useful tokens on both sides.

And we'll kind of come back to that. Yeah. I mean, right now you have to think about whether you should want the $0.10 per million tokens out of a small model or $10 per token, tokens out of a big model, and then how long the chain of thought is likely to be. And that's just cognitive overhead that I don't think actually adds any value. Is the chain of thought included in the price? In what we charge? Most people do. Okay. By the way, the Guinness was not as bad. It definitely not as good as an IPA, but not as bad. It's good. Yeah. I mean, IPA is great, but it's pretty good. We like Guinness. Yeah.

At least in this podcast. Thank you.