OpenAI treats compute as a scarce capital allocation problem

Transcript

From the transcript Odd Lots publishes with the episode.

1. 30:20

   I'm going to move away from safety and alignment for a second, but you're the guy in charge of allocating compute at OpenAI, pretty much, right?

2. 30:29

   So actually, it's funny. I actually try to do as little of the compute allocation as possible. Oh, seriously? Wait, explain more. Well, okay. So I'm someone who has spent a lot of effort to produce the compute. So I spent a lot of effort on the data center side, on the machine learning engineering side, and on the product side, have to you know, ultimately be accountable for how we allocate compute.

3. 30:52

   But actually I try to set up the systems and the processes and then the actual, like a lot of the decisions on where research compute goes market or in Yakup run those a lot of the actual, you know, we basically make a big decision on the split of how much goes to applied versus research. And then within applied, we have some frameworks now where we kind of have buckets for different products. And I would just say that the choice of where compute goes is like, in some ways, the, hardest problem at OpenAI. It's like a capital allocation problem, but it's one that I actually think other people are, you know, I really try to have them, you know, leave it to them as much as possible.

4. 31:28

   Well, how do you think about it from a product standpoint? Because as you say, you can't, I mean, you can't really disaggregate the two too much, right? Ultimately, you have to make decisions about what you want to develop and promote. Yeah.

5. 31:39

   So I do think that you're values and prioritization shine through, through compute allocation, because the world that we're in is one where there just is not enough compute, right? Compute is revenue, right? It's also the production of models. And so you kind of have different timescales of return and you can think about doing more alignment work. You can do really any sort of task. It fundamentally comes down to compute. And there's many people at OpenAI who I would gladly like, well, let me, let me tell, let me tell a story. So, Leading up to the launch of GPT-5.5, we ran a process where we basically looked through all the different places that we had compute that we could potentially squeeze. And we were like, we could cut back on this rate limit. We could shut down this part of the product. We could do this,

6. 32:26

   do that. And we had this whole stack rank of levers because we knew we're going to launch GPT-5.5. People are going to love this model and we're going to run out of compute. And it was such a painful process because You're basically going through every single part of your product and being like, how valuable is this one? How much do we care about these users? And of course you care about all of them. Like we want to deliver value and we built this product for a reason and people are using it. And you're comparing the efficiency of costs to different things. And so what we've tried to do is move towards more of an objective standard and you're never going to be fully there. But there's a second piece of detail as well that's mattered a lot, which is trying to push a lot of the constraint down closer to the people who are in

7. 33:06

   the weeds on a particular product. And so you say, hey, here's your compute allocation. We're about to release another model, new modality, something like GPT Live, right? That's just like a totally different thing. And you need to fit in your existing budget. What do you do? And somehow people always find efficiencies. Sometimes people are like, you know what? Actually, there is this like, you know, kernel efficiency that we didn't roll out yet. Well, I guess we'll roll it out now, right? Or like, oh, it turns out that, you know, there's a peak to trough between your day and night and we could actually pack this new thing on top of that. And so we actually don't need incremental.

8. 33:39

   And so I think that really unleashing the creativity and the co-design of the people who are doing the work is actually maybe the thing that I really focus on. So I really try not to be as much in the like sort of arbiter, like just making these like super high level decisions, but instead really focusing on how do we up-level execution and get the most efficiency down to the metal.