Training a frontier model means committing $100B before you know it works

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 51:39

   together at the very end and then result in these beautiful models. Yeah. How do you build good products when the feedback cycle is so long? Like ultimately you start planning the model in August and you don't know until more than a year later if it's good. And you know, Silicon Valley famously likes to run on metrics, but like ultimately do you just need someone who is the esthete and says, I'm sorry.

2. 52:01

   It's nice when you have metrics, but I think the like state of being an entrepreneur is like how well do you make decisions in an information-poor environment? But there's like that on steroids, right? Mega steroids. I think that this is, and I think Mark gets a huge amount of credit for this, because of how long everything takes, there were many points along over the past year since I've been here, but also, as you mentioned, long before that, where you have to make very large investments or make very big decisions about, you know, how you're going to invest or develop the technology or, you know, what decisions you're going to make long before there's proof points around, you know, whether or not even the thing is working.

3. 52:47

   And so I think this is, this is like, you know, in some sense it's like entrepreneurship on hard mode. It's like the opposite of lean startup. Like lean startup is like, yeah, you like see if you get something working and then you double down and then you see if that works and then you double down. Like you cannot do that in AI. You have to, like, we've been investing into these data centers since, you know, for years and years and years.

4. 53:07

   We've had to... And you'll be radically behind if you waste to see if things are working. So we have these conversations where it's like, at the end of last year, we're like, all right, we are quite excited about the progress. We haven't seen the models yet. Like, it's like, I'm going to like, you know, we're going to write this check for like, I don't know, like $100 billion or $200 billion. I'm like, Alex, you sure this is going to work?

5. 53:32

   It was so extreme that we had the whole exec team. I think Susan scared you a bit. Yeah, well, it's like the joke every time we talked about it would just be like, yeah, and we hope it's all going to work and be good. But there were literally, like the end of last year, other than me giving a vibe report or how our research was going and I could sort of show some plots and whatnot. But it was like, I mean, I was feeling good about it, but it was certainly not clear at all that it was going to work.

6. 54:03

   You guys are saying that the essence of training a frontier model is you have to push your chips all in before you've been dealt the cards. You just have to decide that you're going to develop a frontier model and you don't get to meter the investment. At this scale. I mean, back a few years ago, the clusters didn't have to be this big, but now they do. So I think this is part of the excitement of brooding up a frontier lab now, is basically the ante is extremely high.

7. 54:35

   And I kind of viscerally remember this. We were in Hawaii at the end of last year. It was kind of like the whole exec team got together. And yeah, Susan pulled me aside many times and was like, you know, I just want to tell you if this doesn't work, like, you know, this is going to be really, really bad. it's going to be very very bad in financial terms i really need to impress on you that this model has to work it has to it has to be good if it's good it's all good we're great but uh but if it's bad it's really really really bad and uh susan who you know that was useful motivation it was well it was like uh it was one of these very funny moments where i was like you know obviously the amounts of money are like so like obscenely large what are the amounts of money in training order of math, whatever you can share, but just like, how much of the cost?

8. 55:24

   Well, you can do the basic math. Yeah, no, I can. I mean, a gigawatt cluster, I mean, it's like, I don't know, I think what you'd see out there would be, you know, to operate a gigawatt a year, call it on the order of $15 billion. Okay, so it's roughly a $15 billion effort to train a front-to-your-money. To operate a gigawatt cluster for a year, if you're talking about building the CapEx, it's probably closer to, you know, I mean, you pay for it up front, so it's probably, you know, $50 or $60 billion.