First-principles reasoning can expose hidden cost reductions

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 25:56

   Well, the tools of physics are incredibly helpful to understand and make progress in any field. And first principles obviously just means, you know, break things down to the fundamental axiomatic elements that are most likely to be true, and then reason up from there as cautiously as possible, as opposed to reasoning by analysis or metaphor. And then it just simple things like like thinking in the limit, like if you extrapolate, you know, minimize this thing or maximize that thing, thinking in the limit is very, very helpful.

2. 26:32

   I'd use all the tools of physics. They apply to any field. This is like a superpower, actually. So you can take, say, take, for example, like rockets. You can say, well, how much should a rocket cost? The typical approach that people would take to how much rockets should cost is they would look historically at what the cost of rockets are and assume that any new rocket must be somewhat similar to the prior cost of rockets.

3. 27:01

   A first principles approach would be you look at the materials that the rocket is comprised of. So if that's aluminum, copper, carbon fiber, steel, whatever the case may be, and say, how much does that rocket weigh and what are the constituent elements and how much do they weigh? What is the material price per kilogram of those constituent elements? And that sets the actual floor on what a rocket can cost.

4. 27:27

   It can asymptotically approach the cost of the raw materials. And then you realize, oh, actually, a rocket, the raw materials of a rocket are only maybe one or two percent of the historical cost of a rocket. So the manufacturing must necessarily be very inefficient if the raw material cost is only one or two percent. That would be a first principles analysis of the potential for cost optimization of a rocket.

5. 27:55

   And that's before you get to reusability. You know, to give an AI sort of AI. Example, I guess. Last year, for XCI, when we were trying to build a training supercluster, we went to the various suppliers to ask, this is beginning of last year, that we needed 100,000 H100s to be able to train coherently, and their estimates for how long it would take to complete that were 18 to 24 months. it's like well we need to get that done in six months so then um or we won't be competitive so so then uh if you break that down what well what are the things you need well you need a building you need power you need cooling um we didn't have enough time to build a building from scratch so we've had to find an existing building so we found a factory that was no longer in use in Memphis that used to build electrolytes products, but then the input power was 15

6. 28:57

   megawatts and we needed 150 megawatts. So we rented generators and had generators on one side of the building, and then we have to have cooling. So we rented about a quarter of the mobile cooling capacity of the U.S. and put the chillers on the other side of the building. That didn't fully solve the problem because the power variations during training are very big. So you can have power can drop by 50% in 100 milliseconds, which the generators can't keep up with.

7. 29:26

   So then we added Tesla Megapacks and modified the software in the Megapacks to be able to smooth out the power variation during the training run. And then there were a bunch of networking challenges. The networking cables, if you're trying to make 100,000 GPUs train coherently, are very, very challenging. Almost it sounds like almost any of those things you mentioned, I could imagine someone telling you very directly, no, you can't have that. You can't have that power. You can't have this. And it sounds like one of the salient pieces of first principles thinking is actually, let's ask why. Let's figure that out. And actually, let's challenge the person across the table. And if I don't get an answer that I feel good about, I'm going to not allow that to be, I'm not going to let that know to stand.

8. 30:18

   And is that, I mean, that feels like something that, you know, everyone, if someone were to try to do what you're doing in hardware, hardware seems to uniquely need this. In software, we have lots of, you know, fluff and things that, you know, it's like we can add more CPUs to that. It'll be fine. But in hardware, it's just not going to work. I think these general principles of first principle thinking apply to software and hardware, apply to anything really.

9. 30:43

   i'm just using kind of a hardware example um of of how we were told something is impossible but once we broke it down into the constituent elements of we need a building we need power we need cooling we need uh we need power smoothing and then and then we could solve those constituent elements um but it was and then we and then we just ran the the networking operation to to do the cabling everything um in four shifts 24 7 and and i was like sleeping in the data center and also doing cabling myself um and and there were a lot of other issues to solve um you know nobody had done a training run with a hundred thousand um h100 training coherently last year maybe it's been done this year i don't know but um and then we ended up doubling that uh to 200 And so now we've got 150,000 H100s, 50K H200s, and 30K GB200s in the Memphis Training Center.

10. 31:46

    And we're about to bring 110,000 GB200s online at a second data center also in the Memphis area.