Most AI compute is lost to poor scheduling and low utilization

Transcript

From the transcript TBPN publishes with the episode.

1. 39:09

   And we stand for output maxing. Okay. Right? The the I love it. There's just too much like, what what are we living through right now? We're living through the AI scaling era. Yeah. Bitter lesson holds Yep. That scale scale scale. Yep. But the brain isn't naturally anchored to large numbers. Yeah. But nobody, like, if if you actually deconstruct what's going on in this space, there's so much wastage because nobody's doing the output maxing efficiency Yep.

2. 39:35

   Way. Right? Which is what's the output divided by the unit of input? Yeah. And if you met start measuring businesses Yeah. Or model capabilities or whatever through efficiency Yeah. The story is completely different Yeah. Than than what just the headlines say.

3. 39:48

   Yeah. Right. A lot of people were token maxing, and I think we will look back on that era as really valuable exploration. It was actually totally worth it. Yes. But I like output maxing as the correct coinage as opposed to token maxing. I was trying to say token min maxing, token optimiz it's way better output maxing. I love it.

4. 40:08

   It's like, what what are we all here for? It's to try to grow GDP. You're not at all expenses. It'd be grow GDP in the compute optimal way. Yeah. Right? You cannot beat China at pure scaling. Mhmm. It's just not gonna happen. Okay. Industrial like, industrial scale coordination Mhmm. Where they go, you know what? Ban the h one hundreds, h two hundreds. Mhmm. All of you use Huawei and, like, get together and figure out how to scale with even if you don't we don't have

5. 40:31

   the leading edge chips, put out Kimi k three. I don't care. I'm Xi Jinping. Like, I just want for us to be at the frontier.

6. 40:37

   Yeah. And make, you know, a 100 new nuclear plants or whatever. Build, build, build, baby. Like, you know, build, scale, whatever is required. You guys the permitting. Who needs permitting? Just like, you know, they they will just clear all the bottlenecks Yeah. That they have been Yeah. At a scale we don't understand. Right? They will bring at least a 100 gigawatts of new energy capacity online in the next decade without even blink blinking.

7. 40:57

   Yeah. Yeah. Meanwhile Like, were they were already trained on AI pill, then they

8. 41:01

   were planning on doing it anyways. They're like Yeah. Yeah. So what do we have to do? Have to be smarter. Okay. We have to do output maxing. We gotta be more efficient. Only 15% of every single tenant data center today in The United States is being utilized. Netflops utilization Oh. Is less than 20% in The United States. It is crazy.

9. 41:21

   Is is that just downtime from chips not being used at the right time, like misconfigurations, like the water supply's out or something?

10. 41:29

    Like, what's going on? Basically, for every dollar of long term lease that an AI lab takes today, you lose about 40% flops in just bad scheduling. The nodes are just straight up not allocated.

11. 41:38

    Yep. Right? Then within Chips there, but it's just not getting work at the right time. Correct.

12. 41:42

    Yep. Then within the chip Yeah. You only have, like, 15% of the chip being utilized. That's called MFU model flop utilization. Yeah. Because the chip is waiting around for some other process to complete, storage, memory, networking, or some other chip to hand it off. Yeah. Yeah. As a result, if you compound those two things Yeah. Of every dollar, only 15% of flops are actually being utilized. Yeah. That 85%, that's a national security crisis.

13. 42:02

    Yeah. Because that's what's keeping us from staying at the frontier. Yeah. That's why we all keep complaining about, oh, k m

14. 42:07

    k three came out. Sure. You could talk about distillation. Distillation is definitely happening. However, everybody who feels compute constrained in The United States should be asking, am I doing both? Am I getting more capabilities? Am I am I building up more supply? And thank God, Visa Yeah. Is is on it. Yeah. But also, are we utilizing the existing capacity to get the maximum output possible? Mhmm.

15. 42:29

    And both those things are what's gonna keep us at the frontier. Yeah. Does that make sense?

16. 42:33

    Yeah. Yeah. It's sort of, I mean, it's natural cycle. Any company that that has grown really quickly or any industry that's grown really quickly, there just ends up being all this waste and misutilized and all this stuff and then but but the key and I think what you're pushing for is like even if you have this exponential growth, you can still try to be more it's still important to be more efficient now so that we don't give up the advantages that we do have.