How pooled compute lets startups and public users share GPU capacity

Transcript

From the transcript TBPN publishes with the episode.

1. 54:20

   I'm I'm interested to know about how do you think about the diversification, the spreading of risk across different customers at different levels. That seems like it's important to the philosophy here. Obviously, there's a number of labs and hyperscalers and neo clouds, but then there's also start ups. But how do you think about actually spreading out the long tail of demand across all the potential customers? Is there, like, a portfolio strategy This in

2. 54:49

   is a great question. So if you go to the website, nationalcompute.com, which is where the grid is hosted, you'll see it says the grid is now in public access. Public sector access. You know, we we started with a public mission Mhmm. Which means the the compute, the infrastructure that's being built out has to go on the grid to people who need it the most. Mhmm. Today, that means for us .users, ..gov users, .milusers, we're... As far as we're we're we're one of the only places Mhmm.

3. 55:17

   Where these folks can access live, like hundreds of chips on demand at subsidized rates. You can get an AMD chip there, an MI three fifty five for $3 an hour. And you can get a b b 200 from NVIDIA for about $4. The commercial pool we have, there's over 60 different commercial labs also on the grid. They pay a much higher price. So we've had several millions of dollars being spent on m I three fifty five at about $12 an hour as of yesterday. And so just to give you a sense of the spread, commercial customers do pay higher for on demand access, but public sector users get their compute at subsidized rates. And and the benefit of pooling is we get to essentially load balance across these different customer types and ensure folks who need access the most can get it at prices that don't sort of exclude them from the market. Does that make sense?

4. 56:08

   Yeah. Totally. Last question from my side for now. How... Like demand planning. It feels like the greatest challenge for application layer companies, even obviously the labs as well. But is this something that can be solved? Or like it feels like this is the solution where you just get the most number of chips in a single sort of platform all over the world. And because it just feels like it feels like demand planning for these companies is entirely unpredictable. You have like an instinct moment where you go from obscurity to, overwhelming demand.

5. 56:45

   Or you've seen labs release a new model and it's you get that new capability increase and suddenly you have customers that you weren't even imagining. And so this feels like the only solution, but what are you guys seeing the smartest companies do around around forecasting? Is it really just napkin math and sort of like guessing? Or or is this a new kind of core skill set?

6. 57:09

   I I think each company on their own is doing the napkin math and guessing. And they're looking at their neighbors. Like, they're looking at what is this guy doing? What is that guy doing? And trying to, like, extrapolate that. I think the beauty of what we're doing is is that our grid members are sharing their napkin math with us, so we get visibility across a larger number. And then we're the ones taking the bets on some of the unknowns. Right? Like like the instinct moment you mentioned, you know, we're gonna plan for that so that there is capacity for those that didn't know better than to plan. And the idea is that if you have enough users across it, some may plan too high, some may plan too low, but you should be able to sort of smooth out that demand curve and make sure that the people who need it most have access to GPUs at the time they need it.

7. 57:55

   And and like everybody else, we also use AI. So the first thing you'll see when you log in to nationalcompute.com is our agent called Marshall. Mhmm. Marshall is named after, you know, the the the general who helped unite The United States and Europe post World War two after the Marshall plan. Mhmm. And Marshall is an automated system that input ingests all of that data that Negin was talking about and then actually does demand forecasting for us. And every good member can go talk to Marshall, do... You can kind of think of Marshall as your chief AI compute officer.

8. 58:27

   And we're using reinforcement learning to improve it. And over time, I think this will really unblock a lot of bottlenecks for folks who don't have their own infrastructure teams, their own procurement teams. You know, it's not uncommon today for the the largest labs to have a 100 plus people doing this job. But I don't see why we can't relieve some of that stress through automation.

9. 58:46

   Fantastic. Thank you so much for coming on the way down.