Agent growth could make compute the next infrastructure bottleneck

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 55:01

   So one of the things you've said is there's going to be a thousand X more software in five years. Maybe more, actually. Maybe even you're understanding. Well, I think you're going to have, you know, a hundred times more people writing a hundred times more software. So you have 10,000 times more software. So what breaks in this world in the next five years? Like what breaks first as the world goes here?

2. 55:18

   Yeah, I mean, I think just like compute generally breaks. I think you're already sort of seeing that. Or like I have friends who are just like, oh, yeah, I'm running scraping jobs to find like available EC2 instances for like going and getting my stuff deployed, right? It's not just your one prepper friend or whatever. This is like many, many friends who are just like, oh yeah, like we're running out of compute over here.

3. 55:38

   So I think that that's like a thing that ultimately ends up breaking. Do you think the world, the tech world has made enough investments that it'll be fine in three years? Or you think that we, everybody talks about the compute shortage is coming, coming, coming, but when does it actually hit? Well, we'll have to go in and see. I guess what I'll say is, I think people have invested a lot in the first order effects of AI, but have not invested a lot in the second order effects of AI, right?

4. 56:03

   If code is free and it can, you can generate, you know, tons and tons, like we have so much tokens, so many, so much inference, like coming online, what do you use tokens and inference to go in and generate? Code. Code has to run somewhere. I still think that we are undervaluing just how much that inference is going to come online, how fast these new models are, right? Like you will get to a point and you probably should, you know, arguably be now here now, if you're on the frontier, where you're not looking at most of these things, you're not even like looking and structuring the jobs that these agents are going and fanning off, you're basically just giving them carte blanche to come back with some sort of structure. And then you are beating into shape. And you are setting up, you know, automations to

5. 56:44

   go and receive events that say, okay, every single time we get a ticket or a customer support or thing or whatever, see if you can fix it, if you can go in and do those things, right. And the amount of compute that you need to go in and do that is so, so, so, so high. And so I think we're going to have like we have an inference kind of crunch now and then we'll broad strokes have a compute crunch right right and like i mean what happens when meta with like three billion daily active users like gets really really really good at ai right like what happens when elon who has never come into a market first or lost it because he comes from behind and just dominates this thing what happens when when like these models get really really really good and we go from those like two frontier labs to three frontier labs

6. 57:25

   to like three frontier labs plus China to three frontier labs plus China plus all the open source models to like meta, like all of that stuff coming online. You know, the GLM like 5.3 stuff with like, you know, base tandem fireworks, like running all of this inference, right? Like, yeah, I think it's going to be a very interesting 2027. So everybody talks about compute as a second order effect, but what else?

7. 57:47

   Like, are there other aspects that are going to just, we're going to hit a wall and suddenly you just can't use the CI that we've unleashed? Well, I think it's like permissions and like access and like key ring and stuff like that because you can't just like unleash these things. Like a lot of people are just like, okay, like run it in a sandbox with dangerously skip permissions. And like to some extent that sort of works, right?

8. 58:08

   But like you want to give these things wallets and you want to give these things permissions and you want to give them access to your, you know, like the whole like open claw thing. It was like, everybody was like, here, please take my iMessage, take my emails and all this other stuff, right? And they're just like shoveling all this important information, right? Like, how do you get to a point where you can partially dissolve that barrier to say like, okay, cool, you can have access to this section of emails or you can have access to $1,000 or anything else like that.

9. 58:30

   And you can watch Stripe already do this with like Link or anything else like that. I think Coinbase is doing some stuff with like Agente Crypto with like X402, you know, stuff, right? And so like these things are, again, just like going to go and occur, right? Are there things that agents need from infrastructure that people historically haven't? I think I'm actually more on the side of like the things they need are not new, but it's just like the loops are going to close a lot faster, right? And so internally, we say that software development has been solved, but software engineering has never been more important, right? And so things like, hey, listen, you know, I need you to build this feature. I need you to get it running in development. I need you to get it running in staging. I need you to get it running

10. 59:05

    in production under a feature flag, and I need you to go to roll that out, right? And then go and watch Datadog or observability dashboards or anything to make sure you didn't blow up, you know, this thing, and then listen to customer support tickets, right? You can automate that entire life cycle. It's just completely automatable right now. And we've automated a lot of it internally. And then we'll go in and offer that to like folks. Right. And so I think that like those things, you know, of the new things, there's probably fewer of the new things.

11. 59:30

    They'll probably be very, very impactful. There's just so much low hanging fruit right now that you can basically just say like, well, what was I doing manually and how can I never do that manually again in my entire life? And I think that's what a lot of the kind of folks over at Anthropic are talking about in terms of like loop engineering, like graphs and like all these other things.