A 50% drop in inference costs can drive tenfold usage

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 17:02

   But I'm curious about the training versus inference workloads you're seeing, because you have such great visibility. Do you expect training to be a smaller piece of the pie going forward? And it feels early on inference, but I think everyone's trying to figure out how big is inference going to actually be? I mean, you see stats like 2% of Americans have even paid for AI or, you know, companies are barely using it.

2. 17:24

   And yet inference is exploding. It's like, where does that tap out? And then what does that mean for the training piece of the pie? Yeah. I think, by the way, that stat just seems a little bit misleading. It may be the 2% of the people have paid directly for model usage. But I bet a higher percentage of people like have a Netflix account. Sure. And it recommends a movie to them. Sure. So they actually have used it.

3. 17:44

   So it's like, it is like it's yes and no. Like, it's like how many, what percentage of, you know, Americans have paid for a database? Like probably a really small percentage, but I bet they all like use it on a daily basis anytime they log in somewhere. And so, you know, I think that's a misleading stat into how integrated it is into every single application. People are going to the Amazon website every single day. And when they search for a product, AI is kind of embedded in how they're searching.

4. 18:07

   Sure. They don't pay for it, but they kind of paid for it. And so from that perspective, I think I think inference is a huge business. It's massive. And you've seen in some of the anthropic filings kind of what their revenue looks like. So it's a very good and large business. I think training is a good business too. It's just very concentrated. And so we'll continue to support training clusters. We do support training clusters today.

5. 18:29

   But it's the minority of our business for the most part just because the main source of our business we want to do is support the millions of customers we're building and growing on top of us. How early is inference, do you think? Super early. I think you're almost doing a disservice to say that it's at the 1% of what it's going to be. Not even 1%. I don't think it's even 1%. I think there is a massive runway.

6. 18:53

   And look, every time we see inference get 50% cheaper, we see an order of magnitude more usage. And so people kind of wring their hands. They're like, look how cheap it's getting. And yet the revenue is higher every single year. I think that we're still very early on that path. and it's why we also feel good about the investment we're making with regards to infrastructure, because I just think there is so much more that can and will be done.

7. 19:15

   Do you think about that spend that's driving that inference? Is it coming out of existing budgets, or is it net new? Like, where is that capital? I mean, if you're talking like potentially trillions of dollars in inference that's happening, I'm just like, where is that coming from? Yeah. I think it's all those things. I think that sometimes it's creating new value. Sometimes it's coming out of other budgets.

8. 19:36

   And I do think that there's no one answer to that. I think it's a lot of those things. Sometimes it's allowing you to create products and services and capabilities much faster than you could before, which drives revenue. We'll call it like new sources of revenue. Sometimes it's driving efficiencies where it's supplementing. I don't need to necessarily hire a new person. I can get twice as much work done with the person that I, the thing that I have.

9. 20:01

   And I think both of those things are true.