Inference providers can shift capacity as model demand moves

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:21:37

   There's an interesting structural reason why inference providers should exist separately from model builders. The best model keeps changing a lot. Inference providers keep, can easily reallocate resources between models since overall demand is quite smooth. So, I mean, if you think about all the Chinese labs and, you know, in the future, hopefully more U.S. labs too. every time a new lab releases, you know, the latest and greatest open source models, like you see big shifts in demand, you know, for these models.

2. 1:22:02

   And so I think a challenge with, you know, if all these labs independently like held capacity, they would see when they launch a new model, demand would go up a lot. And then another lab launches a better model and then the demand goes down, you know, but of course, like in the background, they're working on the next version. So like, I think, you know, there's like a structural reason, like I said, like why you're going to start, you're seeing demand kind of shifting to like the latest and greatest Chinese lab. I mean, not like, it's not like, you know, this is like everything just moves overnight because, because a lot of models and prompts and stuff like, like a lot of things that get sort of, there's like a certain stickiness to it. But I do think, you know, there's an argument that it's very easy for modal to like, you know, right today we're

3. 1:22:43

   running, you know, X number of thousand GPUs for one model tomorrow. We need to shift like, you 50% of that to another model, sure. That's just like, that's what the system is built to handle. No one even needs to tell us to do that. It just magically does that. So you agree that basically the LLMs are commodities? And if you agree, then where do you think the business for the big labs is going to be?

4. 1:23:05

   I'm not sure if that's necessarily true, right? Like, is Taylor Swift the commodity? Like, she releases a big album and goes on a tour and makes a lot of money. And then, you know, then there's like a down year when she goes and like records, I don't know if this is the best analogy. And then she releases a new album, you know. So I don't know, like she still has like a lot of brand and there's a lot of, you know, power and like makes a lot of margin.

5. 1:23:25

   So like, as long as the labs like repeatedly can launch new models, and as long as the margin that they have on those models are such that they can sort of pay off, amortize like the cost of training those models, it could still be like a decent business, I think. So you don't think that the intelligence is like plateauing? No, I don't think so. I mean, I think it remains to be seen, right? Like models today are like very good at like doing things that can be trained very efficiently, you know, using reinforcement learning.

6. 1:23:56

   So we've seen that initially with, you know, coding and writing simple snippets. Over time, it's gotten really good at also writing production level code. It's gotten really good at math. It's gotten really good at cybersecurity. Like these are all like very quite verifiable things. I still think it remains to be seen, you know, are models going to be really good at whatever, cooking food or, I don't know, like things that have like more like diffused targets and maybe longer feedback loops or whatever.

7. 1:24:24

   I think humans probably like, my feeling is like will be quite good and quite competitive with models for a long time in those domains. But, you know, certainly, you know, AI models, they're getting very, very good. Cool. As I'm sure everyone can agree with. Yeah. So obviously like the models are getting really good at coding. They're already really good at coding.