Prefill/decode disaggregation remains a case-by-case tool

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 21:51

   So this is like 100% software. I guess a lot of people have talked to me. You know, it feels like this DISAG, you know, splitting pre-fill and decode is very in the air these days, and a lot of people talking about that. Like, is that something you guys think about now, or at least how do you think about that kind of in the future? I think we are in the stage where everyone's talking about that, and no one's doing that properly.

2. 22:11

   Because, like, it is a complex problem to solve. And particularly in inference, you have a lot of other problems, like way lower hanging fruits that you can fix address and hit those SLAs. So we are quite cautious about that. We don't have it, for example, as a go-to solution for every customer, for every workload, because like, first, it doesn't make sense. Second, it's really hard to operate and work with, like provide proper SLAs around that is like proving quite difficult as of today.

3. 22:50

   We have some promising results in our like prototype services, but not as of today. At the same time, we have some like disaggregated setups for the most demanding customers in production right now. It's just more, you know, wide glove manual approach. Yeah. But especially for this newer architectures of like, you know, those gray black walls, whole node that has really fast interconnect, a lot of challenges that you've had before with disaggregated steps are kind of solved if you're able to fit in the footprint of a rack.

4. 23:25

   because, like, this throughput is so much faster, like, interconnect is so much faster, you kind of stop caring and that this is a way of just balancing the perfil and decode nodes to match the particular pattern that you see from the customer. What's your mental model for, like, the workloads that do fit in a rack? I don't know. It's a very case-by-case. Well, these are definitely significant workloads.

5. 23:48

   Like, we are talking, like, thousands of requests per second, right? We are talking like billions of tokens that flow through. But at the same time, because we are the cloud, can for example like share the rack between two workloads and not be that bothered about that right yeah so for example on the cloud side we for this particular product we only allow to consume it like at track level you cannot order two grace blackwell's uh notes it doesn't make sense yeah right like you'll have to allocate the whole rack and then like structure workloads within it but for example when you're a client of such a solution as we do with managed inference for example you can collocate workloads and you can find a way to do that so in a sense you don't need to fit anything in the rack it's just is there a workload that will benefit from this

6. 24:47

   disbalance like intentional disbalance of pre-filling decode nodes yeah and can you fit more than one into a single rack and how do you solve the packing problem? Yeah, that makes sense. You kind of mentioned there's like lower hanging fruit on the inference side than doing that maybe. What are some of the lower hanging fruits today that you still think we're not