Most workloads should use the cheapest GPU that fits

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 16:07

   You were talking about heterogeneous compute and the different places where you're getting compute from and also how the inference stack is changing. How important are the long tail of chips to you guys, the non-NVIDIA chips and things like that, as you think about your supply or what your serving stack looks like over time? I mean, I think NVIDIA is still like the king here. Like, they have the best chips.

2. 16:33

   There's way more demand for NVIDIA versus anything else. Like, you know, we're curious about some of the other chips. Like, you know, we made some experiments, like taking a look at it. But NVIDIA definitely has the best chips. But, you know, like looking at AMD, Terranium, TPUs, et cetera, like starting to be like pretty good. So we're always curious about that. And then, of course, there's a lot of like older chips too.

3. 16:54

   And I think the challenge has been, like, for a lot of the LMs, like, you're kind of constantly pushing, like, you know, how many parameters can you fit? So typically, the latest and greatest LMs, like, they will always require, you know, a full node, maybe even in the future, even multiple nodes, who knows? So, like, eight B300s, like, they will require the latest chip and, like, eight of them, because that's how many you can fit on a node.

4. 17:17

   but there's plenty of models outside of that like some older LMs smaller ones that have they're optimized for performance might be fewer parameters and then outside language there's a lot of stuff around like voice and audio transcription or speech synthesis or like pharma like bio stuff a lot of that stuff actually they don't require the latest and greatest GPUs they actually get more cost efficient serving on hoppers or even amperors or like you know different types of like l40x or rtx pro 6000 stuff like that so we actually i mean we are ramping up like very very aggressively on black walls uh primarily gb's now uh but we see actually like plenty of demand also in the older chips uh which i think is quite exciting there's just so much stuff going on people forget about the hoppers but like they're they are our most common type right now still at modal h100s

5. 18:14

   we run some very like large production use cases based on hoppers how efficient like is model to chip matching today like you know in the industry at large do you feel like most people are using the like optimal you know chip or you know from a price perspective for what they're what they're working or is there still a lot of room for improvement there there's a little bit of room to improve i mean one of the things that we want to do over time is like kind of automate that for people i think in practice it's like less of a problem than it might seem like because often like you're kind of memory constrained so like knowing how much memory you need for a model like you basically pick the cheapest gpu that fits the model that's usually like the heuristic that works in almost all cases so if you have you know if your model is 80 gigs like you can either fit

6. 18:56

   it on a hopper or an older a100 80 gig i guess in that case it depends a little bit on the flops um but but in many cases it's like whatever whatever gpu like is the smallest that fits your model i think over time like there's a bunch of stuff we've been thinking about that where basically you just deploy something in modal and then we manage different chips and maybe automatically give you discounts because the more options we can have to run it on a lot of different chips, that lets us aggregate capacity in a better way and run things at higher utilization because then we can switch between different GPU types as there's underutilization on those.

7. 19:32

   So that's something actually we're quite interested in doing over the next six months is giving people an option to deploy things in a way where we automatically switch types underneath in a way that's still consistent with your dollar per request or dollar per token or whatever. I think there's some incredible cool stuff you can do there. It's super interesting and one thing I imagine is so interesting in your role too is like you're obviously