Inference providers need a platform beyond token serving

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 21:24

   I'm like the eye of the hurricane of like a very exciting, large and competitive market. And so I'm curious as you think about like, you know, what ultimately will kind of differentiate players in this space and like, you know, and, and, and win out. I think there, there's so many true believers in the size of the overall inference market. And I think lots of folks are asking what actually wins out in the end of the day. And I know you have a good thesis on this. So I would love for you to share with our listeners. I mean, I think a lot of the inference market will be quite rough, right? Like I think the sort of straight down the middle, just like, you know, serving tokens, you already see sort of open router basically commoditizing the underlying providers. I tend to think, you know, like the real alpha or like the real like point of

2. 22:06

   differentiation like there will be a lot of differentiation just in like more efficient serving and but but it is a little bit of a race to the bottom and i think and i think there's going to be a lot of opportunities to sort of stay ahead of competition you're going to have to like put a lot of effort into that uh but i think more sort of durable differentiation will definitely be in like you know providing the entire platform of a lot of different things which is like i think something we are quite well positioned to do we have the sandboxes we have reinforcement learning a lot of customers doing also normal fine-tuning um a lot of customers obviously you know deploying other types of models we have batch processing i think i think my view in general is like customers they don't want to buy like single trick ponies like they want to

3. 22:46

   buy like a multi-product set of tools so that to me is like ultimately like probably the biggest source of differentiation is like you know how do you provide that that entire set of tools how do you provide this like very general purpose thing that lets engineers like iterate very fast deliver value and then like just a day-to-day like we're still gonna have to like go and do the knife fighting of like you know staying ahead of your competitors in terms of performance and use the latest kernels and the much more efficient kvcash and there's gonna be a lot of that stuff for a long time because there's so much money in it uh but i'm kind of pointing out i think that's sort of that's like less durable as an advantage i think that you know because over time like people these tricks will sort of spread and people figure out how to how to copy them so the more

4. 23:28

   durable thing, I think, is like being that sort of portfolio of a lot of stuff around inference, not just the inference in itself. You know, having all the models and a lot of, you know, stuff like around reinforcement learning, maybe, or other things. You know, the biggest modal customers are on multiple products. Maybe can you touch on a few of the more interesting ones? Yeah, we have some incredibly cool customers like Cognition is using us at large scale for both for inference and sandboxes, to do a lot of reinforcement learning on modal, these sandboxes for the rollouts.

5. 24:00

   We have customers like Ramp using us for background agents, for the internal coding agent that they actually launched as a standalone product today that they're selling. We have customers like Suno and Runway that use us for regenerative video and audio. Decagon is another customer that uses us for a lot of customer support stuff, voice and text. DoorDash uses us for a bunch of internal applications, mostly LMs, but also all kinds of other stuff.

6. 24:21

   We actually have a lot of customers in computational biotech using us for various types of drug discovery, using, you know, molecular simulations and sequence alignment and all kinds of stuff. We have a bunch of customers in robotics, One X and a few other ones using us to train robot models, like doing a lot of training, but also batch processing of video and things like that. But a lot of customers in the data generation space, so customers who are building environments and running benchmarks in modal, often using sandboxes.