Modal CEO on the Future of Inference, the Chaos of the GPU Market & Origins of the Sandbox Bet

unsolved problems in the inference world or in the inference stack? I think we're still pretty early with inference. I think, you know, a lot of stuff is like, it's like one of those like very empirical sciences. Like I think people are trying different things and seeing what works. I mean, And over time, like people have kind of figured out a lot of stuff, you know, take something like disaggregated pre-fill, right?

Like it's a very non-obvious idea. Like you sort of, you know, do the pre-fill on a set of nodes and then you copy the data over RDMA and do the decode and kind of swap. There's some like very non-obvious things, but like we've invested a lot in those types of things. The KV caching, there's a lot of complexity around like KV caching. Of course, like running more efficient kernels, that's something people do a lot. But there's a lot of these things that, you know, we have a team of, I think, 20, 30 people right now who are extremely focused on just like LM performance optimization. And, you know, it takes a lot of work.

You have to stay up with research. You have to read papers. You have to look at all the software coming out all the time and try and see what works, what doesn't work. It's still kind of an empirical science. Over time, I think, you know, patterns will emerge. I think also the models themselves will change very quickly. like now we have like mixture of experts and more complex things over time like who knows like maybe we'll swap out the transformers for something else and you know and that would change how like kv caching works like there's all these things like that could you know change overnight if someone figures out like a very different paradigm for all these things and maybe they will maybe they won't i don't know uh there's also obviously a lot of stuff around quantization and and other things um uh so so yeah it's it's it's a wild world out there but but yeah we're we're we're playing around with all these things. But it feels like there's so many folks that are working on, you know, different parts of improving inference. And obviously, what's I think been really interesting in the last two, three years is a lot of those optimizations and improvements have been in open source, right? And like, you know, for the broader community, maybe shared via research. As like a

CEO, how do you think about like, the level of what the kinds of problems you want your team to go solve versus, you know, it's like, hey, we're going to benefit from just the entire community, like pushing this space forward? Hi, I'm Anne Maris, and I'm the producer of Unsupervised learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership and see you next episode. As a CEO, how do you think about the level of the kinds of problems you want your team to go solve versus, you know, it's like, hey, we're going to benefit from just the entire community pushing this space forward? We actually love open source because I think a lot of our alpha is in the proprietary side, in the infrastructure layer, like all this cool stuff around autoscaling, for instance.

So I think on other inference providers, we've actually leaned very heavily into open source. We work very closely with both the VLM team and the SGLine team. We contribute very actively to both those repos and do a lot of optimizations with them. And so I think this may be a little bit of a different approach than maybe other inference providers. Of course, there's going to be certain things that you know we we keep internal but but almost everything we we push to to open source and i think it's you know it's because like you have so much alpha in the infrastructure anyway like we might as well just you know work with the amazing open source engines that are out there uh They weren't state-of-the-art a few years ago, but now they are, and that's an incredible thing that we'd love to leverage, frankly.

predicting for like what your compute fleet should be and obviously you see what the workloads are today and you probably have some sense of you know what models.

and you're also trying to predict, like, you know, hey, a year or two from now, like, what, you know, what kind of workloads will we have? I think there's lots of, you know, there's lots of different debates about whether we're going to, you know, most of the, you know, most tokens running will be really large models or actually, you know, we've saturated most tasks and these models will get smaller and smaller. Like, you know, as you think about kind of the evolution of your fleet in the next year or two, like, what, you know, what's your perspective on, like, where you'll see relatively more different kinds of workloads and less?

I think there's been kind of like basically over a trend of like, like larger and larger models over the last three, four years, I kind of, you know, I definitely feel like that trend is going to continue, like we're going to need, like, if B300s had more memory, I'm sure, you know, a Kimi K3 would be bigger. people are kind of like you know training the models in a way that they fit on these these chips so i i think on the at least on the lm side like on the you know lms are in a way like kind of a crude like type of world model like you're trying to like encode as much information as you can in these lms like you need like very big like almost like lookup tables of facts you know anyway it's kind of a conception i think about it and and and that sort of pushes like the memory requirements a lot i mean at some point like it might taper off like who knows but but like for now, like people will make use of every possible memory, uh, like the more memory, like they'll just train more parameters for sure. You obviously sit in like the, the, uh, in some ways

One thing, you kind of mentioned this portfolio of products, and I think that's one of the most interesting parts of the business you're building.

And I imagine, honestly, one of the hardest parts must be just prioritization of all the different things you could build. There's probably so many new things that come up of really exciting tools you can build for developers. And, you know, I guess I'm curious, obviously, you kind of, you know, with sandboxes, you have, you know, incredible products around, you know, coding agents around, you know, reinforcement learning. You know, how do you think about kind of building down that vector? There's also kind of like the fine tuning are all as a service stuff that some of your competitors seem like they prioritize.

Like, how do you think about, I guess, you know, where to prioritize within that portfolio? I think it's an incredibly hard problem at, I mean, most startups, but especially multi-product companies, right? Like the sort of balance of like, you know, first of all, like, when do you go multi-product and then where do you put your bets? Like we're 150 engineers or something like that today. And I struggle with it every time, like every day. Like, you know, I wish we could, you know, do that and, you know, whatever. And like, but I kind of know that even if we had a thousand engineers or 10,000 engineers, like it's all, you know, there's a lot to build.

It's like, you know, I wish we could build a rocket and go to the moon and whatever. Like, it just never goes away. so like you kind of have to be just like ruthless with prioritization and really think about you know what's the highest ROI and like where do you draw the line and you have to be sometimes like pretty aggressive also saying like actually like I know we started on this project I know you're really excited about it but like this other thing kind of came up so like we're gonna have to pause this for a second we'll get back to it later we've had to do that many times too it sucks when it happens but um I really think you know that sort of prioritization and like how do you how do you allocate resources I mean resource allocation It's like basically like, you know, the number one thing that I think a CEO has to really, you know, be ruthless about it, a company and like so much value is being created, like allocating the work to the right resources.

The modal roadmap in each board meeting is extensive and, you know, very daunting. And I'm always impressed by the amount of software that the team can ship and hardcore software while keeping the system running.

obviously had this like very prescient bet in 2023 on on sandboxes right it sounds like you guys had kind of this instinct of like hey this is going to be something that's required for for coding agents and then you know it took a few years for it to really take off and then you know it became this really important primitive i'm curious is there anything else that you guys are working on today that you feel like you're a little early on but like you know you're you're you're equally as excited and and how much of your product portfolio is like those kind of bets versus like okay we know this is a hair on fire like here and now problem jacob are you are you fishing for the next thing we should be investing in a hundred percent a hundred percent i'm nothing if not shameless i i feel like sandboxes they did create a little bit of like a high bar for us like

everyone's like what's the next thing like you know what's the secret sauce you're working on and like what's cooking in the lab you know and and i i wish i had like an easy answer like yeah like 100 confidence you know this thing is it would you have answered sandboxes in 23 or was it just like part of a bunch of things you were doing no because what sandboxes represented back back then was like a little bit of like an asymmetric bat like sandboxes was not like a very hard thing for us to build back then like to getting out like a prototype of sandboxes took like actually like a few days and then i spent a little bit of time you know like improving the sdk and some other people like started building and then we started seeing some customers we rewrote it and then in 2025 when it started exploding then we had to rewrite the system like five times just to handle like more and more scale and make it faster and and and more reliable and things like that so but but the initial investment was actually pretty pretty basic so in that sense like it was a little bit like a sort of asymmetric bat like a kind of a low downside high upside bat i don't know if there's like anything similar today that we're working on

i am i mean like just just kind of call out like a couple of like you know things that i'm quite bullish on and these probably represent like way larger investments uh like you know larger risks in terms of that size, low latency. I think there's going to be a lot of voice-to-voice, a lot of real-time video type stuff. That requires us to rethink a lot of how we manage to compute globally. How do we get down the number of hops? How do we scale up and down in different regions? Almost like building a CDN for inference. So I'm pretty excited about that. I think training is actually kind of an underrated thing.

I think there's a lot of opportunity in training mid-sized models and reinforcement learning that's been a big bet for the next for the last couple years like getting all the rma infiniband type stuff working um but i'll say i think a lot of vm like stuff is quite interesting like i think you know supporting full-blown operating systems like mac and windows and android and iphone could be kind of cool um i think there's a lot of upside there so But I have a lot of those ideas in my head and we. we're like building a lot of those things. I don't know if there's anything that represents the same like sort of explosive potentials like sandboxes.

I mean, maybe like if we have this podcast again in like three years, like we'll kind of look back and... We're going to make it a recurring at this point. So we'll look forward to revisiting that. You know, I guess like obviously, you know, sandboxes have become this fundamental primitive for, you know, coding agents. And it feels like that then brings you into this world where there's like, you know, I imagine you look at the current state of coding agents, you're like, God, there's so many more tools that would be helpful for these agents to make them more effective for people that use them.

How do you think about whether you further build into that world or the sets of things that are interesting for you there? We do think about it, but I think we think about it more from an infrastructure perspective. I think it's very easy to kind of get lost in whatever is the latest today. When I look at the architecture of agent harnesses and how we're building agents and stuff like that, I still kind of feel like we're so early and these architectures are evolving so fast. and like what our job to some extent is like trying to figure out is like what are like the sort of core building blocks that we can put out today that like will last you know like people we need them for the next five years and and that to me is like less obvious like we are interested in like sort of agent orchestration as a category i think there's a lot of interesting stuff there um we're interested in like you know evals and model gateways and all kinds of other stuff around like the whole life cycle of learning from agents and innovating on agents.

But I think it's less clear exactly how the architecture... My view is it will be more cloud-centric and it will be probably more durable objects, kind of workflow engine type driven. And we have some ideas of stuff to build in that space. But I find it harder to lay out exactly what agentic development will look like in five years partly because like i also think like to some extent like all software will like basically be agentic like i kind of hate and hate the word agentic in certain ways because i feel like it's like okay so it's using ai to make decisions like well shouldn't all software do that so like i almost like feel like if you look at it on a five-year horizon like i think like software like all software will become agentic and then what does software look like then like i i don't know but i you know i i hope we can build a lot of the building blocks to support all kinds of aspects of like, you know, in general, just like how software uses AI. Yeah. I mean, it still is like one of the hardest parts of building an infrastructure company, right? Of, you know, across the board for many people, you know, building AI for companies of, you know, what, what actually persists, right? Versus is a pattern, you know, is, is like

rag from two years ago or like, you know, whatever it is. Rag is a good example. Exactly. Like if we had made a bet on like rag, like two years ago, like it would have been kind of a bad thing to do. So, yeah, but, but I would also, so it's hard, but I would also say like one of the benefits of building an infrastructure is like you can take a longer view and you don't have to like listen to all the noise. I mean, you have to listen to the noise, but like you kind of have to like ignore a lot of it and trying to figure out, OK, what are the patterns here? You kind of alluded to some of these other, you know, other workloads that like low latency workloads like voice to voice and real time video. I think there's a lot of chatter in the air about like multimodal inference, robotics inference, these other categories. I'd love to hear you just riff a little bit on like to what extent are those problems, you know, similar to the set of problems the inference markets had before? To what extent are they like different and new?

I think they're all quite similar, but all, I mean, they're like snowflakes, unique in their own way. Robotics is going to be a lot of it on device, of course. So that's a big difference. And you know, I don't sort of envision like we're going to support, you know, most of what robotics looks like, because a lot of it is going to be on the devices themselves. But I think we're going to be able to support a lot of the testing and development and training of these of robotics and a lot of the simulation environments. So that's something I'm very excited about. On the multimodal stuff, I mean, there's audio and music and image and stuff like that.

Like, people don't talk about it as much, but that keeps growing like crazy, too. I feel like LMS have kind of dominated the airspace in the last few years, but I'm just incredibly... you know excited about all the different applications like why are like voice assistants so bad in 2026 like i don't get it like they're just horrible like i have a google home and and like every question it just bungled it does not understand what i'm talking like you know so i just go to chat gpt like why can't you just connect all of it like end to end and train like a system so that that that can you know do like speech to speech i i i think there's a lot of we're going to see a lot of sort of incredibly cool applications in that space i always think about you know you look back like you know four years ago three years ago whatever you know could you have a guess that like things like instinct or suvino or like cognition or like none of those things like are very easy to guess like you know like three years ago so i kind of feel like also like if you look down the road like three years like most people don't have an idea of like what the world is going to look like in three years it's so hard to know but like what i do know is like kind of

judging by the past three years it's going to be like there's going to be some really cool applications i just don't know what it's going to be but they're going to be very cool and hopefully they'll use modal i was looking back at an email you sent me uh with the modal series a pitch deck and this is what you said uh this was after we had signed a term sheet by the way we never actually reviewed a modal deck but here's something we made earlier that i just updated with the latest numbers could be good to review at some point when you sent this So you're at roughly 2 million in run rate. You now, you know, you've crossed a billion run rate, which is insane in three years.

What's been the biggest kind of change to supporting that kind of scale in just, you know, a couple of years? So first of all, I actually realized like the other day, I said on another podcast that I've never made a pitch deck. And then I was actually thinking about it. I was like, maybe I did make a pitch deck after this series A because we had a couple of angels that came in and they were asking about it. So I was like, oh, fine. And I guess I'll put something together. So I realized I did lie. Like I did make a pitch deck, but it was actually really cool. You never showed it to us. So I don't know that it counts.

Yeah. So I don't know if it counts. Like I never like sent a pitch deck to an investor. I sent it to a few angels actually. And that's the only pitch deck I've ever made actually. But in terms of the scale, yeah. I mean, we've grown, you know, 500X, I guess, in terms of revenue, something like that since then over the last three years. So a lot of growth. um i think i think in a way like i don't know maybe i'm like just you know my experience has mostly been at hyperscale startups like i was at spotify for seven years uh and some other places so so for me like there's like fast growth is kind of all i've seen like you know there's like you know every every six months you have to rewrite all the systems you have to kind of rethink everything you're doing you have to kind of change all the processes like you always feel like you're behind like i don't know i i kind of like the mess i guess we would love to you know since since we have your your mind on the podcast we'd love to kind of zoom out maybe talk a little bit about the broader inframarket and some things that have been in the news of late um you know

You know, I feel like on the AI M4Sci, one big category that has gotten a lot of buzz is like this URL as a service market and kind of a bunch of folks, you know, almost forward deploying and like helping companies build their own models. And, you know, you've had folks like Satya saying, hey, every company is going to have their own model.

Like, do you subscribe to that? And like, what do you kind of think of this, of that overall market? No, I think some companies will have their own models. I think that like weighted by dollar revenue, like I think most models will probably be post-strain, right? Right. Like, but but like weighted by company, like in terms of like numbers of companies, like most companies will not have the need to like post do their own post training. Right. But like the large ones, like Cognition is training their own code models and and Cursor did and all these companies are. And like those are the ones also serve a lot of tokens.

So I think when you look at like reinforcement learning and post training, like certain types of parts of it is very attractive. right? Like you can work with companies and serve, you know, better models and do it at lower cost and higher throughput and all of this stuff. Some of it looks like it's kind of like just a service business. So I think a lot of the sort of tension over the next few years, when you look at post training, it's going to be, can you turn this into product? Like, can you sell that product to these companies? This is something we think a lot about is like, how do we create this like amazing product you can just kind of sell and not turn it into just like a service business. And then, you know, I guess last one before we move to the quickfire round would just be like, you know, I think there's this, this question about, you know, how many tools live under like a single infra provider in the AI world versus, you know, our separate categories. I've always struck when we talk to you about like just how much you want to go build and all these kinds of building blocks. I'm wondering if there's categories of AI infra, you know, that you think will be unique and distinct

from the stuff you're building that you're optimistic on and that will kind of like live as still large standalone opportunities. Yeah. I mean, we're not going to go into model training i think like so i i think type safe with jav is going to be you know there's probably going to be a lot of these like custom models uh i'm sure in other verticals too you know other modalities like audio and video and music and like obviously on the application side is going to be a lot of incredible companies um uh and then there's probably going to be a lot of companies like focusing more on like software development and and stuff like that and and you You know, I don't know, like my general view on these things is like there's always like more boxes on the market map than you expect.

But I also do think that overall, every company that is successful in the end, to some extent, has to become a multi-product company, too. So I don't know if that's a tension or not. Like, I think, you know, we obviously want to find, like I said, a cluster of like very adjacent things and do all of it in a way that, like, you know, maximizes the revenue and the shareholder value. um but i you know by no means you know are we going to try to expand into everything i think people kind of make this like false assumption also like you know you look at like anthropic or open ai and they're like why don't they move up to stock and launch their own law firm and launch their own whatever like they're very good at like serving these models and doing that at scale that's how they achieve scale right like and they're very good at doing that like i i don't think it makes sense for every component to do everything and so similar for us like we're not and do everything, but we're going to do a very large cluster of adjacent things and do exceptionally well. And I think there's going to be a lot of room in the market to create a bunch of very large $100 billion or trillion dollar opportunities in the market. We'll end on our quickfire round where we stuff

probably a similar set of questions, what we asked you three years ago, maybe a little bit of an evolution. To start, maybe I'd love just one thing you've changed your mind on in AI in the last year. Well, the opportunity size of sandboxes has obviously been kind of crazy right like i i didn't think it would be anywhere close to you know i mean i thought I knew that in 23, and that's why you made the big bet. Yeah, so that's been kind of incredible. Coding agents have changed a lot of what it means to be an engineer. How, if at all, has it changed how you think about hiring? Not much. I think we're still going to need humans for a long time.

I think the demand for humans is just going up the more productive they get. I think we've seen this for the last 30, 40, 50 years of software engineering. you know, whether it's compilers or high level programming languages or CICD, like every time you make engineers more productive, like the demand just goes up. And so I think we're going to need a lot of different, a lot more engineers. I do think maybe on the short term, it's going to be like a little bit harder for like new grads, like, you know, except for that, the very high side of the spectrum. But in the long run, I'm very bullish on still, you know, computer science.

Actually, ironically, I didn't study computer science, but I'm very bullish on studying computer science in general. Like if my kids wanted to do it, like I would definitely encourage them. What's the biggest mistake you've made in building mobile so far? I don't know. I think one mistake I made in the past is that I think sometimes like you kind of want to sequentially do things. You're like, well, first we need to do this. This is the blocker. We'll get to this other thing later because this other thing is blocked on this. But like in a world where like everything is moving fast like one of the things i've learned is like sometimes you kind of need to do both at the same time like you need to assume that like oh we're doing this thing to unblock it you might as well do this other thing at the same time just in the interest of moving fast that's something i i'm naturally like not mega aggressive which is kind of ironic because like we've run 15x in the last 12 months i sort of had to retrain my brain to like be a little bit more like aggressive and saying like this is a massive market opportunity we need to be even more, you know, like hiring, deploying capital, like whatever it is, like massively just like

paralyzing everything we can. Sweden has a ton of interesting startups that have emerged. What are your thoughts on why that is? And has the Swedish house mafia been replaced by the Swedish tech mafia? Did you know that like one of the Swedish house mafia members was my classmate? It's a fun trivia. Really? I did not know that. Yeah. Back in the early 90s, growing up in Sweden. uh when i was eight he told me my bike was ugly i was very upset uh yeah i mean we've had a lot of tech in in sweden spotify klarna you know a bunch of gaming studios uh i mean more recently obviously lovable and agora stuff like that i think my theory about like a lot of this stuff is like success kind of just creates success like you know a lot of like what makes people going to tech is like they know about some cousin's friend who made a lot of money in tech or something like that so a lot of it is i think it's just like a little bit of a flywheel effect i do think that sweden as a european country has always been pretty entrepreneurial and commercial so i do think there's a little bit of that like we've always been a little sort of maybe going back to the vikings like kind of let's go conquer the world in the past maybe it's been through music

and furniture and stuff like that but now increasingly with tech so you know there's that like I think there's sort of a mentality in Sweden like you can actually Stockholm is like funny because like everyone in Stockholm thinks they're like this like mega global city that's like part of the world in a way where like I actually think they aren't but like I think the fact that they think they're so global is like kind of good like they have this like vision that like everyone cares about Stockholm but I actually kind of like that because it means like they're out there like building you know things and they're saying from day one like we're going to conquer the world like we're going to do this startup it's going to be awesome and I like that like the sort of megalomania I know this vision of like your elementary school reunion and just what a wild group that would be.

That's quite the cast in what I imagine was not a particularly large class. Yeah, yeah, it's kind of a random fact. I guess a shameless question. Do you think GP rental prices 12 months from now will be higher or lower? I think higher. You know, it's funny, like actually. changed my mind on this like in the last 48 hours uh and i'll tell you why i i read the anthropic s1 and i actually had a you know agent summarize it for me uh but it was kind of incredible how profitable that company is now it took to the degree that i hadn't anticipated i thought that like in my mind like 48 hours ago my argument was like oh anthropic like they have really good gross margins but they're spending so much money on training that like as soon as they go public they're gonna have to like slow down and how they grow training and post training and they're gonna you know maybe grow inference as much but but not the other stuff as much but then like i realized like they're basically a profitable company so like they have like zero incentive to slow down on like capacity um capacity acquisition and and i imagine there's a similar dynamic going on in open ai and so i in open ai and anthropic has obviously been like the biggest

two buyers of capacity out there and so the fact that like i don't think they're gonna slow down buying you know uh because of you know clearly like financially they're doing very well i think it will mean that like there's going to be a lot more demand and and i think for that reason uh prices will probably go up on the next 12 months i've heard you say that you've always thought modal you've thought of modal as your 20-year project you're five years in um what do the what do years 10 through 20 look like wow uh i mean hopefully i'll hire a bunch of people that I can do a lot of the sort of, you know, I'm too in the weeds of like everything right now.

So I, but I don't know, like, I just like love this stuff. Like, I love to keep working on this. Like, I think, you know, year 10 to 15, we're going to look more like a conglomerate. We're going to have a lot of different products. We're going to have probably some like crazy bets going to moon and Mars or whatever. But I still hope to be quite involved in this sort of the high level, like resource allocation and what are the things we're betting on. And I'd probably be less involved in sort of day to day, like, you know, let's get a contract from this GPU vendor, like whatever. But like, I think that like sort of like high level, like resource allocation and stuff like that, to me, that's the most fun.

And I hope to stay involved in that for another 20 years, hopefully, maybe longer. Yeah, I mean, if we keep current pacing, at least five more unsupervised learning episodes in that time frame, right? Yeah, exactly. Well, Eric, thank you so much for joining us. We really appreciate it. will give you kind of the last call to action for the audience. If there's, where can they go to learn more about Modal or anything that you would direct them to? Well, you should obviously go to modal.com. You should also run pip install Modal in Terminal and play around with it. You know, there's a lot of people maybe listening to the show who are building startups or thinking about building startups and, you know, would love to see all the cool stuff they're going to build.

And, you know, hopefully at Modal, we can help you with those startups. So let me know if you reach out, by the way. I always love talking to customers, but feel free to play around with modal and looking forward to see what everyone's going to build. I'm Jacob Effron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project, in addition to my day job as an investor at Redpoint.

But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.