Modal built core infrastructure to make cold starts faster

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 10:02

   Maybe you guys will start at the bottom of the stack and work our way up. But going back in time, you touched on it, but you guys invested in some really hardcore technology in the early days. you built your own file system container runtime those types of things how does that show up in the product today um when the the experience developers can have yeah i think i think a massive source of alpha in this market is just like not being afraid to do hard things uh so we went all in when we realized like in order to develop to to in order to to to deliver on the developer experience we wanted to have in the very early days we realized like kind of to build your own file system to do container images really fast so we we started building that in the very early days um and and that become like over the next few years you know it still

2. 10:51

   is like a very massive source of alpha it's been it's been a longer journey for sure like we've had to go a much longer route like building our own file system building our own container runtime our own schedule like we don't use kubernetes and docker like we we use like very little off to shelf software. We used GVisor, which is a fantastic Google product for isolation. Recently started using also KVM for certain tasks, but decided pretty early on that a lot of the stuff we're going to have to build ourselves. And now let's just do some really cool stuff. I think one of the best examples where that shows up is we can do cold starts extremely fast. Turn out like the initial developer experience that we wanted, like very fast feedback loop when you're launching stuff in the cloud. That is also very helpful when you're building auto-scaling

3. 11:37

   applications that need to scale up and down very fast. So first of all, loading the containers very fast, but over time we also launched a couple of really cool features around CPU snapshotting and GPU snapshotting. So we can very fast load data from the disk or from network straight into the GPU and fire up even these large models in a few seconds, which is easy. really cool. And over the last six months, we built this incredibly complex system that basically uses RDMA over peer-to-peer between the nodes to copy data very fast. So I think a lot of this stuff is like kind of, you know, accumulating and really starting to pay off. Like we built some crazy stuff. And now that's like, you know, we're really kind of, you know, reaping the dividends of that, those sort of investments that we made for many, many years. It was a longer

4. 12:27

   route. But, you know, now I think we have some incredibly unique technology that's very hard to replicate. And then kind of moving up to inference, what do you view as, you guys have made, you know, tons of investments there as well and optimizations. What do you view as the