Fast sandbox snapshots could make parallel work practical

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 43:47

   What do you see as the use case for snapshotting? What would that give us that we don't get right now? I think the big thing for me is today, a lot of sandbox workloads are kind of like lift and shift. People think a lot about sandboxes, almost like I have my dev box, now I'm going to shove it in the cloud, I'm going to have an agent control it, maybe I'll run the harness inside the sandbox. I don't know if like long term that's like the right way to think about, you know, on like compute for agents.

2. 44:16

   I think a much better model would be you're going to have this like very long running agents, which almost like to me are like kind of durable execution workflows. Like they're going to run, you know, 24-7. You know, every once in a while they're going to run commands and, you know, having the ability to like spin up a sandbox in, you know, 100 millisecond, run a few commands and then like shut it down.

3. 44:39

   and like snapshot it, I think the faster you can get that like, you know, wake up, you know, go to sleep cycle, the more you can sort of create sandboxes that only run for like a very brief period. And that will make compute much more efficient. And it will also enable you to do some really cool things, you know, like forking and moving things around and, you know, having, like imagine, you know, like clawed code instead of, you know, having sort of a very linear flow, like sequential flow, like kind of branching and, you know, running hundreds of agents at the same time.

4. 45:10

   I think a lot about CI is another illustrative example. For most of the history of CI, talk about GitHub actions and things like that, you've had a very sequential flow. You slowly install a bunch of stuff, then you run the test suite, and then generate some reports or whatever. And it's like, you can break it up into shards, but it's kind of annoying. What I would love to do is take your entire test suite and just run every single test in its own sandbox, right?

5. 45:35

   and just paralyze the shit out of that. Run like a thousand sandboxes. Why can't that be done? I mean, the problem is like there's too much overhead around like, you know, starting and stopping sandboxes. So I think the more like you create like, you know, frictionless like starting and stopping, the more you can enter a world where like you can do this like extremely bursty, like large scale, like run as much as you want in parallel.

6. 46:00

   So, you know, I'm an infra geek, like I said. Like I get very excited about these things. once we figure out all these primitives and the right building blocks, I think it's going to unlock a lot of really cool stuff that we previously could not do with software. Why hasn't that already been built? I would assume even previously people would want to parallelize as much of their CI as possible. You can't, but the overhead, it's like Amdahl's law or whatever.

7. 46:24

   The overhead of parallelization right now is very high. In GitHub Actions, for instance, we parallelize in the modal test suite. But what that basically means is now if you take your test suite and you split it up in two, now each of those shards both need to do all the installs and the test setup. And that takes a long time. It can take minutes. But let's say you can do the test setup once, and then you snapshot the container, and now you clone it, and now you can run the test suites independently.

8. 46:52

   I think a lot of the benefits cannot be, you can't materialize those benefits until you get the friction and the overhead down, the transaction cost.