Multigres aims to prevent data loss during failovers

Transcript

From the transcript Latent Space publishes with the episode.

1. 9:14

   the second one, so scaling up. I see. So that's, that's Mu- that's Multigress. Yes. So Multigress is the second big announcement. Basically, we looked at the Postgres ecosystem, and we looked at all the operators. An operator, for those who aren't familiar, when you run Postgres, you start with a single node, but then with almost any database, you wanna run multiple nodes, and you want things like failover.

2. 9:35

   You want it to manage backups. So usually you have this thing called an operator which can run the entire cluster for you. So there's quite a few operators in the Postgres ecosystem, and what we had found was that under, um, certain conditions, every single one of them could lose data. And so we decided with Multigress to do a ground up rewrite of the entire structure of a, of a operator. And so Multigress is essentially, while our initial goal was to become or give sharding to the Postgres community, what has turned out to be is this entire operating system.

3. 10:07

   And we actually ran it, uh, again through very rigorous testing, and we've done a thou- uh, 1,000 sequential failovers, and it didn't lose any data. And we've tested it in basically every way we can, clobbering it to see whether it will ever lose data. So we're very confident now in its stability, and so we're releasing that one as well tomorrow. It's completely open source. It will be on our platform for customers and, uh, invite only, so people who are running serious workloads can come to us, and we'll start helping them to scale up.

4. 10:38

   Is this the full completion of Multigress being fully sort of parallel to Vitess, or is there still le- something left to go? No, actually, yeah, it was quite interesting. So Suku, um, originally created, uh, Vitess in, for the MySQL ecosystem, but specifically for sharding. And actually, when he came along, he said, "Well, the number one goal that we wanna do for this time," because we're always catching up on compatibility when we use Vitess, you always had to change things as you move to sharded, so change some queries.

5. 11:11

   So he said, "I wanna focus only on compatibility first." And so the way we've designed it is in such a way, this ground up rewrite, is specifically so that we know when we release sharding, you actually won't have to change anything for your database. You'll continue to run the same queries, and Multigress will just be able to handle it. Zero changes to your code. I mean, that's a incredible vision, and the, the fact that you've been able to do it in two years.

6. 11:37

   That's one year. One year. Um, so we haven't released sharding yet. I've seen the early prototypes and the designs for how we're doing it, and it should, should, some of it should land by probably early next year, depending on some of the things that are happening on our platform. We've got a lot going on. But, um, but we're very confident. We're, and we're, we're happy with the progress. It's, it's been pretty good. Yeah.

7. 12:00

   I mean, I think this is the last piece for the total domination of Postgres. Yeah. Right? Yeah, that's, uh, that's it. And for those who don't know, you know, look up Vitess, uh, it powers YouTube, powers, what else? GitHub, uh, everything. Yeah. If workloads were ever to shift away from MySQL, you need this. To, to Postgres Yeah And that's then for Supabase to win. And hopefully it, it's not just for Supabase customers, right?

8. 12:22

   Like, it's fully open source It's community things It's anyone who's- Who's involved... who's running Postgres- Yeah... at scale, um, should be able to use Multigraph. Yeah, I'm sure, like, every coalition, every company that's involved in Postgres in some way is contributing to this thing, right? 'Cause we all want this. Yeah, yeah.