Turso helps Supabase scale down to billions of databases

Transcript

From the transcript Latent Space publishes with the episode.

1. 12:58

   At the moment, we launch about five million databases on our platform every single month, and kinda the future where we know we're going, we think that we're going to have to launch billions per month. And so scaling out is a whole different beast. So we decided that we would have to look at some new architectures and people within the space, the agentic database space, and there's no other leading database provider who does agentic well than Turso.

2. 13:27

   And so actually tomorrow, and when this is announced, it would've come out, where we have announced that we've acquired Turso to help us power agentic databases. Yeah, that's an incredible... I- it's also, like, they're, they're primarily SQLite based, but it doesn't really matter, right? Like, it's, it's more about the architecture of, like, I guess... I don't have the word to describe how basically, like, broadcasts, is that the word? For, for scale out, right?

3. 13:54

   Like, um, there's the idea for them scaling SQLite was that they would have, like, a single writer and, like, multiple readers. Yeah, so, so SQLite itself, it's only a single writer. It's a file format. Um, so generally it's not thought of as a- Like sort of a MVCC for SQLite. Yeah, ex- exactly, exactly. So then they built this Turso architecture, which basically builds MVCC for SQLite so that you can have multiple writers. Yes. And then it's also- And because SQLite is, has a weird license, they had to fork it?

4. 14:23

   They had to fork it. [laughs] They originally started with Libsql, and then they- [laughs]... they came along, and they did this other really cool architecture. And actually now the architecture, actually it was very commonly discussed a few weeks ago, well, on S3. So some of the innovations in the database space are actually coming out of what's happening in object storage. So object storage has added a bunch of new features, which basically mean that you can start using it a little bit like a, a database.

5. 14:48

   And because object storage itself is, I think it's 11 nines of, of data- Durability... durability. Yes Then- Which is not reliability, but whatever... yeah, not reliability. But it, it gives you what you need for, for, um, basically what you would typically have for a cluster of databases. It's kind of like your cluster underneath, so you can start doing some cool things like, yeah, there's quite a few popular databases now, or data use cases that are building directly to- on top of object storage.

6. 15:16

   It's got some limitations, um, in our worldview. It's probably good for these smaller use cases in terms of transactions, because when you've got a transactional database, you've gotta do a lot of reads and writes, and so in this case, you probably don't wanna be doing something at massive scale on this, and that's why we have Multigress. But for these, what we're seeing, where we need to get to, these billions of databases, what we need them to be is a file format. And so that's where SQLite was super interesting. King. It's dominating.

7. 15:47

   And actually, the thing that really piqued our interest, and I reached out to Glover, h- I said, uh, he released a blog post on how they've done their rewrite, basically makes the-- we have a pathway to get Postgres compatibility with their file format. So we're ac- acquiring them, and we're actually gonna keep the Turso brand. So Turso's gonna focus on SQLite, and then we're going to introduce this new, smaller instance type on Supabase called Supabase Spark, and it'll be compatible with all our libraries.

8. 16:17

   And you can have one database for any idea, any prototype that, that you want, and when it takes off, then you just upgrade to full Postgres databases, all the way up to Multigress. I know you've been working on this for a while. This is the dream, right? Yeah, yeah. There's been-- Both scaling up and scaling down has been one of the things that we've been trying to figure out how to do. Yeah.