AI cut a complex data-lake project from six months to ten days

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 18:04

   podcast you and i were just shooting the shit and you were talking about like how you can do products projects now that take a week where in the past would take a team and months can you maybe walk this through a little bit about like how you personally kind of evolved in that we'll do projects and then kind of the gains that you see so like a little bit about like your stack and your process so i use uh cloud code um i'm gonna say something that's actually pretty funny but I use Windows.

2. 18:35

   What? Was that created by this company, Microsoft? And I use WSL. So I use Linux for everything, like development and so on. There's no like X or UI for it. I just use the command prompt. How do you use Windows? Because I just never really wanted to learn the keyboard shortcuts in Max. Are you a gamer? Is it like you tab over to Counter-Strike or whatever? like why else I'm not hitting on Windows I mean you know I used it in the 90s that's really like the only reason is I never wanted to learn the the shortcuts I hate the one button mouse I hate the one button stuff I hate the number of modifiers you have to type for shortcuts and stuff like I just I just never I'm like why why why why waste time okay so I'm like yeah I'll install Linux use Windows yeah And so WSL, I use cloud code.

3. 19:36

   What is WSL? That's Windows subsystem for Linux. Great. So they actually run the Linux. kernel within Windows in a hypervisor and provide you access to it. It's great. They've done a great job. Great. Windows WSL. And Cloud Code. And like my latest, the latest thing that I do is I use Opus 5.5. I tell it. Which is a really good model. It's an amazing model. Really good. And I've shipped in the last two weeks these massive 20 PR things that I just dump on my team to go and review.

4. 20:13

   and they're like, I'm like, don't worry about it. Sounds horrible. It's like the new reporting structure, like the people below review the PRs for the CEO. Is that where we've become? No, everybody's like writing the same thing. Oh, everybody reviews that. So you review other people's PRs too. Yeah, I review my people's PRs. And like everybody's like just PRs flying around. Sprouting in PRs. Yes. But like the thing that has changed a lot in the review cycle is that we started to focus much more on the tests and doing things like golden expectations and are they shifting in different ways and so on.

5. 20:53

   But Opus 5, the trick that I've recently learned about is that instead of using sub-agents, you basically tell Claude to spin up command line sub-agents because those maintain their persistent session over time. And so you can have it spin up these agents separately. You can actually run them from other sessions and so on. And that allows you to be much more efficient from a context perspective. How many agents do you run?

6. 21:19

   How many? How many? How many what? Agents. Agents. Typically, I work on like three or four session windows at the same time because I found that more than that, I just start losing context between the different things because like they start finishing faster and then they wait. And do they interact via the PR? Is that how it works? The sub-agents, like together? No, they interact directly through the command line.

7. 21:49

   Oh, wow, okay. Yeah. So I asked the main thing that does the implementation to have a tech lead reviewer and they basically ask it questions. Does this make sense? Does this not make sense? Review my PR, et cetera. And so they iterate on these PRs together. other um and then they have a pr doc that basically they work on with a design doc and everything in it like the first thing that i do is i ask that agent to do it ask the tech leads to review the pr doc i go through the the prd i go through the design doc make sure it makes sense go through each piece in like detail make sure that all the algorithms work and like all the pieces work the division of labor works the division between prs and their relationships work and then i tell it Just go start doing it.

8. 22:34

   Figure out if there's like parallelization that you can do and then spin up other agents to actually go do them. But ultimately, you have to kind of put it all back in one stack and be able to go through it as a human one by one because otherwise it just gets messy and bushy and you can't really tackle it. So can you give a sense of like a project you do now versus, I don't know, six years ago? like time differences?

9. 23:00

   So the latest project that I shipped was something like... By the way, this is like, to be clear, this is production code. Yeah, this is production code. This is like demo, mock-ups, whatever. So the statement you're about to make is the gains that you get for like shipping production. So I'll give you the example of the last, there's the last PR that I worked on or like the last set of PRs was what we call auto-promotion in a data lake.

10. 23:22

    Yeah. So in a data lake, you store all the data and typically the data is column-oriented, parquet files in Iceberg and so on. And when you get data from logs, they're semi-structured, they're arbitrary columns and so on. In order for... To have the best layout of the data and to be able to query it efficiently and fast, the ideal situation is that every field in the log event becomes a column that you can filter on independently of the rest of the rows.

11. 23:55

    And so we call that auto-promotion or shredding. And doing something like that is incredibly complicated, right? Like you have to go and figure out how does the metadata system work, How do the commits of the metadata system work? How do you coordinate across different writers and so on? And that took maybe about 10 days. In previous days, that would have taken maybe about six months for a single person.

12. 24:27

    Wow. And was this just you? That was just me. It is insane. yeah sort of like factor of 10 factor of 20 improvement yeah for production code for new products yes you know one question I always have is like I can very much understand