Use deterministic rewrites for large codebase changes

Transcript

From the transcript The Pragmatic Engineer publishes with the episode.

1. 26:18

   How do you say like, how do you break it down? How do you plan for it? How do you speed it up? Because you clearly, if you just did it by hand, this would have not fully worked. Right.

2. 26:28

   So one of the things that had enabled any of this and that came out of having been on that low level C++ infrastructure team to start with, Was, okay, if we own the interfaces that are kind of the nuts and bolts that everything at Google is written out of, like way below the database level, just like caches, containers, string manipulation routines, command line flags, like that sort of thing. If we own those things, maybe it's valuable to make them better.

3. 26:55

   But how do you do that when there's 50,000 call sites or 100,000 edits that need to be made to change that API? Yeah. And so luckily I was friends and org chart cousins eventually with the compiler team, especially Clang LLVM people. And so they started doing stuff in the AST transformation sort of space. AST. Abstract syntax tree. So like you take a parse tree and, you know, this is Google, like MapReduce.

4. 27:22

   So if you do a MapReduce over the code base where the mapper parses all of the translation units of your C++, like all of the build targets, And produces, here is the parse tree. Then you can match the parse tree for specific, like, it's not text match. It's, I guarantee you are calling this particular function in this particular context. And then you can apply some code to rewrite based on that. And so we had compiler assisted, you know, semi-automated refactoring to generate all of the text edits.

5. 27:59

   Turns out one of the things that was really hard at first was we could generate the edit, but we couldn't get the white space in the right places. No way. Yeah, seriously. So Clang format exists because I wished for it. This is like the auto formatter for many like C++ sorts of languages at this point. But it's like I literally cannot accomplish this refactoring until someone figures out a way

6. 28:24

   to like hack together a auto white space for C++. And so you do that. You generate one, you know, big edit.

7. 28:31

   It is too large to fit in the codebase because for any given codebase, there's kind of a natural background level of how much change is happening. And if you're trying to edit 10,000 files, 20,000 files, it is likely the case that you will never Be able to test and re-sync before something has gone out of sync. Someone else edited something. Yeah, exactly. Merge conflicts, parallel changes, that kind of stuff.

8. 28:55

   Yeah, exactly. You do not want at a 10,000 plus person company to be like, everyone stop touching the repo for an hour. Let me do this thing. That's not going to scale. And so the tech is you figure out how to do your refactoring so that the old API and the new one can coexist in spelling. Yeah. And then you parallelize, like just cut by like directory structure. Like, all right, I'll send you this batch of updates, this batch of updates, and you wind up, you know, going like that.

9. 29:27

   And then once everything is clean, then there's a race to make sure no one copy and pastes the old thing. And then you delete the old thing. The new thing is the default. You move on with life. You just do that a few hundred times and your code base is in much better shape. Yeah. Wow.

10. 29:42

    And then you're doing this all the while other engineers, they're just doing their thing. They're using these things they don't even know about. They don't want to know about you. I mean, they might be aware of you, but they shouldn't have to change their, as you say, slow down or. Yep.

11. 29:55

    But we did get amazing feedback on some of this, like. At first, people are like, what are you doing? Why are you changing my code? I'm not okay with this. And very quickly, they're like, oh, this is great. I go away. My code just gets better. And I don't need to worry about looking up, is this the right way to do a thing? Because if it was the wrong way to do a thing, you would have gotten rid of it already.

12. 30:21

    When you have someone that is going through and just janitoring the code base like that. Copy and paste becomes the correct answer for how do I do this? Like find someone else that did it once, you copy it, you move on with life and you have evidence that it's probably a reasonable answer.

13. 30:37

    Because if it was wrong, we would have fixed it already. It's very interesting because today, obviously, if you did a big refactoring, the obvious tool that you would reach for is AI, as we've seen with so many companies having lots of success with. Well, in the sense that AI is capable of the challenge that you had is you had this massive asset code base, rewriting it. By hand would have not been feasible, but at Google, you came up with a very clever semi-automated approach, which was based on pretty hardcore language parsing and coming up with new ways where you could formally, maybe informally, or at least prove that this thing will be correct, which is what most teams would not have the availability, capacity, even expertise to do.

14. 31:27

    What everyone has the capability to do right now is like, oh, AI, go and do this and do tests and all this. Where I wanted to get to, like rewriting with AI still feels so much more janky than what you did, which was so much more deterministic. And I wonder if this technique that you used at Google might at some point spread or people will just put up with like AI kind of, you know, like it might or might not work.

15. 31:55

    I think if you're looking for it, it's already out there. In the Java space in particular, tools like OpenRewrite, like you basically... There's a little configuration per run for it. And you say like, this is the pattern I'm looking for in the Java code. This is how the rewrite would look. And it goes through and blasts through all your Java source files and does that update. Turns out AI tools are really, really good at generating open rewrite recipes, which is way cheaper on token consumption and way more accurate and much faster and like auditable.

16. 32:32

    It is a substantially better pattern. And we've seen talks from, I want to say, Netflix and Uber, I could be wrong, specifically on like everyone seems to be coming to the same conclusion of what you need is a deterministic tool that declares the intent of the rewrite. You can have the AI... Write a little skill to train the AI how to create the config for those tools, and then you move on with life.

17. 32:58

    In the C++ space, actually, some good friends might have a startup right now. It's called BrontoSource that is specifically just about that sort of declarative rewrite pattern at a very high level of intent, very powerful for C++.

18. 33:12

    Yeah, I like the idea of using AI to help generate deterministic Software, tools, transformations. Yeah.

19. 33:23

    Kelsey Hightower gave a great talk recently on like zero token architecture and like his whole thesis. And I think everyone is coming to the same conclusion. Don't use AI automatically. For constant inference, use AI to generate the deterministic tool that will actually do the work and then have AI fill in the gaps on the parts of the problem that you can't solve or where to invoke the tool, things like that.

20. 33:50

    Like if your whole architecture is token consumption for all time, it's probably a cheaper and more effective way to do it.

21. 33:58

    I just recently heard a head of engineering tell me that they're using these AI code review tools, which are pretty good, but very expensive and slow. And this person said,