Repeated review comments can become lint rules

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 40:31

   professional as well the review what the way teams used to work and i'm not sure if instagram did this but a lot of other companies that this is you make a pull request you put it up there there's a mandatory human reviewer at google there's actually two because there's one on code quality as well how has this workflow changed how does the hot code team think about code review and how has it changed over time yeah i'll start by thinking i'll start by talking about how code review used to work for me so the way that i used to do it is uh every time i i also used to be one of the most prolific coder viewers oh okay so both yeah yeah right or is it code reviewers that's actually and that's one of the benefits of being in a different time zone like i'm not super human i just didn't have any meetings and the the way

2. 41:13

   that i approach code review is every time that i would have to comment about something i would drop it in a spreadsheet and i would like describe the issue so let's say you know like someone named a parameter you know in a function badly i would like put that in a spreadsheet if someone did some bad react pattern or something, I would put that in a spreadsheet. And then over time, I would just kind of tally up the spreadsheet. And anytime that a particular row had more than three or four instances, I would write a lint rule for it. So just automate it with kind of analysis. And so that's what it used to look like. For me, I've always tried to automate myself away because there's just so many things to do. And this is one of our superpowers as engineers is we're able to automate all of the tedious work. There's very few other fields where you're able

3. 41:54

   to do this thing. This is a thing uniquely that we're able to do. And this is a thing that I've just always enjoyed because it gives me more free time and I get to do the work I actually enjoy. And so today, the way this looks is a little different, but it mirrors this a little bit. So when CloudCode writes code, it generally, it will run tests locally. And this is something Cloud just often decides to do when it's relevant or it'll write new tests.

4. 42:17

   So you kind of do this kind of verification. When we make changes to CloudCode, Cloud will also test itself. so it'll launch itself kind of in a sub process it'll verify itself and it will test itself end to end this is for the look your internal cloud code implementation so you have like this test suite so they can test itself yeah that's right that's right but it'll literally launch itself just in a bash process and kind of just see like hey do i still work okay so it'll do this and this is something that we just didn't code in like it just with opus 4.5 especially it just started spontaneously doing this it just wants to kind of check so so we do this and then we also around Claude-P. So this is the Claude agent SDK in CI. So every pull request at Anthropic is code reviewed by Claude code. And that actually catches maybe like 80% of bugs, something like

5. 43:04

   this. And it's the first round of kind of code review. Claude will automatically address some of these. Some of them it'll leave to a human because it's not sure what to do. There's always an engineer that does the second passive code review. And, you know, there always has to be a person in the loop approving the change so on on the team before anything goes into production if you will an engineer does look at it yes as you're thinking of code review would you do this for every type of project or this is specifically because you now know that this actually has real world impact people depend on it you know there's a lot of users let me put it the other way around like can you see places where you would just not have an engineer review uh code what situations would that be in i think it depends how how it's used yeah i'd agree with that like you know if

6. 43:49

   you're building some personal side project like you can just yolo straight to main you know like even even before ai you would have not reviewed you just trust yourself or you know just ship to production or cessation to production and do some changes that kind of stuff right exactly exactly um the very first versions of quad code that were internal like you know i committed straight to main but then you know as soon as you have users and you know for anthropic our main customer base is enterprises. This is what we care about the most. For us, for safety reasons, security is really important. Privacy is important. These are all related. It's also very important for our customers. And so because this is an enterprise product, it has to be secure.

7. 44:25

   It has to be, we have to make sure that it meets a certain bar. So we definitely use a lot of automation, but at least for now, there has to be a human in the loop just to make sure. One thing that is just known about LLMs is they're non-deterministic. And by putting LLM as a reviewer, Claude, doing a review, it will give good feedback. But how would you deal with the fact that you can't be sure if it's always giving the feedback, you cannot be sure that even if it's capable of catching an issue, that it will necessarily catch that.

8. 44:59

   Are you doing anything in this loop to do deterministic thing? For example, linting is very deterministic, as you will very well know. Like, have you thought of marrying some of these ideas or are using, for example, are using linters on the code base where you found no need for it? Yeah, absolutely. Absolutely. Yeah. This is just, yeah. Yeah. We have type checkers. We have linters. We run the build.

9. 45:18

   Cloud is actually so good at writing lint rules. So actually what I do now, I used to tally stuff up in the spreadsheet. Now what I do is when a coworker puts up a pull request and I'm like, this is lintable, I'll just be at Claude, please write a lint roll for this in that PR on their PR. And we have, you know, you just run like slash, I think it's like setup GitHub or something like this. You can do this in Claude code and it'll install the GitHub app, which then makes it so you can tag at Claude on any pull request, any issue.

10. 45:44

    I use this every single day. So very, very useful. So you want these deterministic steps. Also, though, there are there are ways to get Claude to be a little bit more deterministic. So, for example, you can do best event, you can have it do multiple passes. And this is actually quite easy to do. So, you know, for example, the code review skill that we use internally, it's open source and it's available in the cloud code repo.

11. 46:09

    And so all we do is, you know, we launch parallel agents to do stuff and then we launch parallel deduping agents to check for false positives. But essentially best event, the way you implement it is all you say is cloud start three agents to do this. And that's it. Wors just talked about building that enterprise infrastructure layer, the auth, the permissions,