Uber still requires human review on every PR

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 55:36

   what do you do after that then um um we will go through self-healing ci first one thing that we realized is because our pr volume has gone up year over year oh yeah it must have explored yeah our p50 cycle time p90 cycle time both for inner loop where like you know you are good you are you have the pr and you want to get it to a green state and create the remote pr or once you have your PR created getting the human review getting all the CI things to pass I think reviews become a big bottleneck same as like any other company and we see year over year the delta and we want to like really we're like really focused on like breaking down every aspect of this and like how can we go really deep and optimizing it for the AI era so first thing that we did was we took a whole bunch of like simple CI jobs that used to run once you create the PR and we moved

2. 56:31

   it to inner loop and we kept all of this behind one single command like aifx validate so then when agent creates a pr it can run the linter the coverage rules the build rules the template rules like all of these things so that the probability of this pr in a is in a good condition like goes up then we hooked on ability to do visual validation where i can spin up emulator and test it I can spin up like a web and test it I can spin up like a staging environment where I can deploy this change run the API's and test it and now minion or even my interactive harnesses they know these agents know how to like have the skills for this so then they can go through this verification loop and let's say you create the PR and still something fails then we we have like a self-healing CI you can apply to all the PRs

3. 57:22

   or per PR so this is like an uber specific line that we add to all GitHub PRs that you can link to the issue that you're working on you can see which checks are failing you can have a whole bunch of automations auto request review, auto merge auto fix, essentially everything that we can do to reduce the P50 P90 cycle times and once you do all of this then we have this thing called code inbox this is showing a mock version of that where it will any engineer can go they can see all the prs that are waiting for them all the prs that they created that they're waiting for reviews so if engineer sent this before review i go to this page i can see all the things that we did so far so it makes it really easy for me to get oh the ai code review passed these are the things it checked this is a sanity test screenshot so i know okay i know you can just quickly take a look

4. 58:21

   you have a video as well yeah i can see the the bill checks and all of that and uh we're also adding risk score blast radius of this pr and so on so all of this will give engineers really good signals to figure out okay like is it good to land and which parts of the pr i should focus on and yeah i think we're still uh like we're aggressively like working on this whole bunch of ideas that we have but we want to make sure like you know as we are ready for the influx of PRs we in fact had to like add internal code red for our CI system because year over year the number of CI jobs also doubled or more than doubled so we had to like also like beef up a whole bunch of foundations in this space so that we can we can be ready for the influx of new PRs Yeah, and I guess you're also seeing that there's no way engineers can review all PRs with the same rigor as we did in the past.

5. 59:16

   So I guess you're building tools to signal which ones to pay more attention to, which ones might be low risk in so many ways, maybe in impact, maybe in blast radius. What is your take on what will happen with code review over time? Do you think there will always be human code review or for some things it might go away? Yeah, absolutely. So I think one is for us, like we want to err always on the safer side because the cost of an incident for Uber is like really high.

6. 59:46

   So even with all of this tooling, we like right now, we still require an engineer to review every PR that that goes that goes in. once the more confidence we get in our risk scoring and we can also back test it with years of data right we have all of this data and the more confidence we have in our ai code reviews then maybe for some class of low risk prs we could enable we call it green lane like auto landing but we want to be really really sure about all the dimensions that we want before we go there But right now, the main priority is how do we make it really easy for engineers to get all the signals they need to review faster?

7. 1:00:31

   And if we do a good job there, then I'm sure we can increase the percentage of Autoline PRs over time. But that's kind of P1, P0 is first getting all the signals in the right place. Nice. Yeah, but CodeInbox is nice. I like it. This is a thing that we should touch upon is maybe how I can do sanity testing.