Uber is shifting from diff counts to feature velocity

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:23:14

   do you think about engineering productivity because when i was at uber this was again in like 12 up to 2020 us managers did look at things it wasn't the main thing but we did look at things like diffs per engineer just to show if someone didn't contribute for like two weeks it was like, hmm, what are you doing here? Or if someone did a lot of code reviews, that's actually what we wanted to notice. But it sounds like a lot of those things, the code output, the code artifacts are less relevant potentially.

2. 1:23:39

   What is a productive engineer in your mind right now inside of Uber or inside of your team? Yeah, so I think before the AI world, we were measuring the standard stuff like code throughput and stuff like that. Those are the easy ones. And then we would we also have like every few months we do the developer experience surveys we look at nps you know like sentiment across all kinds of dimensions we look at our code quality dimensions what's our incidence per def what's our code coverage and all of these metrics right um i think now once you go to like yeah i feel like we can leverage like the new set of capabilities to get more nuanced into this discussion like because because now we are internally talking about not necessarily just diff velocity but feature velocity like how much of all of this additional throughput

3. 1:24:25

   is going towards unlocking new new features and how can we even change our organizational processes so that we can reduce the p50 feature time because like you can create 100 diffs but if they are not all feature diffs then like maybe it's not necessarily increasing the value that we can provide to the users right so like one for example one of the things we did is tagging all of our PRs to different categories like on complexity level and is it a feature PR, bug fix, maintenance, testing, documentation on different categories and we could see year over year how is that mix changing. Luckily at least right now I can see our percentage of feature PRs is going up, our percentage of maintenance PRs is going down. So that gives us good indication that we're in the right direction.

4. 1:25:16

   We see the code volume go up and the percentage of PRs that for features going up then we are investing a lot because before even though we had the uplan tool sometimes you might not have the right tracing from this product design dog with this engine design dog to this particular task so it becomes a bit harder to measure what is the cycle time into end reliably across all the features so now we're using ai to like automatically cluster these things and like tag any missing things um so that we can reliably get what is the p50 feature velocity for different complexity of features for every half over the last two years three years and then we can go into is the bottleneck in the prd phase is the bottleneck in the launch phase is the bottleneck in the rollout phase what is going on there and what kind of things we need to invest

5. 1:26:08

   to accelerate the number of features that we can ship per half. And how is your mental model changing of software engineering compared to pre-AI and pre-D's tools? Is it still the same job? Have things changed? I mean, at least in my total tenure in engineering, there's never been this much change to how I work. Like, for example, we say all the engineers at Uber are hands-on. Even distinguished engineers at Uber are hands-on.

6. 1:26:37

   they like they create prs and i used to think i was hands-on engineer but my hands-on definition year over year has completely changed i'm like even personally i'm doing like way more prs than what i was doing last year like it's more easy to do uh all of these things uh all of these things right and i'm able to like explore more number of ideas than what i could possibly do in the past but the job of the engineering is you know in some sense how do you uh build the right context for the agents to produce reliable output like in some sense like you need to get really good at context engineering before i was operating with some context to do some tasks so i'm always thinking about okay if i want to offload some of this work to an agent what kind of context it would need and how do i set up the environment with that right context so that it can produce that work

7. 1:27:26

   then you have to become really good at like you know how do you give the right specs right constraints for the agents apart from building the right context then multitasking has become really like a thing um before if you're like it would take so much time to create a pr so like at least laser focused on that but now you're constantly switching between so many things like so you need to become really good at multitasking across all the sessions figuring how do you scope the how do you scope the work how do you track the work how do you verify the work really quickly because if you're not able to verify the work and like pick any code smells from the agent then um then that's hard then you're shipping like some like some recreations into production right so um like the context engineering ability to multitask ability to

8. 1:28:14

   like review and like verify the outputs these things have become like much more important uh now because you can because you're able to paralyze a lot more than before one thing i'm picking up is maybe a new primitive is we used to like build software thinking how do I build this software and how do I verify it? Whereas now the question that you said is often, how do I set up an agent to be able to build this and to be able to verify it? So it's kind of like almost a meta.

9. 1:28:42

   It sounds like there's more of that, right? Yeah, I'll give an interesting example. I was working with some of the engineers that work on our customer support team. We are going through the journey of having AI agents handle some of our customer support policies. That process would go from oh you have some runbook that human agents would follow so then from that runbook you are going through a whole bunch of steps to create a policy that ai agent can follow then you have to think about okay there is some dynamic elements here then which apis i need to call call to get that data which attributes i need to use then i need to configure my system change bunch of configurations then i need to do manual evals before launching then i need to do continuous evals post-launch. And then in the last few months, we were able to take down

10. 1:29:32

    all of the steps. Okay, how do I now take all of the context and all of the principles that I want to use for the policy conversion and teach agents to be able to do that so that I have a really good v0 policy that an agent can create. Then you automate all of the configuration setup, manual testing, finding the right APIs, finding the right properties so that we can hook the right dynamic elements into this again like that used to take weeks now it's like order of days for any policy then instead of doing like manual evals then how can you like use all of the past context to create synthetic evals on top of that so that you can like you can accelerate you can deploy this and then uh verify this quickly like essentially what we're discussing right which is now you're not instead of doing that work okay how how can you build the right tools and hardnesses on top of

11. 1:30:23

    of this so that you can automate the workflow obviously you need the right guardrails and human verification steps in between but you're designing a system that can do these things instead of like