Production replay lets agents validate fixes against real traffic

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:06:43

   And I'll also talk about one more pretty interesting thing, which is so uber-specific. What is Capri? Yeah. And this is very, very hard to get right. And we had even many versions of this in the past, but I think this is probably our best version and we can really scale it to all the teams. But essentially, the idea is context. because we have, even if you look at our co-rider app, we have hundreds of screens. And the rider app is what most people think is Uber app, right?

2. 1:07:15

   Uber app, exactly. We have different business, different product lines, different cities, a whole bunch of dimensions. If I want to test any new thing, how do I know every combination of these has no regression? And if you were to create all permutations across this, you would have to maintain thousands and thousands of test across the stack and that alone is like again really hard problem in itself so for some of our most complex platforms like we we call it capture replay like essentially you can capture real production traffic with just one middleware because now we have standardized language frameworks across all the services it's very easy to hook this into and it will capture not just the inbound and outbound at that service layer but also any of the downstream apis that service calls those requests

3. 1:08:09

   and those responses are also captured so in some sense you have like full request response and all of your outbounds request response is captured now with all of these captured requests they go through some pipeline and then you can build simple rules that says oh if you see this kind of field in the request this is the value for that dimension so then you can capture all kinds of dimensions that that platform team cares about and once you have all the requests and all the dimensions you can in some sense generate the full set of live dimensions and live combinations that that platform is currently supporting which we call it match guard that i can run pre-deploy in our ci cd that shows for all of the prs that are going in this deploy let me validate against the full combinatorial set of combinations this platform supports

4. 1:09:05

   and show the delta before and after. So then you get really high confidence if am I breaking maybe cash trips only in Brazil with this configuration, which would have been really hard for engineers to manually test or agents also to manually test. So step one is you capture what you want to capture and then you have the system that will rerun it, check that there's regressions. Yeah. so like then i'm going to one of the domains that has this enabled so you can see all the sessions that it captures i can go to any of the sessions um then you can see all of the metadata and tags so essentially it's saying oh this request is tagged as marketplace type eats it's tagged as uh line of business eats it is in this kind of terminal state it knows uh captured all the requests you can also see all of the outbounds that it used if we click on that it will also show you

5. 1:09:57

   the request and response for the outbounds. And if you see at the top, there is slate workflow UID or search environment name. So then let's say if someone, if I'm working on a bug that's only affecting for some reason, San Francisco, Uber Eats trips, I fix that bug. Then I can create a slate environment for that bug for my code changes. I can go here and then say replay just this request on my slate environment and show me the delta here.

6. 1:10:31

   Show me the difference and you're expecting a fix or some change. Yeah, I'm expecting the fix. So because I know the before request, after request, what I'm expecting. And then I can do it for a specific dimension or I can use it like then. And then we also generate what we mentioned like match guard, which is the full set of dimensions possible for that. So then you can create like a, in some sense, a testing automated testing suite for that team.

7. 1:10:56

   and also like more easy way to manually test one specific combination that you are focused on. When I was there, we didn't have this, but we have something called Deputy and Cerberus, but those were, I guess, very early versions. Yeah. So in fact, like what Deputy and Cerberus allowed was ability to route any, what Deputy allowed was ability to route any request from production to your testing instance.

8. 1:11:23

   and Cerberus allowed from your local testing instance back to the staging servers. But now that we have Slate, we don't necessarily have to use Deputy and Cerberus because you can deploy a Slate environment and we don't need to use any of them. I just want to stress because I think it's hard to see from the outside, but it's really hard to build this because every single, again, you have thousands of services, all of them have to support these things from routing, for example, to capturing production, to replaying, right?

9. 1:11:55

   Like this was a lot of work, I assume. Yeah, and now I think the same, like similar to how we exposed like Slate to agents, we can expose all of the Capri functionality to agents so agents can check, oh, I'm trying to fix this. Let me get all of the sessions of this type and then see what request looks like. Can I compose a request of that type on my Slate instance and see, did it fix? Oh, did it not fix?

10. 1:12:18

    Let me go in a loop until I fix this. Yeah, yeah. And now the agent can, even though it cannot see, but it can see the delta and it can tell that nothing has changed, for example. So it can get feedback loops. Exactly, exactly. I think that helps a lot. Now the agent can also do a capture replay on Slate with any of these. We already talked about the Edge Gateway.