Simulations replay production traces to test agent changes

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 9:13

   which you launched today um which is meant to simulate your agent's world world to catch what uh what your changes break so what does it mean to simulate an agent's world it's a good question so you know there was a time like do you remember prompt playgrounds do you remember this okay so like people that were like making chatbots like three years four years ago whatever it's like opening eye and anthropic they have these like prompt playgrounds where it's like you because like a chatbot was essentially just a prompt right it was just this like text you know and then you'd have this little playground and you'd edit the prompt and you'd rerun it and you would see the new result um the thing is that like agents have gotten so you know complicated um and so complex that like there's not some simple prompt that defines their behavior anymore it's like yeah

2. 9:59

   there is a prompt but there's also um skills the tools that they have access to those tools have definitions that explain what the tools are for and like um then there's like uh you you kind of can even statically analyze an agent you kind of have to put it in motion because there's all of whether it's feature flags or conditionals, like what plan the user's on, all these sorts of things that go into assembling the context that's available.

3. 10:21

   And then, of course, there's memory and all these other things. And so what that means is that, like, most teams right now are not able to iterate on their agent in a meaningful way without shipping to production. And so, and I think that production is a very accurate kind of source of truth as far as, like, you know, the problems that really matter and all these sorts of things. But also, like, we want to prevent problems, you know, I think especially as the stakes get higher, it's not enough to just catch problems that are happening in production.

4. 10:49

   You actually have to prevent them in a meaningful way. Yes, yeah, yeah. So I think to date, the way it's been done is, like, just, you know, normal kind of standard evals, which are, like, you define these test cases for every, you know, capability your agent has, essentially, and then you run them. You know, teams will collect, like, thousands of them. um and and nowadays it's just like you asking agent to make the evals um but it's it's very like uh backwards if you think about it it's like um you you know we want agents to do things we have never thought about before and um and what like so defining a test case for every capability is like a little silly and so we took a different approach which is like we want to show you what your change is going to change so the easiest way to do that is if we can make a way which is what

5. 11:39

   we did to re-simulate past things that happened in production that means for you know a specific user was using a personal agent we can simulate all the services that agent was connected to at that point in time their email their calendar you know like they were ordering something on door dash using the door dash mcp and their personal agent like we'll simulate that whole service fill it with a mix of real when it's available synthetic data. What it means is like we can run those simulations and show you like well here's what that if you change the model here's what's going to happen as a result. Oh I see okay okay is there any way that that helps beyond like a model change or can you sort of like rank you know what would have been the best outcome like for the agent or a better decision that the agent

6. 12:20

   would have made? Yeah totally so it's like models, prompts, tools like any hardest change essentially is something that we can um we can simulate yeah i'm curious your thoughts on like that problem of