Agent memory failures need separate storage and retrieval metrics

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 4:58

   So I'm also curious, like, when there's a new model, when there's a model upgrade, what are some of the main pain points that you realize with people that are, like, using agents? Good question. I think every single new model has a new set of, like— Or a switch, even, right? A switch, yeah. From, like, an OpenAI to an anthropic model or something. Yeah. I think that, like, I mean, like, an obvious one from, like, Anthropic to OpenAI, for example, is, like, always, like, writing quality.

2. 5:24

   Like, it just, like, that still seems to be, like, both models have their own problems with writing. I think, like, and so that's always a weird one. I think that right now the main trends we see are people trying to reduce costs and people trying to route correctly. And I think that's a really hard problem still. Yeah. what about like there's i feel like there's just so many different subtle types of um things that can go wrong but like if an agent compacts some like a memory in a way that ends up being kind of like dumb or or wrong or something like that was it was it was in a sense like trying its best yes and not necessarily misaligned and then an agent later on like a instance would read it and then do something wrong with that memory so like how do you think about things like that as well totally um i think this is where like classification actually you know we

3. 6:15

   i think that classification is a really really hard problem and what we think about uh like a lot of what we do is like a classification like a binary classification problem um so when i think about something like memory for example if you're building an agent and um you know on the the first like surface level thing you could track is like people complaining about memory or like issues with memory like the agent says it can't find something the user mentioned or the user is like i already told you that whatever so like you can track that but then the next level is like if you actually want to improve your agent you need to understand if it's from storage or retrieval right so it's like that because you can have issues on both um and uh if you're iterating on on storage is actually a lot harder right because um uh because there it's super long horizon like

4. 6:59

   you can make a change today um but that doesn't fix all the other you know the last like three months or six months of if you have a agent in in production already um and so i think that a lot of it is like getting really granular with like what you're tracking so you have a metric you can actually hill climb like you kind of need a i think a lot of what we're seeing is that when you have like a very good verification loop like you have some way of verifying if your change is good like directionally good or directionally bad um then you can sort of hill climb that and that that works um but defining that well and accurately is like becoming one of the most important problems of the world. I want to shout out our sponsor, Lovable. You know, the app you've been meaning