Agent designs can curate context instead of appending every step

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:17:58

   called KVCache rules everything around me, like C-A-C-H-E. Somehow no one on the internet has said cache rules everything around me beforehand with a C-A-C-H-E. And I can't believe I was the first one to say this. So number one, I have a blog post about why agents are extremely beholden to their kv caches um it's there and and why a lot of the design of an agent very obviously comes from the constraint of knowing that the kv cache is extremely expensive and very sensitive like you don't want to pollute it so things like why don't people use more sub agents why does routing tend to not work all of this i go into like my my blog post i've also released a google doc which i kind of regret because i keep getting share requests for it of people who want to like edit my google doc or something but i had this internal design of how could jf be used for coding agents

2. 1:18:54

   because once upon a time we thought that we would be a bit more stealthy and we thought we would like be building products ourselves and that cat is now totally out of the bag because we decided not to do that but um yeah i i i think that there's So many cool things that can happen if you just assume that you don't need a append-only KVCache. Like, a KVCache is great for saving money, right? But also very constraining. It prevents a lot of, like, cool programming patterns, like fanning out and filtering and reordering and stuff like that.

3. 1:19:30

   And it's... I love the interview question. How would you design a coding agent if KVCaching didn't exist? it really makes you rethink how agents could work and i think it's a really useful intellectual direction for figuring out how to design agents if for example you had like extremely cheap but reliable intelligence on the fly and i think there's so many cool things that can be done um you know the kvcash is a little bit similar to like having global variables everywhere but what if you could be smart about your state why do you need the full view of like every past decision you've ever made um for all the future decisions too because presumably when you're in a moment there's extra information you might want that you could like insert get the decisions out pop it off the stack and like really curate this kvcast or even do super sick things

4. 1:20:23

   like um like log of n lookups within your context nothing stops you from doing like hierarchical searches within your context and that way you don't need to have the same thing over and over and over again. Another gigantic use case is actually tool calls. Tool calls, number one, result in context wrath because they require tons of context up front. Number two, you don't always need them. And number three, because the models have been overfit to their native harnesses, they tend to call the native tools a lot more than the ones you plug in. So the models don't end up calling them that much. What if you could provide logit biases to your tool calls to fine grainly control when things are inserted or not without poisoning your context so there's so many cool things that could be done if i wish i could work on it ah it would be so cool it would have

5. 1:21:11

   been so cool so the agent architecture is potentially very different in a kind of jev greenfield world and you're leaving it as an exercise to the listener or for the listener well i think it's a i think there's an interesting intellectual direction to look there pragmatically we need to care about like what actually works in the real world and then you know there might