The KV cache constrains how coding agents are designed

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 27:03

   great phase i think you said that the tyranny of the kv cache in uh encoding agents maybe explain for our listeners a little bit about like some of the limitations uh encoding agents you know due to the kvcache and like how you think about that problem space i would say almost all design decisions around the ux of coding agents like the stuff that actually functionally changes stuff is built around the kvcache and this is natural because like the biggest wins in ml are always like very hardware aware um this is how things have gotten so efficient recently um or pseudo recently um since gpd4's days gpd4 is very expensive like you just need really need to design around this which is great because it's efficient but it's also quite limited it this is why like routing doesn't tend to work routing absolutely works if you're making the

2. 27:49

   equivalent of like um what we call instruction following calls like you know task in task out you know fuck the cash throw it away uh routing is amazing for that actually you can do way more advanced things than just routing in that case. There's so many advanced patterns you can do when that is your assumption. Like, you could actually try it with a cheap model, use an extremely cheap judge, and if it's sufficient, keep it.

3. 28:09

   If not, try a larger model. Or, you know, try multiple times and pick the best. Like, there's so many things you can do. But when you have, like, this stateful KV cache, you are, like, really constrained to it being kind of append only, and you have this complicated trade-off, kind of like an iPython notebook, where you can never delete the memory because it's, like, a really weird, you know agglomerated state and it turns out that it's very hard to engineer around this because the models are trained with it so if you actually start doing fancy stateful operations with the models having free reign over that state they'll end up looping and stuff like that because they're so tuned to having that kvcache and so efficient to have that kvcache so this prevents you from doing like routing effectively it prevents sub-agents from being really effective

4. 28:53

   because now you have this weird trade-off of like what state what context do i pass to my sub-agent because this requires intelligence in order to know what to pass in. If you pass it all in, this is a huge input token cost instead of a cash read cost. And when your subagent makes an output, how do you know what to pass in? And like what subset is relevant down the line? And the safest thing to do tends to be to append a lot of stuff.

5. 29:15

   And now this is the equivalent of like from a state management perspective. Imagine your state is like global variables. It's passing one giant dictionary around forever until the models break and they start getting dumber over time because context fraud is a real thing. And this is totally cool, just to be clear. I don't believe that there's anything super tragic about this on its own because it's very useful.

6. 29:37

   I think what's tragic is that's the only way to do it. And I think there's so many more advanced patterns that you could do if you could assume that I could just put intelligence all over these operations. And that is the kind of thing to me that I think from an engineering, architecture, design, beauty, power perspective is so limiting. And that is what that tyranny is, because the people who have tried to do those cool algorithms have mostly failed because they've tried to use LLMs for it.

7. 30:05

   And LLMs are not optimized for, you know, routing logic within the software or making discrete little decisions. They're optimized for making human pleasing outputs that are not super reliable running as a sub process. And this then makes people believe this doesn't work. And actually, a lot of what our original shape looks like, it's not too dissimilar from LangChain. There's nothing wrong with LangChain's interface, right?

8. 30:32

   Some would even say that it was maybe perfect for classifiers on top of LMs. I think, I can't remember back in the day, but I'm pretty darn sure it's going to be pretty darn good if someone made a LangChain-compatible wrapper on top of our model. The problem is people didn't like it because it made models somewhat dumber because the models are not optimized for that type of pattern. And this is also a form of tyranny where everyone just thinks Anthropic and OpenAI or whoever else is like number three right now is like doing their best and giving us the best intelligence possible when the reality is there's so many other intelligences possible.

9. 31:09

   And like there's trade-offs there, you know, like there's truly a lot of trade-offs that are made in this design that people can feel but cannot really measure because it's really hard to do so. What's like an example of the shape of what a coding agent could look like, I guess, with like a different intelligence design? One feature that I think would be really sick is what if you could have like multiple agents sharing the same context?

10. 31:35

    And you're like truly like read writing from the same context. That sounds really complicated. Like there's no way that works right now because of the KVCache. But like nothing stops that from happening. And actually a very common pattern people have with agents these days is like they're just cooking and crazy stuff they don't know what's going on and after it does the thing you asked it to do people are like make a visualizer or a dashboard or a learning tool and it's kind of crazy that that has to be like an extension of the content That has all of the trial and error in it. Or starting from scratch, looking at a git commit. Why can't and rereading everything from scratch? Why can't it share the state? And why can't you have like an agent here? That's actually basically read only for the code base, but allows you to write the visualizer, such that as the agent is cooking, you actually get live visualizations of maybe, you know, using like a slightly dumber model, like Sonnet instead of Opus to write out what Opus is doing. So you kind of get a sense of what's going on.

11. 32:29

    instead of having to read the gibbetsing updates or whatever else. I think that would be cool. Another example here is compaction. Compaction makes a lot of sense for a write-once kvcache that is append-only because, like, oh, it's full. We need to restart it. But compaction is a form of compression, and compression is just not totally solvable. Like, it's very hard to compress without priors for compression.

12. 32:56

    But what happens if you compress based on what problem you were solving before. And now you want to actually ask a problem that is different from that one. And it would use information that was compressed away. Like that agent is like Delta, very much losing hand. But what if how the state gets compressed was actually semantic and it could detect, hey, I'm doing a new task. I should actually think about what the context should be for this new task using an ROI calculation, which should be in code.

13. 33:22

    The code should compute expected value of like cash reads and cash writes and everything and recalculate perhaps a new KVCache for this other task and maybe even keep the other one hot. I don't really know. This sounds like cool sci-fi. One more, one more, then I will stop on this topic. But I'm just so excited about this. One thing that I think would be really cool is, hey, I keep cooking on the same code base and I keep reading the same files and having to re-understand my system's abstractions over and over and over again.

14. 33:51

    You know, there's a partial solution to this, which is like maybe the agent should write markdown files, but then how do you keep it in sync blah blah blah you know it's a kind of a pain you know but what if you could look at the history of old models but like not as like a like i feel like right now it's like a little bit of a hack like oh it's a tool that you can use sometimes if the model deems it as necessary but what if like by its nature one of the first steps you do is even a extremely cheap map reduce over all history to find the relevant stuff here and what if that history could also be shared with your co-workers who are using the same code base so you can actually have insights from what they're working on like these are very doable software problems right the problem is that it's like very unfriendly with a kv cache when all the

15. 34:36

    intelligence needs to be append only and there's there's so many other cool things i love it um yeah the creative juices uh flowing i i guess you know another thing i'd be curious about is