Context buckets keep huge histories searchable

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 27:50

   breakthroughs we had and this is one of those things that i'm frankly expecting someone will will publish i guess rlms was like adjacent to that but it's it's We call them context buckets. And the way it started was, if an agent calls an action that returns too much context, like some actions, some MCPs in particular, retrieve like 100,000 tokens or something, okay? You don't want to send that to the agent who's going to get confused, okay?

2. 28:20

   So what you do is that you expose that as a summary of the context bucket. And it's a subagent which contains the entire bucket. It's like, hey, this action returned too much. So I'm here, you know, to stand in for what they actually return. Roughly speaking, this is what it contains, okay? And then the agent can enter into a conversation with the sub-agent, which itself has its own caching, and can manipulate the context using Unix utilities.

3. 28:45

   So right here, you're saving a lot. You're saving a lot of money, and it goes quite fast. Then we went one step further, and we were like, hey, what if we could have recursive context packets? What if context packets could contain other context packets? And what if compaction, because obviously we have compaction, was powered by such recursive context packets? And by that, I mean, like, now when we compact, so the conversation goes, it passes the threshold.

4. 29:12

   I think right now it's like 200,000 tokens, but we keep tweaking it. And at some point, we're like, okay, we're going to compact. We compact. And then the compaction sends all of that stuff into a context packet. So now the agent can query that context packet. So it's not like every compaction is always lossy by nature, and I think compaction operates under the faulty assumption that you never need access to ground truth, which is false.

5. 29:36

   You at some point do need access to ground truth. So with that technique, you have access to ground truth. And then we went a step further, which is, so you have that context packet, which is the compaction of the previous conversation. The conversation keeps going, keeps going, keeps going. You need to summarize again. What you do is that you take all of that, including this context packet right here, and you compact this into a new context bucket.

6. 29:59

   So now you have a context bucket containing a context bucket. And in the end, so you do that and so you have this emerging property where it's like you can have the agent access any point of any infinite number of tokens at arbitrary levels of granularity. Okay? The problem is that if you have a computer science background like this gives you like what's called like ON complexity. Right? So it's like if it wants to access like nine context buckets ago, it's got to go through nine layers of sub-agents and that's really slow and really expensive.

7. 30:28

   And so what we ended up doing is maybe it's getting too technical. Never. Okay, well, have you heard of AVL trees? No, but I'm all ears. Have you heard of red black trees? There is this... I went to the computer science school of hard knocks, so it wasn't formal training for me. I was actually glad to have AVL and black trees because I was like, oh my God, you guys remember your training? You know, this is the moment why don't we use those specking things?

8. 30:58

   Because everybody in school is always like, when do you really use those specking things? Like, we're using AVL and Black Trees, baby. Red Black Trees, they're called. So what is described, if you just do the naive implementation, you get that really nice emerging property for free, which is you have all of those linked context buckets that contain one another. But every context bucket always contains just one other context bucket.

9. 31:19

   And so you end up with like this Russian doll of sorts of context buckets. And like, if you want to open, if you want to get to the bottom, it takes a very long time. So what you do instead is that you have context packets contain multiple context packets, okay? And so you end up with a tree because basically what you want is you want the topmost context packet not to contain the last 100. You want it to contain the first context packet and the last context packet, right?

10. 31:50

    And so you end up with a self-balancing tree. There's these two algorithms. There's the AVL tree, and then there's the red-black tree, which are algorithms that are used to balance a tree. Having one long line, you want to minimize the height of the tree such that going to the bottom takes as few jumps as possible. Technically, AVL is the demonstrably best way to balance a tree because it leads to the lowest height.

11. 32:14

    A red-black tree is better because it takes into account the cost to balance the tree, which is expensive because you need to regenerate a lot of your context buckets and you miss your cash when you do that. It's very expensive. So red-black trees, the canonical implementation of red-black trees is on binary trees, which is just a tree where each node's got two children. We went for, it's called a centenary tree.

12. 32:35

    It's literally just each node's got 100 nodes. So the node below that's going to have like 10,000 nodes, right? And so literally with two jumps, you can have 10,000 context buckets, and you can access like all the contexts in the universe. And that leads to some really surprising behaviors because you're literally no more than two LLM calls away from being able to access 10,000 context buckets. each one of which contains 200,000 tokens, right?

13. 33:02

    So you're at like 2 billion tokens here of context in two LLM calls. And that is what leads to like those really surprising behaviors from those AI agents where you ask them any question and they remember everything perfectly all the time. And that's awesome. That was one really big thing we had to figure out. I hope, please, you know, like copy us. Like I will just, we just don't have the time to publish, but you know, this is, we don't mean for this to be secret.

14. 33:28

    Like, I think this is a really powerful technique, and I have been surprised to not see more communication about it.