Context windows can cover some continual-learning gaps

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 11:41

   For instance, continual learning. We had Dwarkech on a couple weeks ago. Here's how he put it, and he wrote about it in his Substack. The lack of continual learning is a huge, huge problem. The LLM baseline at many tasks might be higher than an average human, but you're stuck with the abilities you get out of the box. So you just make the model, and that's it. It doesn't learn. That seems like a glaring liability.

2. 12:01

   What do you think about that? So first of all, I would say even if we never solved continual learning, Um, even if we never solve continual learning and memory, um, I think that the potential for the LLMs to do, you know, incredibly, uh, incredibly well to, you know, affect things at the scale of the economy will be very high, right? If I think of the field I used to be in, biology and medicine, like, you know, let's say I had a very smart Nobel prize winner and, you know, I, I, I said, okay, you know, you're, you know, you've, you've, you've discovered all these things.

3. 12:34

   You have this incredibly smart mind, but, you know, you can't, you know, you can't, like, read new textbooks or absorb any new information. I mean, that would be difficult, but, like, still, if you had, like, 10 million of those, like, they're still going to make a lot of biology breakthroughs. Like, they're going to be limited. They're going to be able to do some things humans can't, and there are some things humans can do that they can't.

4. 12:53

   But even that, even if we impose that as a ceiling, like, man, that's pretty damned impressive and transformative. And even if I said you never solved that, like, I think, you know, I think people are underestimating the impact. But look, context windows are getting longer, and models actually do learn during the context window, right? So as I, you know, talk to the model during the context window, I have a conversation.

5. 13:16

   It absorbs information. The underlying weight of the model may not change. But, you know, just like I'm talking to you here, and we're having a conversation, and I listen to the things you say, and I think, and I respond to them, the models are able to do that. And from a machine learning perspective, from an AI perspective, there's no reason we can't make the- context length 100 million words today, right, which is roughly what a human hears in their lifetime. There's no reason that we can't do that. It's really inference support. And so again, even that fills in many of the gaps, not all the gaps, but it fills in many of the gaps.

6. 13:58

   And then there are a number of things like learning and memory that do allow us to update the weights. So, you know, there are a number of things around, you know, types of reinforcement learning, learning training, you know, you know, we used to many years ago talk about inner loops and outer loops, right? The inner loop is like, I have some episode and I learned some things in that episode, and I'm trying to optimize for the lifetime of that episode. And kind of the outer loop is the agents learning over episodes. And so I think maybe that inner loop, outer loop structure as a way to learn the continual learning.

7. 14:32

   One thing we've learned in AI is whenever it feels like there's some fundamental obstacle, like two years ago, we thought there was this fundamental obstacle around reasoning. Turned out just to be RL. You just train with RL and you let the model write some stuff down. You know, you let the model write things down to try and figure out objective math problems without being too specific. You know, I think, and we already have maybe some evidence to suggest that this is another of those problems that is not as difficult as it seems, that will fall to scale plus a slightly different way of thinking about things. Do you think your obsession with scale might blind you to some of the new