Pretraining creates digital agents, not evolved animals

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 8:23

   So why shouldn't that be the vision for AI rather than like this thing where we're doing millions of years of training? I think that's a really good question. And I think, I mean, so Sutton was on your podcast and I saw the podcast and I had a write-up about that podcast almost that gets into a little bit of how I see things. And I kind of feel like I'm very careful to make analogies to animals because they came about by a very different optimization process.

2. 8:47

   Animals. are evolved, and they actually come with a huge amount of hardware that's built in. And when, for example, my example in the post was the zebra, a zebra gets born, and a few minutes later, it's running around and following its mother. That's an extremely complicated thing to do. That's not reinforcement learning. That's something that's baked in. And evolution, obviously, has some way of encoding the weights of our neural nets in ATCGs.

3. 9:08

   And I have no idea how that works, but it apparently works. So I kind of feel like brains just came from a very different process. and I'm very hesitant to take inspiration from it because we're not actually running that process. So in my post, I kind of said, we're not actually building animals. We're building ghosts or spirits or whatever people want to call it because we're not doing training by evolution.

4. 9:33

   We're doing training by basically imitation of humans and the data that they've put on the internet. And so you end up with these like sort of ethereal spirit entities because they're fully digital and they're kind of like mimicking humans. And it's a different kind of intelligence. Like if you imagine a space of intelligences, we're starting off at a different point almost. We're not really building animals, but I think it's also possible to make them a bit more animal-like over time.

5. 9:54

   And I think we should be doing that. And so I kind of feel like, sorry, just I guess one more point is, I do feel like Sutton basically has a very, like his framework is like we want to build animals. And I actually think that would be wonderful. If we can get that to work, that would be amazing. If there was a single like algorithm that you can just, you know, run on the internet and it learns everything, that would be incredible.

6. 10:14

   I almost suspect that I'm not actually sure that it exists. And that's certainly actually not what animals do because animals have this outer loop of evolution. And a lot of what looks like learning is actually a lot more maturation of the brain. And I think there's actually very little reinforcement learning for animals. And I think a lot of the reinforcement learning is actually more like motor tasks.

7. 10:34

   It's not intelligence tasks. So I actually kind of think humans don't actually really use RL, roughly speaking, is what I would say. Can you repeat the last sentence? A lot of that intelligence is not motor tasks. It's what, sorry? A lot of the reinforcement learning, in my perspective, would be things that are a lot more like motor-like, like simple kind of like tasks, throwing a hoop, something like that.

8. 10:52

   But I don't think that humans use reinforcement learning for a lot of intelligence tasks like problem solving and so on. Interesting. That doesn't mean we shouldn't do that for research, but I just feel like that's what animals do or don't. I'm going to take a second to digest that because there's a lot of different ideas. Maybe one clarifying question I can ask to understand the perspective. So I think you suggest that, look, evolution is doing the kind of thing that pre-training does in the sense of building something which can then understand the world.

9. 11:23

   The difference, I guess, is that evolution has to be titrated, in the case of humans, through three gigabytes of DNA. And so that's very unlike the weights of a model. I mean, literally, the weights of the model are a brain, which obviously is not encoded in the sperm and the egg, or does not exist in the sperm and the egg. So it has to be grown. And also, the information for every single synapse in the brain simply cannot exist in the three gigabytes that exist in the DNA.

10. 11:52

    Evolution seems closer to finding the algorithm, which then does the lifetime learning. Now, maybe the lifetime learning is not analogous to RL, to your point. Is that compatible with the thing you were saying, or would you disagree with that? I think so. So I would agree with you that there's some miraculous compression going on because obviously the weights of the neural net are not stored in ATCGs.

11. 12:11

    There's some kind of a dramatic compression and there's some kind of like learning algorithms encoded that take over and do some of the learning online. So I definitely agree with you on that. Basically, I would say I'm a lot more kind of like practically minded. I don't come at it from the perspective of like, let's build animals. I come from the perspective of like, let's build useful things. So I have a hard hat on and I'm just observing that, look, we're not going to do evolution because I don't know how to do that.

12. 12:34

    but it does turn out we can build these ghost spirit-like entities by imitating internet documents this works and it's actually kind of like it's a way to bring you up to something that has a lot of sort of built-in knowledge and intelligence in some way similar to maybe what evolution has done. So that's why I kind of call pre-training this kind of like crappy evolution. It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.

13. 13:02

    Just to steel man the other perspective, because after doing this interview and thinking about it a bit, here's an important point here. Evolution does not give us the knowledge really, right? It gives us the algorithm to find the knowledge. And that seems different from pre-training. So if perhaps the perspective is that pre-training helps build the kind of entity which can learn better. It teaches meta-learning and therefore it is similar to like finding an algorithm.

14. 13:25

    But if it's like evolution gives us knowledge and pre-training gives us knowledge, that analogy seems to break down. So it's subtle and I think you're right to push back on it. But basically, the thing that pre-training is doing, so you're basically getting the next token predictor over the internet and you're training that into a neural net. It's doing two things actually that are kind of like unrelated.

15. 13:43

    Number one, it's picking up all this knowledge as I call it. Number two, it's actually becoming intelligent. By observing the algorithmic patterns in the internet, it actually kind of like boots up all these like little circuits and algorithms inside the neural net to do things like in-context learning and all this kind of stuff. And actually, you don't actually need or want the knowledge. I actually think that's probably actually holding back the neural networks overall because it's actually like getting them to rely on the knowledge a little too much sometimes.

16. 14:06

    For example, I kind of feel like agents, one thing they're not very good at is going off the data manifold of what exists on the internet. If they had less knowledge or less memory, Actually, maybe they would be better. And so what I think we have to do kind of going forward, and this would be part of the research paradigms, is I actually think we need to start, we need to figure out ways to remove some of the knowledge and to keep what I call this cognitive core.

17. 14:28

    It's this like intelligent entity that is kind of stripped from knowledge, but contains the algorithms and contains the magic, you know, of intelligence and problem solving and the strategies of it and all this kind of stuff. There's so much interesting stuff there. Okay, so let's start with in-context learning. This is an obvious point,