Dropping human data let AlphaZero beat two years of work in hours

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 27:25

   is a really important goal. I'd love to talk more about this concept of removing human priors. And I think your experience with AlphaGo and then AlphaGo Zero is maybe the best example that anyone can point to of what happens when you remove human priors. Maybe if you take us back to AlphaGo and then AlphaZero, was it surprising to you that removing human priors would result in even smarter intelligence?

2. 27:54

   I wasn't sure, honestly. So I remember, I remember, I always like to do things in research where I think you ask research questions, which are like on a knife edge where it's like 50-50. I think it's not very interesting to do research where you already know the answer and you're like 95% certain this thing's going to work. It's kind of like boring, honestly, to do those things. So, you know, I think research, good research is about asking questions where you're really not sure what the answer is in advance.

3. 28:23

   And in this case, I was about 50-50. And I thought that maybe there was a good chance that it might. But I wasn't confident. And it was very exciting to see it. I should say that you asked about human priors. So we are making a bet, but ineffable. And the bet we're making is that we can go further by giving up on the priors that everyone's relying on today. Like the human data, like the whole internet of knowledge that everyone else is building their systems upon.

4. 28:50

   that's a uh uh an important choice and so we should probably talk about like why we're doing that and i think the reason for me of course it's you know a short-term sacrifice like we we give up a certain amount of performance um in order to access something which is potentially far greater and it comes back to this like you know what does it mean to build a super learner like to me building a super learner again is about building something that that has this like potential for for full depth and breadth. So in the depth direction, we need to get to something which is capable of discovering things which goes so far beyond human knowledge that it goes just far beyond the prior. And the more we lean on human data, the less able we are to actually discover the algorithms that are really going to take us all the way to something that's going to,

5. 29:37

   for example, discover a whole new field of knowledge. And then in the breadth direction, we need to have algorithms that can be placed down and discover things in environments for which language and human data that we started with is just not a good fit. I mean, there are just many, many areas, different kinds of sensors, for example, where human language data or human data in general is not a good fit for some of this description. And so I think by letting go of of this human data, we put ourselves in a position where we can actually discover something which can go far, far beyond it.

6. 30:15

   And so if we come back to the AlphaGo example, what we found there was that the moment we let go of the human data, which we'd spent, by the way, we'd spent about two years building AlphaGo, somewhat similar to the LLM approach, where we built every level of this. We kind of hand put together the next level. We kind of put in a lot of human data and then we were kind of human in the loop, like improving it, where we would kind of take the previous generation and choose how we were going to improve it to give the next generation.

7. 30:44

   And so we were right in the loop of this process. It took us two years to get to AlphaGo and beat LisaDol. When we got AlphaZero right, you know, we gave up on the human data and finally focused on what would it be like to build an algorithm that could discover for itself. So once we got it right, that algorithm, it took just a few hours. like it took a few hours to like outperform the two years of work which we'd done as with human in the loop um so in the in the game of chess it took four hours to beat the world champion the best ever ai that had ever been created in the game of go it took um you know like uh i think it was a couple of days or something but it was incredible progress like and and and so you have to let go of the human data to allow yourself to discover these things that can then

8. 31:27

   shoot past the competition and discover the things that just keep learning and learning and learning forever. So, you know, it has to be an active choice. And if you don't make that active choice, you know, what I see in a lot of the kind of LLM companies today is they're basically, you know, building semi-autonomous, human in the loop kind of learning algorithms with thousands of people in the loop of the learning progress, which is, you know, it works and a lot of respect for how far it gets.

9. 31:53

   But, you know, until we step back from that and really discover the process that's going to learn for itself i don't believe we'll get all the way to something which is truly a super learner