AI self-play works best when success can be objectively tested

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 2:05:07

   there's a race towards getting to superintelligence. And the first company to get to superintelligence is going to reap massive amounts of rewards. How far away do you think we are from achieving that? Well, you know, like I said, like my philosophy tends to be different than I think the mainstream in Silicon Valley. I think that AI is going to be extremely good at, you know, doing labor, extremely good at, and like, you know, chat GPT and being like a personal assistant, extremely good at like, you know, like Replit being an automated programmer.

2. 2:05:43

   But the definition of superintelligence is that it is better than every other human collectively at any task. And I am not sure there's evidence that we're headed there. Again, I think that one important aspect of superintelligence or AGI is that you drop this entity into an environment where it has no idea about that environment, it's never seen it before, and it's able to efficiently learn to achieve goals within that environment.

3. 2:06:23

   Right now, there's a bunch of studies showing, like, you know, GPT-4 or any of the latest models. If you give them an exam or a quiz, that is slightly, even slightly different than their training data, they tank. They do really badly on it. I think the way that AI will continue to get better is via data. Now, at some point, and maybe this is the point of takeoff, is that they can train themselves. And the way we know how AI could train itself through a method called self-play. So the way self-play works is, you know, take, for example, AlphaGo. AlphaGo is, I'm sure you remember Lisa Dole, a game between DeepMind AlphaGo and Lisa Dole, and it won in the game of Go.

4. 2:07:18

   The way AlphaGo is trained is that part of it is a neural network that's trained on existing data, but the way it achieves superhuman performance in that one domain is by playing itself like millions, billions, perhaps trillions of times. So it starts by like generating random moves and then it learns what's the best moves and it's like basically a multi-agent system where it learns, I did this move wrong and I need to kind of re-examine it and it trains itself really, really quickly by doing the self-play.

5. 2:07:53

   It'll play fast, fast games with itself. But we know how to make this in game environments because game environments are closed environments. But we don't know how to make it, we don't know how to do self-play, for example, on literature because you need objective truth. In literature, there's no objective truth. uh, taste is, is different. Right. Um, conjecture, uh, philosophy, uh, there's a lot of things.

6. 2:08:27

   And again, this is, I go back to why there's still a primacy of humans is, um, there are a lot of things that are intangible. Uh, and we don't know how to like generate objective truth in order to train machines in the self play fashion. Uh, but like programming is, has objective truth. Coding has objective truth. The machine can, like, you can construct an environment that has a computer and has a problem. There's a ton of problems. And even an AI can generate, you know, sample problems.

7. 2:09:01

   And then there's like a test to validate whether the program works or not. And then you can generate all these programs, test them. And if they succeed, you know, that's a reward that trains your system to get better at that. If it doesn't succeed, you know, that's also feedback. And they run them all the time and it gets better at programs. So I'm confident that program is going to get a lot better.

8. 2:09:26

   I'm confident that math is going to get a lot better. But from there, it is hard to imagine how all these other more subjective, soft, softer sort of sciences, the AI will get better through self-play. I think the AI will only be able to get better through data from human labor.