Recursive self-improvement does not guarantee runaway growth

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 41:06

   A lot of people are talking about recursive self-improvement. Do you think we're there or what does it mean to you? And does that dramatically change anything when you get there? Yeah, well, I think one thing is like, people talk a lot about recursive self improvement, which is like, you know, the idea that the models get so good, that they start training themselves. And there's this loop where they just get, you know, insanely good. But by definition, recursive self improvement is just getting to a point where the models can literally like quite literally, like improve themselves, but improve themselves is not synonymous with like hyperbolic growth. Like, you could have a model that is recursively self improving, but it's so slow at it, because it's just not as good as humans.

2. 41:43

   And I think that is something that like, you know, sometimes when like the people that are most AGI pill, like AGI, you know, 2027, everything's going to change next year. Sometimes they make jumps as to like, oh, we get RSI and they assume that that RSI is the crazy takeoff RSI. It could be the case that we get to RSI and it's just never as good as human assisted like improvement on these things.

3. 42:07

   Or it could just be really, really slow to actually get to that point. And I think that's the nuance that gets kind of lost. Like right now, these models are making huge contributions to making the models better. But I think the, you know, the hyperbolic growth, I think that is that is not something that we're it doesn't. I haven't seen signs that we're like, you know, humans let go and like unleash it and the AI improving itself.

4. 42:31

   People at the labs, of course, are the most AGI pills because like they are both adopting everything quicker, seeing seeing the most frontier models. But you almost are also bridging to customers that are in like the real world. like banks and all these other giant enterprises and realizing like wow things don't move as fast and like you know bureaucracy is slow and there's all these other truths two questions one what is your AGI timeline and then two what does that actually look like like what happens to the rest of the world like in every other industry when that comes true like how long is that window between oh we've done it in a lab in san francisco and everything has changed at an investment bank or a law firm yeah so i guess agi timeline is a really difficult thing for me just because like

5. 43:18

   i find so much of it is so ill-defined so like what does it mean to you like i i would say we're already there like we're we are post agi right now like we have agi like this is all like we're all doing stuff with agi right now and this what we've seen in the last year this is what life is like post agi which is we get to solve really fucking cool problems that previously we couldn't solve before and i think this is only going to continue i'm like if you can't i'm like very optimistic i think there's a lot of uh work that is kind of on us to do but like it is a very like this is unequivocally going to be a better world and i think generally as these tools get better and as the agi gets smarter as well the scale of ambition that we have for solving problems will also I think the only world in which there's like, there's nothing left for humans to do is if we solve all problems. But my sense is that like, the number of problems that exist is like here. And the number of problems that we're solving is like this.

6. 44:14

   Totally. And it's almost like, as we improve the quality of life, we get to solve different problems. And that's been true throughout human history. And I think the problem in the US isn't necessarily like food scarcity. It's now it's now all these other sort of like higher level. like consider door dash like they solved a great problem which was some people don't have time to go and get this stuff or get their food or groceries if you told like a a human 10 000 years ago about this problem they would be like what that's not a job like that's not a company like what do you mean like just go pick it up right so it's like as the quality of life improves as well new things become problems so for all the problems that we solve to your point that unlocks even more problems that previously we might not have thought about as problems what do you think

7. 44:57

   about the um math conjunctures that have been disproven and like some of the announcements going on of like it's mostly been in math so far uh like using ai to basically look at these these broader theories i think it's incredible it's so fun it must be like it's a very fun time to like work on that stuff it's also funny that now it's probably better if you want to solve math problems to be at an ai company than it is to be in academia which is interesting i also think uh What will it be good at?

8. 45:25

   Because like, I think it seems so far that they've been good at like disproving stuff. Do you think that it will be good in the next few years at actually coming to new discoveries? Yes, I think so. So like right now, it's been really good at, yeah, like the counter examples or like take something from one threshold to a slightly higher threshold, much less good at like defining new areas of research or defining completely new theories.

9. 45:49

   But like what they did with the zeros of the Riemann hypothesis, which are separate podcasts to talk about this. But basically, there's this thing where like, in theory, the theory is proved if you prove a property about 100% of the zeros of the Riemann hypothesis, and remember, the previous threshold was like 47. And like every decade, it would go up by like a couple percent. And they got 20% in one go, which is like unequivocally, like if I'm human on their own did that enormous deal. It is though, very clearly a type of math that is if you ask mathematicians, it's not on the highest order of like beauty like there are certain like beautiful beautiful mathematical proofs and then there's ones that like whoa that's sick you know we got a lot of progress but it's kind of like i don't know what the right yeah it's like very typically like

10. 46:36

    shows counter examples or increases thresholds as opposed to like brand new theory yeah um but it's cool because now we don't need to worry about those problems because these systems are going to get it so we can focus humans on working with ai to build the entirely new theories i want