Exponential compute growth put AGI on a predictable timeline

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 12:08

   And what I was predicting was AGI would happen by 2029. The singularity which really represents a million fold increase would happen by 2045. Now, already the AGI is significantly greater than humans like yesterday. I gave it a book to read, to summarize it, to answer a question. It did that in 40 seconds. Now, humans can't do that in 40 seconds. Already, it's 100 times faster, but not a million times.

2. 12:47

   But AGI will happen by 2029. And I also figured that people would have slightly different definitions of AGI. So there'd be a three year period where people would predict AGI is here and that would start three years earlier, like 2026. Yeah. And indeed we're having people predicting AGI's here already going through 2029 when we really will be very confident that AGI is here. Yeah. So that that that was my prediction back in 1999 actually yeah yeah the thing that really blows my mind about those predictions aside from AGI being pretty much right on the number predicted what 25 30 years in advance in 1999 in 1999 to to I guess 2029 yes 30 years in advance right on the number it's just nutty but a lot of those predictions are based on the on compute and the and the availability of compute and Moore's law continuing and then somewhere in there carbon

3. 13:53

   nanotubes or some future compute substrate needs to exist but what we've done instead is just hammer the transistor on silicon to death and stretch it into massive data centers and stayed right on your curve yeah well if i can show my curve yeah uh sure can we go ahead to the end of the yeah No, not this one. Let me back up a second. So we got, here we go, this one here. Yeah. I mean, in 1939, we were actually increasing relay-based computers.

4. 14:30

   And the exponential growth of relay-based computers is the same as it is today. And NVIDIA and other people are not looking back. it's we'll want to match the exponential growth of relay based 70 years ago yeah but they are the exponential growth has been pretty much the same this is basically a straight line um and this is Increase of 75 quadrillion, so that's 75,000 trillion fold increase in the hardware.

5. 15:08

   We're also making advances in the software. Conservative estimates, we've made about a million fold increase over the 70 years. So the overall increase in computation is equal to the hardware times the software. So we've made a 75,000 million trillion fold increase over this 75 years. That's why we didn't have large language models 70 years ago, or even three years ago, that were effective. It's actually, large language models have only been effective for the last six months.

6. 15:54

   like a year ago really wasn't usable you know what's amazing to me ray is the the raging debate going all the way back to sort of 1980s 1990s when i was working on ai raging debate on whether a parameter in a neural net has anything to do with a synapse in a brain because you could count the synapses you could count the nerve cells and you could you know say okay there's what 300 billion or so and then you count the roughly 100 trillion snaps yeah and about 10 000 10 10 000 to 10 000 synapses per cell uh so you can you can center on about 100 trillion synapses might make a human brain but then there's all this debate on well well a synapse could have anything from quantum effects going on it could be like it could take entire years to simulate who knows it's also going to land on pretty much exact within an order of magnitude exact parity

7. 16:48

   a synapse a parameter in an artificial neural net the iq coming out the other side is one for one no reason to believe that weapon the algorithm is quite different uh the rate uh at which a synapse will form in the human brain is about 200 calculations per second yeah it's very very slow very slow but every every synapse is operating simultaneously so it's massive parallelism so i mean uh back when computers actually did one thing at a time i predicted that we really need to increase the power Parallelism, which we've done, but not we had.

8. 17:34

   We don't have every single synapse happening at the same time, but we have maybe a million to one parallelism. And that's given us the the power that we have today. Yeah. Yeah. You saw it coming. Stephen, do you want to lob a question for right here?