Computing power keeps accelerating across hardware and software

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 20:05

   But, I mean, I've given you a few charts. U.S. personal income. Yeah, let's go ahead and put on the charts one second for those on YouTube, which is most of our audience. Let's take a look at these charts. I mean, in constant dollars, the amount of money per person that a person in the United States makes is 10 times what it was 100 years ago um in constant dollars so we see an x literally an exponential curve between 1774 um up through 2022 and this is just a meteoric rise and the important thing is that these are in constant year dollars they're not inflated dollars yeah um and all this is driven by the exponential increase in computation if you actually that's my most important chart and this is plotted on a log scale which means a straight line each level is 10 times the level below it so this started with the zusa computer he was a german

2. 21:14

   in 1939. It was shown to Hitler. He was not actually in support of Hitler, but it got shown to him. Hitler had no interest in computation. The third computer on this chart is Turing's computer, which was shown to the leader of the Allied forces. We use it to decode the Nazi messages. Anyway, this is a straight line. It means exponential growth. And it started at 0.00007 calculations per second per constant dollar.

3. 21:59

   In 2024, the NVIDIA chip, the B200, is half a trillion calculations per second per constant dollar. So for the same price we can now get 75 quadrillion fold increase in the amount of computation and that's what's driven it. That's why we didn't have large language models in 1931 or even three years ago. We began to have them two years ago. If we compare the things two years ago to today it's remarkable so this constant increase in the amount of computation for the same money uh has driven this ai uh revolution that started two years ago you've you've deemed this the law of accelerating returns moore's law is a segment of it which involves integrated chips it has to do with integrated circuits but this started at relays went to uh tubes discrete transistors integrated circuits and now we've gone beyond integrated circuits and you don't see any

4. 23:06

   variation in this tube it's basically a straight line you don't expect this to slow down or stop no in fact i found this chart 45 years ago and felt it would continue and it has continued exactly as I found it 45 years ago. So I had half of this chart and it's continued onto trillions of calculations per second per constant dollar. So it's pretty amazing. And it's going to continue. And we've made the same progress with software as we have with hardware.

5. 23:42

   So the actual value is the amount of computations we get from software and from hardware both. So it's actually even more expansive than what you see here. Yeah, Elon was saying on our stage last year that he's seeing 100x per year if you include computation and algorithmic efficiencies. Well, that's about what we've done with the kinds of computations we've done with the larger language models. And if we compare what we have today to two years ago, it's something like that.

6. 24:21

   So it's pretty extraordinary. Amazing. I believe that large language models should be called large event models.