Scaling laws argue against a fast takeoff, but cyber risk is real

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 55:04

   I think there's so much spend on inference and that in the short term, we are learning how to corral inference into performance gains. And this is a thing that we just had. And this is what we're seeing now is that people are spending so much and all the employees at the Frontier Labs are spending exponential amounts relative to the improvements, but we're figuring out how to do this and there will be a lot of gains.

2. 55:31

   But I don't think that it's necessarily going to be this fast takeoff when the models build themselves. Human bottlenecks. All the scaling laws are exponential increase in compute for linear increase in performance. and that's just like, it's hard for me to bet against this. I do say that we have Navier-Stokes and I'm of the opinion that math is exceptionally well-studied. Maybe the scientific field that is best suited to this shape of intelligence.

3. 56:04

   And intelligence can take many different forms. And that's not to say that if they do, if an AI model does say discover something that's very different than the transformer that gives very different properties and unlocks a new type of intelligence. Like it is worth updating on this and worth being like, okay, like that is different. But until that happens, I'm like pretty happy with my evidence. The companies are going to do well.

4. 56:29

   They're spending exponentially more to keep this remarkable pace of progress. Some of the Claude system cards say this. They're like, it would take say like 10x more progress to make this a fast takeoff. And it's like for now, they're helping us sustain the pace of progress. and I think it's a good equilibrium but I do think that the labs have people saying this the science is like it's hard to overcome that and it's okay but it's like we don't know exactly what'll go from here.

5. 56:59

   Yeah, I think that's comforting because we know what the inputs and levers are here. There is a very successful autoregressive regime that we've been optimizing and that one just asks for a certain amount of exponential resources which are clearly capped And therefore, open source hopefully keeps it strong as well. It just wouldn't lag behind. There is a dramatic amount of work to do in preparing for infrastructure.

6. 57:23

   There's the AI capability plot, which doesn't necessarily worry me. But the AI capability plot in the pretty much non-existing preparation across public cybersecurity and all of these things, there needs to be more preparation in the assumption that this is coming than there is. And then that's kind of the thing that worries me more than, say, like RSI is coming and unlocking new risks. Like there are risks that are here and that are going to continue to diffuse.

7. 57:51

   And many times I've said, the closed labs get there and then the open labs get there. It's like, we're in this process for a lot of things that are going to cause some problems. It's just like figuring out how to get society to prepare is very hard. And I come of the opinion that we need more eyes, we need more science so that society feels informed. But there are other valid opinions. And like, I do think that not fearing fast takeoff does not mean that you don't worry about risks.

8. 58:19

   I don't think you should. No, I think that's completely... Yeah, maybe I'll send in some people your way that are defected from the labs or are worried precisely about this. Well, thank you so much, Nathan. This was a lot of fun.