Showing powerful AI could change how people respond

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 56:15

   my thinking has been changing is that I now place more importance on AI being deployed incrementally and in advance. One very difficult thing about AI is that we are talking about systems that don't yet exist. And it's hard to imagine them. I think that one of the things that's happening is that in practice, it's very hard to feel the AGI. It's very hard to feel the AGI. We can talk about it, but it's like it's like talking about like the long future like imagine like having a conversation about like how is it like to be old when you're like old and and frail and you can have a conversation you can try to imagine it but it's just hard and you come back to reality well that's not the case and i think that a lot of the issues around agi and its future power stem from the fact that it's very difficult to imagine future ai is going to be different

2. 57:35

   different it's going to be powerful indeed the whole problem what is the problem of ai and agi the whole problem is the power. The whole problem is the power. When the power is really big, what's going to happen? And one of the ways in which I've changed my mind over the past year, and so that change of mind may back, may, I'll say, I'll hedge a little bit, may back propagate into the plans of our company, is that so if it's hard to imagine what do you do?

3. 58:14

   You got to be showing the thing. You got to be showing the thing. And I maintain that I think most people who work on AI also can't imagine it because it's too different from what people see on a day-to-day basis. I do maintain here's something which I predict will happen. That's a prediction. I maintain that as AI becomes more powerful, then people will change their behaviors. And we will see all kinds of unprecedented things which are not happening right now.

4. 58:53

   And I'll give some examples. I think for better or worse, the frontier companies will play a very important role in what happens, as will the government. and the kind of things that I think you'll see, which you see the beginnings of, companies that are fierce competitors starting to collaborate on AI safety. You may have seen OpenAI and Anthropic doing a first small step, but that did not exist. That's actually something which I predicted in one of my talks about three years ago, that such a thing will happen.

5. 59:30

   I also maintain that as AI continues to become more powerful, more visibly powerful. There will also be a desire from governments and the public to do something. And I think that this is a very important force of showing the AI. That's number one. Number two, okay, so then the AI is being built. What needs to be done? So one thing that I maintain that will happen is that right now people who are working on AI I maintain that the AI doesn't feel powerful because of its mistakes. I do think that at some point the AI will start to feel powerful, actually. And I think when that happens, we will see a big change in the way all AI companies approach safety. They'll become much more paranoid.

6. 1:00:23

   I think, I say this as a prediction that we will see happen. We'll see if I'm right. But I think this is something that will happen because they will see the AI becoming more powerful. Everything that's happening right now, I maintain, is because people look at today's AI and it's hard to imagine the future AI. And there is a third thing which needs to happen. And I think this is, and I'm talking about it in broader terms, not just from the perspective of SSI, because you asked me about our company. But the question is, okay, so then what should the companies aspire to build yeah what should they aspire to build and there has been one big idea that actually every that everyone has been locked in locked into which is the the self-improving ai and why why did it happen because there is fewer ideas than companies but i maintain that there is

7. 1:01:17

   something that's better to build and i think that everyone will actually want that it's like the AI that's robustly aligned to care about sentient life specifically. I think in particular, there's a case to be made that it will be easier to build an AI that cares about sentient life than an AI that cares about human life alone, because the AI itself will be sentient. And if you think about things like mirror neurons and human empathy for animals, which is, you know, you might argue it's not big enough, but it exists.

8. 1:01:55

   I think it's an emergent property from the fact that we model others with the same circuit that we used to model ourselves, because that's the most efficient thing to do. So even if you got an AI to hear about sentient beings, and it's not actually clear to me that that's what you should try to do if you solve the alignment, it would still be the case that most sentient beings will be AIs. there will be trillions eventually quadrillions of AIs humans will be a very small fraction of sentient beings so it's not clear to me if the goal is some kind of human control over this future civilization that this is the best criterion it's true I think that it's possible it's not the best criterion I'll say two things I think that thing number one I think that if there is so I think that care for sentient life I think there is merit to it

9. 1:02:55

   I think it should be considered I think that it will be helpful if there was some kind of a short list of ideas that then the companies when they are in this situation could use that's number two. Number three, I think it would be really materially helpful if the power of the most powerful super intelligence was somehow capped, because it would address a lot of these concerns. The question of how to do it, I'm not sure. But I think that would be materially helpful when you're talking about really, really powerful systems.

10. 1:03:34

    Yeah. Before we continue the alignment discussion, I want to double click on that.