A novel virus is the extinction route he can picture

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 37:12

   in the near term you talk about extinction being a real risk human extinction being a real risk of all the the pathways to human extinction that you think are most likely what what is the leading pathway because i know you talk about there being some issue pre-deployment of these ai tools like you know someone makes a mistake um when they're designing a model or other issues post-deployment when i say post-deployment i mean once a chat tpt or something an agent's released into the world and someone hacking into it and changing it and reprogramming it to be malicious.

2. 37:53

   Of all these potential paths to human extinction, which one do you think is the highest probability? So I can only talk about the ones I can predict myself. So I can predict even before we get to superintelligence, someone will create a very advanced biological tool, create a novel virus, and that virus gets everyone or most everyone. I can envision it. I can understand the pathway. I can say that.

3. 38:18

   So just to zoom in on that, then, that would be using an AI to make a virus and then releasing it. And would that be intentional? There is a lot of psychopaths, a lot of terrorists, a lot of doomsday cults. We've seen historically, again, they tried to kill as many people as they can. They usually fail. They kill hundreds of thousands. But if they get technology to kill millions of billions, they would do that, gladly.

4. 38:46

   the point i'm trying to emphasize is that it doesn't matter what i can come up with i am not a malevolent actor you're trying to defeat here it's the super intelligence which can come up with completely novel ways of doing it again you brought up example of your dog your dog cannot understand all the ways you can take it out it can maybe think you'll bite it to death or something but that's all whereas you have infinite supply of resources.

5. 39:16

   So if I asked your dog exactly how you're going to take it out, it would not give you a meaningful answer. It can talk about biting. And this is what we know. We know viruses. We experienced viruses, we can talk about them. But what an AI system capable of doing novel physics research can come up with is beyond me. One of the things that I think most people don't understand is how little we understand about how these AIs are actually working.

6. 39:52

   Because one would assume, you know, with computers, we kind of understand how a computer works. We know that it's doing this and then this and it's running on code. but from reading your work you describe it as being a black box we actually say in the context of something like chat gpt or an ai we know you're telling me that the people that have built that don't actually know what's going on inside there that's exactly right so even people making those systems have to run experiments on their product to learn what it's capable of so they train it by giving it all of data, let's say all of internet text.

7. 40:28

   They run it on a lot of computers to learn patterns in that text. And then we start experimenting with that model. Oh, do you speak French? Oh, can you do mathematics? Oh, are you lying to me now? And so maybe it takes a year to train it and then six months to get some fundamentals about what it's capable of, some safety overhead. But we still discover new capabilities and old models if you ask the question in a different way it becomes smarter so it's no longer engineering how it was the first 50 years where someone was a knowledge engineer programming an expert system ai to do specific things it's a science we are creating this artifact growing it it's like a alien plant and then we study to see what it's doing and just like with plants we don't have 100 accurate knowledge of biology we don't have full knowledge here we kind of know some

8. 41:26

   patterns we know okay if we add more compute it gets smarter most of the time but nobody can tell you precisely what the outcome is going to be given a set of inputs i've watched so many