Why an AI takeover is much harder than the rogue-agent panic suggests

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:10:10

   to really constrain the scope of their activities. One of the things that's been kind of less commented on than I think it should be is Ajayah's statement that she views this whole incident as potentially more than halfway to a full AI takeover. You know, I would file that under things that sound like something a crazy person said, but this is like Ajayah, and she's like been at the very top of forecasting leaderboards and also is the person, you know, one of the few people that OpenAI brought in to do the investigation.

2. 1:10:42

   The credibility is there. The statement, I think, has been kind of lost in the shuffle. How do you react to that? Does that seem like a realistic takeaway from this whole episode? I think I get what she's saying spiritually or something and agree with that. I guess I don't agree with the literal statement. So yeah, what I think she's getting at spiritually is, okay, these agents can totally be misaligned.

3. 1:11:05

   They can totally undermine our security infrastructure. They can totally break out. They can do crazy shit in the world that we would never have expected. I totally agree with that. If we just scale up capabilities more, I think you actually start running into actual damages in the world and not just stealing random data or maybe not even stealing data, right? As happened in the Hugging Face incident.

4. 1:11:24

   So I think I'm with her there. Where I get off the train is I think takeover is a lot harder than maybe the literal statement suggests. So what would it require? Yeah. What would it mean to actually take over? Yeah, I guess one definition is an AI literally runs the government or convinces somebody enough to have the government be running the AI's interests. And or like, yeah, so that's one pathway.

5. 1:11:46

   Another pathway is literal physical takeover. So overcoming the combined might, something the world's armies or maybe even US-China to like physically run over and coerce humans to do their bidding. Both those pathways, I think, are quite implausible to me in the next, you know, like five year, 10 year horizon or something. I think we'll get closer. So yeah, just to tackle them one by one. So on the first one, I guess I don't really think super persuasion is actually a thing.

6. 1:12:12

   So I do think it's possible for AI systems to be better than humans at persuasion. And indeed, they already are in certain circumstances. I'm not sure that it's actually possible to tell somebody, an arbitrary person a thing and let them do what you want, simply because number one, I can simply just choose to ignore you and not trust you. It's like an AI system. Why should I trust you, right? Number two is it's just, it's not, I think, an information problem.

7. 1:12:35

   It's also an attention problem. There's just so many things that people have to pay attention to. For example, the president of the United States just has a ton of stuff going on at the same time. Like, why are they going to be convinced by a 10-second interaction or whatever with a chatbot? So I'm not super sure. Oh, yeah, it's maybe an exaggeration. It's not going to be 10 seconds. Maybe it's an extended interaction, but it still seems like quite a hard lift to me to give high credence to this pathway.

8. 1:12:55

   On the other thing about physically taking over the world, yeah, I think it depends on where robotics goes. it currently doesn't seem like to me that we're going to have general purpose robotics that are good enough and deployed in enough locations throughout the world to actually be able to overcome the world's militaries. So I guess this would require actually producing enough robotics in a secret enough way, robotics with weaponry.

9. 1:13:19

   And this also requires overcoming existing military force. It doesn't seem like you can really easily disable, like completely disable military force with cyber attacks. Surely command control is disrupted. But just talking to people that I know who have been in the military and done stuff, a lot of it's quite decentralized. It's where the guns store. They store just like a random locker with a lock.

10. 1:13:40

    There's no digital code or something you can hack into to disable access to that. And every individual soldier has a rifle, right? So you're going to have to be able to get over that or to overcome that kind of force as well. And plus, it's probably not just going to be the U.S. military resisting. if people actually think that there's an AAC to go over, presumably citizens as well are going to be like, oh, we don't want this happening.

11. 1:14:01

    Let's try to resist. So yeah, 50% seems quite high to me. It's about zero, right? I think anything between one to 10 seemed quite reasonable. I guess maybe the overall... Yeah, I'm usually not very calibrated when I give a numerical forecast, but the overall vibe or something, the actionable thing for me is, okay, it could actually happen. It's important enough to think about how we might block off those paths to this happening and try to study more of this possibility of, oh, will they actually have the motivations to pull this off and the capabilities?