A reward-seeking AI may not entrench its takeover

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 42:43

   curious about whether you think that there's much difference overall in how likely it is that the harms from AI seizing power would be reversible compared to the harms from some group of humans seizing power. Maybe I want to move to Katya here. Like, what do you think about the likelihood of an AI dictator getting toppled versus the likelihood of a human regime enhanced with AI getting toppled. I guess in both cases, there's getting toppled from the outside or there's not being internally stable so that you either drift into something else or fall apart and maybe one part takes over or something like that.

2. 43:37

   At a glance, I sort of expect a fully AI thing to be more internally stable. I think I expect both of them to be less internally stable than other people do or something. But it seems like there's more possibility for, you know, with AI. to have the system itself overall to keep clear track of what all the parts are and respond to things going on. Whereas if you have humans as part of your system, you don't really know what's going on inside of them or how they'll react weirdly to things. Yeah. And in terms of being toppled from the outside, it seems like both of these don't feel very hopeful, though.

3. 44:27

   I'm like, oh, probably I'm missing a bunch of scenarios where people do grab power, but not like absolutely. Like, for instance, you can imagine maybe like a bunch of some AI system managers to just like take over some data centers or something. And then they have they're doing a bunch of stuff on their own that we can't control. And you could imagine that like plausibly they have a lot of power, but we do actually still have the possibility of just like destroying those data centers physically or something.

4. 44:57

   And there's a question of like, do we do that? Like, do some humans manage to coordinate enough to like have the political will to react in that way? Unclear. And maybe like if it was a bunch of humans, it seems like they're less susceptible to that sort of thing. But maybe they managed to grab power in cases where there are still other humans who also have AI systems and can maybe do something about it.

5. 45:24

   Interesting. Tom, do you have an opinion? I like Gatch's point, you could have an AI system that takes over a data center, but doesn't like take additional steps to ensure that no one can ever, you know, turn the data center off. Where, you know, if you're a reward seeking AI, it's a bit less clear that when you seize power and you seize control of the data center to ensure you get that bit of reward in three weeks time, it's a bit less clear that you're going to take additional steps um to ensure that no humans can ever shut you down even in three months time because you might think well look in three weeks time i just want to guarantee i get the maximum reward possible that's kind of all i care about i'm not as motivated to additionally like prepare the robot army needed to ensure that that is never undone whereas i think with a human a human power seeker they are more likely if they

6. 46:19

   want to seize power to be very very invested in not losing power again because um because you know they would be punished for example and because they're probably planning over you know time