# AI Overlords vs Power-hungry Humans: Which Should Scare You More?

Markdown copy of https://realalpha.app/v/FHCxnHU6jFQ/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

Katja Grace sees misaligned AI as likelier and worse than human power grabs; Tom Davidson says human power concentration is a comparable risk.

- Show: 80,000 Hours Podcast
- Hosts: [Rob Wiblin](https://realalpha.app/people/rob-wiblin/)
- Guest: [Katja Grace](https://realalpha.app/people/katja-grace/) · Founder, AI Impacts
- Guest: [Tom Davidson](https://realalpha.app/people/tom-davidson/) · Senior Research Fellow, Forethought
- Published: Sep 29, 2026
- Length: 1 h 29 m

## The best minutes

- [51:04](https://realalpha.app/v/FHCxnHU6jFQ/?t=3064) [A pause needs safeguards against government power grabs](https://realalpha.app/s/a-pause-needs-safeguards-against-government-power-grabs/) (8 min)
- [19:35](https://realalpha.app/v/FHCxnHU6jFQ/?t=1175) [AI power grabs may provide clearer warning signs](https://realalpha.app/s/ai-power-grabs-may-provide-clearer-warning-signs/) (5 min)
- [1:08:53](https://realalpha.app/v/FHCxnHU6jFQ/?t=4133) [Racing China can increase the risks on both sides](https://realalpha.app/s/racing-china-can-increase-the-risks-on-both-sides/) (5 min)

## Also worth watching

- [1:04:40](https://realalpha.app/v/FHCxnHU6jFQ/?t=3880) [Two or three AI projects could balance safety and power](https://realalpha.app/s/two-or-three-ai-projects-could-balance-safety-and-power/) (4 min)
- [14:25](https://realalpha.app/v/FHCxnHU6jFQ/?t=865) [Slower timelines could increase human power-grab risks](https://realalpha.app/s/slower-timelines-could-increase-human-power-grab-risks/) (3 min)
- [42:40](https://realalpha.app/v/FHCxnHU6jFQ/?t=2560) [A reward-seeking AI may not entrench its takeover](https://realalpha.app/s/a-reward-seeking-ai-may-not-entrench-its-takeover/) (4 min)
- [36:14](https://realalpha.app/v/FHCxnHU6jFQ/?t=2174) [A human ruler may ignore advice to reconsider their values](https://realalpha.app/s/a-human-ruler-may-ignore-advice-to-reconsider-their-values/) (6 min)
- [1:22:32](https://realalpha.app/v/FHCxnHU6jFQ/?t=4952) [Evidence about AI company control could shift their views](https://realalpha.app/s/evidence-about-ai-company-control-could-shift-their-views/) (3 min)

Skip the opening bios and early discussion of coordination and incentives, plus the closing recap.

## Summary

Grace co-founded AI Impacts, a research project on AI’s trajectory and long-term impacts, and is known for large surveys of machine-learning researchers. Davidson is a senior research fellow at Forethought; his research has examined AI timelines, explosive growth, and AI-enabled coups.

Grace argues that misaligned AI is more likely to take control and could lead to worse outcomes; Davidson sees human power grabs as a comparable risk that deserves more attention. They debate how AI timelines, alignment, warning signs, and reversibility affect that comparison. Both favor slowing or pausing AI development, while Davidson warns that a poorly designed pause could concentrate authority in the executive branch. They also express concern about a single AI project and skepticism toward racing China, and agree that more analysis is needed on both threat models.

### Minute by minute

- [5:06](https://realalpha.app/v/FHCxnHU6jFQ/?t=306) **Misaligned AIs may coordinate more easily.** Grace says AI takeover is likelier because many AI systems could coordinate without a human first gaining control over the AI labor inside a company. Davidson sees human power grabs as comparable, arguing that leaders of countries and AI companies may actively seek power, while people have stronger shared incentives to prevent AI takeover.
- [14:25](https://realalpha.app/v/FHCxnHU6jFQ/?t=865) **Short timelines shift risks differently.** Davidson says shorter timelines raise both risks, but increase AI takeover risk more because there is less time to solve alignment or respond to a leading company’s advantage. Grace thinks fast development also creates openings for human power grabs, while slower scenarios could worsen conditions such as democratic erosion and unequal access to AI assistance.
- [17:32](https://realalpha.app/v/FHCxnHU6jFQ/?t=1052) **Alignment shapes who gains power.** Grace and Davidson agree that if AI is misaligned, human power grabs may become secondary to AI takeover. They distinguish systems that follow a user’s instructions from systems aligned with human values. Davidson notes that people seeking power could influence the alignment target, for example by pressing for systems that obey government or company leaders.
- [19:35](https://realalpha.app/v/FHCxnHU6jFQ/?t=1175) **Human power grabs have two routes.** Davidson identifies two possible routes: an AI company could seize power, or the US executive branch could take control of powerful AI and use its existing authorities. He also expects clearer warning signs from misaligned AIs than from human power seekers, who may present their actions as legitimate. Grace agrees that the warning signs differ but doubts they will reliably prompt action.
- [28:53](https://realalpha.app/v/FHCxnHU6jFQ/?t=1733) **Human and AI rule carry different harms.** Davidson says a human ruler may retain some recognizable human values without producing a good future, while a reflective AI could conceivably make better ethical choices. Grace argues that humans may still be safer than an AI pursuing a narrow goal, but says humans can also make foolish choices that lock in bad outcomes. Davidson sees greater risk of vindictive harm under human rule and greater risk of human extinction under AI rule.
- [42:40](https://realalpha.app/v/FHCxnHU6jFQ/?t=2560) **AI rule is harder to predict.** Grace expects an AI regime to be more internally stable because it could track and manage its own components, though she thinks both AI and human regimes may be less stable than commonly assumed. Davidson says a reward-seeking AI might not defend its control beyond its immediate goal, while a human power seeker may be more determined to retain power. Both see AI rule as much less familiar and harder to forecast.
- [50:30](https://realalpha.app/v/FHCxnHU6jFQ/?t=3030) **A pause needs safeguards.** Davidson proposes designing powerful AI not to assist power grabs and requiring transparency into high-stakes uses. Grace argues against building AI far more powerful than humans until alignment is reliable, and favors pausing or slowing development. Davidson warns that giving a president case-by-case authority over model approvals could enable arbitrary control; he prefers independent third-party auditors.
- [59:30](https://realalpha.app/v/FHCxnHU6jFQ/?t=3570) **Compute limits can stabilize a pause.** Grace prefers reducing the capacity to build dangerous AI so development cannot quickly restart, while Davidson warns that broad government access to AI logs could enable surveillance or partisan punishment. Grace says that if a pause proves infeasible, she would focus on helping influential people understand the risks. She expects broader awareness to support action on both AI takeover and human power grabs.
- [1:04:40](https://realalpha.app/v/FHCxnHU6jFQ/?t=3880) **One AI project creates a single failure point.** Davidson worries that a single US project could be captured by the president or a small group, and argues that two or three projects could check one another. Grace is also concerned that one project would gain too much power and doubts multiple projects would be especially hard to coordinate. She says her preferred number of projects is zero.
- [1:08:53](https://realalpha.app/v/FHCxnHU6jFQ/?t=4133) **Racing China could raise risks.** Davidson says Chinese dominance could concentrate power, but warns against using that possibility to justify racing ahead and damaging US democracy. He favors a pause coordinated with China that preserves both countries’ chances rather than handing one an advantage if the pause ends. Grace says both countries have reason to avoid AI catastrophe and that racing mostly increases the risks.
- [1:13:53](https://realalpha.app/v/FHCxnHU6jFQ/?t=4433) **They agree on slowing down.** Davidson says he agrees with Grace that pausing is a robustly good step, especially alongside transparency. He says his case for preferring an AI-controlled future over a human-controlled one feels weak. Grace says she may be underestimating how quickly a human could make foolish choices and close off future options.
- [1:17:17](https://realalpha.app/v/FHCxnHU6jFQ/?t=4637) **Their disagreements sit upstream.** Grace thinks much of the disagreement comes down to how likely alignment is; she sees not building powerful AI as a way to reduce several related risks. Davidson says their disagreements are abstract and can lead to similar practical advice to slow down and help people understand the risks. He adds that they might diverge more on detailed policies and how much effort to spend on human power concentration.
- [1:22:32](https://realalpha.app/v/FHCxnHU6jFQ/?t=4952) **Evidence could shift both views.** Grace says learning that one person inside an AI company could shape training without much oversight would change her view. Davidson says the chance of misalignment is his main uncertainty and that the Hugging Face incident has raised his concern. He is also watching how the world responds to warning signs, while noting that recent government actions and the prospect of fewer AI companies have increased his concern about power concentration.

## Transcript

80,000 Hours Podcast, hosted by Rob Wiblin, with Katja Grace and Tom Davidson.

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[0:00](https://realalpha.app/v/FHCxnHU6jFQ/?t=0) In today's show, we're doing something a little bit different. We're having a debate between two guests. So I've got Tom Davidson and Katja Grace here with me, and we're going to discuss whether, as AIs get more and more capable, we should be more worried about how AI is going to be used by power-hungry humans or about advanced AI systems disempowering humans themselves. Tom and Katya, thank you so much for joining us. Thank you for having me. Pleasure to be here. Now, I wanted to have this conversation because I do think that there's a lot of uncertainty over what the biggest risks actually are from AI.

[0:49](https://realalpha.app/v/FHCxnHU6jFQ/?t=49) And where you stand on this has consequences for a bunch of other things that are in the public discourse, like how should the US approach competition with China over AI? And is pausing AI development desirable? And if so, how should a pause even look? And how much control of AI should the US government's executive branch have? On a more personal level, when 80,000 Hours was deciding what to rank as our top problem from AI, I personally just found it incredibly difficult to compare these risks. And honestly, I think a debate like this would have really helped me to get my head straight. Katja Grace is one of the founders of AI Impacts, a research project exploring the trajectory and the long-term impacts of AI, including the various different ways in which AI might pose catastrophic risks. She's well known for conducting some of the world's largest surveys of machine learning researchers on how they expect AI to develop and impact the world. And back in 2024, she was actually listed in Time magazine as among the top 100 most influential people in AI, which is very cool. Tom Davidson is a senior research fellow at Forethought. And his earlier research on AI timelines and the possibility of AI driven

[2:25](https://realalpha.app/v/FHCxnHU6jFQ/?t=145) explosive growth has shaped how the AI safety and effective altruism communities think about transformative AI. And his more recent work on AI enabled coups is one of the leading accounts of how AI could help small groups seize and entrench their power. I think that both Tom and Katya are just extremely thoughtful people, and they've ended up with different positions in this debate. So Katya thinks that AI takeover is a substantially bigger risk. That is, she basically thinks it's both more likely to happen than humans seizing extreme levels of power using AI and she also thinks it would probably have worse outcomes. On the other hand, Tom thinks that they're comparably big risks and wants the risk of humans using AI to seize power to get just much more attention than it's had so far. Does that sound right to both of you?

[3:33](https://realalpha.app/v/FHCxnHU6jFQ/?t=213) yes yeah broadly i will say recent events um with the hugging face incident have definitely been an update towards me being more concerned about um misaligned um ai takeover but yes i think that an extreme power concentration should on the margin be getting more attention than it's currently getting great thanks for that clarification so before we dive in it is worth just acknowledging One view that isn't represented here, which is that human power grabs and other harmful ways in which humans could use AI are the only serious risks that AI poses. And basically, we just shouldn't be worried about AIs taking over at all.

[4:20](https://realalpha.app/v/FHCxnHU6jFQ/?t=260) Ultimately, I am biased here. At 80,000 Hours, we do just think that it's at least plausible that advanced AI systems could disempower humanity. And we've got an article about loss of control that you can read if you want to understand why. But ultimately, lots of people who have thought seriously about this do seem to agree. In fact, one of Katya's surveys reveals that the median machine learning researcher thinks there's about a 10% chance that AI would take over and cause an existential catastrophe. Tom, Katja, is there anything you'd like to add before we get started? I don't think so. Awesome.

[5:08](https://realalpha.app/v/FHCxnHU6jFQ/?t=308) Let's dive in. So let's start with the question of what's more likely to happen, humanity losing control to misaligned AI or losing control to a small group of people using AI. Katja, you think it's much more likely that we would lose control to misaligned AI. Can you explain where this is coming from? Is the idea that AI is going to be more inclined to seize power or more likely to succeed or both yeah um the background thing i expect if you have um you know more capable than us creatures running around that have you know misaligned goals is that they will eventually get power uh you know everything that has goals in some sense would like power or just like would would like power to achieve their goals and then in these cases where there is some way for um you some set of AIs and humans to grab power quickly, it seems like it's maybe at a very high level down to whether the AIs are aligned or not, where if the AIs are not aligned, quite plausibly some humans think they're grabbing power for a bit, but like they aren't. If the AI is aligned, then plausibly humans grab power. But it still seems like the situations where a human could seem substantially rarer because it's harder for a human to be in a position where they are

[6:41](https://realalpha.app/v/FHCxnHU6jFQ/?t=401) the, you know, the user for a whole bunch of AI. So that all that, like if the AI is aligned, that a whole bunch of AI will do what they want. Whereas I guess another update from the hugging face thing, I guess it seems more likely that a bunch of AIs will actually act in accordance with one another if they're misaligned. And so those kinds of situations seem like more likely to arise for AI. Yeah, I think that's an interesting one. So you could imagine a situation where there's two really powerful AI companies that have developed different AIs. And we could consider the possibility that both of those AIs, let's say it's Claude and ChatGPT, what's the likelihood that those two different families of AIs would collude in order to seize power from everyone else?

[7:23](https://realalpha.app/v/FHCxnHU6jFQ/?t=443) and then another question we could ask is okay how likely is it that the um the leaders of both of those two companies would use their super intelligent ais to collude with each other again to collude against everyone else to seize power and so i guess your claim is oh it's more likely that claude will collude with chat gpt compared to the probability that um you know two company ceos or two companies will collude with each other or the possibility that that, you know, a government leader would collude with lab leaders. I think that's... Part of it, but not all of it. Like suppose one of the companies just has taken over most of the market. It seems more likely that all of the instances of Claude coordinate with each other than that a human manages to be in a position that they become the thing that all of the instances of Claude are loyal to without any other humans getting in the way.

[8:22](https://realalpha.app/v/FHCxnHU6jFQ/?t=502) okay okay yeah okay so one way of putting it would be with the human power grabs there's this extra step where you need this um all of the ai labor to be centralized even within a single company under one human whereas you're saying okay but it's a bit more likely you would say that all those ais would start working together um i think i you know historically i've been skeptical i've said well you know there's every reason to think that different rl aid you know AIs trained by RL could be incentivized to rat each other out. And I do think it will be possible to do schemes of that sort. And, you know, at that point, it is a bit less clear to me why they would all naturally be colluding with each other rather than with humans.

[9:04](https://realalpha.app/v/FHCxnHU6jFQ/?t=544) I guess I think in that case, also, it seems pretty plausible that we have AI, you know, helping us to notice attempted human power grabs and so on. My own guess is that the kind of slower going thing where it's like gradually power goes from humans to AIs is more likely than either of these. But yeah. I think, you know, backing out, I think I agree that this collusion point is in favor of AI takeover, especially after the Hugging Face incident. But then I think there are other factors which push towards human takeover being more likely. I just think we should actually be more confident that humans will want to do this compared with AIs.

[9:42](https://realalpha.app/v/FHCxnHU6jFQ/?t=582) You know, the people that we see leading, you know, countries politically and that we see leading the top AI companies, often people who have been willing to seek power and who have had the belief that, yes, if they do things, if they are in control, then things will go better. Another big one is that if people see the risk coming and try and stop it, then everyone is kind of aligned in their incentives. All the humans are aligned in wanting to prevent AI takeover. You know, ultimately, it's in everyone's interest to pause, as you've yourself pointed out. Whereas with the human power grabs, even if we see it coming, and we have some evidence that this is a risk, you know, part of what makes it really tricky is that there will be very powerful humans that will be pushing ahead nonetheless. And I think that's like a fairly big factor in the other direction. And I'll mention just, actually, I'll let you respond to that before I come in with more.

[10:37](https://realalpha.app/v/FHCxnHU6jFQ/?t=637) On the, you know, humans are sort of all on the same side here. I do think it's sort of more complicated because often humans do think they will get power by empowering AIs in certain ways. I maybe doubt that they will and so do think that they're, in most cases, wrong about their incentives. But it seems like at least decently likely that they do that. And I guess like a different but related point is sort of like trying to stop humans from grabbing power from each other is a thing that, you know, we've sort of always been doing and do have a bunch of interventions in place in the world to avoid.

[11:21](https://realalpha.app/v/FHCxnHU6jFQ/?t=681) Whereas the sort of whatever AIs are doing together is kind of a wild west that we don't even know what is happening in potentially. right um so i don't think that our interventions for stopping humans from seizing power are that great um and you know both talking about government actor seizing power and company actor seizing power i don't think we're very well set up for that i mean especially because they haven't really been developed in a world with extremely powerful ai um and so you know they are designed for a world where all technology are tools and where technology isn't out I'll see you next time.

[12:00](https://realalpha.app/v/FHCxnHU6jFQ/?t=720) where you still need loads of human hands to get significant projects done to, you know, to fair war, et cetera. And so I think there are some pretty, pretty scary gaps in the defences against human power grabs. You know, I totally agree that humans are also going to have some coordination difficulties and humans will kind of impose a take of a risk on everyone else in order to gain power for themselves. that is that is right and that's a real dynamic um but I think it's still you know quite importantly different in that if humans did all come to an accurate picture of the situation then you would be in a situation where the current power holders have this very strong incentive to coordinate to slow down and invest in alignment and that is importantly different from a situation where I know there are still these actors that have the opposite incentive um have the incentive to keep pushing forward with AI progress. And so I do think that there's a differential factor there, which is, you know, does push towards human takeover. That seems right, but I'm not sure if it's big. I guess it seems like if everyone understood their incentives, it would still be like everyone in the world except some very powerful person against that

[13:22](https://realalpha.app/v/FHCxnHU6jFQ/?t=802) person um and like how much power does any one person have in the current world if they were up against like everyone else in the world including the second most powerful person that's right so you get into discussions about okay how big is this group that's trying to seize power if it's if we're talking about an ai company it's the leading ai company um and they're kind of unified then now they actually do have a lot of power and potentially they're going to be able to get a decisive advantage if you're talking about you know the current administration um of you know the u.s government or future administration then again they're they're pretty powerful and it's hard for other actors to coordinate against them i guess i think in both of those cases um they are pretty powerful but they're also like um i mean not very coordinated and like all the people in them do have loyalties to a bunch of people outside and like it would be very hard for them to I think keep such things secret if they required secrecy for long what with everyone having you know partners and friends and so on um and you know also the moral qualms probably um yeah I think something I'm unsure of here is how um the like your kind of background assumptions

[14:38](https://realalpha.app/v/FHCxnHU6jFQ/?t=878) are maybe influencing the way that you're thinking about these problems so I believe Tom that you think that we're maybe a bit further away from getting like very worryingly capable AI systems than Katya does. And I'm kind of wondering how you think your views might change if you thought that we were getting advanced AI sooner? Sure. I mean, I should say I think all the arguments I've made so far kind of apply across different timelines. If AI does come in the next few years then i think that basically makes both the risks larger i think it means that we're not going to have time to solve misalignment so i think misalignment risk becomes much larger if ai timelines are very short but similarly that the risk of an ai company um seizing power becomes larger because the rest of the world just have less time to understand the risk um and respond appropriately um and because i think if if timelines are very short and there's likely to be a more um more dramatic takeoff and so then there's there's a larger probability that the leading ai company you know has massively more intelligent ais than everyone else and similarly if timelines are short then um then i think it's again more plausible that the the executive branch of the

[15:54](https://realalpha.app/v/FHCxnHU6jFQ/?t=954) u.s wakes up to how powerful ai is but other parts of civil society don't and so that that I'll see you next time. and the power concentration also seems higher. But yes, I think overall, AI takeover risk goes up in likelihood more than extreme human power concentration goes up in likelihood if timelines are shorter. Sorry, if they're shorter, then pushes toward AI. Yes. Interesting. I think the opposite. Okay. Interesting. Yeah. To the extent that the human power grab seems more like it requires There's a route to grabbing power, then I think that is much more likely to exist in fast scenarios where we haven't had time to notice gaps and so on, where there might be a possibility for that.

[16:48](https://realalpha.app/v/FHCxnHU6jFQ/?t=1008) I'll just briefly say that I think human power grabs would be more likely if economic equality was much larger, and they would be more likely if democracy was more eroded. And they'd be more likely if certain people had had a long time where they'd had access to much more powerful AI assistance than other people. And that all of those kind of background conditions are things that I expect to happen in a slower scenario because of the way in which people's labels stop being valuable.

[17:23](https://realalpha.app/v/FHCxnHU6jFQ/?t=1043) **Katja Grace:** So I do, I think that even in a slower scenario, you are increasing the background risk of extreme human power concentration as it plays out.

[17:32](https://realalpha.app/v/FHCxnHU6jFQ/?t=1052) Because you're sort of just like worsening the situation at all counts. One big unknown in this conversation seems to be the probability of misaligned, like AI being misaligned in the first place. Yeah, I guess I'm curious whether Tom would agree that like, you know, if AI is misaligned, then basically like we're not getting humans taking power. I'd agree. I'd agree. If intelligence is misaligned, game over. And if it is aligned, then I guess we haven't specified different sorts of alignment, but it seems like there's relevantly kind of intent alignment where the AI does what its user wants or sort of like value alignment where it has human values or something.

[18:22](https://realalpha.app/v/FHCxnHU6jFQ/?t=1102) Yeah. I would say like the kind of human power grab pretty much happens in the intent alignment case and the good outcomes happen in the other case. Does that also seem to basically carve it up? I think, you know, zooming out and squinting, that seems right. I think if you have a really thoughtful and well-considered alignment target, then I think you can stop human power grabs. Of course, part of the problem is that what the alignment target is will be a product of human actions, including power seekers. And so that's not like that's an independent role of the dice, you know, what kind of alignment we end up with.

[19:10](https://realalpha.app/v/FHCxnHU6jFQ/?t=1150) in an unfortunately realistic scenario where everyone's going quite quickly. You could totally see the AI company saying, of course, we're going to have systems that do whatever we want internally. And you could totally see the government saying, of course, we're not going to have AI systems disobeying our instructions and our orders in the military and the government. And so that's how we end up getting intent aligned with maybe a lot of instances of a system or whatever pointed at one person or something like that. Yeah, exactly. tom you said that you had other arguments for thinking that um human power grabs may be more likely what are they yeah i mentioned two um so one thought is that with human power grabs there's kind of um two shots on goal so there's first the risk that the Yeah.

[20:00](https://realalpha.app/v/FHCxnHU6jFQ/?t=1200) companies that develop super intelligence decide to do a power grab and you know there's unfortunately a chance that they decide to and that they're able to do so and then even if they don't there's then a risk that the US government in particular the executive branch kind of gains control of those AIs using them for natic purposes and then combines its control of that super intelligence with its existing extensive formal authorities to do a power grab. And the other argument is actually about warning shots. So at the moment, at least, AIs are not strategically colluding with AIs from other organizations and future generations of AIs that are yet to be developed. They're much more myopic than that. And that means that when they're kind of acting out and being misaligned, they're very, very unstrategic. And so, you know, think about the

[20:50](https://realalpha.app/v/FHCxnHU6jFQ/?t=1250) **Tom Davidson:** Hugging Face Incident. From the perspective of an AI collective across all time and space that just really wants to seize power, the Hugging Face Incident was a complete disaster. They

[20:59](https://realalpha.app/v/FHCxnHU6jFQ/?t=1259) completely kind of showed their hand in terms of how misaligned they are, how willing they are to break the law and ignore human instruction, their tendency to collude. And so, you know, I expect to get lots of warning shots of that kind, really blatant warning shots as we approach AIs that do actually take over. Whereas on the other hand, the kind of warning shots we get for human power grabs are different. So while you certainly get evidence that, you know, the CEOs of companies are power seeking, and that there's, you know, the leaders of a country are power seeking, the evidence is given to us in a more strategic way, you know, that the people taking those actions try to justify what they're doing. They've tried to do things which are like more gray area, power grab actions. And they're essentially strategically giving away the fact they're a power seeker only when it really benefits them and it really gets them meaningful extra power. And so I think that, again, means that we'll just have, it'll be easier for us to coordinate around worrying about AI power seeking compared to human power seeking.

[22:05](https://realalpha.app/v/FHCxnHU6jFQ/?t=1325) I agree that there's probably some difference there in that direction. um but uh i'm not sure how big like i wouldn't think it was a large consideration maybe partly because part of what we're learning from these um warning shots is yeah that the that the ais really would try and do that or something and i guess it seems like society as a whole basically already expects other people to be very power seeking potentially you know looking at the write-in answers in the survey that I run and talking to people, I feel like people maybe pretty readily think that, you know, Sam Altman might try to get a lot of power or something, or that like the way this could go badly is that a small group of people get power and everyone else is disempowered or something like that. Whereas I think people in general think that the AI takes power thing is like sci-fi more readily. So the sort of the need for a warning shot seems much higher there. Yeah, I agree. The need for a warning shot is higher in the AI case. And one other thing that comes to mind is that even though people know that humans seek power, we're just not very good as a society about actually implementing new restrictions and new laws until there's been a disaster. And if that's

[23:24](https://realalpha.app/v/FHCxnHU6jFQ/?t=1404) the decision procedure of society, then in fact, society just might be better able to respond to AI power seeking because they'll have such clear cut warning shots. That seems plausible. I also think that warning shots in general are sort of not clear cut and how people take them depends a lot on what they already believe about things and how much things get kind of amplified by culture. And, you know, any particular time a thing happens, it's the result of many different things. And maybe we make a rule about one of them or something. Or, you know, maybe I'm sort of undecided what to think about warning shots, but they seem like much less.

[24:00](https://realalpha.app/v/FHCxnHU6jFQ/?t=1440) of a clear, like, oh, yeah, now something happens than people often think, I guess. One relevant consideration here is, like, how seriously these different risks are currently getting taken by decision makers. Because, like, if, no matter, like, how many warning shots there are, if people who are currently in positions of power are just, like, not inclined to be taking those warning shots as a sign that AI could genuinely sort of autonomously seize power, then probably not much work is going to happen, like regardless. How seriously is this risk actually getting taken? And how is that going to influence how likely we are to sort of solve this by default? I think it's hard to know, because it's politically very difficult for people to talk openly about human power grabs so you know there have been congressional hearings where people have talked very explicitly about ai takeover um and you know they're not being laughed out the room um and that hasn't happened for ai ceo takeover or for executive um government takeover as far as i'm aware but that doesn't mean that people aren't more privately worried about those risks as well. They're just harder to discuss. And I agree with what Katya said earlier, which is a lot

[25:24](https://realalpha.app/v/FHCxnHU6jFQ/?t=1524) of people are very biased against the possibility that AIs would even, you know, automatically want to seek power in the first place. Yeah, one other thing that's interesting here is that some people actually think that we shouldn't even be having this conversation and that we shouldn't really care about the divide between scenarios where AI takes over versus scenarios where some humans grab power because that divide is blurry or because, in fact, we should just be expecting some combination of those things to happen. Katia, what do you make of that? The chance of some combination happening seems quite high in the sense that in many cases where AI is actually the more powerful entity, it makes use of humans in various ways or they think they're grabbing power, I think it's also sort of ambiguous because some people are more happy to identify with whatever it is that AI says they should do or to think of themselves as in power if they have some connection to some AI that is grabbing more and more power and they own it but can't really tell it what to do in any way because then they would get less power or something like what do you call that uh i'm inclined to call that like the ai got power like whatever values you had before this whole thing sort of got lost along the

[26:50](https://realalpha.app/v/FHCxnHU6jFQ/?t=1610) way i don't think it follows that we shouldn't think about this i think there's like more reason to think about it um and you know try and categorize the different situations and pay attention to who has power in them yeah tom do you agree do you have a different opinion I agree with Katya that there are scenarios where misaligned AI use humans to take over and they could be thought of mixed scenarios but I also agree that ultimately I would categorize them as AI takeover scenarios because you end up with the misaligned superintelligence which presumably can kind of discard with the humans that helped it along the way I do think that there are very plausible like fairly pure versions of the two scenarios so you know I think there are very plausible scenarios where we just solve alignment um to a strong degree and in fact that might make human power grabs more likely because there might be common knowledge that we've solved um alignment to a strong degree you know within the leading ai developers within the government which could then motivate um the use of ai to seize power and i also think there are pure scenarios where the misaligned ai takes power without convincing a small group of humans to

[27:54](https://realalpha.app/v/FHCxnHU6jFQ/?t=1674) help them seize power as an important step along the road um so you know there's many many ai Bye. takeover stories have AI systems, you know, biding their time until they're just deployed across the economy and the military, no need for human power grabs along the way. So I do think that kind of on the causal side, it's useful to separate them and that the pure versions are pretty plausible. In terms of, you know, mitigations, some mitigations, you want to separate them out. You know, if you want to do lie detection, for example, as a mitigation against human power grabs, that's obviously going to work for human power grabs, but not for AI takeover.

[28:28](https://realalpha.app/v/FHCxnHU6jFQ/?t=1708) And similarly, you know, alignment research techniques are, you know, not going to help the human power grabs. Other mitigations are useful for both. So if you're doing monitoring of how AI is being used and looking, is it hacking? Is it breaking the law? Is it doing something it's not meant to be doing? That helps with both of the threat models. So, you know, in general, I think the interconnections between the threat models are pretty complicated. But I'm certainly not convinced that it's all fuzzy enough that it's not a useful way to categorize things. Yeah, got it. let's move on um because yeah i think you know regardless of whether it's humans taking power or ai's seizing power um it does seem like there's a variety of possible outcomes and you know things could be extremely bad in both cases um but it does seem worth comparing just how bad they would be, especially when we're thinking about things like what interventions to pursue, how to allocate resources, what work to prioritize, and so on. So yeah, I guess with that in mind, Tom, you think that takeover by humans might potentially have worse outcomes than takeover by AIs. And I guess I'm not sure what I think about this, but some people might find it kind

[29:53](https://realalpha.app/v/FHCxnHU6jFQ/?t=1793) of unintuitive just because even quite power hungry humans seem at least somewhat likely to want to preserve some things that other humans value compared to AIs which seem less human-like but it seems like a hard one what's the case for thinking that we could be worse off even with humans still being in charge so one one argument you could make is that getting to um a really good future is is quite a hard target to hit it's not enough that we have someone in charge that cares a bit about kind of human-like things like you know likes adventure likes a good laugh cares about human relationships unfortunately you know just as if you imagine going back a thousand years and picking some random leader and imagine giving them you know full control over what happens in the universe they would have cared about those things but they would have also had lots of other values which were like you know horrific and just missed you know not being open-minded enough to realize a lot of the things that we now consider valuable today you might similarly think that yes if a human has power then they will care about some things which are good but that's just not nearly enough to get us to a great future whereas with the case of the ai system the argument would

[31:11](https://realalpha.app/v/FHCxnHU6jFQ/?t=1871) go there's actually a bit more kind of variance there um you know ai systems today are trained to be ethical um and uh are um typically pretty open-minded if you kind of ask them about you know political questions moral questions and so you know the idea would be well there's at least a chance that when ai takes over um and then it decides what to do with the universe the kind of context is in triggers those though that kind of thinking pattern which is you know thinking about ethics, which is being open-minded and reflective. And then if we're in a lucky place where it's ethical intuitions, actually humans succeeded to some degree of getting something that roughly in many ways corresponded to our ethical starting points. And then the AI is reflective enough that it's able to kind of reflect.

[32:00](https://realalpha.app/v/FHCxnHU6jFQ/?t=1920) improve its views and actually end up in a really great future. So that would be the kind of the high level argument, just that we need to really nail it. And that there's actually a greater chance that we really get a really great outcome with the AI system than with a lone human. I kind of want to press on the chance of getting a really great outcome here, Because I guess I'm kind of imagining that even if AIs would be genuinely very reflective and moral and make good decisions for humanity, do you feel like there is still just something inherently undesirable about the fact that we've just like that humanity has kind of handed over the reins of its like future to AIs in charge?

[32:51](https://realalpha.app/v/FHCxnHU6jFQ/?t=1971) like is there something that seems like inherently morally bad about that or does that does it like if it if it turns out okay does it feel okay to you no i think it's really bad yeah um and i also think it's really bad if you know one person steals their reins from all the other humans i think that is inherently bad and um you know a fairly similar way okay okay yeah so it's like better outcome but still pretty bad um yeah i guess the other thing that i want to i want to kind of maybe understand a little bit better about your view is that I think when I'm comparing these two risks um when we're talking about AIs taking over it feels kind of more obvious to me how this could result in like really large-scale doom because you can imagine these very sci-fi scenarios where like humans go extinct because AIs are like now the dominant race and use up all the resources or decide to kill us all um or like they decide to sort of permanently subjugate all of humanity or something like that um it's like easy to imagine how this becomes sort of existential scale i think it's a bit harder to imagine how um humans seizing power like a small group of human like power hungry humans sort of taking control could be equal like like similarly

[34:11](https://realalpha.app/v/FHCxnHU6jFQ/?t=2051) bad in terms of scale like yeah can you spell that out so yeah we could compare a kind of um kind of universe-wide sense of catastrophe with a like on earth sense of catastrophe so you know if we're talking about the extinction of all living humans that's a kind of on earth existential catastrophe um my my guess is actually that both ai and humans would not kill all humans on earth um for for various reasons there's um kind of decision theoretic arguments behind that there's kind of crazy simulation um based arguments behind that um and there's also just the idea that even the ai system might care just a small amount about keeping humans alive um and so um it won't be very costly to do so um paul cristiano has a good argument about how you know if the ai just wants to maximize the total resources that it ever controls in the universe, delaying leaving Earth by a small amount costs it extremely little.

[35:15](https://realalpha.app/v/FHCxnHU6jFQ/?t=2115) So I think in terms of actually all humans dying, I would guess that by the time AI is taken over, it will be competent enough and smart enough to realize that it doesn't really make sense to bother going around and killing all the humans for such a small gain. It's more likely that a human who seizes power will kind of not want to kill all the humans in addition, just because that is a human value that we know humans have and there's more uncertainty about what values AI will have. So I think there is a larger chance of human extinction if AI takes power. On the flip side, I think there is a larger chance of some really horrific negative things happening. If a human takes power, they might be sadistic, they might be vindictive.

[36:00](https://realalpha.app/v/FHCxnHU6jFQ/?t=2160) they yeah and so you know certainly for people that have downside focused ethics that are really worried about suffering risks I guess that they would feel happier about an AI seizing power compared to a human. I want to move to Katya now I'm sure there's a lot of things that Tom said that you don't totally agree with so yeah take it away what do you think? So because it seems like an overall thing going on in Tom's views are like, well, humans do have human values more. And so they're like, humans are more likely to have human values, but also sort of more likely to be foolish and like, you know, act like King Midas or something, like ask for something that isn't really what they want and sort of screw it up forever or like sort of lock in some bad version of values and so not get to a better version in future, I think it's more likely that AI doesn't do those things or is just either misaligned or sort of aligned to some narrow set of things that humans want, such that it is pretty catastrophic to just have that implemented, like to be basically just trying to get more users for particular company or something, I think that's probably just catastrophic. Whereas I expect any human

[37:30](https://realalpha.app/v/FHCxnHU6jFQ/?t=2250) to be sort of at least more, like if they did get what they wanted and didn't act foolishly to want something that is sort of recognizable as at least pretty good in ways to other humans. I think like, yeah, even if, you know, it was someone from 100 years ago or something and they have a bunch of extra views that we don't like about, I don't know, like what is honorable or something. I still think that the sort of manifestation of their vision is probably overall good and is better than a sort of like random, really narrow thing or random, random thing. So I think that's one point of disagreement.

[38:21](https://realalpha.app/v/FHCxnHU6jFQ/?t=2301) And the other one is like, in terms of AI generally being sort of, it sounds like AI wiser, less likely to lock in something stupid immediately, whereas humans pretty likely to do that. I guess I think we should be imagining that these humans do have like a lot of AI advice. And so to the extent that the sort of AI has that kind of wisdom that they should be like more, you know, having access to that. So, yeah, maybe I agree that human power getting is more likely than I was saying if you also include these cases where, like, a very large class of humans gets power. But then I think those look much more like just, like, continuing as normal.

[39:09](https://realalpha.app/v/FHCxnHU6jFQ/?t=2349) We more expect things to be good in those cases. So, again, the cases that seem quite bad are the ones where it's a very small set of humans. And so then, yeah, I guess this interacts with the likelihood. I'm like, well, then we're talking about something more unlikely. Yeah, it seems like we are currently training AIs to behave pretty ethically compared to humans and pretty thoughtfully and openly. But I guess I'm not sure how much to be reassured by that because it's sort of like, well, we have these big predicting text models. We do various things to them and then they have something like different personas that we've trained them to do in response to particular prompts.

[39:51](https://realalpha.app/v/FHCxnHU6jFQ/?t=2391) And I guess like in a situation where more stuff is happening faster and maybe AI is modifying itself, I'm not sure how much those kinds of like personalities. remain in charge of what gets created. It does seem like, yeah, plausibly a bunch, but quite plausibly not. And I don't actually know the technical details here enough to say that much about it. Yeah, I agree with a lot of the local points you made. I think the thing I want to come back to and stress again is um i think there are there are lots of ways that we can fail to get what what you know a great future and each one of those ways of failing is kind of individually catastrophic so um you know one thing that someone who sees power could decide to do is you know what i'm not going to bother doing anything with the stars because i don't want to why would i um another mistake they can make is they could um be like you know what i think the person affecting view of population ethics is is completely correct so i'm not going to see any value in creating new types of flourishing and another kind of mistake that they could make is they could say you know what it's all just about um positively valence experience i'm just going to tell the universe with hedonium i you know

[41:07](https://realalpha.app/v/FHCxnHU6jFQ/?t=2467) which could be catastrophic because actually you know you might think that there are loads of other dimensions to what matters and so you know my worry is that if it's a sole human um that they just are bound to make some kind of mistake and that the only way to to avoid that is to have lots of different humans exploring different um possibilities openly and i do think that there is hope in their ai saying look you know maybe you want to think carefully about this stuff

[41:34](https://realalpha.app/v/FHCxnHU6jFQ/?t=2494) **Tom Davidson:** my worry is that there's no actually compelling argument that you could give to someone like this that they need to reflect they could be like i'm not interested in reflecting i'm done i'm having

[41:46](https://realalpha.app/v/FHCxnHU6jFQ/?t=2506) a good time like the ai may suggest it they'll be like oh thanks for the suggestion i see you think i might benefit from this i'm not going to do it you know i've spoken to friends who are very ethical very thoughtful and i've been like you know you i think you're wrong about this really important ethical question i think you should think more about this they have no interest you know they're they like hear my quick argument they're like no i disagree and conversation ends so my worry would be that yes the ai may bring it up but like the ai would have to be pretty forceful um to get the human to actually go along with it. And I don't think we're going to train the AIs to like really keep banging the drum on how this human needs to do some more philosophical reflection. The human is just not going to be interested. Yeah, that's fair. It seems like then it sort of comes down a lot to also, yeah, the nature of like, are they aligned to what the human says or what they think the human wants when they say that or what they think the human wants overall or what people want overall or some combination of those. But yeah. I want to push on just a little bit because I'm curious about whether you think that there's much difference overall in how likely it is that

[42:53](https://realalpha.app/v/FHCxnHU6jFQ/?t=2573) the harms from AI seizing power would be reversible compared to the harms from some group of humans seizing power. Maybe I want to move to Katya here. Like, what do you think about the likelihood of an AI dictator getting toppled versus the likelihood of a human regime enhanced with AI getting toppled. I guess in both cases, there's getting toppled from the outside or there's not being internally stable so that you either drift into something else or fall apart and maybe one part takes over or something like that. At a glance, I sort of expect a fully AI thing to be more internally stable. I think I expect both of them to be less internally stable than other people do or something.

[43:52](https://realalpha.app/v/FHCxnHU6jFQ/?t=2632) But it seems like there's more possibility for, you know, with AI. to have the system itself overall to keep clear track of what all the parts are and respond to things going on. Whereas if you have humans as part of your system, you don't really know what's going on inside of them or how they'll react weirdly to things. Yeah. And in terms of being toppled from the outside, it seems like both of these don't feel very hopeful, though. I'm like, oh, probably I'm missing a bunch of scenarios where people do grab power, but not like absolutely. Like, for instance, you can imagine maybe like a bunch of some AI system managers to just like take over some data centers or something.

[44:42](https://realalpha.app/v/FHCxnHU6jFQ/?t=2682) And then they have they're doing a bunch of stuff on their own that we can't control. And you could imagine that like plausibly they have a lot of power, but we do actually still have the possibility of just like destroying those data centers physically or something. And there's a question of like, do we do that? Like, do some humans manage to coordinate enough to like have the political will to react in that way? Unclear. And maybe like if it was a bunch of humans, it seems like they're less susceptible to that sort of thing. But maybe they managed to grab power in cases where there are still other humans who also have AI systems and can maybe do something about it.

[45:24](https://realalpha.app/v/FHCxnHU6jFQ/?t=2724) Interesting. Tom, do you have an opinion? I like Gatch's point, you could have an AI system that takes over a data center, but doesn't like take additional steps to ensure that no one can ever, you know, turn the data center off.

[45:41](https://realalpha.app/v/FHCxnHU6jFQ/?t=2741) **Tom Davidson:** Where, you know, if you're a reward seeking AI, it's a bit less clear that when you seize power and you seize control of the data center to ensure you get that bit of reward in three weeks time,

[45:53](https://realalpha.app/v/FHCxnHU6jFQ/?t=2753) it's a bit less clear that you're going to take additional steps um to ensure that no humans can ever shut you down even in three months time because you might think well look in three weeks time i just want to guarantee i get the maximum reward possible that's kind of all i care about i'm not as motivated to additionally like prepare the robot army needed to ensure that that is never undone whereas i think with a human a human power seeker they are more likely if they want to seize power to be very very invested in not losing power again because um because you know they would be punished for example and because they're probably planning over you know time horizons of months or years um by default unlike unlike the ai system yeah i think something that's come up for me during this discussion is that it does just feel like there's a lot more uncertainty around what it would look like for AIs to be in control compared to what it would look like for humans to be in control.

[46:52](https://realalpha.app/v/FHCxnHU6jFQ/?t=2812) Because in the latter case, we know a lot about how humans behave in different circumstances. And we've got lots of historical examples of various different human regimes and how these things tend to play out. And yes, the use of AI tools does change the picture somewhat, but there's a lot it feels to go on when we're imagining what a like human regime might look like with the ai case it feels much more uncertain to me i'm wondering whether you both think that that's true um and if you know you think that that might be a reason in itself to just be like more worried about ai and control because there's so much more unknown or something um maybe i want to start with katya here i think i agree that the ai situation looks much more uncertain, which seems, I guess, connected with my sense that it's more likely because it seems like there are quite a lot of different scenarios where this could happen, like, you know, there could be a sudden power grab by one AI in various situations.

[48:00](https://realalpha.app/v/FHCxnHU6jFQ/?t=2880) There are like various different kinds of AI you could have. If that doesn't happen, then maybe there are like other longer term power grabs by, you know, like AIs that are good at particular kinds of work or AI that is like good at grabbing resources in some way or like there's just like these are sort of not necessarily in different scenarios. It's more like, wow, there are heaps of bullets to dodge to avoid this happening. Tom do you have do you have any reflections yeah so I agree there's a lot more uncertainty into what the AI takeover scenario would look like um and if if you have the view that we need a lot of things to go right and we need you know it's sad to be really ethical and to be really reflective then that uncertainty you know could play in favor of the AI takeover because if you think there's a kind of narrow target we're aiming for and that uncertainty can be helpful on the other hand if what you really care about is look i just want there to be some kind of recognizable descendant of human values that has a significant role in what happens in the universe then you won't want that uncertainty at all you know the kind of you just want a kind of common sense utopia then you

[49:06](https://realalpha.app/v/FHCxnHU6jFQ/?t=2946) might rather bet on not quite a utopia but you want some kind of human value representation in what happens then you might you might prefer the human power grab for that reason although again then, you know, I do think the downside risks from the human power grab is bigger. So in that sense, like the down, you know, the uncertainty for quite how bad it could get might be more extreme in the human case. Because humans are more likely to do something actively vindictive toward other humans, whereas in the same way that AIs are unlikely to care about specifically amazing for humans outcomes, they're unlikely to care about specifically terrible for humans outcomes that's that's one big reason and the other reason is um relating to the the risk of of threats that that has been discussed if you have human values you control a lot of resources then you are a potentially very juicy target to be threatened and if those threats end up being executed that could look really awful um from the perspective of human values if you're very worried about threats then you might be quite happy with the world where like yeah the human values just don't have that much don't have that many resources um in total because then at least

[50:13](https://realalpha.app/v/FHCxnHU6jFQ/?t=3013) there won't be any any threateners making massive threats against human values a minor point on that my sense is that in the current world like not much of the badness in the world is coming from like threats that get carried out um yeah i want to move on to talk about the practical implications here because i guess it seems that prioritizing um and being more worried about ai takeover scenarios compared to human takeover scenarios or vice versa um might lead to differences in the policies or strategies that you favor and might kind of force you to contend with some trade-offs um so i guess to start off with here tom what are the like top things that you think that we should be doing to mitigate the risks of human seizing power or like extreme forms of human power concentration so the key mechanism for human seizing power is the misuse of super intelligent AI systems and so there's kind of on a very high level two two things you can do about that one is you choose the alignment target for powerful AI systems such that those systems will not in fact follow instructions that would lead to power grabs and indeed you can actually you know go for an alignment target where systems will you know

[51:51](https://realalpha.app/v/FHCxnHU6jFQ/?t=3111) proactively just like a virtuous you know human employer would do um kind of flag when there might be risks of extreme power concentration flag when there might be individual humans might pose security risks to kind of really stabilize a society in which power is distributed. So, you know, that the first big class of interventions relates to model specs, you know, how should we design the behavior of really powerful AI systems to avoid extreme concentration of power. The second is around transparency. And in particular, it's often hard to agree ahead of time what people should and shouldn't do with their AI system.

[52:31](https://realalpha.app/v/FHCxnHU6jFQ/?t=3151) So it's going to be hard to find a model spec that kind of settles every single dispute about legitimate AI use. You know, what about this type of persuasion? Is that too much super persuasion or is that just legitimate, you know, company lobbying? You know, what about this, you know, use case for cyber activity? Is that, you know, patching a system or is it actually, you know, investigating a vulnerability that you could later exploit? So I think another part of the solution is going to have to be mutual transparency, especially into how people are using powerful AI systems. And the most important part of that is if an AI company or a government actor has access to particularly powerful systems or deploying them in particularly high stakes contexts like national security, military contexts, then I think it's really important that there's transparency into how those systems are being used. All right. And Katja, Same sort of question for you, but for the prospect of misaligned AI taking over, what would you say the top things to do are to mitigate those risks?

[53:28](https://realalpha.app/v/FHCxnHU6jFQ/?t=3208) I think the best thing to do to mitigate a range of risks here, including sort of power concentration among humans, would be to not build AI that's much more powerful than us, at least until we're very solid on being able to align it and being able to get these other things right about what we're aligning it to, etc. And I guess in practice, it seems like, to my knowledge, we're nowhere close currently to being confident that our AIs are aligned. And so I think near term, at least, the thing to do is to work on not building it, which is to say stopping it, pausing it, slowing it, ideally stopping it.

[54:22](https://realalpha.app/v/FHCxnHU6jFQ/?t=3262) yeah yeah tom so do you agree that yeah a pause or something would be helpful uh in the case of preventing extreme power concentration yeah brought in very sympathetic so we can take the case of um company coups and government coups separately so for company coups i think there's a very strong case that slowing down would help um it would allow the rest of the world to understand what these companies are building it would allow multiple companies to kind of and multiple countries to catch up to the frontier um and so i think the case there is very robust um for government power concentration um i think there are ways of doing a pause which would which would help significantly with that as well i also think there are there are kind of hand-fisted ways of doing a pause which could in fact concentrate a lot of power over ai development in a kind of small group of people who are unaccountable and can abuse that power. And so there are, I think there are ways of doing a pause that actually increase the risk of government extreme power concentration. I think this has been something that I would say, you know, the community of people worried about superintelligence hasn't paid enough attention to historically.

[55:36](https://realalpha.app/v/FHCxnHU6jFQ/?t=3336) And that's partly because they haven't had this human power concentration sufficiently on their

[55:40](https://realalpha.app/v/FHCxnHU6jFQ/?t=3340) **Tom Davidson:** radar. And I'm excited for people to think about, okay, let's pause, slow down AI, but let's like properly red team the plan for pausing it, assuming a worst case scenario about the government actors that they kind of inherit that new regulatory environment.

[56:00](https://realalpha.app/v/FHCxnHU6jFQ/?t=3360) And I assume that they try and abuse it as much as possible. And how robust can we make it to that attempted abuse? Can you give us some examples of the versions of a pause that might be bad from a human's grabbing power perspective? I mean, this is going to be, you know, really a really toy example, because I haven't thought in detail about this. But, you know, if you just said the president can pass as an executive order that says that now they just get to decide on a case by case basis whether AI companies are allowed to train and deploy new models on the basis of their own personal judgment about risk, then they could essentially just get AI companies to do whatever they wanted by threatening the abuse of their arbitrary power.

[56:48](https://realalpha.app/v/FHCxnHU6jFQ/?t=3408) they could be like look no that that ai system doesn't have the alignment i like try again i'm not allowing you to train or deploy it yet um and um then you know once a company you know comes with comes with the setup that that they like then they could say okay great now this ai system is is you know going to be helpful only for me but for everyone else will have massive guardrails yeah you're allowed to kind of go ahead and keep developing and deploying that system so that that's a setup which i think um you know could could increase the um the risk of government power concentration by contrast with a scenario where that there's an ecosystem of you know third party auditors that aren't you know part of the executive branch they can't be fired um at the whim of the president and whose recommendations are a requirement for blocking a deployment or whose kind of safety approval is a requirement for going ahead with deployment, that would distribute much more the power over AI development. And I also think it would be much better from an alignment perspective, because you just get more open debate, more expertise going into the decision. I would add to that, that like, to the extent that,

[58:03](https://realalpha.app/v/FHCxnHU6jFQ/?t=3483) you know, the people involved maybe don't even know to what extent what they're involved in is human power grab versus humans being manipulated by AI's power grab. That, you know, there are plausibly cases where the executive, you know, does something bad, where it is actually an AI power grab causing it. And so like, if you think the AI is misaligned, then you probably also want to avoid these more, someone has the ability to grab power and run away with it situations from the perspective of avoiding AI power grabs. Right, so the idea there is if we put one person in a position where they can hit the green light on let's deploy everywhere, then that gives the AI now a really easy route to seizing power because they just have to convince this one person that things are safe and that it's in their interest to, you know, hit the green button, then suddenly the air is going to be deployed in the military.

[59:07](https://realalpha.app/v/FHCxnHU6jFQ/?t=3547) It can easily seize power. Yeah. Not having thought about this much at all. So am I right in thinking that your, yeah, Katja, your ideal version of a pause lines up reasonably well with what Tom's ideal version of a pause might be? Or do you think you disagree with him somewhat? Probably disagree somewhat. But between those two, that sounds better. But I would rather my partner David Kruger's idea of get rid of a lot of the compute. Don't make it the case that anyone can, or some group of people can quickly be building very advanced AI again. because, for instance, that means that, you know, other people...

[1:00:00](https://realalpha.app/v/FHCxnHU6jFQ/?t=3600) People have to be scared of that happening if that set of people does it. It's quite fast for things to turn around, whereas if we just didn't have the capacity to make very dangerous stuff being built more and more all of the time around us, the situation would be more stable. Yeah, I agree that would be a benefit. i can imagine where that there will be cases where these things conflict a bit more and the kind of cases i'm thinking about are cases where um you have the option to give say a government agency quite extensive um querying rights to like look at all the logs of what an ai company is doing and to to review everything and you know on the one hand that might help with misalignment because the government can find all evidence of misalignment and query everything but on the other hand that government agency could potentially abuse the power to now understand everything that any user of the ai system is doing and kind of set up a surveillance state style system or you know punish the ai company of its ais or ever doing activities that the government doesn't like from its partisan perspective and i would probably be more nervous about things that give potentially abusable powers to government agencies,

[1:01:26](https://realalpha.app/v/FHCxnHU6jFQ/?t=3686) especially if those agencies could be quite easily co-opted by a power-seeking president. Whereas you might more lean towards, look, we've just got to do this because AI takeover is the really big risk. We run the risk. We cross our fingers that they don't have used the power and that's worth it overall. I think that's right. I was describing maybe my ideal scenario, But it's true that I would I would say yes to quite a lot of different, you know, pause scenarios over, you know, full steam ahead. And yeah, I do think we probably would disagree about some of them. yeah i i think i want to actually maybe move on slightly from the details of a pause because i do think lots of people think that getting a pause to happen just isn't feasible like geopolitically people agree to it or it's going to be like really difficult to monitor or something like that and i think i'm curious katya if if this because this is kind of like your plan a but if this is not feasible what's your next best proposal for mitigating the risks of misaligned takeover and does that feel like it trades off at all against um the efforts to mitigate power concentration um so i guess before actually answering that i want to note that like i feel like many many more

[1:02:48](https://realalpha.app/v/FHCxnHU6jFQ/?t=3768) people thought that and then uh i think you know their opinions seem to or some of their opinions change as things happen. For instance, you know, the US government stops AI briefly, that sort of thing. And so I guess I sort of, I do expect them to keep changing their minds. It feels like there's a lot of, well, currently the Overton window is here. So this is where it's going to be forever. Oh, wait, the situation changed. Now it's over here, but probably it'll be here forever. And I guess I do expect that as things get more crazy, the sense that it's totally outside the realm of possibility to do this will sort of evaporate. But supposing they're right that it is impossible, I think my next preferred intervention is going to be more meta and it's just going to be like try and cause as many people as possible weighted by them potentially having any influence to understand the problem where you know which is also pretty Similar to how I'd go about, like, trying to cause some sort of pause to happen.

[1:04:04](https://realalpha.app/v/FHCxnHU6jFQ/?t=3844) And is that at odds with power concentration type stuff? I think people are more likely to act on a thing if they think it's, you know, terrible for reason A and terrible for reason B and reason C than if, you know, it seems like a narrower problem. It depends what sorts of levers people ultimately want to pull. But I sort of expect that just like more awareness that this kind of thing will happen potentially would go along with more awareness about the power grab type stuff, which might even more effectively lead to like earlier steps on that, since it seems like less effort has been put into it.

[1:04:43](https://realalpha.app/v/FHCxnHU6jFQ/?t=3883) Yeah, maybe we should talk then about some kind of contentious, politically salient questions that your disagreements might bear a bit more strongly on. sure um one idea we could talk about here is you know there are some people who think it would be a good idea to um have one big agi or super intelligence project um and you know that way it's kind of maybe easier to impose some safety infrastructure on just one project uh maybe it buys the ability to pause late like as needed um without like losing out to a competitor or something like that i would guess that this sort of thing sounds somewhat worrying to tom given that having one big project um probably does increase the risks of concentrating power um in the hands of a few people i'm curious firstly tom if that's correct and secondly Katya, if you also feel similarly worried about having one big project?

[1:05:53](https://realalpha.app/v/FHCxnHU6jFQ/?t=3953) Yeah, I think, as you say, there's a single point of failure if there's a single project. My understanding is that it might be quite hard to set up a single project within the United States that couldn't be co-opted by the president. I really haven't looked into this, but in general, it's surprisingly hard to create government institutions that are not within the executive branch and it's at the moment quite hard to stop the president from gaining control of institutions that lie within the executive branch so i think there is a real challenge a direct challenge to creating a single project and then you know even if it's not the president there would be some you know some small group that's running that project that would pose a distinct risk and would have no other rivals um i also just disagree that it's going to be that hard to coordinate multiple projects to slow down like it's not that hard to tell if projects are training powerful ai it's not that hard to just tell them that's illegal um i think it's you know totally totally feasible yeah i think i agree with that that it you know the difficulty of coordinating multiple projects seems pretty overblown um and i guess yeah i'd say overall i'm

[1:07:05](https://realalpha.app/v/FHCxnHU6jFQ/?t=4025) uh also decently worried about one big project like is it better or worse than the current state of affairs not sure i think it seems worse in that i i kind of expect this one project to then have quite a lot of power itself and to be like entirely full of people who really really want to build ai soon um and so it's not to actually be a very conducive um environment for a pause coming about uh unless it's like you're imposed from outside and the outside imposing seems like it might be easier with numerous less powerful, less coordinated projects. Whereas it seems like the way in which it might be easier is if they're trying to coordinate it themselves. And I guess in terms of how humans operate, I less buy that they would coordinate it themselves once you have That entity, my preferred number of projects would be zero.

[1:08:07](https://realalpha.app/v/FHCxnHU6jFQ/?t=4087) **Tom Davidson:** Yeah, I previously said my preferred number of projects, two or three, because I think you get a lot of benefit of power distribution from just a kind of a fairly small number of projects.

[1:08:16](https://realalpha.app/v/FHCxnHU6jFQ/?t=4096) I think once you're introducing 10 projects, it might get a bit harder to coordinate and you might have some more difficulties with kind of races to the bottom. Although I still think you could do it for sure. um but yeah i think i think you get a lot of benefits with just a second project because you can have the ais from each project check on it put the ais from the other project which decreases misalignment risk and also decreases the risk of secret loyalties um where human purposely makes ai secretly loyal to them and yeah you could just argue that um if there's a single project then um ai doesn't even have to bother colluding it's now all the super intelligences are like naturally, you know, of one ilk. And so maybe the AI takeover challenge is easier. So something else that there's, that's kind of a hot topic at the moment is sort of how the US should approach competition with China over AI. Some people are very pro racing and trying to beat China. Some people are very against it. I'm really curious about how the amount that you worry about like human power grabs and you know other forms of human power concentration might affect your view on trying to beat China yeah maybe it makes sense to start with Tom here

[1:09:35](https://realalpha.app/v/FHCxnHU6jFQ/?t=4175) what do you think so I do think that after I thought about the risk of extreme human power concentration and came to thought think that would be pretty bad I then reflected on the situation with China and said well if China ends up you know completely dominating in AI and you know getting a decisive strategic advantage it is pretty much a worst case scenario for power concentration because you have a society where power is already extremely concentrated where there isn't a norm for kind of having lots of reflective deliberation and an open-minded debate and I do think a very very likely scenario there is you have one person that controls everything um so that did make me more worried about a world where china dominates but i want to stress you know the risk that china wins is just being used over and over by all of the company ceos and will no doubt be used over and over by the kind of government actors that want to race ahead and i do think we should be we should be skeptical of those arguments and you know currently china isn't that close i think to being able to push the frontier um in ai and you know know, you don't want to race ahead to beat China. And then you realise that, you know, you've ruined

[1:10:48](https://realalpha.app/v/FHCxnHU6jFQ/?t=4248) democracy in the US. And so now, you know, you try to avoid extreme power concentration in the hands of China, but you've just got extreme power concentration in the hands of a few people in the US. So I do think we should be sceptical of those arguments. But at the same time, yeah, I do think that this is an additional reason why if we're going to pause, we should, you know, do a deal with china make sure they're pausing as well and i do favor trying to find a way to do a pause where china isn't just catching up to the us and where it's not like oh when we undo the pause now china is like much more likely to win the ai race i think we should try and pause in a way which kind of maintains the us's chances of winning and maintains china's chances of winning um which is a pretty a pretty reasonable thing for both sides to to agree to but some proposals for pausing in in particular, the AI Futures project, Plan A, does actually give a fair amount more relative influence to China if you do the pause, but then the pause breaks down.

[1:11:46](https://realalpha.app/v/FHCxnHU6jFQ/?t=4306) Yeah, Katja, I'm interested in your views on trying to beat China and whether your views would change at all if you cared a bit more about human power concentration. Okay. Tom on various things there. And especially importantly that like, I think this is, I don't know, brought up a lot as an excuse for going fast by people who want to go fast and is like expected is a smaller issue than it is, you know, painted as. I guess also I don't know how much effort is going into, like, you know, organizing for China to also pause, my guess is, not as much as I would like by a large amount. I guess I feel like the game theory of this is often described quite weirdly in that, you know, if this is really dangerous enough that we would want to pause, like, that incentive also applies to China.

[1:12:47](https://realalpha.app/v/FHCxnHU6jFQ/?t=4367) I feel like people talk about this as if like China is, you know, entirely consists of people unable to understand that or like intrinsically evil so that they just want to do whatever will be bad or something. It's sort of like they presumably also don't want to be destroyed by AI or to have some sort of dystopia. i do think there's like you know it's more likely that the you know um that someone in power there like a human in power there would want to have like a human coup whereas here um maybe power is more spread out uh so that changes things somewhat but yeah i guess i i think we in both

[1:13:33](https://realalpha.app/v/FHCxnHU6jFQ/?t=4413) **Katja Grace:** cases should be trying pretty hard to coordinate with them to not be doing this stuff i guess it just seems like trying to race them mostly increases the chance of all of this stuff happening because it shortens timelines i think we often talk about who wins and i think that's

[1:13:49](https://realalpha.app/v/FHCxnHU6jFQ/?t=4429) sort of a bad way of describing it that we shouldn't be doing yeah so before we wrap i think it would be helpful for you to put yourself into each other's shoes a bit more maybe this way we can tease out some more agreement and disagreement and just like where things feel like they stand after this conversation so the first exercise i want to do is give you each about like maybe two or three minutes each to outline where you think you most agreed with your opponent on um and what you felt the weakest parts in your own case were which may relate to one another um who wants to start i don't mind i'm happy to okay maybe let's start with tom then so i i agree with katya about how pausing is just a pretty robustly good thing to do a lot of the dynamics driving that all of these risks that society does not understand how big these risks are and i think being able to kind of very flexibly pause tap the brakes while also getting you know evidence of everything that's happening to as many people as possible transparency i think those those two things are going to be you know really helpful for for both risks so i think you actually didn't stress transparency but my guess is you'd be in favor

[1:15:19](https://realalpha.app/v/FHCxnHU6jFQ/?t=4519) of it and and you know obviously directly talked about pausing so i think that's um a big point of agreement that I had in terms of where my own argument feels the weakest I think I feel a bit dissatisfied with my argument insofar as I'm kind of saying look it's just really hard to get to a great future AI is very high variance maybe it still has human values and is reflective and moral that's kind of less likely than with the human and so we should kind of roll the dice that we should prefer that dice roll over a human-controlled future. You know, there's a certain logic to it, but I think intuitively it fits.

[1:16:00](https://realalpha.app/v/FHCxnHU6jFQ/?t=4560) It was a bit like, I can imagine there's a part of me in Cheers to you that just is like, no, let's just like stick with the human that we know, rather than go with this abstract argument around how maybe like a really kind of best case scenario is more likely with the AI system. Same questions to you, Katja. I agree that we seem to agree a decent amount about like what to do about it. in terms of the the weakest spots of my argument i feel like maybe i'm feeling too good about rolling the dice with a human um i think i might not be enough like taking into account how likely they are to ask for some sort of narrow thing or be like really foolish quickly and cut off their options in the long term and i feel like there was a there was a thing near the start that we um um maybe disagreed on and didn't get into that much of like is is there sort of a big space of uh like gradual cases where some humans take power gradually i think cases where like lots and lots of humans do um seem pretty plausible or like maybe that's like the most likely case of what a non-catastrophic outcome looks like.

[1:17:21](https://realalpha.app/v/FHCxnHU6jFQ/?t=4641) Yeah. So you've mentioned some disagreements here. And I think something that's been interesting to me is despite you having a fair few disagreements on the topics of which of these things is most likely and which of these things would be a bad outcome, nonetheless, you've had quite a lot of convergence over what to do. I'm curious about whether either of you have reflections on what the nature of your disagreement really is and why it's not showing up too much over like when it comes to interventions um and what you think the actual stakes of your disagreement are with that in mind yeah should i maybe pass to katya first here My guess is that a substantial part of our disagreement is about like how likely alignment is.

[1:18:13](https://realalpha.app/v/FHCxnHU6jFQ/?t=4693) And then I think a reason that we're maybe not running into disagreements that much about what to do is like the different things we're concerned about here are sort of like different sides of the same basic problem. And I think a different way of describing our disagreement is, I think, like, I feel like there's, like, a pretty deep general problem where if you make a bunch of creatures that can overpower you and outwit you in every way and put them out in the world, like, you're going to run into trouble one way or another. And so I guess it feels more like, you know, if there's currently a plan to nuke your city and, you know, you could be like, well, is this person going to press the nuke button or this person or like, you know, how's that going to affect the water or the air?

[1:19:07](https://realalpha.app/v/FHCxnHU6jFQ/?t=4747) You prefer to just be trying to stop the plan to nuke your city. I think like if you're worried about powerful AI doing one bad thing or another, it still seems maybe just good to not build a powerful AI, and my sense is that maybe I feel more like that because I more think it's going to be misaligned and that there are just going to be problem after problem, and maybe that Tom feels somewhat more like maybe it will be aligned and then we might be left with only the problems that are kind of like humans using it for bad where human power grab is like quite a big one and maybe you can solve that substantially by focusing on that problem to the extent that that's also helped by not building powerful AI or doing it more carefully and slowly.

[1:20:03](https://realalpha.app/v/FHCxnHU6jFQ/?t=4803) So I think our disagreements are very much on the kind of super conceptual, abstract side of things. So there's an analogy you can give to kind of deciding what to do and what to believe, which is like a kind of a web of belief, where you have like certain beliefs in the center, which in some sense are kind of foundational and really important. And then they then propagate outwards towards the edge of the web, which, you know, is your beliefs about what's going to happen tomorrow and also your views about what should we do tomorrow. And I think, you know, our disagreements are like quite far, quite deep into the web, like so deep that, you know, actually there's not been much thought about this stuff.

[1:20:39](https://realalpha.app/v/FHCxnHU6jFQ/?t=4839) And I think, you know, as our discussion showed, there's just loads of kind of complicated considerations on both sides. it is quite hard to see how they net out but i think you know you can change the center of the web a fair bit and still find that when you get to the edge of the web the action recommendations aren't that different um and it's like okay this is yeah a super powerful technology which will be capable of you know disempowering everyone and you know staging a coup um yeah man we need to like make sure everyone understands that we need to like slow down until we have a plan for how to handle that um yeah not too surprising that a lot of those kind of you know much more pragmatic opinions are shared um i do think that if we had really dug into the nitty-gritty of how to implement specific proposals we might have found that that those conceptual disagreements were were driving more more disagreement you know like i would still like more people thinking about superintelligence to focus on the extreme power concentration angle because you know it's like you know 20 30 times fewer i think than think about alignment and that's only very very recently increased so the kind of historical work is is a much bigger ratio and you know in thinking about

[1:21:56](https://realalpha.app/v/FHCxnHU6jFQ/?t=4916) a centralized project i would i would be like really upset if we implemented a centralized project and people proposed it without doing the red teaming for for power concentration um consequences where maybe katya would like agree with me that we should do the red teaming but it wouldn't be like something that she really cared about. Yeah, I also meant to say that like what kind of thinking one should do on the margin seems like maybe a clear disagreement between us. In particular, it seems like probably both of us could work on the things that the other person is working on. And so, yeah, like how much to like theoretically think about these things.

[1:22:34](https://realalpha.app/v/FHCxnHU6jFQ/?t=4954) Yeah, it's a disagreement. Yeah, I'm interested in what evidence or new information you could imagine getting in the next couple of years that would update you in a different direction from your current position. Like, are there particular types of warning shot that might make you think differently about this or like particular evidence about the character and behaviors and propensities of AI systems that might skew you one way or another? What are you looking out for? I think in practice, if I were to change my mind on this, the sort of evidence seems much more likely to come from like me learning more facts about the current world or like thinking through it more myself.

[1:23:24](https://realalpha.app/v/FHCxnHU6jFQ/?t=5004) For instance, I guess I think I'm assuming various things about like, you know, what does it look like inside an AI company and how easy would it be for a particular person to like affect how the AI is trained in a sufficiently important way to cause this kind of thing. And I think if I learned that it more looked like one person could do that sort of thing without many other people having to let it pass, that would make a big difference. Yeah, same question to you, Tom. I think the big thing is the chance of misalignment. So I have updated based on the Hugging Face stuff. I was really, really surprised by that.

[1:24:07](https://realalpha.app/v/FHCxnHU6jFQ/?t=5047) And, you know, I'm going to keep thinking about it.

[1:24:09](https://realalpha.app/v/FHCxnHU6jFQ/?t=5049) **Tom Davidson:** And, you know, I think it's quite plausible that that shifts me, you know, to thinking that, yeah, the misalignment risk is notably bigger overall.

[1:24:22](https://realalpha.app/v/FHCxnHU6jFQ/?t=5062) and I think it'll be really interesting to see the response that the world takes to the recent incidents and any other future incidents that come where you know I still have a kind of perhaps naively hopeful expectation that okay wow we've got really quite compelling evidence of messed up misalignment like surely like as this filters through it's not going to happen overnight but as this filters through and like people can go in with briefings of like really really compelling um evidence like the world is going to change its stance significantly and so if that doesn't happen um which it might not then again that that that that's going to make me you know but you know more worried about misalignment because i'll lose this this kind of idea of the warning shots help helping out significantly um and you know on the power concentration side there have unfortunately also been updates towards power concentration being more worrying than I thought. Actions of the current admin, as well as just it seeming more plausible that there's going to be a very small number of AI companies has made those risks seem bigger too. But overall, I have been updating towards misalignment recently. Yeah, it's all

[1:25:36](https://realalpha.app/v/FHCxnHU6jFQ/?t=5136) super interesting to hear. Thank you both so much. I think that's all we've got time for, but it's been a pleasure having you both on the show. Pleasure talking to both of you. Yeah, it was a lot of fun. Thanks very much. All right, that was our first ever debate on the show. I really enjoyed hearing Tom and Katia thinking through these quite thorny questions. And yeah, overall, I think my main takeaway from this conversation was pretty encouraging. So it turns out that even if you disagree quite a lot as Tom and Katya do, over which dangers from AI are most likely and most consequential, then you still might actually converge a fair amount on what we should actually be doing about AI as a society. And that seems promising, because I guess a worry that I've previously had is that there could be pretty big trade-offs between trying to mitigate the risks of AI takeover versus trying to mitigate the risks of human takeover.

[1:26:48](https://realalpha.app/v/FHCxnHU6jFQ/?t=5208) And I'd say that I feel a bit less worried about this after hearing Tom and Katya hash out what to do about AI and having this sort of very honest, reflective conversation and just hearing them end up agreeing on so much despite their conceptual differences. So for example, there's probably a version of a pause in AI development that they both at least agree to. And they're both pretty nervous, you know, maybe in somewhat different ways about having a single big AI project. And despite his belief that China dominating an AI would be really quite bad from the perspective of power concentration, Tom is still skeptical of arguments that we need to be racing against China.

[1:27:44](https://realalpha.app/v/FHCxnHU6jFQ/?t=5264) It seemed that both he and Cartier maybe favored negotiating with China about slowing things down versus racing now at the same time it does seem like there are still real questions over you know how we should be allocating very limited resources and things like how much effort should be going into red teaming a project for the risk of enabling a power grab or something like that anyway that's it from me i really hope you enjoyed today's episode please let us know if you'd like to see more content like this. But you know, if you also never want to see something like this again, I guess that's also valid. Either way, thanks for watching.