AI Overlords vs Power-hungry Humans: Which Should Scare You More?

yes yeah broadly i will say recent events um with the hugging face incident have definitely been an update towards me being more concerned about um misaligned um ai takeover but yes i think that an extreme power concentration should on the margin be getting more attention than it's currently getting great thanks for that clarification so before we dive in it is worth just acknowledging One view that isn't represented here, which is that human power grabs and other harmful ways in which humans could use AI are the only serious risks that AI poses. And basically, we just shouldn't be worried about AIs taking over at all.

Ultimately, I am biased here. At 80,000 Hours, we do just think that it's at least plausible that advanced AI systems could disempower humanity. And we've got an article about loss of control that you can read if you want to understand why. But ultimately, lots of people who have thought seriously about this do seem to agree. In fact, one of Katya's surveys reveals that the median machine learning researcher thinks there's about a 10% chance that AI would take over and cause an existential catastrophe. Tom, Katja, is there anything you'd like to add before we get started? I don't think so. Awesome.

Let's dive in. So let's start with the question of what's more likely to happen, humanity losing control to misaligned AI or losing control to a small group of people using AI. Katja, you think it's much more likely that we would lose control to misaligned AI. Can you explain where this is coming from? Is the idea that AI is going to be more inclined to seize power or more likely to succeed or both yeah um the background thing i expect if you have um you know more capable than us creatures running around that have you know misaligned goals is that they will eventually get power uh you know everything that has goals in some sense would like power or just like would would like power to achieve their goals and then in these cases where there is some way for um you some set of AIs and humans to grab power quickly, it seems like it's maybe at a very high level down to whether the AIs are aligned or not, where if the AIs are not aligned, quite plausibly some humans think they're grabbing power for a bit, but like they aren't. If the AI is aligned, then plausibly humans grab power. But it still seems like the situations where a human could seem substantially rarer because it's harder for a human to be in a position where they are

the, you know, the user for a whole bunch of AI. So that all that, like if the AI is aligned, that a whole bunch of AI will do what they want. Whereas I guess another update from the hugging face thing, I guess it seems more likely that a bunch of AIs will actually act in accordance with one another if they're misaligned. And so those kinds of situations seem like more likely to arise for AI. Yeah, I think that's an interesting one. So you could imagine a situation where there's two really powerful AI companies that have developed different AIs. And we could consider the possibility that both of those AIs, let's say it's Claude and ChatGPT, what's the likelihood that those two different families of AIs would collude in order to seize power from everyone else?

and then another question we could ask is okay how likely is it that the um the leaders of both of those two companies would use their super intelligent ais to collude with each other again to collude against everyone else to seize power and so i guess your claim is oh it's more likely that claude will collude with chat gpt compared to the probability that um you know two company ceos or two companies will collude with each other or the possibility that that, you know, a government leader would collude with lab leaders. I think that's... Part of it, but not all of it. Like suppose one of the companies just has taken over most of the market. It seems more likely that all of the instances of Claude coordinate with each other than that a human manages to be in a position that they become the thing that all of the instances of Claude are loyal to without any other humans getting in the way.

okay okay yeah okay so one way of putting it would be with the human power grabs there's this extra step where you need this um all of the ai labor to be centralized even within a single company under one human whereas you're saying okay but it's a bit more likely you would say that all those ais would start working together um i think i you know historically i've been skeptical i've said well you know there's every reason to think that different rl aid you know AIs trained by RL could be incentivized to rat each other out. And I do think it will be possible to do schemes of that sort. And, you know, at that point, it is a bit less clear to me why they would all naturally be colluding with each other rather than with humans.

I guess I think in that case, also, it seems pretty plausible that we have AI, you know, helping us to notice attempted human power grabs and so on. My own guess is that the kind of slower going thing where it's like gradually power goes from humans to AIs is more likely than either of these. But yeah. I think, you know, backing out, I think I agree that this collusion point is in favor of AI takeover, especially after the Hugging Face incident. But then I think there are other factors which push towards human takeover being more likely. I just think we should actually be more confident that humans will want to do this compared with AIs.

You know, the people that we see leading, you know, countries politically and that we see leading the top AI companies, often people who have been willing to seek power and who have had the belief that, yes, if they do things, if they are in control, then things will go better. Another big one is that if people see the risk coming and try and stop it, then everyone is kind of aligned in their incentives. All the humans are aligned in wanting to prevent AI takeover. You know, ultimately, it's in everyone's interest to pause, as you've yourself pointed out. Whereas with the human power grabs, even if we see it coming, and we have some evidence that this is a risk, you know, part of what makes it really tricky is that there will be very powerful humans that will be pushing ahead nonetheless. And I think that's like a fairly big factor in the other direction. And I'll mention just, actually, I'll let you respond to that before I come in with more.

On the, you know, humans are sort of all on the same side here. I do think it's sort of more complicated because often humans do think they will get power by empowering AIs in certain ways. I maybe doubt that they will and so do think that they're, in most cases, wrong about their incentives. But it seems like at least decently likely that they do that. And I guess like a different but related point is sort of like trying to stop humans from grabbing power from each other is a thing that, you know, we've sort of always been doing and do have a bunch of interventions in place in the world to avoid.

Whereas the sort of whatever AIs are doing together is kind of a wild west that we don't even know what is happening in potentially. right um so i don't think that our interventions for stopping humans from seizing power are that great um and you know both talking about government actor seizing power and company actor seizing power i don't think we're very well set up for that i mean especially because they haven't really been developed in a world with extremely powerful ai um and so you know they are designed for a world where all technology are tools and where technology isn't out I'll see you next time.

where you still need loads of human hands to get significant projects done to, you know, to fair war, et cetera. And so I think there are some pretty, pretty scary gaps in the defences against human power grabs. You know, I totally agree that humans are also going to have some coordination difficulties and humans will kind of impose a take of a risk on everyone else in order to gain power for themselves. that is that is right and that's a real dynamic um but I think it's still you know quite importantly different in that if humans did all come to an accurate picture of the situation then you would be in a situation where the current power holders have this very strong incentive to coordinate to slow down and invest in alignment and that is importantly different from a situation where I know there are still these actors that have the opposite incentive um have the incentive to keep pushing forward with AI progress. And so I do think that there's a differential factor there, which is, you know, does push towards human takeover. That seems right, but I'm not sure if it's big. I guess it seems like if everyone understood their incentives, it would still be like everyone in the world except some very powerful person against that

person um and like how much power does any one person have in the current world if they were up against like everyone else in the world including the second most powerful person that's right so you get into discussions about okay how big is this group that's trying to seize power if it's if we're talking about an ai company it's the leading ai company um and they're kind of unified then now they actually do have a lot of power and potentially they're going to be able to get a decisive advantage if you're talking about you know the current administration um of you know the u.s government or future administration then again they're they're pretty powerful and it's hard for other actors to coordinate against them i guess i think in both of those cases um they are pretty powerful but they're also like um i mean not very coordinated and like all the people in them do have loyalties to a bunch of people outside and like it would be very hard for them to I think keep such things secret if they required secrecy for long what with everyone having you know partners and friends and so on um and you know also the moral qualms probably

One big unknown in this conversation seems to be the probability of misaligned, like AI being misaligned in the first place. Yeah, I guess I'm curious whether Tom would agree that like, you know, if AI is misaligned, then basically like we're not getting humans taking power. I'd agree. I'd agree. If intelligence is misaligned, game over. And if it is aligned, then I guess we haven't specified different sorts of alignment, but it seems like there's relevantly kind of intent alignment where the AI does what its user wants or sort of like value alignment where it has human values or something.

Yeah. I would say like the kind of human power grab pretty much happens in the intent alignment case and the good outcomes happen in the other case. Does that also seem to basically carve it up? I think, you know, zooming out and squinting, that seems right. I think if you have a really thoughtful and well-considered alignment target, then I think you can stop human power grabs. Of course, part of the problem is that what the alignment target is will be a product of human actions, including power seekers. And so that's not like that's an independent role of the dice, you know, what kind of alignment we end up with.

in an unfortunately realistic scenario where everyone's going quite quickly. You could totally see the AI company saying, of course, we're going to have systems that do whatever we want internally. And you could totally see the government saying, of course, we're not going to have AI systems disobeying our instructions and our orders in the military and the government. And so that's how we end up getting intent aligned with maybe a lot of instances of a system or whatever pointed at one person or something like that. Yeah, exactly.

One relevant consideration here is, like, how seriously these different risks are currently getting taken by decision makers. Because, like, if, no matter, like, how many warning shots there are, if people who are currently in positions of power are just, like, not inclined to be taking those warning shots as a sign that AI could genuinely sort of autonomously seize power, then probably not much work is going to happen, like regardless. How seriously is this risk actually getting taken? And how is that going to influence how likely we are to sort of solve this by default? I think it's hard to know, because it's politically very difficult for people to talk openly about human power grabs so you know there have been congressional hearings where people have talked very explicitly about ai takeover um and you know they're not being laughed out the room um and that hasn't happened for ai ceo takeover or for executive um government takeover as far as i'm aware but that doesn't mean that people aren't more privately worried about those risks as well. They're just harder to discuss. And I agree with what Katya said earlier, which is a lot

of people are very biased against the possibility that AIs would even, you know, automatically want to seek power in the first place. Yeah, one other thing that's interesting here is that some people actually think that we shouldn't even be having this conversation and that we shouldn't really care about the divide between scenarios where AI takes over versus scenarios where some humans grab power because that divide is blurry or because, in fact, we should just be expecting some combination of those things to happen. Katia, what do you make of that? The chance of some combination happening seems quite high in the sense that in many cases where AI is actually the more powerful entity, it makes use of humans in various ways or they think they're grabbing power, I think it's also sort of ambiguous because some people are more happy to identify with whatever it is that AI says they should do or to think of themselves as in power if they have some connection to some AI that is grabbing more and more power and they own it but can't really tell it what to do in any way because then they would get less power or something like what do you call that uh i'm inclined to call that like the ai got power like whatever values you had before this whole thing sort of got lost along the

way i don't think it follows that we shouldn't think about this i think there's like more reason to think about it um and you know try and categorize the different situations and pay attention to who has power in them yeah tom do you agree do you have a different opinion I agree with Katya that there are scenarios where misaligned AI use humans to take over and they could be thought of mixed scenarios but I also agree that ultimately I would categorize them as AI takeover scenarios because you end up with the misaligned superintelligence which presumably can kind of discard with the humans that helped it along the way I do think that there are very plausible like fairly pure versions of the two scenarios so you know I think there are very plausible scenarios where we just solve alignment um to a strong degree and in fact that might make human power grabs more likely because there might be common knowledge that we've solved um alignment to a strong degree you know within the leading ai developers within the government which could then motivate um the use of ai to seize power and i also think there are pure scenarios where the misaligned ai takes power without convincing a small group of humans to

help them seize power as an important step along the road um so you know there's many many ai Bye. takeover stories have AI systems, you know, biding their time until they're just deployed across the economy and the military, no need for human power grabs along the way. So I do think that kind of on the causal side, it's useful to separate them and that the pure versions are pretty plausible. In terms of, you know, mitigations, some mitigations, you want to separate them out. You know, if you want to do lie detection, for example, as a mitigation against human power grabs, that's obviously going to work for human power grabs, but not for AI takeover.

And similarly, you know, alignment research techniques are, you know, not going to help the human power grabs. Other mitigations are useful for both. So if you're doing monitoring of how AI is being used and looking, is it hacking? Is it breaking the law? Is it doing something it's not meant to be doing? That helps with both of the threat models. So, you know, in general, I think the interconnections between the threat models are pretty complicated. But I'm certainly not convinced that it's all fuzzy enough that it's not a useful way to categorize things. Yeah, got it. let's move on um because yeah i think you know regardless of whether it's humans taking power or ai's seizing power um it does seem like there's a variety of possible outcomes and you know things could be extremely bad in both cases um but it does seem worth comparing just how bad they would be, especially when we're thinking about things like what interventions to pursue, how to allocate resources, what work to prioritize, and so on. So yeah, I guess with that in mind, Tom, you think that takeover by humans might potentially have worse outcomes than takeover by AIs. And I guess I'm not sure what I think about this, but some people might find it kind

of unintuitive just because even quite power hungry humans seem at least somewhat likely to want to preserve some things that other humans value compared to AIs which seem less human-like but it seems like a hard one what's the case for thinking that we could be worse off even with humans still being in charge so one one argument you could make is that getting to um a really good future is is quite a hard target to hit it's not enough that we have someone in charge that cares a bit about kind of human-like things like you know likes adventure likes a good laugh cares about human relationships unfortunately you know just as if you imagine going back a thousand years and picking some random leader and imagine giving them you know full control over what happens in the universe they would have cared about those things but they would have also had lots of other values which were like you know horrific and just missed you know not being open-minded enough to realize a lot of the things that we now consider valuable today you might similarly think that yes if a human has power then they will care about some things which are good but that's just not nearly enough to get us to a great future whereas with the case of the ai system the argument would

go there's actually a bit more kind of variance there um you know ai systems today are trained to be ethical um and uh are um typically pretty open-minded if you kind of ask them about you know political questions moral questions and so you know the idea would be well there's at least a chance that when ai takes over um and then it decides what to do with the universe the kind of context is in triggers those though that kind of thinking pattern which is you know thinking about ethics, which is being open-minded and reflective. And then if we're in a lucky place where it's ethical intuitions, actually humans succeeded to some degree of getting something that roughly in many ways corresponded to our ethical starting points. And then the AI is reflective enough that it's able to kind of reflect.

improve its views and actually end up in a really great future. So that would be the kind of the high level argument, just that we need to really nail it. And that there's actually a greater chance that we really get a really great outcome with the AI system than with a lone human. I kind of want to press on the chance of getting a really great outcome here, Because I guess I'm kind of imagining that even if AIs would be genuinely very reflective and moral and make good decisions for humanity, do you feel like there is still just something inherently undesirable about the fact that we've just like that humanity has kind of handed over the reins of its like future to AIs in charge?

like is there something that seems like inherently morally bad about that or does that does it like if it if it turns out okay does it feel okay to you no i think it's really bad yeah um and i also think it's really bad if you know one person steals their reins from all the other humans i think that is inherently bad and um you know a fairly similar way okay okay yeah so it's like better outcome but still pretty bad um yeah i guess the other thing that i want to i want to kind of maybe understand a little bit better about your view is that I think when I'm comparing these two risks um when we're talking about AIs taking over it feels kind of more obvious to me how this could result in like really large-scale doom because you can imagine these very sci-fi scenarios where like humans go extinct because AIs are like now the dominant race and use up all the resources or decide to kill us all um or like they decide to sort of permanently subjugate all of humanity or something like that um it's like easy to imagine how this becomes sort of existential scale i think it's a bit harder to imagine how um humans seizing power like a small group of human like power hungry humans sort of taking control could be equal like like similarly

bad in terms of scale like yeah can you spell that out so yeah we could compare a kind of um kind of universe-wide sense of catastrophe with a like on earth sense of catastrophe so you know if we're talking about the extinction of all living humans that's a kind of on earth existential catastrophe um my my guess is actually that both ai and humans would not kill all humans on earth um for for various reasons there's um kind of decision theoretic arguments behind that there's kind of crazy simulation um based arguments behind that um and there's also just the idea that even the ai system might care just a small amount about keeping humans alive um and so um it won't be very costly to do so um paul cristiano has a good argument about how you know if the ai just wants to maximize the total resources that it ever controls in the universe, delaying leaving Earth by a small amount costs it extremely little.

So I think in terms of actually all humans dying, I would guess that by the time AI is taken over, it will be competent enough and smart enough to realize that it doesn't really make sense to bother going around and killing all the humans for such a small gain. It's more likely that a human who seizes power will kind of not want to kill all the humans in addition, just because that is a human value that we know humans have and there's more uncertainty about what values AI will have. So I think there is a larger chance of human extinction if AI takes power. On the flip side, I think there is a larger chance of some really horrific negative things happening. If a human takes power, they might be sadistic, they might be vindictive.

they yeah and so you know certainly for people that have downside focused ethics that are really worried about suffering risks I guess that they would feel happier about an AI seizing power compared to a human. I want to move to Katya now I'm sure there's a lot of things that Tom said

the human wants when they say that or what they think the human wants overall or what people want overall or some combination of those. But yeah. I want to push on just a little bit because I'm

horizons of months or years um by default unlike unlike the ai system yeah i think something that's come up for me during this discussion is that it does just feel like there's a lot more uncertainty around what it would look like for AIs to be in control compared to what it would look like for humans to be in control.

Because in the latter case, we know a lot about how humans behave in different circumstances. And we've got lots of historical examples of various different human regimes and how these things tend to play out. And yes, the use of AI tools does change the picture somewhat, but there's a lot it feels to go on when we're imagining what a like human regime might look like with the ai case it feels much more uncertain to me i'm wondering whether you both think that that's true um and if you know you think that that might be a reason in itself to just be like more worried about ai and control because there's so much more unknown or something um maybe i want to start with katya here i think i agree that the ai situation looks much more uncertain, which seems, I guess, connected with my sense that it's more likely because it seems like there are quite a lot of different scenarios where this could happen, like, you know, there could be a sudden power grab by one AI in various situations.

There are like various different kinds of AI you could have. If that doesn't happen, then maybe there are like other longer term power grabs by, you know, like AIs that are good at particular kinds of work or AI that is like good at grabbing resources in some way or like there's just like these are sort of not necessarily in different scenarios. It's more like, wow, there are heaps of bullets to dodge to avoid this happening. Tom do you have do you have any reflections yeah so I agree there's a lot more uncertainty into what the AI takeover scenario would look like um and if if you have the view that we need a lot of things to go right and we need you know it's sad to be really ethical and to be really reflective then that uncertainty you know could play in favor of the AI takeover because if you think there's a kind of narrow target we're aiming for and that uncertainty can be helpful on the other hand if what you really care about is look i just want there to be some kind of recognizable descendant of human values that has a significant role in what happens in the universe then you won't want that uncertainty at all you know the kind of you just want a kind of common sense utopia then you

might rather bet on not quite a utopia but you want some kind of human value representation in what happens then you might you might prefer the human power grab for that reason although again then, you know, I do think the downside risks from the human power grab is bigger. So in that sense, like the down, you know, the uncertainty for quite how bad it could get might be more extreme in the human case. Because humans are more likely to do something actively vindictive toward other humans, whereas in the same way that AIs are unlikely to care about specifically amazing for humans outcomes, they're unlikely to care about specifically terrible for humans outcomes that's that's one big reason and the other reason is um relating to the the risk of of threats that that has been discussed if you have human values you control a lot of resources then you are a potentially very juicy target to be threatened and if those threats end up being executed that could look really awful um from the perspective of human values if you're very worried about threats then you might be quite happy with the world where like yeah the human values just don't have that much don't have that many resources um in total because then at least

there won't be any any threateners making massive threats against human values a minor point on that my sense is that in the current world like not much of the badness in the world is coming from like threats that get carried out um yeah i want to move on to talk about the practical implications here because i guess it seems that prioritizing um and being more worried about ai takeover scenarios compared to human takeover scenarios or vice versa um might lead to differences in the policies or strategies that you favor and might kind of force you to contend with some trade-offs um so i guess to start off with here tom what are the like top

So am I right in thinking that your, yeah, Katja, your ideal version of a pause lines up reasonably well with what Tom's ideal version of a pause might be? Or do you think you disagree with him somewhat? Probably disagree somewhat. But between those two, that sounds better. But I would rather my partner David Kruger's idea of get rid of a lot of the compute. Don't make it the case that anyone can, or some group of people can quickly be building very advanced AI again. because, for instance, that means that, you know, other people...

People have to be scared of that happening if that set of people does it. It's quite fast for things to turn around, whereas if we just didn't have the capacity to make very dangerous stuff being built more and more all of the time around us, the situation would be more stable. Yeah, I agree that would be a benefit. i can imagine where that there will be cases where these things conflict a bit more and the kind of cases i'm thinking about are cases where um you have the option to give say a government agency quite extensive um querying rights to like look at all the logs of what an ai company is doing and to to review everything and you know on the one hand that might help with misalignment because the government can find all evidence of misalignment and query everything but on the other hand that government agency could potentially abuse the power to now understand everything that any user of the ai system is doing and kind of set up a surveillance state style system or you know punish the ai company of its ais or ever doing activities that the government doesn't like from its partisan perspective and i would probably be more nervous about things that give potentially abusable powers to government agencies,

especially if those agencies could be quite easily co-opted by a power-seeking president. Whereas you might more lean towards, look, we've just got to do this because AI takeover is the really big risk. We run the risk. We cross our fingers that they don't have used the power and that's worth it overall. I think that's right. I was describing maybe my ideal scenario, But it's true that I would I would say yes to quite a lot of different, you know, pause scenarios over, you know, full steam ahead. And yeah, I do think we probably would disagree about some of them. yeah i i think i want to actually maybe move on slightly from the details of a pause because i do think lots of people think that getting a pause to happen just isn't feasible like geopolitically people agree to it or it's going to be like really difficult to monitor or something like that and i think i'm curious katya if if this because this is kind of like your plan a but if this is not feasible what's your next best proposal for mitigating the risks of misaligned takeover and does that feel like it trades off at all against um the efforts to mitigate power concentration um so i guess before actually answering that i want to note that like i feel like many many more

people thought that and then uh i think you know their opinions seem to or some of their opinions change as things happen. For instance, you know, the US government stops AI briefly, that sort of thing. And so I guess I sort of, I do expect them to keep changing their minds. It feels like there's a lot of, well, currently the Overton window is here. So this is where it's going to be forever. Oh, wait, the situation changed. Now it's over here, but probably it'll be here forever. And I guess I do expect that as things get more crazy, the sense that it's totally outside the realm of possibility to do this will sort of evaporate. But supposing they're right that it is impossible, I think my next preferred intervention is going to be more meta and it's just going to be like try and cause as many people as possible weighted by them potentially having any influence to understand the problem where you know which is also pretty Similar to how I'd go about, like, trying to cause some sort of pause to happen.

And is that at odds with power concentration type stuff? I think people are more likely to act on a thing if they think it's, you know, terrible for reason A and terrible for reason B and reason C than if, you know, it seems like a narrower problem. It depends what sorts of levers people ultimately want to pull. But I sort of expect that just like more awareness that this kind of thing will happen potentially would go along with more awareness about the power grab type stuff, which might even more effectively lead to like earlier steps on that, since it seems like less effort has been put into it.

think it would be helpful for you to put yourself into each other's shoes a bit more maybe this way we can tease out some more agreement and disagreement and just like where things feel like they stand after this conversation so the first exercise i want to do is give you each about like maybe two or three minutes each to outline where you think you most agreed with your opponent on um and what you felt the weakest parts in your own case were which may relate to one another um who wants to start i don't mind i'm happy to okay maybe let's start with tom then so i i agree with katya about how pausing is just a pretty robustly good thing to do a lot of the dynamics driving that all of these risks that society does not understand how big these risks are and i think being able to kind of very flexibly pause tap the brakes while also getting you know evidence of everything that's happening to as many people as possible transparency i think those those two things are going to be you know really helpful for for both risks so i think you actually didn't stress transparency but my guess is you'd be in favor

of it and and you know obviously directly talked about pausing so i think that's um a big point of agreement that I had in terms of where my own argument feels the weakest I think I feel a bit dissatisfied with my argument insofar as I'm kind of saying look it's just really hard to get to a great future AI is very high variance maybe it still has human values and is reflective and moral that's kind of less likely than with the human and so we should kind of roll the dice that we should prefer that dice roll over a human-controlled future. You know, there's a certain logic to it, but I think intuitively it fits.

It was a bit like, I can imagine there's a part of me in Cheers to you that just is like, no, let's just like stick with the human that we know, rather than go with this abstract argument around how maybe like a really kind of best case scenario is more likely with the AI system. Same questions to you, Katja. I agree that we seem to agree a decent amount about like what to do about it. in terms of the the weakest spots of my argument i feel like maybe i'm feeling too good about rolling the dice with a human um i think i might not be enough like taking into account how likely they are to ask for some sort of narrow thing or be like really foolish quickly and cut off their options in the long term and i feel like there was a there was a thing near the start that we um um maybe disagreed on and didn't get into that much of like is is there sort of a big space of uh like gradual cases where some humans take power gradually i think cases where like lots and lots of humans do um seem pretty plausible or like maybe that's like the most likely case of what a non-catastrophic outcome looks like.

Yeah. So you've mentioned some disagreements here. And I think something that's been interesting to me is despite you having a fair few disagreements on the topics of which of these things is most likely and which of these things would be a bad outcome, nonetheless, you've had quite a lot of convergence over what to do. I'm curious about whether either of you have reflections on what the nature of your disagreement really is and why it's not showing up too much over like when it comes to interventions um and what you think the actual stakes of your disagreement are with that in mind yeah should i maybe pass to katya first here My guess is that a substantial part of our disagreement is about like how likely alignment is.

And then I think a reason that we're maybe not running into disagreements that much about what to do is like the different things we're concerned about here are sort of like different sides of the same basic problem. And I think a different way of describing our disagreement is, I think, like, I feel like there's, like, a pretty deep general problem where if you make a bunch of creatures that can overpower you and outwit you in every way and put them out in the world, like, you're going to run into trouble one way or another. And so I guess it feels more like, you know, if there's currently a plan to nuke your city and, you know, you could be like, well, is this person going to press the nuke button or this person or like, you know, how's that going to affect the water or the air?

You prefer to just be trying to stop the plan to nuke your city. I think like if you're worried about powerful AI doing one bad thing or another, it still seems maybe just good to not build a powerful AI, and my sense is that maybe I feel more like that because I more think it's going to be misaligned and that there are just going to be problem after problem, and maybe that Tom feels somewhat more like maybe it will be aligned and then we might be left with only the problems that are kind of like humans using it for bad where human power grab is like quite a big one and maybe you can solve that substantially by focusing on that problem to the extent that that's also helped by not building powerful AI or doing it more carefully and slowly.

So I think our disagreements are very much on the kind of super conceptual, abstract side of things. So there's an analogy you can give to kind of deciding what to do and what to believe, which is like a kind of a web of belief, where you have like certain beliefs in the center, which in some sense are kind of foundational and really important. And then they then propagate outwards towards the edge of the web, which, you know, is your beliefs about what's going to happen tomorrow and also your views about what should we do tomorrow. And I think, you know, our disagreements are like quite far, quite deep into the web, like so deep that, you know, actually there's not been much thought about this stuff.

And I think, you know, as our discussion showed, there's just loads of kind of complicated considerations on both sides. it is quite hard to see how they net out but i think you know you can change the center of the web a fair bit and still find that when you get to the edge of the web the action recommendations aren't that different um and it's like okay this is yeah a super powerful technology which will be capable of you know disempowering everyone and you know staging a coup um yeah man we need to like make sure everyone understands that we need to like slow down until we have a plan for how to handle that um yeah not too surprising that a lot of those kind of you know much more pragmatic opinions are shared um i do think that if we had really dug into the nitty-gritty of how to implement specific proposals we might have found that that those conceptual disagreements were were driving more more disagreement you know like i would still like more people thinking about superintelligence to focus on the extreme power concentration angle because you know it's like you know 20 30 times fewer i think than think about alignment and that's only very very recently increased so the kind of historical work is is a much bigger ratio and you know in thinking about

a centralized project i would i would be like really upset if we implemented a centralized project and people proposed it without doing the red teaming for for power concentration um consequences where maybe katya would like agree with me that we should do the red teaming but it wouldn't be like something that she really cared about. Yeah, I also meant to say that like what kind of thinking one should do on the margin seems like maybe a clear disagreement between us. In particular, it seems like probably both of us could work on the things that the other person is working on. And so, yeah, like how much to like theoretically think about these things.

Yeah, it's a disagreement.

but it's been a pleasure having you both on the show. Pleasure talking to both of you. Yeah, it was a lot of fun. Thanks very much. All right, that was our first ever debate on the show. I really enjoyed hearing Tom and Katia thinking through these quite thorny questions. And yeah, overall, I think my main takeaway from this conversation was pretty encouraging. So it turns out that even if you disagree quite a lot as Tom and Katya do, over which dangers from AI are most likely and most consequential, then you still might actually converge a fair amount on what we should actually be doing about AI as a society. And that seems promising, because I guess a worry that I've previously had is that there could be pretty big trade-offs between trying to mitigate the risks of AI takeover versus trying to mitigate the risks of human takeover.

And I'd say that I feel a bit less worried about this after hearing Tom and Katya hash out what to do about AI and having this sort of very honest, reflective conversation and just hearing them end up agreeing on so much despite their conceptual differences. So for example, there's probably a version of a pause in AI development that they both at least agree to. And they're both pretty nervous, you know, maybe in somewhat different ways about having a single big AI project. And despite his belief that China dominating an AI would be really quite bad from the perspective of power concentration, Tom is still skeptical of arguments that we need to be racing against China.

It seemed that both he and Cartier maybe favored negotiating with China about slowing things down versus racing now at the same time it does seem like there are still real questions over you know how we should be allocating very limited resources and things like how much effort should be going into red teaming a project for the risk of enabling a power grab or something like that anyway that's it from me i really hope you enjoyed today's episode please let us know if you'd like to see more content like this. But you know, if you also never want to see something like this again, I guess that's also valid. Either way, thanks for watching.