A human ruler may ignore advice to reconsider their values

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 36:17

   that you don't totally agree with so yeah take it away what do you think? So because it seems like an overall thing going on in Tom's views are like, well, humans do have human values more. And so they're like, humans are more likely to have human values, but also sort of more likely to be foolish and like, you know, act like King Midas or something, like ask for something that isn't really what they want and sort of screw it up forever or like sort of lock in some bad version of values and so not get to a better version in future, I think it's more likely that AI doesn't do those things or is just either misaligned or sort of aligned to some narrow set of things that humans want, such that it is pretty catastrophic to just have that implemented, like to be basically just trying to get more users for particular

2. 37:23

   company or something, I think that's probably just catastrophic. Whereas I expect any human to be sort of at least more, like if they did get what they wanted and didn't act foolishly to want something that is sort of recognizable as at least pretty good in ways to other humans. I think like, yeah, even if, you know, it was someone from 100 years ago or something and they have a bunch of extra views that we don't like about, I don't know, like what is honorable or something.

3. 37:59

   I still think that the sort of manifestation of their vision is probably overall good and is better than a sort of like random, really narrow thing or random, random thing. So I think that's one point of disagreement. And the other one is like, in terms of AI generally being sort of, it sounds like AI wiser, less likely to lock in something stupid immediately, whereas humans pretty likely to do that.

4. 38:36

   I guess I think we should be imagining that these humans do have like a lot of AI advice. And so to the extent that the sort of AI has that kind of wisdom that they should be like more, you know, having access to that. So, yeah, maybe I agree that human power getting is more likely than I was saying if you also include these cases where, like, a very large class of humans gets power. But then I think those look much more like just, like, continuing as normal.

5. 39:09

   We more expect things to be good in those cases. So, again, the cases that seem quite bad are the ones where it's a very small set of humans. And so then, yeah, I guess this interacts with the likelihood. I'm like, well, then we're talking about something more unlikely. Yeah, it seems like we are currently training AIs to behave pretty ethically compared to humans and pretty thoughtfully and openly.

6. 39:33

   But I guess I'm not sure how much to be reassured by that because it's sort of like, well, we have these big predicting text models. We do various things to them and then they have something like different personas that we've trained them to do in response to particular prompts. And I guess like in a situation where more stuff is happening faster and maybe AI is modifying itself, I'm not sure how much those kinds of like personalities.

7. 40:01

   remain in charge of what gets created. It does seem like, yeah, plausibly a bunch, but quite plausibly not. And I don't actually know the technical details here enough to say that much about it. Yeah, I agree with a lot of the local points you made. I think the thing I want to come back to and stress again is um i think there are there are lots of ways that we can fail to get what what you know a great future and each one of those ways of failing is kind of individually catastrophic so um you know one thing that someone who sees power could decide to do is you know what i'm not going to bother doing anything with the stars because i don't want to why would i um another mistake they can make is they could um be like you know what i think the person affecting view of population ethics is is completely correct so i'm not going to see any value in creating new types of flourishing

8. 40:58

   and another kind of mistake that they could make is they could say you know what it's all just about um positively valence experience i'm just going to tell the universe with hedonium i you know which could be catastrophic because actually you know you might think that there are loads of other dimensions to what matters and so you know my worry is that if it's a sole human um that they just are bound to make some kind of mistake and that the only way to to avoid that is to have lots of different humans exploring different um possibilities openly and i do think that there is hope in their ai saying look you know maybe you want to think carefully about this stuff my worry is that there's no actually compelling argument that you could give to someone like this that they need to reflect they could be like i'm not interested in reflecting i'm done i'm having

9. 41:46

   a good time like the ai may suggest it they'll be like oh thanks for the suggestion i see you think i might benefit from this i'm not going to do it you know i've spoken to friends who are very ethical very thoughtful and i've been like you know you i think you're wrong about this really important ethical question i think you should think more about this they have no interest you know they're they like hear my quick argument they're like no i disagree and conversation ends so my worry would be that yes the ai may bring it up but like the ai would have to be pretty forceful um to get the human to actually go along with it. And I don't think we're going to train the AIs to like really keep banging the drum on how this human needs to do some more philosophical reflection. The human is just not going to be interested. Yeah, that's fair. It seems like then it sort of comes down a

10. 42:27

    lot to also, yeah, the nature of like, are they aligned to what the human says or what they think