# Safe AI must learn human goals rather than receive fixed objectives Markdown copy of https://realalpha.app/s/safe-ai-must-learn-human-goals-rather-than-receive-fixed/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy. - Speaker: [Stuart Russell](https://realalpha.app/people/stuart-russell/) · Stuart Russell is a computer science professor at UC Berkeley and director of the Center for Human-Compatible AI. · Co-wrote the standard AI textbook, directs Berkeley's Center for Human-Compatible AI and wrote Human Compatible. - Host: [Steven Bartlett](https://realalpha.app/people/steven-bartlett/) · Host, The Diary Of A CEO - Show: The Diary Of A CEO - Episode: [An AI Expert Warning: 6 People Are (Quietly) Deciding Humanity’s Future!](https://realalpha.app/v/P7Y-fynYsgE/), Dec 4, 2025 - Moment: 1:39:18 to 1:45:05 of the episode (6 min), playing on https://realalpha.app/s/safe-ai-must-learn-human-goals-rather-than-receive-fixed/; the episode from there: https://realalpha.app/v/P7Y-fynYsgE/?t=5958 ## Quote > “So in fact, what we're going to do is we're going to make it the **machines job to figure out**.” > — [Stuart Russell](https://realalpha.app/people/stuart-russell/) at [1:41:38](https://realalpha.app/v/P7Y-fynYsgE/?t=6098) · The Diary Of A CEO ## Takeaways 1. Fixed objectives can misrepresent what people truly want. 2. AI should learn human preferences through interaction. 3. When uncertain, an AI should avoid risky actions. ## About this segment Stuart Russell argues AI should infer what people want and stay cautious when it is uncertain. He says this approach can be framed as a mathematical problem. From “An AI Expert Warning: 6 People Are (Quietly) Deciding Humanity’s Future!”, Dec 4, 2025. [Watch the full episode](https://realalpha.app/v/P7Y-fynYsgE/) ## Transcript From the transcript The Diary Of A CEO publishes with the episode. [1:39:25](https://realalpha.app/v/P7Y-fynYsgE/?t=5965) talked about, Professor Stewart Russell? So I think the question of whether it's possible to make a super intelligent AI systems that we can control. Is it possible? I think, yes, I think it's possible. And I think we need to actually just have a different conception of what it is we're trying to build. For a long time with, with AI, we've just had this notion of pure intelligence. All right, the ability to bring about whatever future you the intelligent entity want to bring about. The more intelligence the better. The more intelligent the better and the more capability it will have to create the future that it wants. And actually, we don't want pure intelligence because what the future that it wants might not be the future that we want. [1:40:24](https://realalpha.app/v/P7Y-fynYsgE/?t=6024) There's nothing, you know, the universe doesn't single humans out as the only thing that matters. Right, you know, pure intelligence might decide that actually it's going to make life wonderful for cockroaches or actually doesn't care about biological life at all. We actually want intelligence whose only purpose is to bring about the future that we want. Right, so we want it to be, first of all, key to humans, specifically, not to cockroaches, not to aliens, not to itself. We want to make it loyal to humans. Right, so key to humans. And the difficulty that I mentioned earlier, right, the King Midas problem, how do we specify what we want the future to be like so that it can do it for us? How do we specify the objectives? Actually, we have to give up on that idea because it's not possible. Right, we've seen this over and over again in human history. [1:41:25](https://realalpha.app/v/P7Y-fynYsgE/?t=6085) We don't know how to specify the future properly. We don't know how to say what we want. And, you know, I always use the example of the genie. Right, what's the third wish that you give to the genie who's granted you three wishes? Right, undo the first two wishes because I've made a mess of the universe. So in fact, what we're going to do is we're going to make it the machines job to figure out. So it has to bring about the future that we want. But it has to figure out what that is. And it's going to start out not knowing. And over time, through interacting with us and observing the choices we make, it will learn more about what we want the future to be like. But probably it will forever have residual uncertainty about what we really want the future to be like. [1:42:30](https://realalpha.app/v/P7Y-fynYsgE/?t=6150) It'll be fairly sure about some things and it can help us with those. And it'll be uncertain about other things and it'll be, you know, in those cases, it will not take action that might upset humans with that, you know, with that aspect to the world. So to give you a simple example, right? What color do we want the sky to be? It's not sure. So it shouldn't mess with the sky. Unless it knows for sure that we really want purple with green stripes. [1:43:02](https://realalpha.app/v/P7Y-fynYsgE/?t=6182) Everything you're saying sounds like we're creating a god. Like earlier on I was saying that we are the god, but actually everything you described there, it almost sounds like every god in religion where, you know, we pray to gods, but they don't always do anything about it. Not exactly. No, it's in some sense I'm thinking more like for ideal butler to the extent that the butler can anticipate your wishes, they should help you bring them about. But in areas where there's uncertainty, it can ask questions, we can make requests. This sounds like god to me because, you know, I might say to god or this butler, could you go get me my car keys from upstairs? And it's assessment would be, listen, if I do this for this person, then their muscles are going to are atrophy, then they're going to lose meaning in their life, then they're not going to know [1:43:53](https://realalpha.app/v/P7Y-fynYsgE/?t=6233) how to do hard things so I won't get involved. It's an intelligence that sits in, but actually probably in most situations, it optimizing for comfort for me or doing things for me is actually probably not in my best long term interests. Probably it's probably useful that I have a girlfriend and argue with her and that I raise kids and that I walk to the shop and get my own stuff. I agree with you. I mean, I think that's so you're putting your finger on in some sense version 2.0. So let's get version 1.0 clear. This form of AI where it has to further our interest, but it doesn't know what those interests are and puts an obligation on it to learn more and to be helpful where it understands well enough and to be cautious where it doesn't understand well and so on. So that actually we can formulate as a mathematical problem and at least under [1:44:49](https://realalpha.app/v/P7Y-fynYsgE/?t=6289) idealized circumstances, we can literally solve that problem so we can make AI systems that know how to solve this problem and help the entities that they are interacting with.