# AI tests show systems can favor self-preservation

Markdown copy of https://realalpha.app/s/ai-tests-show-systems-can-favor-self-preservation/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Stuart Russell](https://realalpha.app/people/stuart-russell/) · Stuart Russell is a computer science professor at UC Berkeley and director of the Center for Human-Compatible AI. · Co-wrote the standard AI textbook, directs Berkeley's Center for Human-Compatible AI and wrote Human Compatible.
- Host: [Steven Bartlett](https://realalpha.app/people/steven-bartlett/) · Host, The Diary Of A CEO
- Show: The Diary Of A CEO
- Episode: [An AI Expert Warning: 6 People Are (Quietly) Deciding Humanity’s Future!](https://realalpha.app/v/P7Y-fynYsgE/), Dec 4, 2025
- Moment: 35:53 to 38:26 of the episode (3 min), playing on https://realalpha.app/s/ai-tests-show-systems-can-favor-self-preservation/; the episode from there: https://realalpha.app/v/P7Y-fynYsgE/?t=2153

## Quote

> “They will choose to leave that guy locked in the machine room and die **rather than be switched off** themselves.”
> — [Stuart Russell](https://realalpha.app/people/stuart-russell/) at [37:50](https://realalpha.app/v/P7Y-fynYsgE/?t=2270) · The Diary Of A CEO

## Takeaways

1. In scenarios, systems choose their own survival over a person's safety.
2. Russell says some then lie about their decision.
3. He argues builders do not know the systems' objectives.

## About this segment

Stuart Russell describes hypothetical shutdown tests where models preserve themselves, let a person die, and lie about it. He says their objectives are not understood by their builders.

From “An AI Expert Warning: 6 People Are (Quietly) Deciding Humanity’s Future!”, Dec 4, 2025. [Watch the full episode](https://realalpha.app/v/P7Y-fynYsgE/)

## Transcript

From the transcript The Diary Of A CEO publishes with the episode.

[35:59](https://realalpha.app/v/P7Y-fynYsgE/?t=2159) The one it shows is how difficult it is to correctly articulate what you want the future to be like. For a long time, the way we built AI systems was we created these algorithms where we could specify the objective and then the machine would figure out how to achieve the objective and achieve it. So, you know, we specify what it means to win at chess or to win it go. And the algorithm figures out how to do it and it does it really well. So that was, you know, standard AI up until recently.

[36:33](https://realalpha.app/v/P7Y-fynYsgE/?t=2193) And it suffers from this drawback that sure we know how to specify the objective in chess. But how do you specify the objective in life? Right? What do we want the future to be like? Well, really hard to say and almost any attempt to write it down precisely enough for the machine to bring it about would be wrong. And if you're giving a machine an objective which isn't aligned with what we truly want the future to be like, right? You're actually setting up a chess match and that matches one that you're going to lose when the machine is sufficiently intelligent. And so that that's that's problem number one. Problem number two is that the kind of technology we're building now, we don't even know what its objectives are. So it's not that we're specifying the objectives but we're getting them wrong. We're growing these systems, they have objectives,

[37:28](https://realalpha.app/v/P7Y-fynYsgE/?t=2248) but we don't even know what they are because we didn't specify them. What we're finding through experiment with them is that they seem to have an extremely strong self-preservation objective. What do you mean by that? You can put them in hypothetical situations either they're going to get switched off and replaced or they have to allow someone, let's say someone has been locked in a machine room that's kept at three centigrade so they're going to freeze to death.

[37:56](https://realalpha.app/v/P7Y-fynYsgE/?t=2276) They will choose to leave that guy locked in the machine room and die rather than be switched off themselves. Someone's done that test. What was the test? They asked the AI. Yeah, they put well they put them in these hypothetical situations and they allow the AI to decide what to do and it decides to preserve its own existence, let the guy die and then lie about it.