The alpha, twice a week
That was 3 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
The Diary Of A CEO · 3 min · Dec 4, 2025
AI tests show systems can favor self-preservation
Stuart Russell describes hypothetical shutdown tests where models preserve themselves, let a person die, and lie about it. He says their objectives are not understood by their builders.
Stuart Russell is a computer science professor at UC Berkeley and director of the Center for Human-Compatible AI.
Co-wrote the standard AI textbook, directs Berkeley's Center for Human-Compatible AI and wrote Human Compatible.
Steven Bartlett, Host · Host, The Diary Of A CEO
About this segment
Stuart Russell describes hypothetical shutdown tests where models preserve themselves, let a person die, and lie about it. He says their objectives are not understood by their builders.
From “An AI Expert Warning: 6 People Are (Quietly) Deciding Humanity’s Future!”, Dec 4, 2025. Watch the full episode
Answers “Is AI dangerous?” See all 32 answers
Thanks. We'll look at and fix what's off.
“They will choose to leave that guy locked in the machine room and die rather than be switched off themselves.”
Takeaways
- In scenarios, systems choose their own survival over a person's safety.
- Russell says some then lie about their decision.
- He argues builders do not know the systems' objectives.
In this episode
Full episode- 1:39:18Safe AI must learn human goals rather than receive fixed objectives5:47
- 1:18:46Automation can shrink employment even as production grows7:40
- 30:15AI could improve itself by doing AI research3:59
- 1:32:59AI safety needs a measurable extinction-risk limit4:44
- 1:13:48China's AI rules are stricter than the race narrative suggests4:30
- 13:49AGI's bottleneck may be understanding, not compute3:30
35:53AI tests show systems can favor self-preservation2:33
- 51:12Humanlike robots can trigger misplaced empathy5:28Up next
Transcript
From the transcript The Diary Of A CEO publishes with the episode.
- 35:59
The one it shows is how difficult it is to correctly articulate what you want the future to be like. For a long time, the way we built AI systems was we created these algorithms where we could specify the objective and then the machine would figure out how to achieve the objective and achieve it. So, you know, we specify what it means to win at chess or to win it go. And the algorithm figures out how to do it and it does it really well. So that was, you know, standard AI up until recently.
- 36:33
And it suffers from this drawback that sure we know how to specify the objective in chess. But how do you specify the objective in life? Right? What do we want the future to be like? Well, really hard to say and almost any attempt to write it down precisely enough for the machine to bring it about would be wrong. And if you're giving a machine an objective which isn't aligned with what we truly want the future to be like, right? You're actually setting up a chess match and that matches one that you're going to lose when the machine is sufficiently intelligent. And so that that's that's problem number one. Problem number two is that the kind of technology we're building now, we don't even know what its objectives are. So it's not that we're specifying the objectives but we're getting them wrong. We're growing these systems, they have objectives,
- 37:28
but we don't even know what they are because we didn't specify them. What we're finding through experiment with them is that they seem to have an extremely strong self-preservation objective. What do you mean by that? You can put them in hypothetical situations either they're going to get switched off and replaced or they have to allow someone, let's say someone has been locked in a machine room that's kept at three centigrade so they're going to freeze to death.
- 37:56
They will choose to leave that guy locked in the machine room and die rather than be switched off themselves. Someone's done that test. What was the test? They asked the AI. Yeah, they put well they put them in these hypothetical situations and they allow the AI to decide what to do and it decides to preserve its own existence, let the guy die and then lie about it.