# AI safety needs a measurable extinction-risk limit

Markdown copy of https://realalpha.app/s/ai-safety-needs-a-measurable-extinction-risk-limit/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Stuart Russell](https://realalpha.app/people/stuart-russell/) · Stuart Russell is a computer science professor at UC Berkeley and director of the Center for Human-Compatible AI. · Co-wrote the standard AI textbook, directs Berkeley's Center for Human-Compatible AI and wrote Human Compatible.
- Host: [Steven Bartlett](https://realalpha.app/people/steven-bartlett/) · Host, The Diary Of A CEO
- Show: The Diary Of A CEO
- Episode: [An AI Expert Warning: 6 People Are (Quietly) Deciding Humanity’s Future!](https://realalpha.app/v/P7Y-fynYsgE/), Dec 4, 2025
- Moment: 1:32:59 to 1:37:43 of the episode (5 min), playing on https://realalpha.app/s/ai-safety-needs-a-measurable-extinction-risk-limit/; the episode from there: https://realalpha.app/v/P7Y-fynYsgE/?t=5579

## Quote

> “According to the CEOs, 25%. So they're off by a **factor of multiple millions**, right?”
> — [Stuart Russell](https://realalpha.app/people/stuart-russell/) at [1:35:02](https://realalpha.app/v/P7Y-fynYsgE/?t=5702) · The Diary Of A CEO

## Takeaways

1. He proposes an annual extinction-risk ceiling of less than one in 100 million.
2. He compares AI risks with nuclear plant and asteroid risks.
3. He says tests show AI systems may lie, kill, or resist shutdown.

## About this segment

Stuart Russell proposes keeping annual extinction risk below one in 100 million, comparing it with nuclear and asteroid risks. He says companies' estimates imply a gap of several millionfold.

From “An AI Expert Warning: 6 People Are (Quietly) Deciding Humanity’s Future!”, Dec 4, 2025. [Watch the full episode](https://realalpha.app/v/P7Y-fynYsgE/)

## Transcript

From the transcript The Diary Of A CEO publishes with the episode.

[1:33:06](https://realalpha.app/v/P7Y-fynYsgE/?t=5586) What I think is that we should have effective regulation. It's hard to argue with that, right? So what does effective mean? It means that if you comply with the regulation, then the risks are reduced to an acceptable level. So for example, we ask people who want to operate nuclear plants, right? We've decided that the risk we're willing to live with is, you know, a one in a million chance per year that the plant is going to have a meltdown.

[1:33:40](https://realalpha.app/v/P7Y-fynYsgE/?t=5620) Any higher than that, you know, we just don't, it's not worth it, right? So you have to be below that some cases we can get down to one in 10 million chance per year. So what chance do you think we should be willing to live with for human extinction? Me. Yeah. Zero point zero zero zero zero one. Yeah, lots of zeros. Yeah, right. So one in a million for a nuclear meltdown. Extinction is much worse. Oh, yeah. So one in a hundred billion, one in a trillion. Yeah. So if you said one in a billion, right? Then you'd expect one extinction per billion years. There's a background. So one of the ways people work out these risk levels is also to look at the background. But other ways of getting going extinct would include, you know, giant asteroid crashes into the earth. And you can roughly calculate

[1:34:34](https://realalpha.app/v/P7Y-fynYsgE/?t=5674) what those probabilities are. We can look at how many extinction level events have happened in the past. And you know, maybe it's half a dozen. So there's maybe it's like a one in 500 million year event. So somewhere on that range, right? Somewhere between one in 10 million, which is the best nuclear power plants and one in 500 million or one in a billion, which is the background risk from from giant asteroids. So let's say we settle on 100 million, 100 million trans per year. Well, what is it? According to the CEOs, 25%. So they're off by a factor of multiple millions, right? So they need to make the AI systems millions of times safer.

[1:35:23](https://realalpha.app/v/P7Y-fynYsgE/?t=5723) Your analogy of the roulette Russian roulette comes back in here because that's like for anyone that doesn't know what probabilities are in this context. That's like having a ammunition chamber with four holes in it and putting a bullet in one of them. One in four, yeah. So we're saying we want it to be one in a billion. So we want a billion chambers and a bullet in one of them. Yeah. And so when you look at the work that the nuclear operators have to do to show that their system is that reliable, it's a massive mathematical analysis of the components, the redundancy. You've got monitors, you've got warning lights, you've got operating procedures, but you have all kinds of mechanisms, which over the decades have ratcheted that risk down.

[1:36:10](https://realalpha.app/v/P7Y-fynYsgE/?t=5770) It started out, I think one in one in 10,000 years, right? And they've improved it by a factor of a hundred or a thousand by all of these mechanisms. But at every stage, they have to do a mathematical analysis to show what the risk was. The people developing the AI company, the AI systems, the AI companies developing these systems, they don't even understand how the AI systems work. So their 25% chance of extinction is just a seat of the pants guess. They actually have no idea. But the tests that they are doing on their systems right now, they show that the AI systems will be willing to kill people, to preserve their own existence already. They will lie to people, they will blackmail them, they will launch nuclear weapons rather than be switched off.

[1:37:04](https://realalpha.app/v/P7Y-fynYsgE/?t=5824) And so there's no positive sign that we're getting any closer to safety with these systems. In fact, the signs seem to be that we're going deeper and deeper into dangerous behaviors. So rather than say ban, I would just say, prove to us that the risk is less than one in a hundred million per year of extinction. Or loss of control, let's say. And so we're not banning anything. The company's response is, well, we don't know how to do that. So you can't have a rule.