The alpha, twice a week
That was 4 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
The Diary Of A CEO · 4 min · Sep 4, 2025
OpenAI canceled its alignment team months after a four-year pledge
He points to an alignment team that OpenAI said would solve the problem in four years, then canceled within months. He argues that demonstrating control is impossible could deter people from pursuing it.
Roman Yampolskiy is a Professor of Computer Science and Engineering and AI safety researcher.
Coined the term AI safety and directs the University of Louisville's Cyber Security Lab.
Steven Bartlett, Host · Host, The Diary Of A CEO
About this segment
He points to an alignment team that OpenAI said would solve the problem in four years, then canceled within months. He argues that demonstrating control is impossible could deter people from pursuing it.
From “The AI Safety Expert: These Are The Only 5 Jobs That Will Remain In 2030! - Dr. Roman Yampolskiy”, Sep 4, 2025. Watch the full episode
Answers “Is AI dangerous?” See all 39 answers
Thanks. We'll look at and fix what's off.
“The day they announced it, I think they said they're going to solve it in four years. Like half a year later, they canceled the team.”
Dr. Roman Yampolskiyat 50:40 · The Diary Of A CEO50:40Takeaways
- He says the team was canceled about six months after its launch.
- He wants claims of indefinite control to face scientific scrutiny.
- He favors useful narrow AI tools over general agents.
In this episode
Full episode- 2:43AI capability is advancing faster than safety research5:03
- 21:51Humanoid robots could automate physical work by 20304:07
- 37:00A novel virus is the extinction route he can picture4:40
49:01OpenAI canceled its alignment team months after a four-year pledge3:38
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 49:14
So what should I do? What should the person sat at home listening to this do you you talk to a lot of people who are building this technology ask them precisely to explain some of those things they claim to be impossible how they solved it or going to solve it before they get to where they're going do you know i don't think sam altman wants to talk to me i don't know he seems to go on a lot of podcasts maybe he does he wants to go online i wonder why that is i'd love to speak to him but i don't think he wants to i don't think he wants me to uh interview him have an open challenge maybe money is not the incentive but whatever attracts people like that whoever can convince you that it's possible to control and make safe super intelligence gets the prize they come on your show and prove their case
- 50:09
anyone if no one claims the price or even accepts the challenge after a few years maybe we don't have anyone with solutions we have companies valued again at billions and billions of dollars working on safe superintelligence we haven't seen their output yet yeah i'd like to speak to ilia as well because i know he's he's working on safe superintelligence Notice a pattern, too. If you look at history of AI safety organizations or departments within companies, they usually start well, very ambitious, and then they fail and disappear.
- 50:48
So OpenAI had superintelligence alignment team. The day they announced it, I think they said they're going to solve it in four years. Like half a year later, they canceled the team. And there is dozens of similar examples. creating a perfect safety for superintelligence, perpetual safety as it keeps improving, modifying, interacting with people, you're never going to get there. It's impossible. There is a big difference between difficult problems in computer science and pre-complete problems and impossible problems.
- 51:21
And I think control, indefinite control of superintelligence is such a problem. So what's the point of trying, then, if it's impossible? Well, I'm trying to prove that it is specifically that. Once we establish something is impossible, fewer people will waste their time claiming they can do it and looking for money. So many people go and give me a billion dollars in two years and I'll solve it for you.
- 51:42
Well, I don't think you will. But people aren't going to stop striving towards it. So if there's no attempts to make it safe and there's more people increasingly striving towards it, then it's inevitable. But it changes what we do. if we know that it's impossible to make it right, to make it safe, then this direct path of just build it as soon as you can becomes suicide mission. Hopefully fewer people will pursue that.
- 52:05
They may go in other directions. Again, I'm a scientist. I'm an engineer. I love AI. I love technology. I use it all the time. Build useful tools. Stop building agents. Build narrow superintelligence, not a general one. I'm not saying you shouldn't make billions of dollars. I love billions of dollars. But don't kill everyone, yourself included.