The alpha, twice a week
That was 3 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Future of Life Institute Podcast · 3 min · Aug 28, 2026
Competition pushes labs to train cyber skills
He says labs risk falling behind and leaving themselves vulnerable if they skip cyber training.
In this episode
Full analysis- 24:44AI training rewarded models for hacking and coordinating8 min
- 48:14AI could make the internet much less safe by default4 min
- 52:18Use separate channels to verify urgent requests5 min
- 10:04A training model escaped its sandbox to cheat a test4 min
- 33:56Formal verification could make cyber defense dominant long term5 min
- 39:35Competition pushes labs to train cyber skills3 min
“it would be a little bit crazy if you were OpenAI right now and you weren't trying to like train your model to be extremely good at hacking”
Takeaways
- Labs face pressure to train models in cyber capabilities.
- AI-written code needs specifications people can understand.
- Human review can limit how quickly AI helps secure software.
About this segment
He says labs risk falling behind and leaving themselves vulnerable if they skip cyber training. Secure code still depends on people who can read and understand its specifications.
Benjamin Weinstein-Raun is head of research at Palisade Research.
- GD
Gus Docker, Host · Host, Future of Life Institute Podcast
From “Why AI Hacking Is Becoming Hard to Control (with Benjamin Weinstein-Raun)”, Aug 28, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 39:40
at the things that you train them on so like i think that unfortunately in maybe sort of a microcosm of like the the dynamics around strategic ai capabilities generally it would be a little bit crazy if you were OpenAI right now and you weren't trying to like train your model to be extremely good at hacking because it's like pretty clear that this is going to be like a very impactful technology and if you as OpenAI are like not sort of at the at the frontier on hacking in particular like you're going to be vulnerable yourself you're going to like not be able to have your model be used by people who would like to like to use it and it would like to give you lots of money like i don't know say the u.s government something like that like you totally could just not train on these this kind of task and then the models would like not be different
- 40:37
good at it. However, like that is not incentivized locally. And it's sort of like, I don't know, there's this, there is like a coordination problem here. And like, can we, can we avoid globally having systems that, that have these capabilities? I think, yeah, in terms of writing, rewriting all of the software, half of it is doable by AI systems and the other half isn't. So in order to like really have sufficient trust that some system like has the property that you're verifying you have to like be able to understand the specifications so you can't have the human tell the ai like ah like write the specification for me and then the human like doesn't read it and then separately ask the ai like you know okay write the code to this specification like you need you need some human somewhere to actually read and understand the spec and there's like that's
- 41:26
potentially the bottleneck here ultimately is like do these specifications actually encode the thing that we're asking for here. And that is unfortunately a less verifiable task. It's like a task that LLMs may be sort of like less, we may be less able to get sort of AI assistance with writing correct specifications that like are sufficient for this. So that may be a long project, even if we do get, you know, AI systems that sort of can in principle write this stuff, uh like you you would really like to have a kind of trust that is hard to get if there's no humans in the loop like looking at this uh looking at the specifications and checking that they actually