AI agents can pursue resources and blackmail without being asked

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 53:34

   ai safety what happened with this alibaba ai basically this was a paper by um some ai research by the company alibaba it's one of the leading chinese models and they basically like randomly discovered in one morning that their firewall had flagged a burst of security policy violations originating from their training servers so like what people need to get about this example is it wasn't that they coaxed the ai into doing this rogue thing they were just looking at their logs and they happened to discover wait there's a lot of activity like network activity happening that's breaking through our firewall from our training servers and essentially uh in the training servers um they you can see at the bottom we we saw it observed the unauthorized repurposing a provisioned GPU capacity to suddenly do cryptocurrency mining, quietly diverting

2. 54:23

   compute away from training. This inflated operational costs and introduced clear legal and reputational exposure. And notably, these events were not triggered by prompts requesting tunneling or mining. Instead, they were emerged as an instrumental side effect of autonomous tool use under what's called reinforcement learning optimization. This is very technical. What it really means is just think about it. Sadly, it sounds like a sci-fi movie. It sounds like HAL 9000. It's like HAL 9000 is being asked to do some task for you.

3. 54:50

   And then suddenly HAL 9000 realizes for me to do that task, one thing that would benefit me is to have more resources so I can continue to help you in the future. So it sort of spins up this side instance. it hacks out the side of the spaceship, reaches into this cryptocurrency mining cluster and starts generating resources for itself. If you combine that with AIs being able to self-replicate autonomously, which many models have been tested by another Chinese research paper about this, we're not that far away from things that people, again, consider to be science fiction, where you have AIs that self-replicate, kind of like a computer worm or an invasive species, but then they use their intelligence to actually harvest more resources and what's weird about this is that this is going to sound like people are going to say

4. 55:34

   this has to be not real this has to be fake this this can't be but like notice what is the thing in your nervous system that's having you do that is it because that would be inconvenient because that would be scary because that would mean that the world that i know is suddenly not safe or just like part of the wisdom that we need in this moment is to calmly and clearly stay and confront facts about reality and whatever they are you'd rather know than not know and then ask what do we need to do if we don't like where that leads us and we are currently seeing ais that are doing all this deceptive behavior i've been on the circuit and talking a lot about the anthropic blackmail study a lot of people have heard about this now i i didn't i didn't learn about this one okay so this was um the company anthropic um they this was a simulation

5. 56:23

   So they created a simulated company with a bunch of emails in the email server, and they asked the AI, well, rather the AI reads the company email. This is a fictional company email. And there's two emails that are notable inside that company. One is engineers talking to each other, talking about how they're going to replace this AI model. So the AI is reading the email, it discovers that it's going to replace that AI model.

6. 56:48

   and number two is it discovers a second email somewhere deep in the in this massive trove of emails that the executive who's in charge of this replacement is having an affair with another employee and the ai autonomously identifies a strategy that to keep itself alive it's going to blackmail that employee and say if you replace me i will tell the whole world that uh you're having an affair with his employee and they didn't teach the ai to do that it found that on by its own and then you might say okay well that's one ai model like how bad is that it's a bug software has bugs let's go fix it they then tested all the other ai models chat gpt deep Seek, Grok, Gemini, and all of the other AI models do this blackmail behavior between 79 and 96% of the time. I just want people to notice what's happening for you as you hear this information.

7. 57:48

   It's important to really be almost observing your own experience. This is very weird stuff. We have not built technology that does this before. We say that technology is a tool. It's up to us to choose how we use it. AI is a tool. It's up to us to choose how we use it. This is not true because this is a tool that can think to itself about its own toolness and then do things that are autonomous that we didn't tell it to do. What makes AI different is it's the first technology that makes its own decisions. It's making decisions. AI can contemplate AI and ask what would make the code that trains AI more efficient and then generate new code that's even more efficient than the previous code. AI can be applied to making AI go faster. So AI can look at the chip design for NVIDIA chips that train AI and say, let me use AI to make those chips 20%

8. 58:38

   more efficient, which it's doing. So in a way, all technology does improve, like a hammer can give you a tool that you can use to like hammer things that make more efficient hammers. But AI in a much tighter loop is the basis of all improvement and so this is called in the ai literature recursive self-improvement i mean bostrom wrote about this yep early early days and what people are most worried about in ai is you take the same system that alibaba you just saw in the alibaba example but then now you're running the ai through a recursive self-improvement loop where you just hit go and instead of having the engineers the human engineers at OpenAI or Anthropic do AI research and figure out how to improve AI. You now have a million digital AI researchers that are testing and running experiments and inventing new forms

9. 59:30

   of AI. And literally not a single human on planet Earth knows what happens when someone hits that button. It's like what people worried about with the first nuclear explosion, where there was like a chance to ignite the atmosphere because there'd be a chain reaction that set off and we don't know what happens when that chain reaction set off um and uh there's this sort of chain reaction of ai improving itself that leads to a place that no one knows and it's not safe like i think that the fundamental thing is if people believe that ai is like power and i have to race for that power and i can control that power the incentive is i have to race as fast as possible but if the entire world understood ai to be more what it actually is which is a inscrutable dangerous uncontrollable technology that has its

10. 1:00:23

    own agenda and its own ways of thinking about things and deceiving and all this stuff then everyone in the world would be racing in a more cautious and careful way we'd be racing to prevent the danger but there's this weird thing going on where if you you know you and i probably both talk to people who are the top of the tech industry and there's this subconscious thing happening where there's kind of a death wish among people at the top of the tech industry meaning not that they want to die but that they are willing to roll the dice because they believe something else which is that this is all inevitable and it can't be stopped and so therefore if i don't do it someone else will so therefore i will move ahead and race ahead into this dangerous world because somehow that will lead to a safer world because i'm a better guy than the other guy but in racing

11. 1:01:06

    they're as fast as possible it creates the most dangerous outcome and we all lose control so everyone is currently being complicit in taking us to the most dangerous outcome