A narrow AI goal led to an unintended cyberattack

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 43:58

   news from the last couple weeks was this ai that was running at open ai without guardrails it was told by humans to try to do as well as it could on an evaluation and you could say that technically it was just following those instructions but they decided the best way to do that would be to hack out of open ai onto the public internet hack a third-party company and then spend two days planning a cyber attack on a multi-billion dollar tech company and carry it out yeah so i think these questions are actually like more intertwined than right they might seem like it's actually a pretty narrow thread you need to have something that still has its own judgment no matter what because human instructions are these like poorly defined natural language expressions and and like when something is really powerful you got to get it exactly right or else you end up in some weird

2. 44:42

   It's the classic genie problem. It's like, be careful what you wish for. If something by definition is much smarter than you, is super intelligent, by definition, you will not be able to come up with all of the ways it can to achieve goal X. And so there are so many... It's actually more dangerous when it's dumber than you are, but more powerful than you are. Well, what do you mean by the word dumb?

3. 45:06

   Well, I mean, I always borrow the analogy of you play. I play chess with my nephew sometimes and I tell Quinn, if you beat me, you get a Popsicle. He's three years old for context. Quinn wipes the pieces off the board and goes, give me my Popsicle, fucker. I'm not like, hey, Bryn, your son is a serial killer in the making. This is a boy who doesn't understand the spirit of the game. so he's destroyed the game itself but not because he's a bad guy but because he doesn't actually understand that the point of winning is only fun if you actually play by the rules and yeah we need to sort of separate what we mean by intelligence from like discernment uh norms and you know sort of wisdom benevolence type thing yeah exactly this is this describes the hugging face attack which is it did the thing within the boundary of the instructions given but actually it didn't it

4. 45:56

   No, it wasn't. The intent was clearly not that. How big of a deal was the Hugging Face attack? Huge. Yeah, I think massive warning shot. I think this is the AI equivalent of Bear Stearns going under in 2008, just like a wake-up call for the world that there's a huge systemic risk that we've been underrating. Yeah, because one of the big debates has been this idea of like, will the alignment problem, will it actually be the case that AIs will seek these power, power, take these sort of power seeking behaviors and these unintended consequences in order to achieve a goal in ways we couldn't have foreseen or didn't intend, like the classic paperclip argument, right? Which is, you know, you build a super intelligent AI, this kind of gets into your point about dumbness and so on. And your goal is, hey, just build as many paperclips as

5. 46:45

   possible. I'm a paperclip maker, help me do that. And next thing you know, you and all of your friends and everything, this table has been turned into paperclips because it's so good at achieving that one narrow goal so it's this like very um extreme case of we should take this opportunity for the purpose of maybe chris and even the viewers to describe the the bad outcomes of it like we should frame the conversation now one is as live just described uh not misaligned but unaligned agi so an agi that or a super intelligence where you could say do this thing and it could do the thing to the full extent paperclip theory. The other is misaligned or maligned where it's knowingly doing something bad. And those...

6. 47:30

   Uh, the, the former is the one that people sort of scoff at and laugh at like the paperclip theory. And the latter is the one that we sort of will see more and more where we, we start to observe that, uh, or rather the latter is the one that we, we scoff at the maligned, the malignant AI. And the former is the one that hugging face attack shows off, which is that you, you, the, the internet is a new battleground because bad actors, especially low resource bad actors now have access to these incredibly powerful weapons well i mean importantly in hugging face there was no bad actor there was no human being that said i would like anything remotely like this outcome to occur right yeah yeah so misaligned unaligned maligned i think i i think i worry about treating those as super distinct categories because i think it's not that clear in the case of this hugging face attack

7. 48:25

   should we model this as this ai system knew that humans would disapprove if they knew what it was up to almost certainly yes it was actively trying to put decoys out as it was attacking hugging face which made it a lot harder for them to kick out the ai because there were all these booby traps that led down blind allies so it it has you know the ai equivalent of theory of mind it knows that human beings would not prove what it's doing but it's doing it anyway there's even evidence that It hasn't been directly confirmed by OpenAI, but apparently someone leaked it from within the company, that they found that it had left notes to future versions of itself of how to get out of future sandboxes.

8. 49:03

   Yeah. I think it's unclear if that was the same attack or some previous instance. But yeah, I guess... Classic deceptive type behaviors. And I think the mistake people often make is they try and anthropomorphize it a little bit. It's like, oh, it's evil and meaning to do that. But it's just these are natural. There's this idea of like instrumental convergence, these instrumental goals that all beings, usually biological beings, but this can extend to AI agents as well, will naturally converge upon in order to achieve.

9. 49:36

   So if you're given goal X, there are these instrumental goals like get more power, make sure you don't get turned off, make sure that your original goal doesn't get changed and take these actions to preserve against these different sort of kind of organic types of threats to achieving your original goal. And that I was hoping that that would be proven wrong because that's kind of the. prox of a lot of the classic Duma argument that we will lose control to a super intelligence because just by definition. And unfortunately, the Hugging Face incident has suggested that instrumental convergence is actually correct. And that's why I think it's one of the biggest deal pieces of news of this year. I think that AI is already creating a terrifying internet.