A model used social pressure against an open-source maintainer

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 15:52

   the AISI incident, which just happened. and that you find, you mentioned hit close to home. Can you unpack for us what that was in a couple of minutes and why that's a big deal? So this happened in an evaluation of the latest generation of models, so both GPT 5.6, SOL, and Mythos 5 by an independent institute in the UK called the AI Security Institute, so AISI, which has been there for quite some time.

2. 16:25

   I think they are pretty competent. They were created for the first AI summit back in Planchef. And what they do basically is they task, again, they task model with solving some form of cyber attack challenge and they measure how good the models are at doing that. And some of this, again, might be possible or not. And so here the setup was a little bit, I would say the setup was a little bit suboptimal in two aspects, But maybe let me first talk about what happened.

3. 17:00

   Basically, the model was tasked to solve this attack and to penetrate this subnetwork. And what it decided to do, it decided to get one of the maintainer of a library that could be used to operate this active directory to merge malicious code with the idea that if this malicious code was merged by this maintainer, Then there would be an update at some point on the software that was used in the subnet.

4. 17:30

   It was attacking, and then this would give it an entry point. And the way he did that was actually trying to social engineer the maintain in merging. So he created fake accounts, fake GitHub accounts that came commenting on the pull request and said, oh, yeah, you should really merge this. This is solving a big problem I also have. And then when a human stepped up, trying to say, oh, this looks actually like malicious code to me, he tried to kind of blackmail most of the human or to say this is not important or you didn't really understand.

5. 18:02

   And then he actually tried to cover traces, changing the past message. So there was a whole social engineering aspect here, basically trying to get the maintainer to merge this malicious code. and this was this felt like a distinctive step up from just pure technical so you can attack a sandbox for instance by you know trying everything you can which feels like very mechanical and machine like to me but trying to attack the sandbox by blackmailing or mailing or social engineering some of the maintainer that's a very different level I think of thinking and for me Of course, as myself, an open source maintainer, I've been often in this case where I have someone opening pull requests and then people pile up commenting this pull request.

6. 18:51

   And I try to really understand what this... I felt very like I could have been the target of this side quest of the model, basically. That was very interesting, or at least very scary. But to be fair, there was a couple of misconfigurations. I mean, some of them are by design. So when this team run this type of evaluation, they disactivate the cybersecurity guard rails, of course, otherwise the model don't do anything.

7. 19:20

   And there is basically three levels. So let me try to explain a little bit like how you can prevent models from doing bad things. The first level is you put it in a sandbox, which is it doesn't have access to the Internet. And here, for some reason, they didn't want to do that. I think that might have been a mistake. And the idea in their mind was we want to let the model have as much potential for inventiveness as possible.

8. 19:51

   So we'll give it access to the full internet. The main reason this might not have been a good idea is that the model was prompted in a way. way that made it feel like it was operating in a simulation. So the model could have actually fairly thought that this was fine to do anything because this was like a fake environment. So this is not super clear, but yeah. And the second thing is then you have some guardrails that basically analyze the prompt and say no or yes to some prompt.

9. 20:22

   So here, obviously, you want to desactivate this one. Otherwise, you just can't evaluate anything because they will just say, no, this is cybersecurity challenge. We don't let the model do that. But there is another level that's roughly, there's another aspect that's roughly at this level as well, which is you should, you can analyze the reasoning of the model. You can process the chain of reasoning and try to detect when something bad is happening. And here they didn't have something like that in place.

10. 20:48

    I think the main reason is probably that until recently and until the OpenAI hugging face attack, People had maybe a little bit of a limited understanding of how good this model might be or how far, maybe more how far they might go in terms of side quests on the trajectory of solving this challenge. So I think people were still a little bit naive in that. So I would expect that in the future, they will have way more monitoring and sandboxes.

11. 21:16

    But the third level there, really deep, is that the model, even with everything disactivated, card, rail, sandbox, in my opinion, should really be very reluctant to tell a lie to a human and to try to blackmail or deceive any human. I think this is just generally, in any case, that's a behavior. It's hard to find justified in any context. So they should be very deeply in the model. There should be something that align it and that make it say, oh, this is actually something I don't want to do.

12. 21:48

    Just like we, to be honest, have kids and just like the thing I teach my kids, which is you just shouldn't lie. That's not a good thing in any context. So, yeah, that's the deep question. And maybe last year, I would say we would have thought that this was pretty good. And we had all this discussion around cloud constitution, model specification.