Open models helped when closed tools refused to respond

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 7:08

   unnoticed. So to unpack some of this, so you alluded to the fact that to be able to defend yourself, the closed source models were not available. And that's the sentence you tweeted that. And that's one of the key aspects of this, which is so fascinating, which is like the thing you said, the first autonomous AI attack was carried out by a closed model and defended against with an open one which is basically the reverse or what everybody thought so can you can you unpack that what what uh what did you guys do how did you go about it and what does that mean for open source yeah i mean the the so what happened in the so so we have a couple of like traditional cyber security protections so like wii's or amazon like we use a range of them but we also have a stack like many people who is mostly based

2. 7:59

   the right on Cloud Code right now, which we use for many things. We use that for deploying, we use that for coding, but we use also for operating and processing. And in this case, it's not only that Fable told us, I'm not allowed to touch cybersecurity, but also Opus, which was the fallback, was saying, no, I'm also not touching these things. So basically the end was just say, we won't process anything about that, but you're welcome to apply to our cybersecurity program with a link to an application form.

3. 8:29

   But the thing you have to realize there, and I was mentioning also earlier, is when somebody is penetrating in your infrastructure, they start to what we call move laterally, which is usually you have an entry point, but the destination is quite far. So they kind of find a way to compromise some of the credential there to get progressive access to more and more of your infrastructure. You kind of have to move fast.

4. 8:53

   It's a matter of at least hours and even more minutes so that you can stop them as soon as you can so that basically the access and the blast radius stay localized. So you don't have time to apply for a cybersecurity program. It's not the moment you want to fill in like a Google form or something and just have someone take time to vet if you're supposed to be given access or if it's not or if it's too dangerous and maybe interview you.

5. 9:18

   That's just definitely not the way this is going to work. And I think in the future world where cybersecurity is going to be a big topic And I think it will keep being a more important topic. It's a little bit naive, I think, just to think that every company is going to be part of the same, you know, vetted cybersecurity program by just one of the two big labs. I think it's like a little bit like crazy to think that you're going to have, you know, 100,000 vetted companies that progressively apply.

6. 9:47

   So anyway, in this case, we say, well, we have to stop this now. So we basically tried all the open source model that we had. And GLM, which is close to the state of the art right now, which just before Kimi that this happened, now probably Kimi K3 is the closest to the state of the art, but GLM 5.2 is actually really good as well, was just very good to process this. And basically we could extract some of the pattern and we could understand basically the hacker here was trying to access mostly the data set.

7. 10:18

   So we just reboot this part of our infrastructure. we have a very simple, we have a very flexible way to respawn quotes and notes. So this was how we just ultimately stopped. But I think that was very ironic, but because I think one year ago, roughly around the summer, most of the discussion about open source was this very simple mapping where open source was equal to unsafe and closed source was equal to safe. And that seems very obvious in the mind of everyone. And there was this idea that if we only have closed source, we'll be just fully safe. And if we only had open source, we'll be very unsafe.

8. 10:59

   Well, everything that's been happening in the past month has been, I think, basically contradicting this very simple mapping. I think closed source models are less easy to control than we think they are. On the other hand, open source model, for some reason, it might change in the future, but currently are not trained so much on actually, I would say bad behaviors like that. So they're pretty bad at cyber attack or like deceptiveness, if you look at that. So it's a little bit hard to understand exactly where does this come from, in part because well, open weights model tend to come with a very extensive technical report that explains how they are trained. Close source model, we can only try to guess. So it's quite funny also as well, because I was seeing a lot of people trying to understand why Mythos was behaving like that. And they were

9. 11:48

   using Kimi K3 as an example of how we should be trained. So they use like this supposedly like open source and say it's very bad, dangerous thing to try to understand why the good thing that we don't know anything about is being trained. but yeah that's how the world is right now so I think it's interesting more generally I think in the future and to be honest and I would say I'm careful like optimistic around that and I'm not specifically against closed source model or ultimately pro open source model I just think both of them are necessary just like we like to have closed and open source software we like to I mean, I'm happy to run on the Mac right now, which is, you know, kind of a mix of both.

10. 12:31

    It's based on the Unix kernel that was open source, but then there's a company over there. They're closed source, and that's great because I'm also very happy I'm not on the Ubuntu right now. It's super easy to record this podcast with you for this reason. Well, my former Ubuntu spent a lot of time, like many of us, just connecting a microphone or whatever and trying to watch a movie with my girlfriend.

11. 12:49

    My girlfriend was, when are we going to watch the movie? I was like, I'm almost there. I'm open. They're still just installing the code. So I think both of them are advantage and equivalent and drawbacks. And I think the world where we are, where the frontier is closed and there is like not too far open source model that you can use as well for many things is actually a pretty good middle ground solution.

12. 13:15

    To make sure I got it right.