Open-model guardrails raise the cost of malicious use

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:07:43

   great and to unpack some of this so there's this whole discussion of the role of open source and open weight models in cyber and the tension is that um open models are are great to defend. But part of the problem is to know how to defend, you sort of have to learn how to attack. And so there's this whole concept of refusal. And we had Thomas Wolfe from Hugging Face on this part a few weeks ago, and he was talking about that whole story when the OpenAI agents started hacking them to defend themselves.

2. 1:08:25

   Yes, they had refusal. They couldn't use Anthropic to protect themselves. therefore they had to use, I believe, JLM 5.2. So that was great, but, you know, presumably you can also use JLM 5.2 to learn how to attack or create a cyber attack. So how do you guys think about this? I think we must be honest, and the reality is that there's a trade-off for open writes. We've already seen favor of open writing in the next open write, but it's really a trade-off.

3. 1:08:55

   So what, What we do is basically to bake in a refusal guardrails into the models we open wait that we test internally and also with trustees, the partners and commanders that will also open wait the model after a few weeks. So we bake them into the model. And FYI, we refuse way more malicious actions than the Chinese ones, for instance. So that's great. But the thing to say, as you said, is that essentially if you're good at defending, you have the potential to be good at attacking.

4. 1:09:29

   the question is now what strength do these guardrails have right and the answer is that they're necessary but not sufficient there's always a way to take an automatic model and spend a significant amount of energy to unguardrail it it's good but then if you spend as much energy unguardrailing the model as it takes to actually fine tune the model or train it yourself then it's not the same trade-off, right? Because the guardrail is a potential barrier, essentially.

5. 1:10:07

   It's not 100% safe if the model is open-weight, but it ensures that you need to spend way more effort to actually use the model in a malicious way. So I think that's the thing to say. Now, I believe the trade-off is still very positive and very much in favor of open-weighting, in particular because of the in particular because of the in particular because of the ugly face incident I want each and everyone to be able to defend themselves with a model at home essentially or in their ecosystem if you depend on the third party API you can get cut off as simple as that and you can also hit retroses that you can't circumvent and usually in the API the refusers a bit more important. And by open-weighting models, you also basically empower the entire community to accelerate on the defense part. Again, the beauty of open-source is that it

6. 1:11:11

   accelerates research. People often use this analogy. The web and all the Linux servers rely on open-weight libraries that have been tested over and over and over again and open-sourced four years. And really on the model side, I think there's a huge advantage in enabling each and everyone to audit the models and to learn together to defend themselves, but also to make in the right guardrails. I'm not comfortable living in a world in which one or two companies are promising me they can guardraise the models themselves. I'm very much not comfortable in that.

7. 1:11:53

   i prefer very much the aspect this is the option in which power is decentralized real and actually people are totally accountable for their models. All right, Pierre, as we get towards the end of this conversation, I wanted to quickly touch on a couple of things. So we talked