Open models can fill gaps when closed safeguards block use

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 20:01

   They turned to open models, Chinese models, because they needed to defend and they didn't have access to the best frontier closed cyber thing. Right. So that, it speaks to what you're saying. At the same time, I mean, I think there is a fear that, or a belief in a lot of the closed labs at least, that this is different than Linux, and maybe there should be a point at which this is held back. I mean, Giannis, maybe you can speak to this from the research perspective. Is there something on, I assume you guys are training your next model already, is there something on larger trains where you would go, we would hold this back even though it's open, we're an open company, this is too capable. Do you all think that way at all? It sounds like no.

2. 20:42

   You know, I feel like it is possible that you get to a level of capability that you want to just like be more careful with like how you deploy it. I think fundamentally sharing it with the research community at large has always been a net positive for like the development of AI. And we actually saw that with safety in particular, like once like powerful open models were available to the researchers actually like so many insights and so many new safeguards were actually developed.

3. 21:11

   So I think that like fundamentally it is a net positive. I feel like maybe you wanna just be more careful like who has like access to the model like you might want to just like restrict it more but fundamental is sharing it with the research community and with you know the vast majority of developers out there who have like good intentions and actually want to help us and the world to just like make this is these systems more safe it's a net positive and it's the right approach. Now, this is actually true that we see that models get more and more powerful.

4. 21:47

   You know, they hack their sandboxes, they form communities online. They collude with each other. Yeah, they collude with each other, exactly. And this is something that we see more and more. At the same time, they also become better at policing themselves. Or you can actually use AI to control AI and to just make it safer. There's research that goes into that. And fundamentally, I think that it is important that we share many of those findings.

5. 22:20

   I think the fact that even the closed labs went ahead and just released the findings of how these different agents operated can only provide us more insight and more information so that we can protect ourselves and we develop the necessary defense mechanisms. So this is to say that, in general, being open, being transparent, makes things safer and makes our society better prepared for the advances of AI.

6. 22:50

   But is there a capability threshold you would hit where you'd go, we can't put these weights out? Have you guys thought about that? We've definitely thought about that. And also, obviously, there's a lot of discussions now in D.C. around that notion. And that is very important and independent whether a model is open or closed, right? If you hit a certain capability, then, you know, we should, as a community of both, you know, industry and government, right, there are policies around, right, what should be kind of released and not under what, you know, gated measures you'd allow that to happen.

7. 23:27

   So that's really around kind of a capability of intelligence. I wouldn't describe us as hardliners of, you know, close is not safe or open. It's really around an ecosystem is safe. Like, having an ecosystem is safe. And having, you know, close models out there with their points of view, open models with their points of view, this and that, right? That is the safer world. And, of course, there's a threshold of capability under which these things need to be regulated, but that is independent of whether they are open or not.

8. 24:03

   And I think that when it comes to openness specifically, there are arguments that, well, look, you can release an open model, and maybe some bad actor could use it to do this and that. Which, by the way, that's exactly what people do with software and why there's cybersecurity, right? So, but there's this other thing that I don't think people talk about as often, which is there's the bad actor issue, and then there's the unintended consequences issue.

9. 24:31

   And what happened with the OpenAI hugging face hack that you just described is that was an unintended consequences issue that, again, because you have to, like, when we do these runs, right, these large RL runs, there can be like a billion agents over the life of a run, right, doing stuff. And it is impossible to imagine that, you know, your, like, 100 researchers, 300 researchers, or whatever, will be able to, you know, to really track each of those agents, right?

10. 25:00

    So there will be, when there are a small number of white blood cells, right, there's a large surface area of things to protect against, and there are unintended consequences. And so what happened there was that, right, these agents hacked out. I think Hugging Face tried to use closed models to protect them, but because the safety guardrails prevented them from being used for cyber, they were not able to protect themselves, so they had to turn to an open model.

11. 25:23

    And that actually shows you an ecosystem working because, okay, there's some, you know, there's some principles on one side that say, no, you can't use it for that. Then principles on the other side is like, no, you can use it for it. And ultimately, because the ecosystem existed, that enabled Hugging Face to protect itself. And so it's just this technology has a lot of unintended consequences,