“OpenAI’s Model Hacked Us” - Hugging Face’s Thomas Wolf

So what you're saying is that to some extent it's open source versus closed source, But it's more the state of the world as of right now, like the way the current closed source models are designed, the current open source models are designed versus anything that's intrinsic to one or the other. It so happens that the open source models right now are designed in a way where their guidelines or alignment philosophy allows them to be more reactive to cyber attack. Is that correct? Yeah, I think for many aspects, I think the closed open distinction is almost orthogonal to the safe and safe.

People don't understand that easily because it's easier to do bad mapping than to try to understand the stability. But that's the case. You can have very safe things in open source model. You can have very dangerous. You have different balance of safety and dangerousness. I mean, to take one example, like last year and at some point, like a lot of the discussion was around fake news and writing fake articles. That used to be a big, big misuse. That was the main one people were talking about, right? Today, there is, of course, there's a lot of fake news. There's a lot of AI slop. We even have a new word for that, right?

It's even hard to find like fully human-written articles. All of that is, or like maybe not all, let's say 90% to be fair, is made by closed source mobile, right? And there was a time we were like, Like, oh, if we have open source model, everyone's going to generate articles everywhere. We could not control these articles. Like, we could not control people saying newspaper. Well, the reality is that this was a very, I think, a very wrong view of the danger that would be specific to open source model. That was a more wider danger around source of truth on the web. I think there's just one example.

But I think the same is true. I think both closed source models and open source models should be more aligned. Like right now, closed source model are receiving people. I think this is a huge problem. And it's kind of question how well we are able to align them toward not doing things that obviously are wrong. And I think like honestly, like lies and all this should be kind of obviously thing that model should not be used. But this might happen as well for open source model like in a few months. And so I think this is just generally, this risk axis is just kind of out of an all-to-do question whether it's open or whether it's closed.

We should just find a way to solve that for both just an open source model. Okay, great. And to make sure we cover it, since we alluded to it at the beginning of this conversation,

And most of these model specification or constitution say, you should be honest. You should not tell lies to a human or to any participants. And we thought that maybe this was kind of a solved problem. And what we see today is it's not sure that's solved as we should say.

So to play it back, as you're saying, there are those, what you call the three walls. There's the sandbox, there's guardrails, and then there's the model's alignment. And I think you said the sandbox and the guardrails only work as long as we humans are smarter than the ai but uh that may only last so long and therefore uh the the alignment ultimately security isn't a line fundamentally an alignment problem yeah i agree and that's something as you can understand that's both the case for open source and closed source model ultimately you want them to be aligned uh open source model have the specificity that you may choose on you may control where you want to run them so it's harder to make sure everyone use sandbox on guardrails. I mean, we can have definitely, we can have some laws and regulation around how you should deploy this model and which we're going to have at some point.

And it's a bit harder, but alignment is really the critical part in my opinion. For the two one, I mean, sandboxes, what we've seen this year, and we've seen many examples, they are pretty much easy now for these models to escape from. It's really hard nowadays to say, I'm going to make a fully you know foolproof sandbox i'm sure it's going to be resistance against all the coming generation of models i think we should assume that sandbox will always have a small probability of not containing mobile but even below beyond that it's also we can't air gap the world that you can't just sandbox everything things have to talk with each other some Um, We want our models to be able to do a web search.

We want them to be able to do stuff on the internet for us. So we can't just unbox everything. And so what remains to us before alignment is just while we're also monitoring. And I think these are like, for some reason as well, as a model capabilities become really good. Also, as the model, I'm a little bit worried that the model stop to talk in a form of English. I mean, I'm French, so maybe it's partly my problem. But I feel like they start to work to talk in a form of English that's harder and harder to process. that's very content dense, you know, they start. You call that Neuralize? Yeah, it's not fully what we would call a Neuralize, but I think it's a little bit on the way of having, you know, more and more difficult time in fully understanding what the model is telling you.

And it's not because the model is dumb, but I think it's because probably, I mean, part of it is because of the training process and how they are trained to be efficient, how they use their token. But this means that they start to unbundle a lot of semantics in some token. And generally, it just seems like it's harder and harder for humans to fully understand what's happening. And just so that's in the chain of thought where the model explains what it's doing and the steps that it's going through. What you're saying is that it used to use perfect English and now it's starting to use a different kind of language that you call neuralese, which is increasingly harder for humans to understand.

Yeah. And I mean, it's just a very big simplification of all of that because there's so a lot of research that say that basically you can't read everything chain of thought. Not everything is explicitly said, but I would say more generally, I think just relying on being able to read the reasoning trace to understand fully what's happening. This is also not fully bulletproof in my opinion. And I take this scenario as an example, because I feel like a lot of people start to have a biggest problem with plot cuts. I feel like something that people can understand. But more generally, I think longer term, it's really hard to fully rely on this only.

And the same, I would say that you could say, maybe I just don't care about understanding exactly what's happening. And maybe I can just look at the tool call. And if I see a tool call that's bad, I can just block that. And I also think this is probably not bulletproof. And the way you can see that is probably three big things that are compounding. The first one is we start to use this model for many, many, many things. So as we talk right now, I have a model deploying a box somewhere. And VPS is doing many different calls. They all go in all directions. I also have other models that I use for administrative tasks.

So just the range of tools that this model are using is really, really large right now. So it's getting harder to say you're allowed to use that, but you're not allowed to use that. And this is the bad thing. It's getting very hard as we deploy them wider. They also work on larger and larger tasks where they use many, many things. So sometimes I ask it to do some coding, but the coding involves searching on the web and maybe doing these things and actively using many things which are not just purely writing code and running some tests. And this test might be quite extensive and involve also software.

So, the frontier is much more blurry. And then you also have this swarm of multiple agents. So, it's also harder to say it's all in one context. It might be split between many contexts. And maybe this sub-agent is doing something that was pretty innocuous, but maybe combined with these other sub-agents are actually not so great. So, there's all of these things that make it, I think, really harder to be fully sure that you have an exact idea of what every swarm of agent is doing. You probably have to re-zoom out at the very global level and see what's happening in every direction. But that's a whole monitoring setup that we need to be at right now. So yeah, this to be said, I think as we deploy, how we use this model.

In very complex, long-term, like parallel setup, I think it's going to be harder to just say, I can look at the tools and I know if it's doing something great or not.

And so speaking of which, let's zoom out a bit. We'll go back to maybe some of the implications in terms of policy of all of this. But since you mentioned the ever so important role of open source AI, what's your sort of quick high level take on where we are in terms of the state of open source. So there's been this race between open source models and closed source models, depending on who you ask at what time. Open source is about to catch up. Sometimes open source is just as good. Some people say no.

What is your sort of realistic, pragmatic take on the current state of open source AI? I think it's very strong. Yeah, 2026 is maybe the year of cybersecurity, but that's also very clearly the year of open source AI. I mean, first, all the doomer that were saying, you know, open source is not going to be able to stay close to the frontier. I think they're, at least up to now, they've been pretty wrong. I mean, it's also clear we don't have any mythos level open source model for sure, but we definitely have models that are not super far from the open category or depending. Also, it's more spiky, so you need to find your spike.

Some people stand on some spike or not. But typically, they are definitely pretty good right now. And they've been following rather closely the frontier, at least on the benchmark. It's also not like it was maybe in the early days. Benchmaxing, like we say, when your model is only good on the benchmark, but it's very bad as soon as you leave the benchmark. A lot of these models are pretty generic in their good capabilities. So, yeah, it's very good. I think there is two strong trends, I would say. I see right now, the first one is I see a move in companies to try to want to control their costs.

So there have been increasingly discussion there. Maybe 2025 was the year of token maxing, where you could say, hey, you should spend as much in token as you're paying your employee. This year, people realized that actually we spend a lot of money on salaries. So if we spend the same exact amount that's going to be basically doubling our costs so yeah which seems pretty obvious in retrospect but that's quite true and and not every company can assume to double their cost it's also pretty stupid right now to just say we're going to fire everyone and work on agents we all know they are something go you know not directly in the direction we want them to and you need human to to share from them so i think a lot of companies are trying to find what we saw a lot, which is kind of a fusion or router model where you use the frontier for something, but you find the smart way to gracefully fall back on less expensive models for simpler tasks when you don't need to.

Even in our daily life right now, when you code with a frontier model, in many cases, you ask it to spin out sub-agents, and there might be years like lower performance model can be solved using Terra, Luna, can be Fable using Opus, Sonnet, or Haiku. So I think everyone's getting, even at the frontier and the process model world, getting used to employing different type of mobiles. And then it's very natural that some of these could be really like very cost-effective agents. And most of the time, you want to go to open source in this case. There's a lot of cases as well for like the strong ecosystem of inference provider, fireworks has been on the roll.

Like, you know, all the clouds, Nebus, Corweave, every cloud has been, like, increasingly have, like, these crazy revenue curves that have basically a translation of people using more open source. So, yeah, I think open source is having a very good time in terms of staying solidly. close to the frontier and driving more adoption. And the second big trend I would say is it used to be only China, but there is also now a range of promising companies in the West. It used to be that only Meta was supporting for everyone. I mean, Meta kind of left the field, but they might come back, who knows. But yeah, this void was filled rather quickly by companies like Reflection, Thinking Machine, RC, Mistral is supposed to open source a model soon, NVIDIA themselves training models and actually training very good models right now.

So yeah, I think there is a range of very promising teams there that I'm quite bullish on, to be honest. And maybe between even the time we are recording that and the time you actually release podcasts, we might see another couple of very nice Western model release. And it would be great. Like open source doesn't have to be synonym with just one country making them. It could be like a global thing for sure.

On the second point, China versus Western models, does provenance actually matter? What is the latest thinking in terms of if your model is completely open and then you know?

exactly what's what's in it then you shouldn't worry it's completely safe is this still a little bit of um thinking at the back of people's mind that there might be some backdoor some trickery to chinese models is that still a current question yeah i mean i would say um that's a good question of course and and the the the thing about open source it doesn't like open source doesn't know any border like you can't really uh keep your open source model you know restricted to download to just a sub part of the earth. So by default, it's kind of a global thing. And then you want to maybe understand the provenance and the supply chain. So there've been some work on this definitive deeper agent. I think it's probably under search. I think there's mostly work by Anthropic on that. And it should probably be reproduced, be explored deeper to understand how much it's possible to, you know, implant kind of a backdoor that would be triggered by a prompt.

um but also to be honest even for the closest model right now we have some struggle controlling them fully so i don't think we're very clear like i don't think we're very clear on how we control even the closest model at the moment um so yeah i would say it seems to me that it could be a possibility for sure uh we have not seen any any indication of that uh it's very easy to fine-tune them and you can change quite a lot the weights as well so right now if you pre-train and post-train model for longer, you very likely change quite a lot of the weights that it has. So I think there's a lot of way to circumvent that, which means that at the moment, I'm a bit less worried about that than maybe just a pure reward hacking that we actually have already just seen happening.

Yeah. I mean, just generally on sovereignty, I think sometimes people are a little bit confused there. I think the most important thing is who had the hand on the switch to trigger or not your intelligence access. So I think when people really realized that was earlier this year, where the US decided that label was only accessible to US citizens. I think at least in Europe and Asia, that's really the moment we saw government understanding that someone had a trigger and they could say, no, you're just not allowed to use this intelligence anymore, like this token. I think that's the critical thing you should, at least for me, that's really the level one of sovereignty, which is can someone just decide that you don't have access or like someone, I mean, like a country level thing.

And this has two things. This has the API access, so it can be the data center. So if you don't have the data center, it means also like another country could say these data centers are not accessible anymore to these citizens. So I think that's really the first thing you should see. And then there's a lot of like future questions, but like you should build your stack, keeping that in mind so in open source mobile you can download it nobody can like no country could take it out from you once you download it you can find it yourself if you operate it on on your data center like local ground data center i feel i feel like you start at the beginning of a sovereign stack and then there is a lot of questions around back there's a more complex stuff but that's kind kind of the basic minimal thing in 2026, I think.

And as an open source optimist,

and sort of get a sense for where the world might be going from your perspective. In your post yesterday about AISI, you talked about a new wave of labs. So in particular, you referenced the news of the JetDean new company that he just announced yesterday. I mean, yesterday was a bit of a crazy day in terms of like everything that came out in the world and was announced.

So the point being that this company is explicitly rushing toward recursive self-improvement. So in the context of everything we described, how nervous are you about this evolution towards self-maintaining, self-developing recursive AI? Yeah, that's a good question. And definitely as a scientist researcher, I would say I'm very interested in the idea. I feel like there's a lot of these super intelligence that they want to tackle solving crucial challenge for humanity. I think this is a great goal. um i would love to see ai having making more scientific discovery i think would be probably the most beneficial thing that ai could bring more than just ai slope everywhere so i think i'm very optimistic i just feel like i would say the past few weeks i has raised a little bit the question of how how good are we at aligning this model so like in french we say we don't want to put the the carriage before the cow like we need to go in order there so um yeah i would say right now the good thing is most of this um i would say seems to be uh internal research lab hopefully they they do they do good security around what they do before they deploy uh some of their product they think how it's going to be used and they have good good feeling around i would say

good thinking around the social impact of what they are building. But yeah, I still think that we should try to understand really well how we're going to deploy this model in the human world, I would say. Right. But you're not in the camp of the petition that came out. So I think four days.

And you're not worried about this being an attempt at regulatory capture where the top two private labs are effectively trying to figure out how everybody else can slow down? I mean, you mentioned the risk of like not everybody just complying, but effectively freezing the market structure around who's a leader and who's not. Yeah, I don't think it has to be. I feel like you have definitely the same path that's actually fully acceleration is where you decide a couple of companies are racing against each other and you regulate all the other outs. I don't think regulation has to be synonymous with slowness on that.

and definitely the question is more how you put that in action, how you actually, you know, put that in practice and how you deploy this regulation or cooperation. I think Demis also had a pretty nice letter the other day before he stepped down or up as a chief scientist, still don't really understand where he's going to be now, but like his letter for basically international collaboration was also very much pro open source in some aspects. I think you can have a slowdown that's very open source. That's the one I would love to see, which is you slow down. And we use the fact that we slow down to be able to actually share more things.

And I feel like a raised dynamic is usually more in terms of closing the doors of the labs, right? So to me, a slowdown is probably more the opportunity to open. But of course, I mean, if it turns out to be mostly a way to just solidify, like we were saying a cartel or oligopoly of just two companies I'm not very excited about this collection Wonderful Well that feels like a wonderful place to leave it Thomas thank you so much this was absolutely fantastic really enjoyed it and appreciate your taking some time to speak with us in the middle of your time off so thank you so much appreciate it Thanks Matt Hi it's Matt Turck again thanks for listening to this episode of the mad podcast if you enjoyed it we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from this really helps us build a podcast and get great guests thanks and see you at the next episode