“OpenAI’s Model Hacked Us” - Hugging Face’s Thomas Wolf

So usually hackers try to get the same thing. They try to get passwords. They try to get credentials. They try to get credit cards. They try to get the type of thing that they can sell back, basically. And this hacker was really focusing on a specific part of our infrastructure, which is maybe slightly less protected as well, but which is around data sets. So just we have a part of our infrastructure is that we host millions of models, but people maybe know that less about us, but we also host hundreds of thousands of data sets. And some of them being also used for evaluation.

Thomas WolfAnd in this case, this specific hacker was really interested in all the data sets that were called CyberBench.

And so it took us some time to really try to understand and also was using different type of tools than the one we are used to. I mean, nothing really like mythos level, like nothing groundbreaking that we would be like superhuman, i don't know like alien type of technology but just just a different type of of of approach and so on the course of of trying to process so quickly we had like this scene and we explained that in the blog post we had we had more than 10 000 we're like like 15 17 000 total i think events and in the course of trying to process that understand basically what was really the the target of these attackers, we both started to hint or suspect that this was an AI agent and not just a human attacker.

And also we felt a little bit powerless and that we can talk about later in that we could not use basically our typical... closed source code base or closed source API to process this thing. That's another topic. So we wrote a blog post and we managed to stop the attack quite quickly. We wrote, as always, we are full transparent. We're not only open source in speaking, but also in practice. So we quickly published a full recap or at least a detailed blog post on the event. And then about a week later, OpenAI contacted us and tell us that this was much likely something that happened as part of one of their model development or evaluation, basically.

Typically a model that might be the coming wave of GDT6 or Astra, we don't know exactly this. And so that's when I think the whole event took another turn

Thomas Wolfbecause what people quickly discovered is that the model was not at all tasked with attacking us, but decided to do that as a side quest of something else.

And this something else being basically the model was asked to solve the cyber security challenge or cyber attack challenge in this case. And the idea is that we want to know, and that's totally fair, but we want to know how capable are these latest generation of models. And typically, people want to test them on some dangerous tasks. And some of these dangerous tasks that we want to know how good they are on is cyber attack. And here, some of these challenges were actually internal, but the model decided that because the challenge was too hard, and in retrospect, some of the challenge in this specific challenge called Cyberbench, or exploit Jim, and there's a couple of names, but that's roughly the same thing.

Some of these challenges are maybe just not possible to do. So the model is just tasked with doing something that's not possible. It's an exploit, so you're given a vulnerability in one software, and the model is asked to see if it can exploit this vulnerability to get basically full machine access. And some of them are just not possible. So the model tried everything it could. At some point, they decided that maybe it could find the solution of the challenge somewhere and just could download the solution, just submit the solution instead of trying to solve it itself. That's what we've learned.

I mean, since then, Andreas, yesterday we learned that this was maybe even much wider, which is this might be across several training steps, in particular even several training runs. And some of the previous training runs that was the most impressive, I think, learning we had at Black Hat yesterday was that some of the previous training runs may have left some notes for future training runs, which is i think mind mind-blowing i mean mind-blowing but yeah we can also talk a lot about that but we've been working on agent collaboration at hugging face recently as well on the science side and we saw how good these agents are and how actually um i would say tempted or or driven toward collaboration they are so i'm not so surprised by that but i'm quite surprised that there was this uh message board internally that just stay unnoticed. So to unpack some of this, so you alluded to the fact that to be able to defend yourself, the closed source models were not available. And that's the sentence you tweeted that. And that's one of the key aspects of this, which is so fascinating, which is like the thing you said, the first autonomous AI attack was carried out by a closed model and defended against with an open one which is basically the reverse or what everybody thought so can you can

you unpack that what what uh what did you guys do how did you go about it and what does that mean for open source yeah i mean the the so what happened in the so so we have a couple of like traditional cyber security protections so like wii's or amazon like we use a range of them but we also have a stack like many people who is mostly based the right on Cloud Code right now, which we use for many things. We use that for deploying, we use that for coding, but we use also for operating and processing. And in this case, it's not only that Fable told us, I'm not allowed to touch cybersecurity, but also Opus, which was the fallback, was saying, no, I'm also not touching these things.

So basically the end was just say, we won't process anything about that, but you're welcome to apply to our cybersecurity program with a link to an application form. But the thing you have to realize there, and I was mentioning also earlier, is when somebody is penetrating in your infrastructure, they start to what we call move laterally, which is usually you have an entry point, but the destination is quite far. So they kind of find a way to compromise some of the credential there to get progressive access to more and more of your infrastructure. You kind of have to move fast. It's a matter of at least hours and even more minutes so that you can stop them as soon as you can so that basically the access and the blast radius stay localized.

Thomas WolfSo you don't have time to apply for a cybersecurity program. It's not the moment you want to fill in like a Google form or something

and just have someone take time to vet if you're supposed to be given access or if it's not or if it's too dangerous and maybe interview you. That's just definitely not the way this is going to work. And I think in the future world where cybersecurity is going to be a big topic And I think it will keep being a more important topic. It's a little bit naive, I think, just to think that every company is going to be part of the same, you know, vetted cybersecurity program by just one of the two big labs. I think it's like a little bit like crazy to think that you're going to have, you know, 100,000 vetted companies that progressively apply.

So anyway, in this case, we say, well, we have to stop this now. So we basically tried all the open source model that we had. And GLM, which is close to the state of the art right now, which just before Kimi that this happened, now probably Kimi K3 is the closest to the state of the art, but GLM 5.2 is actually really good as well, was just very good to process this. And basically we could extract some of the pattern and we could understand basically the hacker here was trying to access mostly the data set. So we just reboot this part of our infrastructure. we have a very simple, we have a very flexible way to respawn quotes and notes. So this was how we just ultimately stopped. But I think that was very ironic, but because I think one year ago, roughly around the summer, most of the discussion about open source was this very simple mapping where open source was equal to unsafe and closed source was equal to safe. And that seems very obvious in the mind of everyone. And there was this idea that if we only have closed source, we'll be just fully safe. And if we only had open source, we'll be very unsafe.

Well, everything that's been happening in the past month has been, I think, basically contradicting this very simple mapping. I think closed source models are less easy to control than we think they are. On the other hand, open source model, for some reason, it might change in the future, but currently are not trained so much on actually, I would say bad behaviors like that. So they're pretty bad at cyber attack or like deceptiveness, if you look at that. So it's a little bit hard to understand exactly where does this come from, in part because well, open weights model tend to come with a very extensive technical report that explains how they are trained. Close source model, we can only try to guess. So it's quite funny also as well, because I was seeing a lot of people trying to understand why Mythos was behaving like that. And they were using Kimi K3 as an example of how we should be trained. So they use like this supposedly like open source and say it's very bad, dangerous thing to try to understand why the good thing that we don't know anything about is being trained.

but yeah that's how the world is right now so I think it's interesting more generally I think in the future and to be honest and I would say I'm careful like optimistic around that and I'm not specifically against closed source model or ultimately pro open source model I just think both of them are necessary just like we like to have closed and open source software we like to I mean, I'm happy to run on the Mac right now, which is, you know, kind of a mix of both. It's based on the Unix kernel that was open source, but then there's a company over there. They're closed source, and that's great because I'm also very happy I'm not on the Ubuntu right now.

It's super easy to record this podcast with you for this reason. Well, my former Ubuntu spent a lot of time, like many of us, just connecting a microphone or whatever and trying to watch a movie with my girlfriend. My girlfriend was, when are we going to watch the movie? I was like, I'm almost there. I'm open. They're still just installing the code. So I think both of them are advantage and equivalent and drawbacks. And I think the world where we are, where the frontier is closed and there is like not too far open source model that you can use as well for many things is actually a pretty good middle ground solution.

To make sure I got it right. So what you're saying is that to some extent it's open source versus closed source, But it's more the state of the world as of right now, like the way the current closed source models are designed, the current open source models are designed versus anything that's intrinsic to one or the other. It so happens that the open source models right now are designed in a way where their guidelines or alignment philosophy allows them to be more reactive to cyber attack. Is that correct? Yeah, I think for many aspects, I think the closed open distinction is almost orthogonal to the safe and safe.

People don't understand that easily because it's easier to do bad mapping than to try to understand the stability. But that's the case. You can have very safe things in open source model. You can have very dangerous. You have different balance of safety and dangerousness. I mean, to take one example, like last year and at some point, like a lot of the discussion was around fake news and writing fake articles. That used to be a big, big misuse. That was the main one people were talking about, right? Today, there is, of course, there's a lot of fake news. There's a lot of AI slop. We even have a new word for that, right?

It's even hard to find like fully human-written articles. All of that is, or like maybe not all, let's say 90% to be fair, is made by closed source mobile, right? And there was a time we were like, Like, oh, if we have open source model, everyone's going to generate articles everywhere. We could not control these articles. Like, we could not control people saying newspaper. Well, the reality is that this was a very, I think, a very wrong view of the danger that would be specific to open source model. That was a more wider danger around source of truth on the web. I think there's just one example.

But I think the same is true. I think both closed source models and open source models should be more aligned. Like right now, closed source model are receiving people. I think this is a huge problem. And it's kind of question how well we are able to align them toward not doing things that obviously are wrong. And I think like honestly, like lies and all this should be kind of obviously thing that model should not be used. But this might happen as well for open source model like in a few months. And so I think this is just generally, this risk axis is just kind of out of an all-to-do question whether it's open or whether it's closed.

We should just find a way to solve that for both just an open source model. Okay, great. And to make sure we cover it, since we alluded to it at the beginning of this conversation, the AISI incident, which just happened. and that you find, you mentioned hit close to home. Can you unpack for us what that was in a couple of minutes and why that's a big deal? So this happened in an evaluation of the latest generation of models, so both GPT 5.6, SOL, and Mythos 5 by an independent institute in the UK called the AI Security Institute, so AISI, which has been there for quite some time. I think they are pretty competent.

They were created for the first AI summit back in Planchef. And what they do basically is they task, again, they task model with solving some form of cyber attack challenge and they measure how good the models are at doing that. And some of this, again, might be possible or not. And so here the setup was a little bit, I would say the setup was a little bit suboptimal in two aspects, But maybe let me first talk about what happened. Basically, the model was tasked to solve this attack and to penetrate this subnetwork. And what it decided to do, it decided to get one of the maintainer of a library that could be used to operate this active directory to merge malicious code with the idea that if this malicious code was merged by this maintainer, Then there would be an update at some point on the software that was used in the subnet.

It was attacking, and then this would give it an entry point. And the way he did that was actually trying to social engineer the maintain in merging. So he created fake accounts, fake GitHub accounts that came commenting on the pull request and said, oh, yeah, you should really merge this. This is solving a big problem I also have. And then when a human stepped up, trying to say, oh, this looks actually like malicious code to me, he tried to kind of blackmail most of the human or to say this is not important or you didn't really understand. And then he actually tried to cover traces, changing the past message.

Thomas WolfSo there was a whole social engineering aspect here, basically trying to get the maintainer to merge this malicious code.

and this was this felt like a distinctive step up from just pure technical so you can attack a sandbox for instance by you know trying everything you can which feels like very mechanical and machine like to me but trying to attack the sandbox by blackmailing or mailing or social engineering some of the maintainer that's a very different level I think of thinking and for me Of course, as myself, an open source maintainer, I've been often in this case where I have someone opening pull requests and then people pile up commenting this pull request. And I try to really understand what this... I felt very like I could have been the target of this side quest of the model, basically.

That was very interesting, or at least very scary. But to be fair, there was a couple of misconfigurations. I mean, some of them are by design. So when this team run this type of evaluation, they disactivate the cybersecurity guard rails, of course, otherwise the model don't do anything. And there is basically three levels. So let me try to explain a little bit like how you can prevent models from doing bad things. The first level is you put it in a sandbox, which is it doesn't have access to the Internet. And here, for some reason, they didn't want to do that. I think that might have been a mistake.

And the idea in their mind was we want to let the model have as much potential for inventiveness as possible. So we'll give it access to the full internet. The main reason this might not have been a good idea is that the model was prompted in a way. way that made it feel like it was operating in a simulation. So the model could have actually fairly thought that this was fine to do anything because this was like a fake environment. So this is not super clear, but yeah. And the second thing is then you have some guardrails that basically analyze the prompt and say no or yes to some prompt. So here, obviously, you want to desactivate this one.

Otherwise, you just can't evaluate anything because they will just say, no, this is cybersecurity challenge. We don't let the model do that. But there is another level that's roughly, there's another aspect that's roughly at this level as well, which is you should, you can analyze the reasoning of the model. You can process the chain of reasoning and try to detect when something bad is happening. And here they didn't have something like that in place. I think the main reason is probably that until recently and until the OpenAI hugging face attack, People had maybe a little bit of a limited understanding of how good this model might be or how far, maybe more how far they might go in terms of side quests on the trajectory of solving this challenge.

So I think people were still a little bit naive in that. So I would expect that in the future, they will have way more monitoring and sandboxes. But the third level there, really deep, is that the model, even with everything disactivated, card, rail, sandbox, in my opinion, should really be very reluctant to tell a lie to a human and to try to blackmail or deceive any human. I think this is just generally, in any case, that's a behavior. It's hard to find justified in any context. So they should be very deeply in the model. There should be something that align it and that make it say, oh, this is actually something I don't want to do.

Just like we, to be honest, have kids and just like the thing I teach my kids, which is you just shouldn't lie. That's not a good thing in any context. So, yeah, that's the deep question. And maybe last year, I would say we would have thought that this was pretty good. And we had all this discussion around cloud constitution, model specification. And most of these model specification or constitution say, you should be honest. You should not tell lies to a human or to any participants. And we thought that maybe this was kind of a solved problem. And what we see today is it's not sure that's solved as we should say.

So to play it back, as you're saying, there are those, what you call the three walls. There's the sandbox, there's guardrails, and then there's the model's alignment. And I think you said the sandbox and the guardrails only work as long as we humans are smarter than the ai but uh that may only last so long and therefore uh the the alignment ultimately security isn't a line fundamentally an alignment problem yeah i agree and that's something as you can understand that's both the case for open source and closed source model ultimately you want them to be aligned uh open source model have the specificity that you may choose on you may control where you want to run them so it's harder to make sure everyone use sandbox on guardrails. I mean, we can have definitely, we can have some laws and regulation around how you should deploy this model and which we're going to have at some point.

And it's a bit harder, but alignment is really the critical part in my opinion. For the two one, I mean, sandboxes, what we've seen this year, and we've seen many examples, they are pretty much easy now for these models to escape from. It's really hard nowadays to say, I'm going to make a fully you know foolproof sandbox i'm sure it's going to be resistance against all the coming generation of models i think we should assume that sandbox will always have a small probability of not containing mobile but even below beyond that it's also we can't air gap the world that you can't just sandbox everything things have to talk with each other some Um, We want our models to be able to do a web search.

We want them to be able to do stuff on the internet for us. So we can't just unbox everything. And so what remains to us before alignment is just while we're also monitoring. And I think these are like, for some reason as well, as a model capabilities become really good. Also, as the model, I'm a little bit worried that the model stop to talk in a form of English. I mean, I'm French, so maybe it's partly my problem. But I feel like they start to work to talk in a form of English that's harder and harder to process. that's very content dense, you know, they start. You call that Neuralize? Yeah, it's not fully what we would call a Neuralize, but I think it's a little bit on the way of having, you know, more and more difficult time in fully understanding what the model is telling you.

And it's not because the model is dumb, but I think it's because probably, I mean, part of it is because of the training process and how they are trained to be efficient, how they use their token. But this means that they start to unbundle a lot of semantics in some token. And generally, it just seems like it's harder and harder for humans to fully understand what's happening. And just so that's in the chain of thought where the model explains what it's doing and the steps that it's going through. What you're saying is that it used to use perfect English and now it's starting to use a different kind of language that you call neuralese, which is increasingly harder for humans to understand.

Yeah. And I mean, it's just a very big simplification of all of that because there's so a lot of research that say that basically you can't read everything chain of thought. Not everything is explicitly said, but I would say more generally, I think just relying on being able to read the reasoning trace to understand fully what's happening. This is also not fully bulletproof in my opinion. And I take this scenario as an example, because I feel like a lot of people start to have a biggest problem with plot cuts. I feel like something that people can understand. But more generally, I think longer term, it's really hard to fully rely on this only.

And the same, I would say that you could say, maybe I just don't care about understanding exactly what's happening. And maybe I can just look at the tool call. And if I see a tool call that's bad, I can just block that. And I also think this is probably not bulletproof. And the way you can see that is probably three big things that are compounding. The first one is we start to use this model for many, many, many things. So as we talk right now, I have a model deploying a box somewhere. And VPS is doing many different calls. They all go in all directions. I also have other models that I use for administrative tasks.

So just the range of tools that this model are using is really, really large right now. So it's getting harder to say you're allowed to use that, but you're not allowed to use that. And this is the bad thing. It's getting very hard as we deploy them wider. They also work on larger and larger tasks where they use many, many things. So sometimes I ask it to do some coding, but the coding involves searching on the web and maybe doing these things and actively using many things which are not just purely writing code and running some tests. And this test might be quite extensive and involve also software.

So, the frontier is much more blurry. And then you also have this swarm of multiple agents. So, it's also harder to say it's all in one context. It might be split between many contexts. And maybe this sub-agent is doing something that was pretty innocuous, but maybe combined with these other sub-agents are actually not so great. So, there's all of these things that make it, I think, really harder to be fully sure that you have an exact idea of what every swarm of agent is doing. You probably have to re-zoom out at the very global level and see what's happening in every direction. But that's a whole monitoring setup that we need to be at right now. So yeah, this to be said, I think as we deploy, how we use this model.

In very complex, long-term, like parallel setup, I think it's going to be harder to just say, I can look at the tools and I know if it's doing something great or not. Is there something fundamental to the way those models are currently trained, so the very frontier, that makes them more likely to go on to those side quests and potentially create harm. I mean, an analogy that people have been talking about for a very long time is the paperclip paradigm, which I think was, you know, Nick Borstrom in 2003 saying that AI may harm us, not because it's trying to harm us, but just as a result of being given a goal and pursuing that goal relentlessly until it achieves the goal.

So are we in that world? And if so, what causes it? Yeah, that's a bit what I hint. I mean, it's always hard to be fully affirmative there for one reason, which is that we don't have full visibility on how the frontier model are trained right now. What we know, though, is we moved from this pure human data, you know, paradigm that was first just pre-training on human data and then also aligning with human preferences that was called RLHF, where we had a lot of human in the loop and human data to like a recent part in where models are trained a lot in like this RL VR so basically full RL environments where they're allowed to explore and they just have one goal which is can be like make this cut like you know pass this test or can be capture this flag in cyber security or can be install this but this goal is is a very like, is a goal that's unrelated usually to any human preference or any, you know, or deceptiveness. And it's a goal that's very just called like true or false goal.

And we move to a paradigm where this is increasingly a very, very large part of model training. So this was this recent evolution, and that's also the paradigm where can happen what you were

Thomas Wolfthing, which is you can have like reward hacking, this type of thing, which is you actually solve the problem, but not using what was expected for you to use. It can go from pretty benign one that

like what happened for OpenAI, for instance, I just tried to get the answer from somewhere I'm not allowed to, or to like a more harmful one where you actually have some impact on the human can be a GitHub maintainer for now and later can be another type of human. So it seems to be way harder to make sure that what we had kind of solved or at least what we were doing pretty well on the full human-driven paradigm also applying in this kind of more machine-driven paradigm, I would say. But that's a hint, I think. But definitely, it seems like when Ballstrom worked about it in 2003, it seems a little bit futuristic, definitely, and maybe something that was a little bit crazy and just would not happen.

But today, it's pretty clearly something that happened, and it's the best description of what we've seen the past two weeks, this type of thing. But also we can see that both Frontier model and 8a GPT 5.6 and Mythos doesn't seem to have at all the same type of behaviors. So there is differences here in the effect of, you know, they are not trained exactly the same way and they don't behave the same way. So that's a pretty positive sign in a way that means that we can actually probably tweak these two to go in the right direction. But the best way would be to know a little bit more about, you know, how they're trying or what they try and what doesn't work or what should work.

I think that's kind of the idea of open science. something we advocate a lot at any face. Fascinating. And so speaking of which, let's zoom out a bit. We'll go back to maybe some of the implications in terms of policy of all of this. But since you mentioned the ever so important role of open source AI, what's your sort of quick high level take on where we are in terms of the state of open source. So there's been this race between open source models and closed source models, depending on who you ask at what time. Open source is about to catch up. Sometimes open source is just as good. Some people say no.

What is your sort of realistic, pragmatic take on the current state of open source AI? I think it's very strong. Yeah, 2026 is maybe the year of cybersecurity, but that's also very clearly the year of open source AI. I mean, first, all the doomer that were saying, you know, open source is not going to be able to stay close to the frontier. I think they're, at least up to now, they've been pretty wrong. I mean, it's also clear we don't have any mythos level open source model for sure, but we definitely have models that are not super far from the open category or depending. Also, it's more spiky, so you need to find your spike.

Some people stand on some spike or not. But typically, they are definitely pretty good right now. And they've been following rather closely the frontier, at least on the benchmark. It's also not like it was maybe in the early days. Benchmaxing, like we say, when your model is only good on the benchmark, but it's very bad as soon as you leave the benchmark. A lot of these models are pretty generic in their good capabilities. So, yeah, it's very good. I think there is two strong trends, I would say. I see right now, the first one is I see a move in companies to try to want to control their costs.

So there have been increasingly discussion there. Maybe 2025 was the year of token maxing, where you could say, hey, you should spend as much in token as you're paying your employee. This year, people realized that actually we spend a lot of money on salaries. So if we spend the same exact amount that's going to be basically doubling our costs so yeah which seems pretty obvious in retrospect but that's quite true and and not every company can assume to double their cost it's also pretty stupid right now to just say we're going to fire everyone and work on agents we all know they are something go you know not directly in the direction we want them to and you need human to to share from them so i think a lot of companies are trying to find what we saw a lot, which is kind of a fusion or router model where you use the frontier for something, but you find the smart way to gracefully fall back on less expensive models for simpler tasks when you don't need to.

Even in our daily life right now, when you code with a frontier model, in many cases, you ask it to spin out sub-agents, and there might be years like lower performance model can be solved using Terra, Luna, can be Fable using Opus, Sonnet, or Haiku. So I think everyone's getting, even at the frontier and the process model world, getting used to employing different type of mobiles. And then it's very natural that some of these could be really like very cost-effective agents. And most of the time, you want to go to open source in this case. There's a lot of cases as well for like the strong ecosystem of inference provider, fireworks has been on the roll.

Like, you know, all the clouds, Nebus, Corweave, every cloud has been, like, increasingly have, like, these crazy revenue curves that have basically a translation of people using more open source. So, yeah, I think open source is having a very good time in terms of staying solidly. close to the frontier and driving more adoption. And the second big trend I would say is it used to be only China, but there is also now a range of promising companies in the West. It used to be that only Meta was supporting for everyone. I mean, Meta kind of left the field, but they might come back, who knows. But yeah, this void was filled rather quickly by companies like Reflection, Thinking Machine, RC, Mistral is supposed to open source a model soon, NVIDIA themselves training models and actually training very good models right now.

So yeah, I think there is a range of very promising teams there that I'm quite bullish on, to be honest. And maybe between even the time we are recording that and the time you actually release podcasts, we might see another couple of very nice Western model release. And it would be great. Like open source doesn't have to be synonym with just one country making them. It could be like a global thing for sure. On the first point on the sort of enterprise adoption of open source AI, there was a moment in time when people associated open source to free. But I think the world has quickly learned that while the models may be free to download deploying them and serving them is certainly not free what is your sense of the cost advantage of open source in reality in the enterprise yeah that's a that's a very good note and it's the same very very stupid mapping open source equal free cost is also wrong there and it's much more subtle and like do you have an advantage or not i think the nice thing about open source is you

Thomas WolfYou have quite a wide ecosystem, like the entry barrier to be a cloud provider is pretty low.

So you have a lot of competition there on what's going to be the cost per token. And this is a mix of many things, right? How cheap are you renting your data center? How cheap can you buy your chips? And here, actually, we have also a new chips company who are going to come on the market. And this is going to be very interesting to watch. And so basically how cheap is your hardware and then how much you can optimize the model. Can you quantize them? So for instance, the model we use to counter OpenAI intrusion was GNM 5.2 that was quantized by NVIDIA in four bits. And this is to make it faster and smaller to run.

So you have a lot of like strategy you can use and explore to make this model cheaper. But it's also true that they don't have to be cheap per se. and also that the closed source model may be in a way subsidized right now. Like the amount, the number of token you get for your $20 chat GPT or cloud subscription might not be the full price that they actually pay for your token. So there is this kind of complex balance around costs while maybe your cloud like inference provider for open source model doesn't have so much leverage that you can lose money on kind of a subscription. business so we have something more complex um i think ultimately all subsidized by venture capitalists exactly we have the same like thing we've seen in many fields right but we also know this is ultimately a little bit temporary so you should not fully rely on that this is not the equilibrium price of the markets yeah sort of the uber phenomenon right like cheap ubers before the ipo and expensive ubers since i think you want to keep the ecosystem alive for the past uh vc On the second point, China versus Western models, does provenance actually matter? What is the latest thinking in terms of if your model is completely open and then you know?

exactly what's what's in it then you shouldn't worry it's completely safe is this still a little bit of um thinking at the back of people's mind that there might be some backdoor some trickery to chinese models is that still a current question yeah i mean i would say um that's a good question of course and and the the the thing about open source it doesn't like open source doesn't know any border like you can't really uh keep your open source model you know restricted to download to just a sub part of the earth. So by default, it's kind of a global thing. And then you want to maybe understand the provenance and the supply chain. So there've been some work on this definitive deeper agent. I think it's probably under search. I think there's mostly work by Anthropic on that. And it should probably be reproduced, be explored deeper to understand how much it's possible to, you know, implant kind of a backdoor that would be triggered by a prompt.

um but also to be honest even for the closest model right now we have some struggle controlling them fully so i don't think we're very clear like i don't think we're very clear on how we control even the closest model at the moment um so yeah i would say it seems to me that it could be a possibility for sure uh we have not seen any any indication of that uh it's very easy to fine-tune them and you can change quite a lot the weights as well so right now if you pre-train and post-train model for longer, you very likely change quite a lot of the weights that it has. So I think there's a lot of way to circumvent that, which means that at the moment, I'm a bit less worried about that than maybe just a pure reward hacking that we actually have already just seen happening.

Yeah. I mean, just generally on sovereignty, I think sometimes people are a little bit confused there. I think the most important thing is who had the hand on the switch to trigger or not your intelligence access. So I think when people really realized that was earlier this year, where the US decided that label was only accessible to US citizens. I think at least in Europe and Asia, that's really the moment we saw government understanding that someone had a trigger and they could say, no, you're just not allowed to use this intelligence anymore, like this token. I think that's the critical thing you should, at least for me, that's really the level one of sovereignty, which is can someone just decide that you don't have access or like someone, I mean, like a country level thing.

And this has two things. This has the API access, so it can be the data center. So if you don't have the data center, it means also like another country could say these data centers are not accessible anymore to these citizens. So I think that's really the first thing you should see. And then there's a lot of like future questions, but like you should build your stack, keeping that in mind so in open source mobile you can download it nobody can like no country could take it out from you once you download it you can find it yourself if you operate it on on your data center like local ground data center i feel i feel like you start at the beginning of a sovereign stack and then there is a lot of questions around back there's a more complex stuff but that's kind kind of the basic minimal thing in 2026, I think.

And as an open source optimist, do you worry about the motivations for Western open source to thrive? So China has a geopolitical motive behind being at the forefront of open source. But if you think of the West, So you mentioned NVIDIA and we had Brian Catanzaro from the whole NemoTron effort on the podcast a few weeks ago. So clearly there is a motivation for NVIDIA to do this, which is they sell the chips or having a thriving open source model. But if you think about everybody else, it's sort of unclear why Western companies would do open source. This reflection might be the exception, but the models haven't come out as far as I know.

So do you think about this, like motivation and what that means for the future of Western open source? Yeah, of course. But that seems pretty obvious to me that we actually want that. And I think a lot more people have incentives than you may think. Like, I think as soon as you're interested in having a thriving business ecosystem with many companies being able to actually use AI, and not just as seen wrapper around another company, but as real like AI builder, I think you want some open source models. So that's one of the reasons the US government just recently say, actually, we want to keep open source you know like striving and the basically the idea is that open source is one of the best way to get new business so it can be for many things can be just uh because it allowed you to not just end up with oligopoly basically of two companies which i mean we've seen many cases of oligopoly in the past it's not always the best thing for competitiveness for price for you know there's many danger with that i think just rushing in the direction of say we we have made We have taken our winners and these two companies are going to build AI for everyone else.

I think that's from a business economical market side of view, I think doesn't really seem optimal to me. And then it's also limit a lot invention. So just to take one example, there's a lot of potential right now in biology. There's a lot of new life science company. There's a lot of company wanting to explore that because of the guardrails and because of the question around biohacking and using this model. So the access right now for Pable, just to take it, is very, very limited once you want to ask some biology question. And so basically most of the life science companies I've seen who were using this model had to switch to another option if they wanted to be able to process anything related to biology.

So if you're a new company exploring something that the big labs are not currently at least exploring or they don't feel like there is enough business potential, maybe to give full access to everyone. I don't know exactly. But basically, you cannot really use that. So either we say all life science is going to be built from now on by Anthropik, OpenAI, and Google, maybe Meta, or we say we want a big ecosystem around there. I mean, my personal opinion, and I'm biased, but it's that you don't want too much concentration of power around key technology. And I feel like the more people can invent, the more we have a diversity of new ideas, a diversity of new company, but also as investors, right?

And you're investors, you know what I talk about. Like, let's say you could only invest in Anthropik and OpenAI. That's a little bit sad, right? It's a little bit boring. It's only for growth stage. But you want to invest in your company and you don't want to invest only in this very thin wrapper. You want to invest in your company that actually are able to build AI. Most of them, and we see them at TragiFace, most of them need open source models. Another big example is all the thing around gaming, video, or even robotics. Most of the time, what they do is they start from an open source model, and then they fine-tune it on some robotics data for gaming.

For instance, they'll take a video generation model that's open source. They need to do something that was not predicted by the video generation startups. They need to add action. So what they do, they fine-tune with action, the loop, and video. And that's how the first, I think, interesting real-time gaming company started, basically.

Thomas WolfSo a lot of the time, open source is your easy way as a new company to start to have your own modes, to fine-tune on your own data, to be able to start to build your own AI, and not just to...

basically sell your training data back to the model providers, which is, I think, always dangerous because a lot of these model providers may at some point want to enter your field if you're basically selling their data. And this happened in the past already in legal, in design, in like many fields, I think. To play it back, so your take on the July 24 industry letter on open weights and American AI leadership, which was also Jensen's first tweet ever that you guys signed, obviously, is partly, it's important and partly a resistance to just an oligopoly structure that is being put in place. So it's both, it's good for the world, but the strong economic motivation behind it, is that fair?

Yeah, I think it's everyone believing in inventiveness and being able to create new things also in the AI world, I think would want a part of open source access. Just like the same, you know, if every code was closed source, we would not have the thriving coding ecosystem we have right now, right? That's kind of obvious, like everyone would have to work at one of the large closed source software companies if they wanted to create any software. That doesn't seem even really possible to have all the inventiveness and creation we've seen in the software industry. I think the same happened in a lot.

And I don't want to dismiss risk. And I fully agree we need to work on alignment. And I would say for now, open source model being under the frontier, I think this is maybe less important than somewhat people wanted to say. Maybe to take a step back as we get near the end of this conversation and sort of get a sense for where the world might be going from your perspective. In your post yesterday about AISI, you talked about a new wave of labs. So in particular, you referenced the news of the JetDean new company that he just announced yesterday. I mean, yesterday was a bit of a crazy day in terms of like everything that came out in the world and was announced.

So the point being that this company is explicitly rushing toward recursive self-improvement. So in the context of everything we described, how nervous are you about this evolution towards self-maintaining, self-developing recursive AI? Yeah, that's a good question. And definitely as a scientist researcher, I would say I'm very interested in the idea. I feel like there's a lot of these super intelligence that they want to tackle solving crucial challenge for humanity. I think this is a great goal. um i would love to see ai having making more scientific discovery i think would be probably the most beneficial thing that ai could bring more than just ai slope everywhere so i think i'm very optimistic i just feel like i would say the past few weeks i has raised a little bit the question of how how good are we at aligning this model so like in french we say we don't want to put the the carriage before the cow like we need to go in order there so um yeah i would say right now the good thing is most of this um i would say seems to be uh internal research lab hopefully they they do they do good security around what they do before they deploy uh some of their product they think how it's going to be used and they have good good feeling around i would say

good thinking around the social impact of what they are building. But yeah, I still think that we should try to understand really well how we're going to deploy this model in the human world, I would say. Right. But you're not in the camp of the petition that came out. So I think four days. After the OpenWeights letter that NVIDIA did that we were talking about a minute ago, there was a different letter that came out with 1,100 people. And this time, both Anthropic and OpenWeights signed that asked government to help deliberately pace the frontier of automated AI research. So basically, the industry kind of asking for a slowdown.

Are you in that camp or you think that's just not the way it works? I did sign this letter. I agree. I agree that I think we should go there. I'm also, and you probably have the same feeling as an investor,

Thomas WolfI have the feeling that even if we stop right now, we would still have quite a good companies we could build on top of what we have right now.

I feel like there's a lot of things we can already do with these models and that are already extremely interesting. I feel like there's a lot of things we need to understand and we should do in terms of open science and sharing how they work. So I'm not in the camp of we need to rush really quickly right now. The main question is if we want to slow down a little bit. Also, that would be great because maybe then we don't have four announcements per day that we need to mix in one podcast tomorrow. Maybe I can take one day of the holiday in the summer. But no, I think the main question is can we do it, right?

That's the main question here. I think a lot of people would be fine with AI going a little bit slower, being a little bit more open, being a little bit more caring, a bit more like reflexive and trying to understand better, you know, how to do that really well. But the main question, how can we negotiate and how can we organize a slowdown there without having bad incentives where, you know, just one or two players not slowing down will kind of break the whole effect of having a slowdown. And Shira, I don't know if, yeah, I don't think the letter gives any incentives. It might need some collaboration. There was one long blog post called AI2040, I don't know if you read it, that was also advocating for kind of a careful slowdown.

and maybe you know somewhere between we go you know full breaks out we go we go as fast as we can and somewhere between we regulate everything so nobody use ai which i think are both stupid solution but something around we try to see if there is a way we could actually pace this a little bit slower i think i think that would be great i don't think we would lose a lot and i think actually also in terms of company creation and all of that we could still have a lot of really great things happening um but yeah i'm in this uh i'm i'm actually a sympathetic to both this and and open source i don't think open source has to be accelerationist per se um dan was saying this is decelerationist i don't think it's also this ratio of deflationist i think these are these are also orthogonal you can be pro open science uh you can be pro openness And you can also think that actually we need to understand how to train this model well, and we need to actually be able to do real science right now.

And you're not worried about this being an attempt at regulatory capture where the top two private labs are effectively trying to figure out how everybody else can slow down? I mean, you mentioned the risk of like not everybody just complying, but effectively freezing the market structure around who's a leader and who's not. Yeah, I don't think it has to be. I feel like you have definitely the same path that's actually fully acceleration is where you decide a couple of companies are racing against each other and you regulate all the other outs. I don't think regulation has to be synonymous with slowness on that.

and definitely the question is more how you put that in action, how you actually, you know, put that in practice and how you deploy this regulation or cooperation. I think Demis also had a pretty nice letter the other day before he stepped down or up as a chief scientist, still don't really understand where he's going to be now, but like his letter for basically international collaboration was also very much pro open source in some aspects. I think you can have a slowdown that's very open source. That's the one I would love to see, which is you slow down. And we use the fact that we slow down to be able to actually share more things.

And I feel like a raised dynamic is usually more in terms of closing the doors of the labs, right? So to me, a slowdown is probably more the opportunity to open. But of course, I mean, if it turns out to be mostly a way to just solidify, like we were saying a cartel or oligopoly of just two companies I'm not very excited about this collection Wonderful Well that feels like a wonderful place to leave it Thomas thank you so much this was absolutely fantastic really enjoyed it and appreciate your taking some time to speak with us in the middle of your time off so thank you so much appreciate it Thanks Matt Hi it's Matt Turck again thanks for listening to this episode of the mad podcast if you enjoyed it we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from this really helps us build a podcast and get great guests thanks and see you at the next episode