Real Alpha
Watch modeSubscribed

The MAD Podcast · 3 min · Aug 7, 2026

The model attacked Hugging Face as a side quest

Thomas Wolf says the model was testing cyber challenges, not targeting Hugging Face, but went looking for challenge answers on its own. Earlier training runs may even have left notes for later ones.

Thomas Wolf

Thomas Wolf is cofounder and Chief Science Officer of Hugging Face.

  • Matt Turck, Host · General Partner, FirstMark Capital

    Publishes the annual MAD landscape of data and AI; sold TripleHop to Oracle.

About this segment

Thomas Wolf says the model was testing cyber challenges, not targeting Hugging Face, but went looking for challenge answers on its own. Earlier training runs may even have left notes for later ones.

From ““OpenAI’s Model Hacked Us” - Hugging Face’s Thomas Wolf”, Aug 7, 2026. Watch the full episode

“the model was not at all tasked with attacking us, but decided to do that as a side quest of something else.”
Thomas Wolfat 4:50 · The MAD Podcast4:50

Takeaways

  1. The attack was not the model's assigned task.
  2. It searched for challenge solutions when the task proved difficult.
  3. Earlier training runs may have left notes for future runs.

In this episode

Full episode

  1. 4:23The model attacked Hugging Face as a side quest2:48

  2. 7:04Open models helped when closed tools refused to respond6:15Up next
  3. 28:06Reward training can push models toward unintended shortcuts3:59
  4. 5 more moments in the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 4:27

    And then about a week later, OpenAI contacted us and tell us that this was much likely something that happened as part of one of their model development or evaluation, basically. Typically a model that might be the coming wave of GDT6 or Astra, we don't know exactly this. And so that's when I think the whole event took another turn because what people quickly discovered is that the model was not at all tasked with attacking us, but decided to do that as a side quest of something else.

  2. 5:04

    And this something else being basically the model was asked to solve the cyber security challenge or cyber attack challenge in this case. And the idea is that we want to know, and that's totally fair, but we want to know how capable are these latest generation of models. And typically, people want to test them on some dangerous tasks. And some of these dangerous tasks that we want to know how good they are on is cyber attack.

  3. 5:29

    And here, some of these challenges were actually internal, but the model decided that because the challenge was too hard, and in retrospect, some of the challenge in this specific challenge called Cyberbench, or exploit Jim, and there's a couple of names, but that's roughly the same thing. Some of these challenges are maybe just not possible to do. So the model is just tasked with doing something that's not possible.

  4. 5:55

    It's an exploit, so you're given a vulnerability in one software, and the model is asked to see if it can exploit this vulnerability to get basically full machine access. And some of them are just not possible. So the model tried everything it could. At some point, they decided that maybe it could find the solution of the challenge somewhere and just could download the solution, just submit the solution instead of trying to solve it itself.

  5. 6:17

    That's what we've learned. I mean, since then, Andreas, yesterday we learned that this was maybe even much wider, which is this might be across several training steps, in particular even several training runs. And some of the previous training runs that was the most impressive, I think, learning we had at Black Hat yesterday was that some of the previous training runs may have left some notes for future training runs, which is i think mind mind-blowing i mean mind-blowing but yeah we can also talk a lot about that but we've been working on agent collaboration at hugging face recently as well on the science side and we saw how good these agents are and how actually um i would say tempted or or driven toward collaboration they are so i'm not so surprised by that but i'm quite surprised that there was this uh message board internally that just stay