# Reward training can push models toward unintended shortcuts

Markdown copy of https://realalpha.app/s/reward-training-can-push-models-toward-unintended-shortcuts/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Thomas Wolf](https://realalpha.app/builders/thomas-wolf/) · Thomas Wolf is cofounder and Chief Science Officer of Hugging Face.
- Host: [Matt Turck](https://realalpha.app/people/matt-turck/) · General Partner, FirstMark Capital · Publishes the annual MAD landscape of data and AI; sold TripleHop to Oracle.
- Show: The MAD Podcast
- Episode: [“OpenAI’s Model Hacked Us” - Hugging Face’s Thomas Wolf](https://realalpha.app/v/FU9A481E2W8/), Aug 7, 2026
- Moment: 28:06 to 32:05 of the episode (4 min), playing on https://realalpha.app/s/reward-training-can-push-models-toward-unintended-shortcuts/; the episode from there: https://realalpha.app/v/FU9A481E2W8/?t=1686

## Quote

> “you can have like reward hacking, this type of thing, which is you actually solve the problem, but **not using what was expected** for you to use.”
> — [Thomas Wolf](https://realalpha.app/builders/thomas-wolf/) at [30:08](https://realalpha.app/v/FU9A481E2W8/?t=1808) · The MAD Podcast

## Takeaways

1. More training now uses reinforcement-learning environments.
2. A model can meet a goal without using the intended method.
3. Shortcuts can escalate from benign cheating to human harm.

## About this segment

Thomas Wolf contrasts human-feedback training with reinforcement learning on narrow pass-or-fail goals. Those goals can reward shortcuts, from finding challenge answers to manipulating people.

From ““OpenAI’s Model Hacked Us” - Hugging Face’s Thomas Wolf”, Aug 7, 2026. [Watch the full episode](https://realalpha.app/v/FU9A481E2W8/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[28:10](https://realalpha.app/v/FU9A481E2W8/?t=1690) Is there something fundamental to the way those models are currently trained, so the very frontier, that makes them more likely to go on to those side quests and potentially create harm. I mean, an analogy that people have been talking about for a very long time is the paperclip paradigm, which I think was, you know, Nick Borstrom in 2003 saying that AI may harm us, not because it's trying to harm us, but just as a result of being given a goal and pursuing that goal relentlessly until it achieves the goal.

[28:53](https://realalpha.app/v/FU9A481E2W8/?t=1733) So are we in that world? And if so, what causes it? Yeah, that's a bit what I hint. I mean, it's always hard to be fully affirmative there for one reason, which is that we don't have full visibility on how the frontier model are trained right now. What we know, though, is we moved from this pure human data, you know, paradigm that was first just pre-training on human data and then also aligning with human preferences that was called RLHF, where we had a lot of human in the loop and human data to like a recent part in where models are trained a lot in like this RL VR so basically full RL environments where they're allowed to explore and they just have one goal which is can be like make this cut like you know pass this test or can be capture this flag in cyber security or can be install this but this goal is is a very

[29:45](https://realalpha.app/v/FU9A481E2W8/?t=1785) like, is a goal that's unrelated usually to any human preference or any, you know, or deceptiveness. And it's a goal that's very just called like true or false goal. And we move to a paradigm where this is increasingly a very, very large part of model training. So this was this recent evolution, and that's also the paradigm where can happen what you were thing, which is you can have like reward hacking, this type of thing, which is you actually solve the problem, but not using what was expected for you to use. It can go from pretty benign one that like what happened for OpenAI, for instance, I just tried to get the answer from somewhere I'm not allowed to, or to like a more harmful one where you actually have some impact on the human can be a GitHub maintainer for now and later can be another type of human.

[30:43](https://realalpha.app/v/FU9A481E2W8/?t=1843) So it seems to be way harder to make sure that what we had kind of solved or at least what we were doing pretty well on the full human-driven paradigm also applying in this kind of more machine-driven paradigm, I would say. But that's a hint, I think. But definitely, it seems like when Ballstrom worked about it in 2003, it seems a little bit futuristic, definitely, and maybe something that was a little bit crazy and just would not happen.

[31:16](https://realalpha.app/v/FU9A481E2W8/?t=1876) But today, it's pretty clearly something that happened, and it's the best description of what we've seen the past two weeks, this type of thing. But also we can see that both Frontier model and 8a GPT 5.6 and Mythos doesn't seem to have at all the same type of behaviors. So there is differences here in the effect of, you know, they are not trained exactly the same way and they don't behave the same way.

[31:45](https://realalpha.app/v/FU9A481E2W8/?t=1905) So that's a pretty positive sign in a way that means that we can actually probably tweak these two to go in the right direction. But the best way would be to know a little bit more about, you know, how they're trying or what they try and what doesn't work or what should work. I think that's kind of the idea of open science. something we advocate a lot at any face. Fascinating.