# Game data can pretrain robots, but real-world tuning still matters

Markdown copy of https://realalpha.app/s/game-data-can-pretrain-robots-but-real-world-tuning-still/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Pim de Witte](https://realalpha.app/builders/pim-de-witte/) · Pim de Witte is CEO of General Intuition. · Co-founded General Intuition, a frontier AI lab for spatial-temporal reasoning, and previously co-founded gaming platform Medal.
- Host: Eric Newcomer
- Show: Newcomer Pod
- Episode: [The Startup Building World Models From a Billion Video Game Clips](https://realalpha.app/v/zsgnonEeAiQ/), Aug 3, 2026
- Moment: 17:04 to 25:34 of the episode (9 min), playing on https://realalpha.app/s/game-data-can-pretrain-robots-but-real-world-tuning-still/; the episode from there: https://realalpha.app/v/zsgnonEeAiQ/?t=1024

## Quote

> “we have models that are pre-trained on mostly video game data that **transfer over to novel robotics tasks** in minutes worth of fine-tuning data.”
> — [Pim de Witte](https://realalpha.app/builders/pim-de-witte/) at [18:32](https://realalpha.app/v/zsgnonEeAiQ/?t=1112) · Newcomer Pod

## Takeaways

1. Game clips pair visual frames with the actions players take.
2. Models transfer to new robot tasks with minutes of fine-tuning.
3. Negative examples help evaluate and align models.

## About this segment

De Witte explains how game clips with recorded actions help models learn spatial reasoning and transfer to robotics. The games provide broad pretraining; real-world data and safety checks are still essential.

From “The Startup Building World Models From a Billion Video Game Clips”, Aug 3, 2026. [Watch the full episode](https://realalpha.app/v/zsgnonEeAiQ/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[17:13](https://realalpha.app/v/zsgnonEeAiQ/?t=1033) foundation models. So I want to, yeah, I want to engage with that head on. Like, is that, true that video games will sort of get you to the real world i think you know we have in in you know self-driving cars there was that you know training in like grand theft auto is sort of like simulation but then self-driving cars clearly have had to do a ton of real world stuff i mean the thing i most get about what you're trying to do is that by understanding controller inputs controller we can use a controller for robots yeah that is pretty one-to-one and so getting really good at the types of inputs really constrains which is a far too open-ended problem so put we can talk about that more but like putting aside the actual like controller piece and speaking speaking um sticking to like the modeling of the world how to what extent do

[18:04](https://realalpha.app/v/zsgnonEeAiQ/?t=1084) you think modeling out video game worlds is really going to translate to the real world when they are pretty different yeah um so we've gotten there uh we we have um a physical robots currently functioning on our foundation models uh and they work incredibly well there's drawbacks to both real world data collection and game data collection the end of the day you just need both um our bet is not that we are going to solve all robotics on video games data it's just that like the same reason the same way that lms are currently hill climbing scientific problems you just need a lot less data of the new type in order to generalize to the next task.

[18:40](https://realalpha.app/v/zsgnonEeAiQ/?t=1120) So we have models that are pre-trained on mostly video game data that transfer over to novel robotics tasks in minutes worth of fine-tuning data. And so I'll give you some of the benefits and I'll give you some of the drawbacks. The benefits is, for example, there's so much variation in different lighting conditions or unexpected events unfolding in video games. like we have as many uh car crashes in video games every day that actually happen in the united states for example it's like in fact we also have uh you're saying there are more car crashes in the video games than in the real world we've counted the amount of clips uploaded with car crash labels surpasses the amount of physical car crashes that happen in the united states every day uh on metal and so um we also have more people at any given time controlling

[19:23](https://realalpha.app/v/zsgnonEeAiQ/?t=1163) with steering wheels than waymo has cars on the road um so for pre-training this is all really important uh you still need to fine tune into specific behaviors but for pre-training this is very very helpful to have that level of skill and then for instance in the real world it's really really hard to collect data under tons of different lighting conditions where if you if you transfer it over to a new environment that the model doesn't get mixed up by just pixels looking very very different from uh the way it was before and so games for instance because of the amount of variance and expressivity um i would argue for example if i were to explain to you the concept of a bridge, and I would show you every single bridge.

[20:00](https://realalpha.app/v/zsgnonEeAiQ/?t=1200) in New York City versus I would show you every single way that somebody has laid out a bridge in different type of video games. The video games, you would actually have a deeper understanding of the essentials of a bridge than if you actually looked at a bridge in New York City, because conceptually, you've seen every single way that somebody has tried to express that, which is more likely to then when you encounter a new bridge that you- Are there video games that try to cheat the physics where it's like, it doesn't actually add up if you really looked at it from every angle and did the math?

[20:29](https://realalpha.app/v/zsgnonEeAiQ/?t=1229) this is why fine tuning is important you still need that piece and do you score video games that are more sort of realistic in their physics do they get more weight than ones that aren't the beauty of our approach is that you have to every single game teaches the model a different thing some are useful for their realism some teach very good spatial reasoning some teach really good interface use so there isn't like a one you know the the model recipe and how much of of um of what you use is kind of uh and also like where you use non-game data right for example is kind of what we do every day in terms of cheating physics like maybe i'll try to explain it this way when um when you learn something for a job or even as a kid you tend to simulate out you tend to play out scenarios even though it's not going to be the real scenario um and then when you actually can

[21:27](https://realalpha.app/v/zsgnonEeAiQ/?t=1287) in contrast scenario for real you're more prepared you understand it better i think it's the same so you essentially have a reference frame in the models and therefore the fine tuning just happens a lot faster than um uh than before and a lot of times especially when you're when you're in pixel space you're like i've seen this before it's easy to learn it yeah exactly and so you need to get a few things right like for example right um in different countries you drive on different parts of the road and uh so if you if you have a bunch of fine-tuning data from the uk you have a bunch or UK drivers, they might drive very differently from the American one. So if you're working on a self-driving use case, and by the way, like metal has this separation, right? Like we can just tell. And so it's more so that we view it as pre-training. I think that the specific

[22:12](https://realalpha.app/v/zsgnonEeAiQ/?t=1332) implementation behaviors you're describing is more what happened at the post-training and fine-tuning stages of the models. Do you worry, this is going to be an out there question, but an AI out there questions feel uh justified I mean you know Grand Theft Auto or whatever you maul down people as sort of like part of the fun of the game yeah we're building these systems to be sort of all purpose like do you worry that like core to the system you're building is a machine that's been trained that like mauling down pedestrians is like a good fun thing to do yeah yeah uh so the answer is it's incredibly important at pre-training that the model understands cause and effect, including negative cause and effect, because that's actually how you align them. So as for, so for instance, when we were approached for acquisition by all the labs,

[22:58](https://realalpha.app/v/zsgnonEeAiQ/?t=1378) a lot of it was due to them needing our data for alignment problems, because you need a lot of negative examples of what not to do in order to actually evaluate models on how they're going to do in the real world. So the data having certain biases is a double-edged sword. You also see, you needed to solve it. You also want it in the pre-training data so that it understands how to predict the impacts of its future negative actions, right?

[23:23](https://realalpha.app/v/zsgnonEeAiQ/?t=1403) And so there is, so this is also, for instance, why world models are so important, right? So our core business is general agents that just take in pixels and predict actions. But world models is how you can really steer the behaviors and the environments towards the specific types of behaviors that you want in fine-tuning versus at pre-training, it's less important. But how do we get models to hold sacred the real world in a way that they don't – the digital world, right?

[23:52](https://realalpha.app/v/zsgnonEeAiQ/?t=1432) I mean I guess there's this worry that it's like, oh, they're all worlds. They're all sort of simulated. There isn't this like bright line between – This type of world and that type of world, and therefore, the very different moral worlds of the digital world and the real world aren't obvious to these machines. Is that your business? Where does that occur? For sure. We take it super seriously. We look at what happened with Cruise, for example, and we viewed this as a zero-tolerance problem.

[24:21](https://realalpha.app/v/zsgnonEeAiQ/?t=1461) Cruise basically decimated almost as a company by having this terrible accident that they didn't handle very well. Exactly. And so you have to be on top of this stuff. Our first set of models, so we're very, we work with the partners that implement these models to understand their safety guardrails. And in a lot of cases, what you do is you use our models to predict actions, but then you pair it with like the more deterministic or sensor based systems like LiDAR to add like rule based systems on top of that, that might invalidate some of those actions if they do come too close or if they might put something at harm.

[24:58](https://realalpha.app/v/zsgnonEeAiQ/?t=1498) And then we also have lots of evaluations and benchmarks to understand exactly how our models might perform in these types of behaviors. And again, how do you do this? You actually put the model in a situation where something violent happens and you just check whether the model predicts the same actions as the human. Right. And if it does, that's a bad outcome. Right. So the data having or including negative events is actually critical to to solve.

[25:23](https://realalpha.app/v/zsgnonEeAiQ/?t=1523) um i actually anticipate that we'll open up this data to a lot more labs uh because it's just the