Data strategy matters more than the world-model label

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 33:26

   So maybe, you know, speaking of YAN, probably the, I was going to say easiest, but certainly one of the hottest areas of, I would say, company formation and hence company, I would say investments have been world models, right? World models as applied to robotics, world action models. What's your, like, how does, and, you know, again, And world models is a small term with like at least like, you know, 20 different definitions of how kind of people end up like, you know, applying it to the thing that they're doing.

2. 33:57

   But how do you think about almost that kind of like strand, if you will, or that kind of like, you know, node of kind of the research tree and how it intersects with how you think about the world at like, you know, physical intelligence? I'll give a little bit of a roundabout answer because it's not that there's anything inherently good or bad about world models. it's that the most important thing for building foundation models paradoxically is not the model itself it's the data and uh if you want a foundation model that really works especially in a new area um the most important thing to figure out is what should the data be where is it going to come from the long-term trajectory about how the model will get better and better meaning where the additional data will come from and you kind of have to subordinate the other

3. 34:44

   choices to that. So I talked in the beginning when you asked me about Pi Zero, about why it's based on LLM and diffusion action generation mechanism. I don't think those choices are fundamental. I think the fundamental thing is figuring out the trajectory for data. And I think that trajectory is going to be a bit complex. Like it's something where perhaps we start off with a lot of data that is produced by humans, eventually move towards more like intervention style data, eventually more autonomous data. And whether the thing that we train on that data is a video model, a world model, an LLM, a diffusion-based action generation mechanism, is sort of a downstream choice. Now, my own hypothesis is that actually the ultimate models will do all of those things. That they will have a world model component because that is how you do

4. 35:29

   reasoning that is below the semantic level of representation. So if you want to cook a recipe, maybe you think about it semantically. You say, oh, I'm going to add the vegetables. I'm going to add the salt. If you are rock climbing, you're not thinking, I'm going to grab the stone that is to the right of that thing over there. You're thinking of a spatial concept. And you probably need a robotic foundation model to be able to imagine things visually. And that is basically a world model. But again, these choices are downstream. The big question is how you're going to get the data that will enable all this to function.

5. 36:00

   function and that's why i think that you know some of the world model focus is maybe a little bit distracting in that it kind of shifts more of the focus to what kind of model you have away from the most important question every almost successive era of um company that comes out of