General-purpose robots need diverse data

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 31:13

   If you had to think forward a few years and you guys are not developing all the like models or whatever for your robots. And there's kind of like this almost mod store where people can create things and create like new actions or tasks that a robot can do. And then someone can just like download it onto their Neo and suddenly a new skill is unlocked. How do you like do that? i think it's going to be essentially exactly the same as you see in large models today so there will be a few dominant models just because they have the best data and evals there's like this not so well kept secret in ai that there are no secrets in ai it's all just data and evals like why are the best models the best models is because Because these companies have the best data.

2. 31:58

   They were early, had huge... User bases, they got enormous amount and still get like an enormous amount of data. And since they have so much user data, they are able to train better models. And when you have better models, you get better user data because you get more users and it's kind of like the self-enforcing flywheel. There are other models where you can kind of like specialize. You can download an open source model, you could try to like do some fine tuning on it to like get it better on your specific task. But it has to be a very hard, very niche task before this makes more sense than just using data.

3. 32:30

   model and i don't think this is going to be different we see the same trends the same scaling laws the same patterns uh it's just robotics has until now been like kind of like a toy problem everyone's been working on extremely small sets of data and they've been working mostly on like robotics data where it's like oh i gathered 4 000 hours of like these specific behaviors with my robot and like this is a very big data set no it's not like 4 000 hours is nothing even like a few hundred thousand hours is nothing and if you see the models that people train now of like equipping people with sensors right and then like having that happen at scale and then they're saying like oh we trained this foundation model at 200,000 hours of data which is huge compared to what used to be in robotics it's actually still tiny like general intelligence emerges

4. 33:17

   at like hundreds of millions of hours a day so you're not gonna go gather that data that's why I deeply believe that you need to use these machines as kind of like the bridge between human data and machines. And I do hope that through doing this, the models that we train can operate a lot of robots, not just Neo. And you can generally go in that direction, right? You can take a large, complicated, like very capable model that's trained on this enormous diversity of data, and you can distill it down on a smaller problem.

5. 33:48

   And then that model will be very good at the smaller problem. You can't go the whole way around. you can't take this model from like this smaller problem that only saw that data and distill it up to solve a general problem that doesn't work it's a one-way thing so you kind of have to have the most general system that has the most diverse data and like has experienced the most things and then you can take that intelligence and distill it down to like more specific applications so the question is when that happens right because we're not there yet and in the meantime there will be very good applications for a lot more kind of like specialized models and that's what we're seeing today right if you want to fold laundry then actually if you go and just gather a lot of data on folding laundry with our robot and trade in the VLA that will work better than our

6. 34:29

   robot but it won't be good at doing anything else it it doesn't have a general understanding of how the world works or like it's essentially intelligent it won't exhibit kind of like social behavior and like understanding how to interact with people it'll just fold the laundry and that's kind of like the regime we are in today but it's quite quickly evolving so I think 2027 will look very different. Maybe not 2026. I think 2026 to specialized models will still be better. But I think 2027, you're going to see a shift towards more like general intelligence.

7. 34:59

   So initially, you kind of have to use like teleoperation in some form or another to like, I don't know, bridge edge cases and stuff like that. You need all the data. That's a simple answer. So like you use the web data, which is going to 99 of the data you use simulation synthetic data you use sensors that you put people with to get some of that data you use eccentric video data you're going to use robot data with teleoperation and you're going to use robot data where the robot is just learning by doing uh you use all of it um i think the specific data of like oh i have an intervention and in the sense of like the robot failed and like now i correct the robot um not that important that scale it's extremely important that small scale like if you want to like solve a specific problem with a very small amount of data

8. 35:49

   this data is essential and the technical term here would be that your distribution is very small so like you know how to be within this kind of realm here Then the robot knows what to do. If it gets outside, you need something that tells it how to get back on distribution, so it knows what to do. But if you have a general model that is very capable and diverse, then actually, it's very hard to get out of distribution, and then you don't need this.

9. 36:16

   You can always kind of like find your way back to where you want because you're still in distribution. It has to do with the robustness of the model. first you train on the internet data then you get a little bit of your own data but eventually in order to really get the data that you need to train a super genius robot you just have to have a whole bunch of these um in operation like if you think about if you have 50 000 next year suddenly and they're operating let's say 16 hours a day uh you're getting 16 times 50 000 every single day of new data i assume when you're thinking about like your bottlenecks for scaling your data is like one of the biggest bottlenecks, just we need more robots in the field.

10. 36:55

    Yeah, and they need to be in very diverse environments and very diverse applications. In reality, you're almost never data-bound. You're diversity-bound.