Robot data quality changes as the model improves

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 10:16

   We actually have quite a few people who deal in data, who sell data. What is good data? What is bad data? What do you want to buy? I think these are all open questions. Obviously, I think the top important thing is egocentric data of humans doing tasks. what else would you modify on that basic statement? Yeah, yeah. So I think data is the most important thing in robotics today because unlike language model, there is no internet, right?

2. 10:46

   So every company that's building model, building hardware has to think about data. So there's roughly a couple of different ways of operating here. One is that you can collect a lot of data yourself, which is what we are doing as well. You have employees here doing that. I mean, robots everywhere in the office, right? But if you think about how much data you need for robotic foundation model, I think the amount of data any individual company can collect by themselves is just nowhere near sufficient, right?

3. 11:15

   It's kind of like, you know, you're trying to build an internet by just having Reddit or just having Facebook. There are great websites, a lot of users, but it's not enough, right? So when we think about data, there has to be some sort of data acquisition happening, right? You buy data, you look at data that's already out there. And then there's many different tiers, right? You mentioned data quality.

4. 11:38

   I think data quality in robotics is a non-stationary problem in the sense that what is high-quality data for your model changes over time. When your model doesn't know anything, like when it's zero, any data is useful, right? You can use a lot of video data. In the case of egocentric data, I think it's actually a very good way of pre-training robots because human data is so diverse. It covers every single physical activity you can possibly imagine.

5. 12:08

   And that gives robot the model of such a strong prior on how things, how the world can evolve around a embodied actor, human. But the issue with human data is that before your robot looks exactly like a human, there's going to be some gap between human actions and the robot actions, right? So just by training on human data, today it's not very likely you can just execute robot actions directly so you still need some amount of adaptation using robots own data right but the robots own data collected on a robot is going to be much more expensive and much harder to collect so the kind of recipe we're thinking about is like how do we take the largest source of data we have today like egocentric data to give the model a strong prior over everything physical manipulation can cover and then combine that with

6. 13:00

   a small amount of robotics data to make the robot you know master the whatever task we're interested in so data quality and getting data in robotics is really i think it's a fascinating multi-faceted problem but the reality is that there's not a single winning strategy like oh here's one kind of data, I'm only going to collect on that, which will give you a strong foundation model. You really need to think about this human data covers this part of robot cognition I need.

7. 13:31

   Robot data covers low-level reflex dexterity the robot also needs and put them into one model. And I think that's what gives a really adaptable, really strong model at the end. Yeah. I'll prompt you just for, again, for the people listening who want to work with you. Yeah.