General-purpose robots can bootstrap their own data

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 21:14

   Kind of the common concern slash complaint that you hear about robotics is that the training data is too high dimensional to kind of be able to collect the amount of data that you need in sufficient kind of like, you know, quantity to be able to train these models. And you're already kind of like essentially starting from on the back foot because you don't have kind of, you know, existing repository of data much like, you know, we had for text.

2. 21:42

   So help us kind of shape this, right? Like what are your latest views on either the scarcity or perhaps actually like, you know, relative abundance of data for robotics, right? Like how should one think about kind of that general thought? So I'll give maybe like a frustratingly big picture answer to this. Go from big. Yeah, please. The universe is really big and almost none of the universe is inside computers.

3. 22:09

   Almost none of it is on the Internet. The entirety of the universe, effectively up to a rounding error, is the physical world. Which means that if you want a machine that understands the universe, that understands physical phenomena, it has to be a robot. There's no other way to get that information. You will not find it on the internet because it is not there, because the internet is very small and the universe is very big.

4. 22:32

   Now, if we bring that big picture principle down to something a little bit more concrete, insofar as robots become useful insofar as anybody wants a robot around them well there's probably going to be lots of robots like you know cars are useful and there's lots of cars robots become useful there'll be lots of robots when there are lots of robots they are all collecting data they are trying things succeeding failing recording what's going on around them so even if we step back from the from how big the universe is and just consider you know economically relevant work that somebody wants done, if some fraction of that is done by robots, the data coming from robots, dwarfs everything else.

5. 23:09

   Like it's not even a close call, it's just enormous. Now you can backpedal from that even more, you could say, well, okay, maybe they're not so useful. There's billions of them, but there's only nearly thousands. And you can do that calculation, you still end up with robots at any kind of commercial scale producing enormous amounts of data. So now that's very, very good news. That means that the sort of asymptotic picture for this looks extremely good.

6. 23:32

   So then we have a bootstrapping problem. Like, how do we get there? Well, a bootstrapping problem is, that's like a good problem to have. If you know the asymptotic thing looks good and you just have to somehow lift yourself up there, okay, do the- You're an optimist. This is why you're a startup founder. This is great. Do the engineering stuff, do the industrial stuff, and you'll get there and you already know that like, that place you're going to get to is a really good place.

7. 23:52

   So, you know, it's interesting. So we had Dimitri from Waymo, who's a co-CEO of Waymo here a couple of weeks ago. You know, a lot of what we were talking about is that, yeah, it's- It's great to have Waymo operate, I think, I don't know, in 15 cities today. But, I mean, I think there's probably both data to support your argument because the number of cars that Waymo is running is still not that large.

8. 24:11

   Like maybe like 5,000, 6,000 across all of its cities, right? So it's not, it's actually like low thousands, if you want to, mid thousands, I guess. But on the flip side, their bootstrapping took them 17 years to kind of get to that point. So I, you know, it's kind of, I guess, making an argument for your asymptotically, you don't need a high end count to be able to collect the amount of data. And, you know, cars are operating in the most, in some ways, high risk environments alongside humans.

9. 24:41

   So that's good. But how do you kind of cut down the bootstrap type? Because cars are too specialized. So if you were a person and you're, you know, maybe you're a child, you're growing up, you're learning all your physical skills. If the only thing you were allowed to do in life is drive cars, it'd be very difficult to acquire all the physical skills you need. If you're, you know, not even a baby, if you're like a five-year-old, a seven-year-old, and you get put behind the wheel of a car, okay, we all know what's going to happen.

10. 25:08

    Like, that's not a good, you know, child development strategy. The good thing about robots is that they are less specialized. Because, like, it seems counterintuitive. Like, a robot needs to be able to do more stuff than a car. but it's also a virtue because it can do more stuff than a car therefore it has more things to learn from some that are higher stakes some that are lower stakes and those lower stakes things will bootstrap the higher stakes things just like they do for people like that is not at all a radical idea so the problem with cars is that they're too specialized such a good answer because you're right that like actually certainly operating a car as a human is pretty unnatural to kind of like have the interplay of using your hands and your legs to kind of do something there's not much prior art that you can kind of in your own personal experience that you can draw

11. 25:54

    from whereas for a robot you can kind of reuse and kind of recombine a lot of the the core underlying concepts so that makes sense yeah also a car can't like learn about the world by playing around because you're going to get stopped by the cops or crash right something yeah the rl is hard um well yeah walk us through like some of the accelerating feedback loops that you have seen when you are able to deploy your models on hardware into settings that then allows them to start collecting data in a particular setting, which then in some ways accelerates the capabilities of those models?

12. 26:30

    So both deployments that we've done ourselves and deployments that other companies using our models have done, oftentimes the theme there is actually pretty clear-cut that you get to a level of autonomy that's sufficient, that you can deploy something under human oversight. human interventions, and the interventions that fix the mistakes provide exactly the data that you need to get the model to very high performance. That's not a novel strategy at this point. This is something that a lot of folks working on robotic learning understand very well.

13. 27:00

    I think, however, in the story arc of this stuff, that's actually just the beginning. I think eventually what we'll get to is a situation where not only can you deploy and use human interventions to fix up the mistakes and provide exactly the right data, but the robots only experience and also provides additional data to improve over time. And this is something that is not yet widely used in deployment, but it is working in laboratory experiments that we're doing. And it actually makes, it's a very natural transition because when you have a robot deployed under partial autonomy with interventions, what the interventions do is they correct mistakes. So the robot is maybe sorting something at a logistics facility.

14. 27:38

    It drops an item. The intervention is to pick up the item and put it back. So now the robot knows how to fix the mistake, but it doesn't know how to avoid that mistake in the first place. And that's where the reinforcement learning will come in and address those issues. So initially, the interventions give you a way to fix mistakes. That makes you more autonomous. So now you can be clumsy but autonomous. And then the RL comes in and fixes the clump.

15. 28:00

    And that's a slower process and is enabled by greater autonomy. So maybe to zoom out for a second, right?