Robot training is converging on a mixed data recipe

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 32:02

   Is it fair to say that the robotics industry has kind of converged on a training data mix that is, you know, internet scale data from YouTube and the like, plus a bunch of egocentric data, plus tele, UMI, and then plus tele-op, and like, maybe there was like a triangle of like amount of data on each. I think Jim Phan had like a really nice visual on this at a Sequoia event. Is that kind of like the mix that you see?

2. 32:26

   And then amongst your customers, has everybody kind of converged on a very... very similar mix? It's fun to see that even a year ago, many folks would argue and say, oh, no, it's only UMI data that's going to work. It's only egocentric data that's going to work. And you slowly realize, to your point, all of this is essentially a data pot of soup. And in that pot of soup, all the ingredients are going to be important at some point. I think what we're seeing is a convergence across what that recipe might be today. And that does look like some version of a million hours of human data across egocentric and different types of modalities, some tens of thousands of hours of UMI data, and then some last mile teleoperation data and in-domain, in-distribution data in just a couple hours, even of that demonstration.

3. 33:20

   I think we are seeing that. But over time, and why we don't subscribe aggressively to even one religion here is that things like tactile data haven't even entered the mix yet. And if there is this belief that these models ultimately will need to learn how much force it takes to apply towards the door handle, for example, these will have to come into the data pot of soup. Simulation will come into the data pot of soup, and they all will be important to converge on. One of the, I guess, objections that I hear most frequently to the types of data being collected are, you could watch millions of hours of tennis data and you still wouldn't know how to beat Roger Federer. And I'm curious what you think of that and what it means for where we might be headed in terms of the way robots learn. I love the analogy of AlphaGo ultimately. When you talk about robots playing

4. 34:20

   tennis, you got to talk about robots playing Go. And I think with Lisa Dole and all the work that DeepMind did, at the very beginning, it was large-scale behavior cloning of different experts playing Go. Then the next step was the self-play version of with the rules of the game, can you ultimately simulate all the different ways until you come up with the magic of Move 37 ultimately, which was so out of scope of what humans could have thought in that moment on the go for us though in our case of robotics for for go the rules of the game are the is the entire circumference of the world that you will operate in.

5. 35:06

   In robotics, you have the incredible rich detail that the physical world has, and we ultimately don't have a world simulator that can recreate that today. And so to close that gap, a lot of the egocentric data today is to, one, help us ground the basic physics and the basic behavior cloning that you have to get to. It is also contributing to creating what this world simulator might look like. And then somewhere along the way, I think we will be able to create and have robots playing tennis like Roger Federer.

6. 35:40

   Do you think they'll mostly be doing self-play in simulation or in the real world? I could see a hybrid, ultimately.