Image-based plans transfer manipulation skills across robots

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 31:02

   So that's the current state of things. Now there is a little bit of a kind of surprisingly positive read on that, which is even though you need data from these robots, the amount of special stuff that the model is doing is kind of minimal. When we started doing all of this, I had a big long list of all the cool research I wanted to do to better accommodate different morphologies, like can you factorize the model's representation in some way so that there's a six-degree freedom arm head and a seven-degree head and a gripper head, et cetera.

2. 31:32

   We didn't do any of that. The model just outputs a big vector of numbers. If the robot has less degrees of freedom than the number it outputs, it just zero pads it. There's just like nothing fancy, and that's it. And then it just train on all the robots and outputs the correct actions based on what it's seen through the camera, essentially. But now, to your point about whether you can handle new robots.

3. 31:55

   So, so far, the thing that we focused on, and I think this is... Thank you. Showing some promise is to be able to transfer skills between robots. And this is actually where the particular choices in how the model works seem to matter. For example, you can have a model that does some intermediate thinking. And that thinking can be done in different modalities. So you can think in text. And thinking in text is really good for transferring high level behavioral structure.

4. 32:27

   So that's basically how you understand that, hey, if I want to like clean the kitchen and put away like the silverware, first open the drawer. Like that's kind of a semantic inference. And you can transfer that very well because obviously that's like largely agnostic to any embodiment or anything like that. But even lower level things can be transferred if you use the right representation. So one experiment we did is we had a thinking stage that is expressed in images where you basically like dream up an image of the next milestone in the task.

5. 32:55

   And with that, we could actually get a robot, the UR5 robot, to fold a t-shirt, even though we didn't have any t-shirt folding data on the UR5. Because while getting the arm motions correct is very hard, because the robot basically requires very different joint angles to do the task, cooking up an image of what it looks like for it to fold a shirt is not that hard, because you've seen the robot arm in all sorts of different poses, you've seen the shirt in all different stages of being folded and unfolded, you know roughly where it should hold it, so getting a good generative model to cook up that image is pretty straightforward.

6. 33:25

   And once you have the image, then from that, backing out the correct actions is easy, too, because you can just look at the synthesized arm angle and just back out what the angle should be. So it's not changing the problem, but it's just introducing this intermediate step that makes it easier to solve. Just like if you're solving a math problem, if you figure out the right intermediate step, the answer is obvious from that intermediate step.

7. 33:47

   And I think that's really exciting because now that shows that this level of generalization across robots, and I'm sure other generalization too can be facilitated with thinking just like in LLMs but with a twist that you have to think in the right modality interesting so it outputs a I guess that image is what its video sensor is seeing and it's like the next step yeah you can almost think of it like image editing you can do the same thing with video you can do it with video prediction but the key is to like imagine what it would look like to progress on this task that seems like a very human thing to do yeah yeah