Reasoning and better annotations help robots generalize

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 9:38

   So maybe to move on from Pi 0 to Pi 0.5, I think one of the cool things about the second model, it was the second model, right? Yeah. Was that it started to demonstrate some amount of generalization, right? It started to kind of like work in environments where it had not been explicitly trained. It cleaned kitchens and bedrooms that I'd never seen before. Also, it just did more long horizon kind of like work that was more dexterous.

2. 10:05

   So what is a key ingredient that kind of unlocked the capability kind of like step function change or I guess generalization from Pi 0 to Pi 0.5? So there are quite a few things, but a few of the really important principles for robot foundation models, these principles are not actually all that surprising because they look just like the principles for other foundation models, but they have to be mapped onto the robotics problem in the right way.

3. 10:28

   So the first and most obvious one is that you need the right data. And getting the right data is not just an industrial problem. It is also a research problem. In fact, that's why it's such a difficult problem is that it combines economics, industrial scale operations, and research. You have to figure out the right thing, scale it up in the right way. You usually can't answer what the right thing is until you scale it up to some level.

4. 10:50

   So that's the data piece. You also need the model to be able to understand complex instructions and break them down into manageable pieces. So this is where the LLM part actually becomes really important. So the Pi-05 model actually had an internal reasoning component. So if you tell it to clean a kitchen, it doesn't just try to map clean the kitchen to robot actions. It breaks it down into thoughts, and the thoughts might be like pick up the dish, pick up the towel, wipe the spill, and then from there grounds that into actions.

5. 11:22

   And this is tremendously powerful. This is the same reason as why coding agents work, right? Coding agents today, you don't just tell them what you want and they spit out lots of code. They actually make a plan. They break it down into steps. And Py05 was our first model that did something like that. And since then, we've extended it. We can do reasoning in other modalities. We can do reasoning of variable length and so on.

6. 11:41

   And these are very important things for doing really complex tasks. How much of the capabilities increase, whether it be Py0.5 or 0.7, is coming in some ways of the afterglow of what we are seeing of like, you know, kind of frontier, more standard LLM kind of research or I guess capabilities. versus, hey, it's great that we figured out reasoning models. Maybe we'll figure out how to get, you know, swarms or kind of like, you know, tens of or thousands of essentially like agents kind of collaborating to do things and then figuring out how to apply it to robotics.

7. 12:15

   I mean, I guess, do you think of that as a kind of like, you know, concrete strategy at Pi, which is that like, it's great. We have, you know, OpenAI and Enthropic with like hundreds of billions of dollars worth of kind of research capabilities going in and we'll be able to kind of draft off of that kind of like, you know, for more specific domains. So the trouble is that the ideas are not actually the same.

8. 12:37

   Like there's analogies to be drawn, but the analogies from my experience often end up being more obvious in hindsight than they are in foresight. So for example, one of the things that turned out to be really important to get great zero shot generalization in PIO7, you know, PIO7 is a very powerful model. It can actually generalize to entirely new tasks. You can give it like a kitchen appliance that the robot has never used and it can kind of muddle through and figure out how to use that kitchen appliance, one of the important things for that is to have the right annotations for your robot examples.

9. 13:05

   Now, in hindsight, this is blindingly obvious. Like, the biggest innovation for video models has been prompt expansion. Prompt expansion is what enables video models to actually, not just to take complex instructions, but to fit complex data. But the trouble is that for a robot task, obviously, you can't do the same thing as you do for video models. Because for video models, a prompt expansion, You tell it, like, you know, make a video of a dog running through the grass, and it'll say, you know, there's beautiful green grass with, like, dandelions growing on it, and the dog has, like, flowy golden fur.

10. 13:36

    Like, the robot doesn't care about any of that. The robot needs something else. It needs something that is much more tied to the behavior that it needs to exhibit. So the kind of the principle is kind of shared. Why does prompt expansion work? Because more detailed prompts have higher mutual information with the output that you want. So that's the same principle. But you still have to figure out how to instantiate that principle in a robotics context so that it actually works.

11. 13:56

    Talk a little bit about Pi 0.7, your latest model, right? Like what is a big delta between 0.5 and 0.7? And insofar as you're able to share, is the next one going to be 1.0? So Pi 0.7 is actually the latest model that we wrote a blog post about. There is a secret even later model that, of course, some people might actually already be using through a beta test that we have. But Pi 0.7 is the last model where we, the most recent model that we actually described in detail.

12. 14:26

    So we still need to figure out how to present all the newer stuff. BIO7 is a model that is really designed around getting zero-shot generalization out of the box. So in some ways, this kind of parallels the evolution that we saw in LLMs, where initially we had models that were really good starting points for fine-tuning. Then we had models that could be kind of prompt-engineered to do something. And then eventually we had instruction-tuned models where you could just tell them what they want.

13. 14:51

    So robotics is not at the stage yet where you can just tell it exactly what you want it will do anything, but the PIO7 model does do a wide range of things out of the box, and you can explain tasks to it. You can basically coach the robot in language to acquire new skills, and you can do that without additional action supervision, because the model can break down complex skills into steps, so it's enough to just supervise the steps rather than the actual actions. It also reasons in different modalities, so it can reason in language as well as with images, and that's very important because for a lot of tasks, you can transfer more knowledge from web scale pre-training in the visual modality as opposed to just in language. So if you, for example, tell the robot, you know, I'd like you to push the power button on this appliance, just

14. 15:39

    saying, oh, the power button is like three inches to the left of the right edge of the toaster or something that's like not so useful, but generating a picture where the robot finger is touching the button, that tells it exactly what to do. So this kind of visual reasoning, it's not that it's better than textual reasoning. It's, again, it's like the complement. It gives you exactly the stuff that is hardest to get out of text.

15. 16:00

    A bunch of other things, but the principle really is to move towards this out-of-the-box, zero-shot generalization without requiring the fine-tuning for every task you want to do. Oh, that's super interesting. So is the model then actually doing intermediate visual reasoning steps? So in the Pi-07 model, the image generation is actually done by a separate model. And this is very similar to how you would, you know, if you load up like Gemini or ChatGPT and you ask it to generate a picture, that's going to be calling into a separate model.

16. 16:30

    But the input to that image generation model is produced by the LLM. So it's kind of a similar principle. I think ultimately we'll see integration of all these things where there will be one model that does everything. But, you know, with where the tech is now, the image generation part is usually separate.