Real Alpha
Watch modeSubscribed

Minus One · 2 min · Oct 22, 2026

Robot models need prior knowledge beyond demonstrations

Sergey Levine

Co-founder, Physical Intelligence

  • Aditya AgarwalHost · Partner, South Park Commons

    One of Facebook's first engineers, he helped build its first search engine; later CTO of Dropbox.

About this segment

Levine says Pi Zero pairs robot observations and actions with language, while an LLM supplies knowledge demonstrations lack. That division explains why robot data alone is not enough.

From “How Close Are We to Robots That Can Do Anything? | Sergey Levine, Physical Intelligence”, Oct 22, 2026. Watch the full episode

“you need to give it the stuff that the robot data doesn't have, and that's the prior knowledge.”
Sergey Levineat 8:44 · Minus One8:44

Takeaways

  1. Robot data grounds actions but contains limited general knowledge.
  2. An LLM adds knowledge that demonstrations cannot supply.
  3. Video and human data can fill in the initial model outline.
Watch 2:24 from 7:17

In this episode

Full episode
  1. 9:32Reasoning and better annotations help robots generalize7:16
  2. 21:08General-purpose robots can bootstrap their own data7:02
  3. 33:21Data strategy matters more than the world-model label2:58
  4. 28:02Robotics may advance faster by solving the whole field5:27
  5. 1:50Robot-only data taught grasping, not broader physical skills2:38
  6. 16:39Robotics researchers must trace failures across the whole stack4:36

  7. 7:17Robot models need prior knowledge beyond demonstrations2:24

  8. 37:21Robotics teams need expertise and shared stack context2:42Up next

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 7:23

    So a robot foundation model needs to be able to take as input the things that the robot is observing. So those are typically going to be camera images. You could use depth, you could use LiDAR, but in the end, it's some sort of image-structured data. And it needs to produce actions. And again, you could have a few choices for actions, but they're all kind of the same. Something continuous that represents either positions of the robot's end effectors or joint angles or joint motor commands, torques, whatever.

  2. 7:50

    Basically, a vector of numbers, roughly 30 to 50 numbers. So that's the basics. Now, if you want it to be a foundation model, you need one more thing, which is you need to tell it what you want it to do. Thank you. and, you know, the most obvious way to do that is to tell it in language, so therefore it also gets language as input. There are other ways to tell it what to do as well that are a little bit more involved, but language is a fairly basic one.

  3. 8:11

    So that gives us, like, the starting outlines of this thing. It needs to produce continuous actions, it needs to take in lots of images, and it needs to take in language. So now you have a few choices. One very obvious choice is train the model from scratch on all of your robot data. So that's basic imitation learning. The trouble with this is it's very difficult from just the robot data you have to get everything that you need for the system to be successful.

  4. 8:34

    So that's that prior knowledge piece that I mentioned before. So then the next logical step is to say, well, if you need prior knowledge, let's start it off from a model that kind of provides the complement to your robot data. So you don't want to start from a model that's as close as possible to robot because you need to give it the stuff that the robot data doesn't have, and that's the prior knowledge.

  5. 8:53

    So that's why the LLM is a really good starting point because the LLM kind of has, like, exactly what you lack. It has lots of knowledge, but it is not grounded. So the LLM is a good starting point. Now, I don't think that's the only way to do it. I think that once you have that, that's kind of like the outlines of a thing, but then you can fill in, you can color in the outlines and put in more stuff.

  6. 9:13

    So you could, for example, and I think we've already seen some of this, and we'll see more of it in the future, you can include other data in training. You can include videos. You can include, you know, data from humans, all this other stuff. But the Pi Zero model really was not intended to be the ultimate thing that does everything. It was intended to be the outline. And once we draw the outline, then we can color in the stuff inside.

  7. 9:32

    We can add in reasoning, chain of thought, other data sources, and so on. That's a lot of what we've been doing since then. Okay.