Sergey Levine: Humanoid Robotics Results, Chinese Labs & Future Timelines

So when we're doing kind of more fundamental technology development, the key is to understand, like, what are those scalable levers? So figure out the design, figure out, like, roughly the mixtures, and that by itself does something pretty cool, but not like, but that's not the thing that actually changes the world. It's like when you start pulling that lever that things actually change. So with LLMs, when the first GPT models came out, with GPT-2, it did some stuff, but it was sort of like a parlor shook. So you could get it to synthesize a story about unicorns in Peru or something, and it was coherent English, but it wasn't a thing that would solve lots of real-world problems.

But the folks that worked on this kind of recognized that, hey, there's something kind of magical that happens because as you add more data and you make the model bigger, this stuff gets more coherent and more effective so they could see that if we do a lot more of that then it'll become a lot more powerful so to come back to your question what I would say about robotics is that it's not in the in the like GPT-4 and GPT-5 stage where it's like an industrial scale effort to kind of make the model bigger and and get more capability out it it's in that stage where we're establishing the fundamental technologies and because of that what one should expect to see right now is not necessarily that each month the model gets bigger and more powerful by some predictable kind of scaling curve.

It's that the scaling properties themselves are evolving as we develop the right technologies. So to bring this back to something closer to reality, I'm very happy with the demos that we're doing here at Physical Intelligence, and I think that a lot of the results that other people are coming out with are really cool. But these are, to put them in context, we should not expect these to be the things that are actually illustrating the power of scale. We should expect them to be developing the fundamental technologies that will be scaled up after that. So where I think we're at now is that we're actually kind of getting all those puzzle pieces in place.

And I think it's actually very close. I think a lot of the puzzle pieces are falling in place. But it's not like, what makes it so hard to prognosticate about where the technology is going to go is that it's not yet at that predictable scaling stage. It's at the stage where we're figuring out the puzzle pieces, which I think is really exciting. But it means that it's also very hard to foresee what the coefficients on that will be. What are the most astonishing emerging capabilities you've seen so far? That is the thing that is the most fun. And certainly, we've seen a lot more of that happening as we progress.

In the very beginning, it was kind of little things, but they were kind of magical because in robotics, basically, prior to 2024, the stuff never happened. So the little things that happened, and this was maybe at this point, about two years back, we would see things like, OK, we train our policy for folding laundry, and it takes out individual shirts out of the hamper and tries to fold them. And then one very vivid memory I have in late 2024 is we were watching one of the evals, and it takes out like two shirts at the same time. And I'm watching this, and I'm like, OK, it's done for. Like, there's no way it can possibly do this.

And then it puts the two shirts on the table, disentangles them, puts one of them back, and then starts folding the other one. And it's like, wow, like that, OK, like in retrospect, you can do some like detective work and figure out where it got that from some piece of training data. But that was like one of those moments where I don't think anybody watching that eval thought that like the robot was going to do this It's like a little thing. It's like exhibiting the common sense you expect people to have but actually the thing that I find more interesting recently is some of the mistakes because one of the things that's That was pretty remarkable about LLMs is that once they got good enough Even the mistakes kind of made sense in the sense that they weren't like crazy mistakes were just outputs like like ZZZ all the time, but there are mistakes that sort of semantically are sensible.

We had an evaluation last year for Pi05 where the robot was cleaning up a kitchen and it's told, like, put away all the utensils, like there's like some spoons, spatulas, et cetera. And it tries to open the drawer where it thinks the silverware goes and it can't really get the drawer open. So it slides over and opens the oven, which is right next to it, and then starts putting this stuff in the oven. It's like, you know, you can sort of imagine that if you ask like a child to clean stuff up and put it away, they might decide to do that because, okay, it's like a container and you can put stuff there and nobody sees it.

Another experiment we had is washing all the plates. So it would pick up the plates, wash them with a sponge and put them on the drying rack. This was an experiment on memory because it has to keep track of everything that it's doing. It has a scratch pad kind of memory where it's writing down like, hey, I had three plates, I cleaned the gray one, I cleaned the green one, and then it drops one of them on the floor and drives the base over it so you can't see. I was like, okay, I've cleaned the gray plate. It's done. So, I mean, obviously, like, these are not the things that we want to see, but it's kind of interesting that some of the mistakes, they're almost like what you would associate with, like, a child trying to do the task.

So now it just needs to grow up. You mentioned the advancements. They're kind of happening all over the industry. And I know there's a lot of people building humanoid robots. There's Figure, there's Tesla, and there's many other competitors. And you have the expertise of what is hard and what isn't. When you look at the competitors, has there been any advancement or achievement where you think, oh, that's really admirable and that's impressive? I actually think that one of the most inspiring things to me in the industry is to see the kind of takeoff that autonomous driving systems have had. because, you know, like one of the criticisms that is sometimes leveled against robotics researchers is, you know, it's like, you know, same as like nuclear fusion, like it's the technology of the future, but it's always in the future.

But that's what people said about autonomous driving too. And now like, you know, we're in San Francisco, you can go outside, you can take a Waymo and it will actually like take you to your destination and there was no driver sitting there. So I... Like, you know, without getting too much into the technical details, I think what's really inspiring about that is just like this case in point that, yes, you can actually have one of these technologies of the future and it actually does land. And I think it's not an accident that it's landing now in the mid-2020s because a lot of the puzzle pieces for large-scale ML are getting to the level where we can put them together with like actual physical systems. And there's a lot of differences between driving and robotic manipulation, of course. But I think that the illustration that like, yeah, we can actually land learning-based technologies in the real physical world, I think that's really inspiring. If OpenAI or Anthropic started investing more heavily into robotics, how do you think that would impact the industry? Do you think competitors would be worried about that? Robotics is an area where maybe to be fair, like the ecosystem hasn't been as healthy as it has in other areas of machine learning. And what I mean by that is that

like computer vision and NLP are things that sort of lend themselves naturally to a machine learning based ecosystem because there's freely available data. People kind of have a general acceptance that they're going to be using learning. There aren't concerns, as severe concerns about safety, at least physical safety. People rightfully are concerned about AI safety, but it's not the same as a physical device causing some physical harm. And because of that, I think it's a bit easier to spin up a very serious large-scale ML effort in those areas. Robotics is not like that. Robotics traditionally is not a discipline that really embraces sharing of data, for example.

So I think that the more activity there is around learning in robotics, the more I think it'll shift people's thinking towards this kind of future where we accept that robots will be controlled by learned models, not by hand-designed controllers, that there will be data, that data will need to be shared because there's no way that somebody can build out a true foundation model in a single vertical, and basically kind of like shift the entire thinking around robotics to look more like how we think about vision and NLP, as opposed to traditional factory automation. So I think in that sense, like, you know, much as I'm like proud of the work that we're doing at physical intelligence, I think it'll take more than one company to kind of like shift everyone's thinking in that direction.

a decade. A lot of people talk about timelines, and it's kind of more nebulous. And I'm wondering

let's say it's a humanoid robot that's one step in assembly line in manufacturing and my understanding you're less interested in that because that's not really a step towards that general intelligence. What I would say, it's not that I'm less interested in it. It's that I think that the real world has these leaky abstractions that make that kind of stuff a lot more complex than it seems. Let me try to explain this with an analogy. So in the 90s, when people really started working full steam on autonomous driving, there was this idea that

we could avoid a lot of the hard problems by kind of instrumenting the environment a little bit. Like we'll have like magnetic sensors along the highway and so on. And cars will have like a little, you know, sensor on them and a little transmitter so they can tell where each of the cars are. Kind of like the way you do it with aircraft, basically. And then people thought, well, we don't really need like fancy AI. We'll just like have these sensors and it'll just work. And that just basically didn't go anywhere because the real world has so many messy exceptions and special cases that even if 99% of the time, like the magnetic sensors and all that stuff just allow the car to drive, the 1% when like someone steps in the middle of the road or there's like a piece of trash or whatever, it just messes everything up. So the thing that actually worked was when Waymo said like, hey, we're going to not try to avoid the hard problem. We're going to actually try to deploy our cars not in like middle of nowhere but in San Francisco. Like very messy and let's just deal with it head on.

And that allowed that community to make progress. And I think robotic manipulation is going to be the same way, that past the fully structured world of the factory, if you want to go even a little bit outside of that, even if 99% of the time it's all pretty straightforward, that 1% when something weird happens means that you really need the full scope of the problem basically to be addressed.

OpenAI, Anthropic, Cursor, and Vercel all use this product to make their lives better. And the problem it solves is when you're building SaaS or an AI product and you want to sell to other companies, there's all these requirements you need to meet. There's SSO, there's SCIM, there's RBAC, there's audit logs.

These are all things that take time to integrate but aren't the main focus of your app. WorkOS is an API layer that lets you meet all of these requirements in just a few lines of code. So let's say you have a new SaaS product and you want to sell to other companies. WorkOS will solve all of these critical feature gaps for you. You can check them out at workos.com to learn more and get started. And I appreciate them for supporting my work and sponsoring this podcast.

model has data that was collected using that specific set of motors specific set of joints? So far we've obviously put a lot of effort into cross embodiment models that can handle many different robot types, but generally you do need data of the robot you're going to be deploying on to get good performance.

So kind of the metric of generalization there is not can you zero shot a new robot, but it's mostly can you get away with less experience from the new robot and transfer skills from other robots.

Some things you plan semantically, and some things you plan spatially.

Like if you're doing rock climbing, you're probably not thinking like, hey, left arm to rock, 37 centimeters to the left. You're probably more like imagining your arm reaching for the rock. Earlier in the conversation, you mentioned that Waymo was very inspiring. And their kind of path to productionization is proof that you can do real-world generalized robotics. And if I recall correctly, when I was a lot younger, it was kind of this early promise of this is going to happen. And then in reality, it took a lot longer. And so I guess my question is, in the case of humanoid robotics, what would make you say single digit years it's coming versus a long tail and policy challenges as well, I could imagine?

I think one big difference between how robotic foundation models address the problem and how more traditional engineered systems address the problem is that the stack is really thin. So it's not easy to train a foundation model. You need to, obviously, you need to get the right data. There's a lot of work that goes into curating, labeling, all that other kind of stuff. But the actual software that runs on the robot is very, very simple. So, you know, you might have some kind of thinking or reasoning stage. You might have You might have the model produce actions. It needs to be fast enough But the like just if you think about in terms of raw lines of code It's much much lower than a more traditional AV stack And, you know, partly that's because modern autonomous vehicles, the work on that started a lot earlier with very different technologies and evolved over time.

Partly it's also because the problem is more safety critical. Like, yes, you don't want a robotic manipulator to, like, drop a fragile object. But at the end of the day, that's a lot less bad than having a car hit somebody. So that is not to say that the safety challenges with robots are not real. They're very real, and it's very important to tackle them. In fact, it's probably one of the harder ends of the problem. But they are not as much of a hard stop to practical deployments because you can come up with tasks and environments and domains and also physical hardware where those problems are a lot less severe.

So I think that combination, radically simpler software stack plus less drastic software challenges, I think actually make it a lot easier. And because, you know, to your earlier point, that there was this kind of flywheel effect, that there's a positive feedback loop that, you know, started to get things out in the world, even under some constraints, will actually facilitate getting them out more and more. I think a lot of people are familiar with this idea of post-mortem, you know, looking back on why something failed.

with these AI companies, which is, if they're wildly successful, it creates this, I guess, worry or new set of things. So for instance, when Anthropic had a very powerful model, then the government comes in and there's these worries about safety and risk and all that.

And I'm curious how you think about that. If physical intelligence this year had a a phenomenal, incredibly capable, generalized model. How do you think about those kinds of topics that might come up? Working on AI safety is not a new thing. My colleague at UC Berkeley, Stuart Russell, was talking about this stuff like over a decade ago, and lots of people spent a lot of time working on it. It's just that the trouble is when the technology moves so fast, the important problems are not just a function of like, you know, the core principles, also a function of how society reacts to what kind of tools are adopted and so on and I think that's very very hard to anticipate so I don't have like a very satisfying answer here in terms of how we are approaching it our philosophy around all this stuff is basically one of empirical experimentation like let's get stuff out there let's see what happens in the real world let's see what goes right and what goes wrong, so that we have as much of a preview for what the technology can do, what are its weaknesses, what are its strengths, and so on. But at the end of the day, you kind of have to just keep your eyes open, see what happens, and adjust as you go. It's very hard to anticipate.

And I think your question, though, is very spot on, even though I don't have a great answer for you because like, yeah, if we're having like this much concern. And issues with AI systems that are basically limited to using computers, we're presumably going to have strictly more concerns and issues with AI systems that can do everything in the physical world that we can do, right? So the issues are real. It's just, you know, you kind of have to like see what happens and then adjust. And that's kind of in the scary path. In the happy path, if everything goes well and we have incredibly capable models and robots, in 10 years, is the North Star that that's the end of human labor?

I believe it's a mistake to think of robots as mechanical people, right? Like, you know, computers at some level are kind of like mechanical brains, but when personal computers like really took off in the 90s or the 2000s, etc., it's not like the first thing that happened is that, you know, people replaced their brains with computers. Rather, what we saw is actually a proliferation of very different kinds of computers. We saw kind of like ubiquitous computing. So you would have a computer on your desk, but you might also have one in your pocket. You might have one in your refrigerator and in your car.

Like, because computing became so accessible, you could have a little bit of computing in everything. And, you know, I don't think that that's what, like, the people that first started thinking about this stuff in the 40s and 50s would have imagined. They would have imagined, like, you know, room-sized computers, whose job it is to, like, control the, you know, the policy of an entire country or something, rather than, like, a little bit of computer in everybody's refrigerator. So, I think it's, you know, by analogy to that, we might imagine there might be, like, a little bit of physical actuation in everything.

and it might just be like lots of everyday things that you have to do yourself. Now you get like a little bit of help with it. I think the other example that's worth thinking about is modern coding agents, right? So I think that, you know, this is something where, of course, the jury is still out as to what the end game of coding agents is. But certainly from the experience of software engineers today, like it kind of seems like probably fair to say that most would consider coding agents to be more empowering them rather than like, you know, somehow causing them to have a panic. I mean, some people might have a panic, but in general, at least from the software engineers that I've talked to and from my own experience, it's more empowering to kind of be able to amplify how much work you can do with AI tools. So, I think from that, and that maybe is like a pretty direct analogy because that is straight up an example of an actual real job where AI has entered into it and has actually provided more leverage to the people doing that job. So I think that's another example that we can look to. But the truth is that I think it remains to be seen. So in LOMs, there's a few seminal papers that if you read those papers, you kind of get a sense of the lineage of the

breakthroughs that mattered and understanding where we are today. In the robotics industry, are there a set of top papers that you really think kind of show the breakthroughs that people should know about if they're curious about the state of the art in terms of humanoid robotics? One thing I would point out, and this is like partly a shameless plug because I am a co-author on that paper, though candidly like 99.9% of the work on this was done by Tony, who was the lead is the original ACT paper, the Aloha paper. It's an interesting example because in some ways the ideas weren't really that new, but they were illustrated in a really nice way.

And the idea was that, hey, if you set up the right kind of low-cost robot setup, in his case it was based on these robot arms from Trosson Robotics that are like $7,000 hobbyist arms. He set them up in a bimanual setup with a leader follower to the operation device. And he showed that actually if you do it right without really any particularly fancy tricks, you could easily collect to the operation data. Extremely dexterous tasks that people had previously thought would require like very sophisticated hardware and all sorts of like really expensive stuff and then set up like a fairly straightforward transformer based model and it could actually do a lot of those tasks.

And it's kind of like an interesting thing because usually in academic research we put a big premium on like, you know, do you have some like sophisticated new mathematical thing or some sophisticated like technical insight. And in that paper, which I think at this point has been hugely influential, the insight is really just like, yeah, just put together the right pieces and have a little bit more faith in what a simple robot could do, so to speak, equipped with a good intent learning system. And he showed things like replacing batteries in a remote control. He even got like a little mannequin foot and he showed that you could put a shoe on it for like an assistive task sort of.

Like, you know, some people need help getting their shoes on, so that's good. But what people found, I think, so interesting about that paper is just how far you could get with relatively simple building blocks. And at this point, he open sourced the code for it, and the ACT code has been used by lots of people. If someone wants a very basic starter kit for robotic learning, that's usually what they grab. And I think that it's worth for somebody who wants to get into the field to go through that paper and really understand what's going on there, because even though in some ways it's not that sophisticated, I think it provides a bit of calibration on what matters.

The details matter, but the details don't have to be complicated. Before all this large model robotics kind of wave, prior to that, Boston Dynamics had these really impressive demonstrations and tons of mindshare. I guess I wasn't even in the field of saying, wow, they're really doing incredible robotics. And then in the last, I don't know how many years, I don't really hear about them much anymore. Is there some shift in the industry that made that so? Or, you know, is that something you could explain? So the way I would explain it is this, that there are, you know, robotics at some level is about building complex systems.

So even though it's very tempting to say like, oh, there's different areas of AI, there's like LLMs and vision and robotics, one of those is not like the others because for robots, you actually need like all the parts, everything from like how you wire up the robot, what the power source is, what does the actuator look like, all the way to how does it do like high-level planning to determine like what tasks to do next. And even though we could look at these things and say, like, oh, all of these different videos and different companies and different demos, they're all robotics, they're really kind of different parts of the stack.

A lot of what the classic Boston Dynamics results show is very sophisticated hardware, very carefully designed hardware, with a traditional control approach, with very smart controls engineers setting everything up, but with comparatively less emphasis on the kind of decision-making aspect. And I think that at a particular point in time, that actually made a lot of sense because if we can't build a physical body, it doesn't matter what kind of decision-making system is running on it. But to our earlier discussion about generalization, at this point, we're at a stage in the development of these things that even though we can do more on hardware, in many ways it's good enough.

And the big challenge is how to have the decision-making loop that actually works and that reacts intelligently to everything in the environment. And the place where I would draw the dividing line between those is, like, decision-making loop doesn't mean symbolic decisions. It could mean... level decisions, the question is, do you need to take the rest of the environment into account, or are you just dealing with a robot? So if you want to do a backflip on flat ground, you most have to deal with a robot. But if you want to pick up a coffee cup off of a table, even though that's maybe in some ways simpler than doing a backflip, you really have to understand what's going on in the rest of the world rather than just your own body.

And that dividing line, I think the way that technology has panned out, I think it's fair to say that that is the dividing line between AI and controls. Controls is when you have to control the robot body. AI is when you have to take into account what goes on outside of the robot. And I think that's why you see this divide, because I think a lot of the demos where you mostly needed to deal with the robot itself and not the rest of the world, really good controls could allow you to, could admit a very good solution there. And, you know, another thing I would say here is, like, okay, if there's a lot of controls work that goes into doing some particular skill, Well, there is actually something to learn from that because if you can hand design a controller that performs a sophisticated behavior, very likely you can also learn that controller.

So like just that proof of existence that the thing is possible and not only possible but also simple enough that a person could build it because remember people are, you know, at the end of the day, even with code these days, the kind of complexity that people can handle is not as high as the kind of complexity the AI can handle. So if a person can hand design something to do a backflip or do some acrobatics, So that's a really great proof of existence that there exists some relatively parsimonious control law for doing that skill. And parsimonious does mean generalizable. So if it's simple enough for a person to design, probably there's something fairly general in there.

And if you can learn it and automate it without having to have the human controls engineers in the loop, that's good news. And then last question for you is, if you could go back to when you just entered the industry and give yourself some advice, knowing everything you know now, what would you say? One thing that I've learned over the last few years, which I think is a little different than kind of my original mindset, is I think that addressing robotics effectively requires using very broad prior knowledge. And I think that there's this idea that a lot of people in robotic learning have, which I think I shared initially, that since people learn things kind of from scratch, maybe robots should learn things from scratch, too.

For example, in some of our early work on large-scale robotic learning at Google, we had this what we call the arm farm project. We set up a bunch of robot arms in a conference room, actually, because we didn't have a proper lab, but it was a conference room, and we had them all grasping objects. The idea was, well, if they grasp millions of objects, they'll learn very general grasping strategies. It basically worked. Like, they could learn to grasp objects, but it was very hard to, like, take it further from that to the next level. So, okay, now it can pick up anything, but, like, so what? It didn't serve as a very good stepping stone for more complex skills.

And I think part of that was that we were approaching this, like, very blank slate. Like, let's start from zero and see if knowing nothing in advance, the robot could start picking up behaviors. But I think that it's much, much more practical to get all this to work if you can combine robot experience with knowledge that you can pull in from other sources. Like, for example, I was very skeptical initially about the utility of language, and I think scientifically this is defensible, which is that, like, hey, you know, like, animals can do some pretty impressive things, like monkeys can do really cool stuff, but monkeys, as far as I know, can't speak, at least not very eloquently.

So maybe our robots should also be able to do stuff, and they don't necessarily need to understand language. But I think the subtlety there is what's important is not language, it's prior knowledge, that you can put in as a scaffold in your learning process. And you can pull in that knowledge in all sorts of ways, like, you know, humans don't necessarily pull it in entirely through language. Humans and monkeys certainly. We don't. They do it from observation, from observing other people, other creatures, and so on. So there's lots of sources of prior knowledge, but the point is that you've got to get that prior knowledge in there. Otherwise, you're actually faced with a harder problem than what humans and animals have to solve. Because like, you know, if a person had to figure out how to like assemble IKEA furniture, but they've never actually encountered any article of furniture in their entire life, like, okay, that would be like pretty difficult because they don't even know what like, what the point of this is or what the end game looks like. So yeah, prior knowledge is important. And while I'm still a big fan of learning things through experience, I think that my advice to myself would have been take prior knowledge more seriously.

Awesome. Well, thank you so much for your time, Sergey. I really appreciate it. Yeah, thank you for your questions. Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Liskov, Mike Stonebraker, mark brooker these were all people that i brought on because someone left a comment on another note aside from the podcast i'm working on building the ergonomic keyboard that i wish existed here's a glance at the prototype it's a split keyboard so there's two sides this is in the case but yeah we launched on kickstarter and we hit our goal within eight hours of launching i really appreciate it if you were one of the people who grabbed one of the early units we're now working on the long journey of building the tooling now and so if you still want to pick one up i've left the late pledges open on kickstarter so you can grab one there i'll put a link in the description thank you again for watching the podcast and i'll see you in the next episode