Robot-only data taught grasping, not broader physical skills

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:56

   And I figured, well, if this stuff that I'm building makes virtual characters perform complex physical tasks, perhaps it could also work for getting physical robots to perform complex tasks. So I went and interviewed with Peter Abiel at UC Berkeley for a postdoc position because I figured I need a little bit more education to figure out how to actually do this. Peter called me a couple days later and told me that he already felt like his lab was doing too many different things and all this weird character animation stuff I talked about doesn't really fit. But then a week later, I don't know what happened. He called me again and said that he wanted to offer me a postdoc job.

2. 2:27

   So I never actually found out why there was that change of heart, but I'm very glad that it happened because then I got to learn how to actually make stuff work on robots, which was in some ways like surprisingly similar, in other ways, surprisingly different. And, you know, from there, I kind of realized that there's this wide open world of robotics where I could actually apply some of the stuff I'd figured out, but those were some new challenges. And that was a lot of fun. After that, I worked at Google for a while. The right idea, but also a bit of a naive idea, which is that to make robotics work, you need to get the robots to collect lots of data. And what we did is we set up a whole bunch of robots in a lab. It was actually a repurposed conference room.

3. 3:09

   And we had them just grasping objects all the time. Why grasping objects? Well, because I figure that's like a very basic building block for robotic behavior. So you could just launch lots of robots, have them try it, and they'll collect experience. And that kind of worked, but it hit a wall. And this was kind of the mistake in hindsight. The wall that it hit is you could get robots that are really good at grasping stuff, but you couldn't get them to be good at anything else. And the reason is that all the other stuff doesn't just take lots of experience. It also takes lots of knowledge. And having the robots discover that knowledge from scratch is very difficult. I mean, even people wouldn't discover from scratch. If you take a newborn infant you lock them in a room by themselves with nobody around they're well first

4. 3:45

   they will fall on their faces they will not have a good time and they're certainly not going to figure out lots of physical skills so the missing piece and this is coming to the end of this story is that uh the missing piece is not just robots collecting experience but also endowing them with prior knowledge and prior knowledge is something that, in the meantime, the AI community had figured out a really great way to get from data on the web by training foundation models.

5. 4:06

   So that idea of putting lots of robots in a room, I think there is something to it when it is combined with the right source of prior knowledge from foundation models. And perhaps, you know, I'm still not done with this adventure of trying to figure out how to replicate human intelligence, but perhaps now I'm on to something that feels more promising to me than the video game stuff from the beginning.