Robotics researchers must trace failures across the whole stack

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 16:45

   What does it mean to be a researcher, like in kind of the robotics field? What is the day to day of like a kind of like a researcher at physical intelligence? Like, you know, I think it's interesting, right? Like a lot of folks who either have computer science backgrounds, or kind of work as software engineers, we have a particular mental model of like, you know, the day to day job. But I'm curious, always, like, what does it mean specifically to be a researcher? Like, what is like, what are you doing on a day to day basis?

2. 17:15

   Um, so robotics is very, um, uh, I guess heterogeneous, right? So to get a robot to work, uh, you need to do lots of things that will range from like tightening the screws that attach the gripper to the arm to, uh, figuring out how to train a model on, you know, thousands of GPUs. There isn't really one answer because, uh, every day is different, which is one of the things that makes it so much fun.

3. 17:37

   Um, but you know, honestly, if there was like one, one trait that I think is really important for a robotics researcher, it's attention to detail. And the reason you need such attention to detail for robotics, even more so than other disciplines, is because the stack is really deep. And if something isn't working, it could be not working at the level of, like, someone didn't attach the connector that goes into the camera properly, all the way up to the highest level of, like, well, you know, our model is, you know, it's like 16 billion parameters, but it needs to be 32 billion parameters, something like that. So it could be anywhere in the stack and having a really good understanding of many different levels of abstraction is really important. You know, one of the kind of the classic question that I get, you know, from people outside of Silicon Valley is like,

4. 18:24

   why are AI researchers so highly paid? And the answer is pretty simple. It is not that many of them, right? Like relative to kind of like the impact of these foundation models is the basic supply demand problem. So I'm curious, how many great robotics researchers do you think there are in the world right now that could have that level of attention to detail and, you know, could like Like, is it like a few hundred?

5. 18:45

   There's just not that many people who have done this, right? It's a tricky question because I think there's a big difference between the, you know, saying like, how many people have a robotics PhD from a top university and like a publication record and the willingness to get stuff to actually work and have that attention to detail as opposed to how many people have the mindset necessary to get this right.

6. 19:09

   And there's a big difference. Like, there's certainly people that have a strong publication record, PhD from top school, but just don't have quite the mindset to go to work. And on the flip side, there are people that have the right mindset, don't have the credentials, but have the ability to get there in the right environment. So it's a very hard question to answer. And probably the main lesson for me is not to prejudge these things and to really pay attention to how somebody approaches the problem, what is their actual thought process, and what is their, I guess, life philosophy, for lack of another word.

7. 19:39

   So I guess maybe to kind of put you on the spot here, like, do you think it is possible to get hired at physical intelligence as a robotics researcher without having a great publication record, but having a great mindset? It's like, listen, I am a great software engineer. I care about this. I can learn. I have attention to detail. I've kind of done, you know, kind of at least the basics of how all these models work.

8. 20:00

   Do you think that's a possibility? Yeah, absolutely. I think the biggest challenge with that is the signaling aspect of it, right? That in any such situation, if you want a job, the other party needs to have some way to evaluate that. Yes, yes, yes. And that's very, very hard. And, you know, I wish there was a way to do that better. But, you know, certainly the most important thing is not the PhD or the credentials.

9. 20:25

   It's the right mindset, the willingness to learn things, to admit that there are some things that you don't know. And most importantly in robotics, the willingness to have the attention to detail. And the attention to detail at many levels in the stack, it's like easy to say, but it's hard to do because people, especially smart people with an engineering mindset, often have a natural tendency to specialize.

10. 20:47

    And that is a bit antithetical to having that attention to detail to realize that the reason your model is not running is not because you don't have the right learning rate. It's not because you didn't use the right data set. It's because you ran it on a robot where the fingers have a little gap between them when you command them to close fully because somebody didn't attach the connector correctly.

11. 21:04

    So you have to go there, plug in the connector properly, test it yourself, and then go back to training your models. Maybe that's a good segue, right?