Egocentric data gave Mecka a scalable starting point

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 8:35

   Maybe for our audience, just say a word on what is egocentric data? And I think you really started your company kind of making a bet behind it. Obviously, it's much more distributed now, but what is egocentric data? Why did you decide to bet the company on it at the beginning? Egocentric data is pretty much video data from humans demonstrating and doing tasks from a first-person POV. The high-level idea was that from the gap of human data to robotics data, you could close a lot of the vision gap.

2. 9:04

   Where robotics experience the world from POV, can we also show it what it is like to interact with the world from first-person as well? Why we bet on it was pretty much when we explored the space. A lot of what we saw was robotics data at the time was teleoperation, which was humans controlling robots to do demonstrations. It was very hard to scale at the time. And from what we saw across GPT and a lot of the principles of scaling up, we had to find ways to scale up data much more significantly.

3. 9:39

   And I think some of our early intuitions were that humans and robots share a lot of similarities. humanoids were just starting to take off humans and humanoids look the same why couldn't we turn humans into this fleet style learning that tesla has to be able to all contribute into understanding and ground the physics of the world. What does that look like? You're mounting an iPhone on somebody's forehead and just letting them go about their day? What does that look like in practice? When we first started, our very earliest pilot was we created a mobile app. We got all the Chinese aunties and uncles to record themselves cooking, cleaning, doing tasks. That obviously evolved a lot. We tried a number of things. We did UMI data. We did all the sorts at the beginning. That's evolved quite a bit into

4. 10:26

   all the custom hardware that we have to produce now, and all the different partnerships that we have to have across different industries. And ultimately, we're sensorizing humans. And that's the broader category. How important is it to get force data from the sensors? Because one of the things I've always thought about is in self-driving, you don't really need force data as much, right? Whereas with robotics, especially with manipulation, you really care about how hard am I squeezing that door handle or picking up that egg. And it seems to me that video doesn't quite capture that. You might be able to approximate it, but it doesn't really capture it. How important is capturing force data, and how do you think about getting that? I think some of the substitutes for force data that we've seen would be approaches like wrist cameras currently across both humans

5. 11:09

   and robots, and being able to see the contact time between the hand and the different objects that you interact with. That obviously is kind of a hacky way, ultimately, to get to understanding how much force in newtons it takes to pull open a chair or twist a bottle cap, I think force data will be incredibly important. It's actually one of the products that we've been building out in that when you iterate with many different robotics foundation labs, you realize to close this gap, one, we have to create new hardware.

6. 11:42

   So we've been developing things like gloves for humans to wear, and in those gloves, being able to collect the information and how much force it takes to do your daily life and different tasks.