Video models may be learning how the physical world works

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 52:17

   so this is a text to video generation so you put in a text prompt you get you get out a video which sounds easy turns out to be very hard um it's very magical um there there so is uh you one of the more impressive ones at OpenAI there, you know, we, we, there are others, you know, we have, we have a bunch of others and there's a bunch of others underway. It's a competitive space, but you know, the, the, the, the class of technologies is very impressive and Sora is very impressive. I mean, one is just like, how amazing is it to live in a world in which you can literally, you know, say a text prompt of, you know, I, you know, a hobbit living in a hobbit town with, you know, the dragon shows up and then like, literally it will render that for you. Right. And if you want the dragon to be rendered out of used car parts, you just say

2. 52:52

   dragon made out of used car parts and it renders that for you. Right. And so it's, it's like, I mean, that, that, that's just amazing to start with. Um, uh, the even more amazing thing about that, um, and opening, I put out a paper about this. Um, it's very interesting. So the more amazing thing about that is, um, to generate, um, video that passes the sniff test of the human eye looks at it and thinks that it real thinks that it's real. Um, you can't, you can't just sort of copy from other videos, you know, and that's part of what's happening is, you know, this is your training a system like store is trained on millions of hours of video you know from like all over the world and all this open source stuff and everything and old you know out of old movies and all this stuff and and you know then there's copyright disputes over what else is in there

3. 53:32

   and so forth but um you know so it's trained on lots of video but it's not sufficient to just train on lots of two you know because video is all 2d it's not sufficient just train on 2d video and then generate a new 2d video that actually looks to the human eye like it's a representation of the real 3d world and if and if you look at these videos coming out from sora if you look at them carefully. Basically, what you see is like multiple sources of lighting in different parts in 3D space coming together. You see reflections coming off of reflective surfaces that are actually correct. You see translucency that's correct. And then you get combinations of these factors. So like, for example, if you do something where like man walking through a puddle at night, you'll get the splashing effects of the water. The water has to splash in a way that is

4. 54:17

   physically realistic. The light has to come through, refract through the water droplets in the correct way. The water droplets have to reflect the image of the man's shoe in the right way. So, and the AI term for this is world model. This is not only a model for video, this is a model that actually understands the real world. It's a model that actually understands 3D reality, right? And it understands light, right? And it understands surfaces and textures and materials and motion and gravity and gravity and shapes and right exactly all these things right exactly um you know a stiff surface versus a spongy surface um yeah um you know definitely you know give me a close-up give me a close-up of a um of a baseball bat hitting a baseball you know like in slow motion it has to deform the baseball in the way that in the real world it does you know

5. 55:06

   when it hits the bat and like for this story it has to do all those things and in fact it is able to do all those things like the if you just look at the results you can see all that's happening and so so what that means backing up to that the implication is that model is not just a video model it's what's called a world model meaning it actually understands 3d physical reality the implication of that is that we may we may have basically just solved the fundamental challenge of robotics um the fundamental challenge the fundamental challenge of robotics is how do you get a physical robot to navigate the real world without screwing everything up right so how do you get a robot waiter to come navigate through a busy restaurant without stepping on anybody's foot, right? Without tripping over anything, without spilling water on the table, um, to

6. 55:47

   understand everything that's happening in real time to be able to adapt, you know, very similar to the self-driving car challenge. Like how do you do that? But how do you do that for basically everything? And how do you put machines up close with people in such a way that it's like completely safe? And it turns out one of the things you need to do to do that is you need to The robot needs to have a comprehensive understanding of physical reality so that it can understand what's happening.

7. 56:06

   And so when things change, because, you know, the robot's seeing primarily visual, right? It's just like you're seeing visual, and so you have to map the visual into an internal representation of the 3D world. Up until now, building a world model like that has been difficult or impossible, and it now appears that that's actually starting to work. So, Westworld. It's happening.