Video models could train robots and power real-time characters

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 39:06

   original idea of runway was always around how do we use technology to augment human creativity and meant humans in some way. And the way we started was, well, we have a beginning of an idea of, let's start with the creative industries, because it feels like the most obvious one when I create pixels on a screen, when I create moving pixels on a screen, the first thing people think is, well, movies and ads, and it's the obvious kind of first line of sight.

2. 39:35

   But then if you really think about the technology that we're powering and building behind it, We're making visible and possible to create moving pixels, realistically animated, whatever shape or form you want. And those pixels can be used for way more things, right? So one of them is entertainment. So can AI models be used for storytelling? Yes, of course, 100%. That question is answered. Can AI models be used for...

3. 40:00

   What I call boring content. Boring content is you want to watch a movie of yourself in a spaceship. I think that's fun, entertaining. Would you watch six hours of cars driving on the street? No, probably not. But if you look at it from a technical standpoint, those two things are very similar. They're just pixels on a screen. Just moving things and things are happening. Now, we attribute from a human perspective value to one more than the other.

4. 40:21

   You're willing to wait in line and buy a ticket and sit for two hours to watch number one. You're not going to do that for number two. but someone else is willing to pay and spend and watch that for a long time number two which is autonomous systems or robots or drive self-driving cars that's how you learn all in the world you learn by watching videos so instead of me not only recording the videos i can generate the videos and you can learn from that and so for us it's like well if we're solving we're answering the questions of like these models are used for this and they're helpful and that answer the question this answer, we're now kind of realizing, and other people also realize, that if you scale these video models to a significant point, these models not only encode storytelling, they encode and they

5. 41:06

   figure out the patterns of the world at large. The physics, the concession, like the occlusion of things, how things move, gravity, there's a lot of things that models are learning just by observing. And the implications of that are, well, you can start thinking of all these different applications, robotics. And then the other question, the other kind of interesting perspective is once you have a world model, a model system that can understand the world, maybe in the same way that we might have understood, we understand the world is that you can technically run them in real time. You can make them react and condition them on things that are happening on a real time basis. And that opens a whole new avenue of like opportunities. It opens a whole new avenue of opportunities in in entertainment as well you can think about having conversations with characters moving in

6. 41:51

   open worlds similar to a game you can think about conversational systems we released this protocol characters really recently that's based on our world models systems and models that allows you to have open-ended conversations with avatars that look extremely hyper real and they're happening in real time like is there a chris that your employees talk to there's many versions of many different people including chris uh but the thing is that you can create one for anything you want. I'm using one to learn French. And I have a philosopher now. I actually connected two of them and they can chat with each other. And that for me is very interesting because it obeyed maybe to the point of something we mentioned in the beginning was it creates a new market.

7. 42:30

   I don't think we have a name for that because people might think about it as a game or a film, but it's not really either of both. We actually don't have a language and a word to describe what it is. You're interacting with your screen, your pixels moving, you're being part of it, and it's entertaining. You're going to spend your people spend hours on it. What do we call it? I don't know. We'll figure it out.

8. 42:51

   But it's just really interesting. Well, we are excited to see where you take this and what kind of things we can do with the products, you know, in the future.