Game engines can supply context AI video lacks

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:55:17

   and what an AI system can create in Unreal Engine. I think a lot of people in the industry are overly optimistic about the rate of progress of AI for video and other things like that. The real problem is consistency. Like, spurting out an image is really high quality. But with video, over the course of seeing all the AI approaches have consistency issues going from one place to another. And I don't think that those will just be remedied easily. Fundamentally, AI just doesn't have anything resembling an understanding of the entire scene they're in, the entire arc of the movie or plot they're in, and the entirety of the world around them and how it might affect the scene.

2. 1:55:59

   whereas game engines have that exactly where they need to be and so I think what we're going to see in the space of world-class high-quality productions isn't just everybody moves to AI and a large part of the human creatives contributing to that are obsolete I think what we're going to see is AI becoming a multiplying force on the power of human creatives making them able to create better stuff more quickly and with higher quality and results I think, unlike the fields of generative 2D art and generative text, I think the future of AI is much more complex and nuanced.

3. 1:56:37

   And I think your interview with Mark Zuckerberg conducted in VR was a really good first example of this. So you did this VR discussion. It was capturing your faces and then rendering a completely 3D computer graphics model of your faces. and then the end result was patched up by an AI image enhancer that was able to add an awful lot of the missing subtleties that are lost by normal computer graphics rendering.

4. 1:57:02

   And that's the first step. You can imagine the output of Unreal Engine being enhanced by an AI pixel shading post-processor is one thing. You can imagine creation of objects being enhanced, especially mashing up high-quality objects that have already been created. Epic's Quixel team went around the world and scanned tens of thousands of real-world objects at extremely high levels of quality. They have everything from rocks to trees to archaeological finds and so on, all captured there.

5. 1:57:33

   And we have an awesome library of them on the fab content site. What's missing is the ability to create arbitrary amounts of new content. And I think using data like that and AI to create completely new trees that meet your specification from all of the knowledge it has built up of high-quality scan trees is going to be a really valuable thing. But, you know, I don't see this reducing the need for people or the role of people.

6. 1:57:58

   Rather, I think it actually is probably an enhancer on that. I can't help but think when I go on Amazon and Netflix to watch a movie, there's an awful lot of linear content, and most of it isn't very good because, you know, of the limitations of the media and the budgets and of other things. If we can use AI as an enhancer on that, then, you know, everybody's going to have even more opportunity than they have now.

7. 1:58:22

   Every single technological revolution has changed the way that people work, but it's ultimately created more opportunity for people. And, you know, there are pundits predicting that this might be the last, but I think just the opposite. I'm an optimist on this, and an optimist that it's going to create opportunity for everyone. Do you think it will be possible to generate, so use generative AI to create dynamic objects, like you mentioned, trees in the Unreal Engine world, so create meshes and textures and empower the creator to create faster.

8. 1:58:58

   Use meta knobs, like hyperparameters versus very nuanced, where you can control much faster the look of a face, the look of a tree, all that kind of stuff. Yeah, I think that's the central challenge of the next decade. of game engines and AI for, you know, content creation of all sorts. Because you have two very different models of the world that are emerging. There's the scene graph, the technical term we use to describe the set of all the objects in the world in a 3D world maintained by Unreal Engine or another engine.

9. 1:59:30

   You know, so in the videos you saw, it's the rocks and the trees and the snow and the bridge and the people and all of these things. And each one has enormous amounts of data attached to it. Some are texture maps, some are sound files, some are animation files, and enormous amounts of detail all stored there in that procedure. In this precise computer graphics representation that enables rendering it from any perspective with any settings and so on, it's a completely general system that has complete context about the state of the world at any point.

10. 2:00:02

    And so you can always precisely reproduce it. If you play the same scene 10 times in a row, it's always the same. It's never randomly changing. you're like, oh no, why did this character's face change midstream? But it's also rather limited because you have to build everything manually. And it's costly and it's time-consuming and it requires expertise. And then you have this other model of the world, which is what AI sees or thinks. If we could peer into what's really happening and its parameters, there's something like the mushy connections of neurons in a brain. It has a vast amount of knowledge about the world and about graphics and about images and about people and about everything else.

11. 2:00:39

    It's stored in a human, incomprehensible form, but it can be extracted through queries like asking it to produce an image from a prompt or a video from a prompt or whatever. But the huge problem with that is it's very mushy data. We don't know how to give it a command that will give us a precise result. And if it produces one image one time and we change our prompt slightly, it might produce something completely different. We were unable to art-direct it.

12. 2:01:06

    And so it's this completely untamed tool. And I think when we figure out more and more ways to merge these and connect these two together, you can imagine AI enhancing the process of content creation in a traditional scene representation. You can imagine the scene representation being shared with the AI so the AI not only sees the prompt, but also, here's a list of all of the objects in the world and their characteristics and so on. It can learn more about how those objects should move and interact.

13. 2:01:33

    So if you get a constant feedback cycle going back and forth between an engine and AI, then I think you can get the best of both worlds, stable scenes, but also the higher productivity of being able to get content out, and the ability to select specific parts of it and arc-direct those, and to have those arc-directions stick and be recognized as part of this permanent scene representation. Yeah, I can't wait until AI can operate not in the space of pixels, but in the space of scene graphs.

14. 2:02:02

    creating objects in the scene graph, whether it's like you mentioned audio