Detailed scene descriptions help smaller models create better designs

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 16:38

   several breakthroughs, but one on top of the other that I think made the difference. Can you give us some examples of the kinds of things you learned that really moved the needle for you? Without getting too much into the weeds, we underestimated how detailed of a scene description you can give to a diffusion model actually if the diffusion model is above some number of parameters can handle like incredible detail of the scene so you can the size of our scene description really grew in the last six months significantly as we realized oh actually you can be a lot more accurate a lot more precise and so we expanded basically not only the initial prompt of course but like the description of the seen is now to a level of detail that I think no human will want to actually go and look through and edit, you know, unless it's like visually editing, right?

2. 17:35

   And so there's so much detail and by increasing a little bit the size of our diffusion model that we trained from scratch, we were actually able to capture all that detail, all those nuances and really, you know, squeeze the maximum quality out of it. And is that a unique property of your particular diffusion model that you trained? Or is that just a general observation about diffusion models? I think it can be generalized for more cases.

3. 18:07

   In our case, I think because we understand so well the coupling between the generation of the scene and the diffusion model, You know, we really work to make sure, you know, we didn't over generate on the scene description or, you know, or under train on the diffusion model. And so the two are like perfectly calibrated to be the input and the ideal input and output model of each other. Are the scene descriptions only unstructured text or are there also some degree of structure, coordinates, bounding boxes, sizes, things like that?

4. 18:44

   Yeah, oh, they are absolutely structured. Yeah, it's a fully structured, it's still text-based, but it's fully structured with all the details that you just said, bounding box, relative position, transfer position, like all that. We have found that the more detail we can provide to a composition, really the better. And we have the ability to basically render all that detail in the decision model. Also, one thing that I want to mention is which is great for the creative industry overall, I think.

5. 19:19

   There is this race to build the most capable LLM out there, right? And now we are already in the tens or hundreds of trillions of parameters. But for creative tasks, I think we have found that, let's say, 100 billion, it's already a pretty large number that can, And if properly tuned and post-trained and fed the right data, a model of that size can already achieve so much. And we are seeing also a lot of other models, also video models that leverage LLM as an input to create that structured data like the MiniMax.

6. 20:00

   H3 is another model that has a similar approach, even though it's a video model. But we are seeing the size of the basically vision language model or LEM that is the input, you know, not growing 10x every three months anymore, right, but being kind of stable. In an industry where like everything grows, you know, 10x every month, like that number has shown that like, you know, you don't need more than 100 billion parameter to do most of the creative tasks, I would say. Probably video understanding maybe needs just a little bit more because it's probably the most complex type of task, but for everything else, below 100 billion parameters, you can already achieve wonders. And so to be clear on what you're describing there, in this two-stage model where you have an LLM like thing or an LLM period

7. 20:58

   generating a scene description you're saying there's kind of diminishing returns beyond 100 billion and smaller models do just as well as larger models and kind of nailing that scene description and providing the right input to the diffusion model. That's correct and while being you know a lot faster and a lot cheaper right and so just the economics they don't work if you use like the latest frontier model to do that task, then, you know, it will be like a dollar per image generated or something like that. And so this, I think, together coupled with the ability of people to train those smaller models in very efficient ways, I think really unlocked a lot of value and a lot of usefulness of AI in the creative