Try Pro free

Latent Space · 3 min · Nov 16, 2026

A video model learned enough 3D to fine-tune in 1,000 steps

Andreas Blattmann describes turning a model trained on ordinary 2D video into an explicit 3D model with about 1,000 fine-tuning steps.

The episode’s best minutes

Full analysis
  1. 57:12A small dataset unlocked general image editing in hoursPro pick 4 min
  2. 34:53A video model learned enough 3D to fine-tune in 1,000 steps3 min
  3. 40:12Training should focus compute on informative denoising stepsPro pick 5 min
  4. 5 more moments in the full episode
“I think, 1K fine-tuning steps on 3D data, and we got a 3D model out of it.”
ABAndreas Blattmannat 36:51 · Latent Space
36:51

Takeaways

  1. The video model was pretrained on plain 2D video.
  2. About 1,000 steps on 3D data produced an explicit 3D model.
  3. The result points to video as a source of spatial understanding.

About this segment

Andreas Blattmann describes turning a model trained on ordinary 2D video into an explicit 3D model with about 1,000 fine-tuning steps. The experiment suggests video pretraining can encode useful spatial structure.

AB

Andreas Blattmann

Black Forest Labs

  • Shawn Wang (swyx), Host · Co-founder, AI Engineer

    Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.

From “Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs”, Nov 16, 2026. Watch the full episode

Transcript

From the transcript Latent Space publishes with the episode.

  1. 34:59

    But importantly, this, this work was a lot about investigating the effects of data filtering, which, as we felt, or as I personally felt back in the day, were like in research, and they still are very much underappreciated. Like data [laughs] is just like, when, when, when we, when we think about what these models are doing, effectively we're estimating a probability distribution, and the data set is what gives rise to this probability distribution, right?

  2. 35:23

    And if you have the perfect algorithm, the best algorithm in the world that, that just learns anything you want, but your data is very bad and you have like on average blurry videos or images and like, like not a lot of motion in the videos, your model will learn that no matter how good your algorithm is. So what comes out of the model will actually be very bad and like, like- Mm-hmm. Yeah... not nice. So the data is one of the most important levers that you have as one who, who wants to train these models.

  3. 35:50

    And in the entire research field, there was never much talked about data because a lot of people from the research community, they just worked on benchmarks and, uh, like, like tweaked the algorithms. But if you talk about frontier models, obviously we know that by now that data is extremely important, right? And, and that is what we wanted to be really like sharing a bit, like sharing more context on.

  4. 36:10

    So we investigated how much data... Like, and then you have this, this trade-off, I wanna be training on a, a lot of data. Quantity versus quality, basically. Exactly. Where is the- Mm... where's the sweet spot? That was, uh, what we investigated there. And then also a very interesting chart- Is there, is there a chart we should pull up? Uh-... co- coming, coming. Ah. We, we can actually t-take it. The, the, the most- Yeah...

  5. 36:26

    exciting experiment that we did was not the data one. But actually if you go down a bit, um- We then-- And it comes back to what we discussed earlier. We then showed that just pre-training on plain 2D videos allows you to fine-tune a explicit 3D model extremely fast from it. So if you go to the, to the last paragraph here of the, of the paper. Here we go. The- these are these 3D, uh, 3D, um, views here of, of the model.

  6. 36:53

    So what we actually did is we took this video model that we trained, and then we had, I think, 1K fine-tuning steps on 3D data, and we got a 3D model out of it. Mm. And, and th-that, that comes back to the question, okay, what, what do you actually need to, to kind of understand the world in 3D, right? Only 1K [laughs] [laughs]... steps of fine-tuning that shows that this video model, although only trained on, on- It was learning something... plain video- Mm-hmm...

  7. 37:14

    actually had this insane amount of 3D understanding, and you just took a bit of a, a, a, a focus to get a 3D model out of it. Mm. And that, that is really like, that was one of the early experiments where we saw, okay, this is actually kind of-- You can get world understanding through- Got it... training on plain video. And like from a research perspective, this was like, I think, the most interesting experiment, um, in this paper. Also pretty short paper. There's like a section here where you even explain the pipeline, right?