Try Pro free

Latent Space · 5 min · Nov 16, 2026

Autoregression is more data-efficient; diffusion can sample faster

Andreas Blattmann contrasts autoregressive models’ training efficiency with diffusion’s ability to distill sampling into a few steps.

In this episode

Full analysis
  1. 57:12A small dataset unlocked general image editing in hoursPro pick 4 min
  2. 34:53A video model learned enough 3D to fine-tune in 1,000 steps3 min
  3. 40:12Training should focus compute on informative denoising stepsPro pick 5 min
  4. 1:24:25BFL aims to crack one-shot robot learning by 20278 min
  5. 1:13:06Autoregression is more data-efficient; diffusion can sample faster5 min
  6. 27:39The SDXL refiner came from a different, unreleased model6 min
  7. 47:01BFL was founded to capture value its models had created4 min
  8. 1:05:17More than ten image references are hard to review5 min
“I would say autoregressive models are way more data efficient.”
ABAndreas Blattmannat 1:14:14 · Latent Space
1:14:14

Takeaways

  1. Autoregressive models get more training signal from each example.
  2. Diffusion models can distill many denoising steps into a few.
  3. The guests say research on both approaches is still active.

About this segment

Andreas Blattmann contrasts autoregressive models’ training efficiency with diffusion’s ability to distill sampling into a few steps. The trade-offs help explain why neither approach has won.

AB

Andreas Blattmann

Black Forest Labs

  • Shawn Wang (swyx), Host · Co-founder, AI Engineer

    Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.

From “Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs”, Nov 16, 2026. Watch the full episode

Transcript

From the transcript Latent Space publishes with the episode.

  1. 1:13:12

    One more part of the text tree, and then we want to go into robotics in action. Mm-hmm. Um, autoregressive- Yes... image gen. It seems like a big trend with both Gemini and ChatGPT doing it. Yeah. Uh, like- Is that as big an impact in, in your mind as it is in ours? You know what I mean? Like, I, I-- it seemed like a very big change because then you could conversationally edit. It is the, like, the next obvious step from conte-uh, i- FluxContext, where instead of editing with a image and a, and, and text prompt, you're conversationally working with a, with a model.

  2. 1:13:45

    That I would say is not a, a feature of autoregressive models- Yeah. Mm-hmm... uh, only. It's just based on the data you train on it. And like you, you, you can get language capabilities into a diffusion model as well, right? Yes. Which we, which we know is, is, uh, working. And, um, I said in, in long term, this is where we believe this will, this will also go into and where we're working towards to also get more modalities in, including this language model, right? Mm-hmm.

  3. 1:14:08

    So it's not, it's not, it's not a question of the, of the, of the algorithm there. Yeah. Um, it's just a question of like, what do you focus on, basically. I think both approaches are extremely exciting and have a lot of strengths and a lot of like shortcomings. I, I would say autoregressive models are way more data efficient. You can train them way better in data because every, like, you have the-- for, for diffusion models, you have this trajectory from data to noise, right? And every training example you're training on is effectively because you have this continuous path from data to noise in between.

  4. 1:14:37

    It's giving you r-rise to infinitely many training examples because you can pick every possible spot to denoise from, from this trajectory between data and noise for one example. Obviously, the model would overfit if you would only train on, on, on this one, uh, example. Theoretically, it's just like one training example is giving or one data point is giving rise to infinitely many training examples.

  5. 1:15:02

    Whereas autoregressive models can just train on all the tokens in parallel, basically, and get the maximum gradient from each training example. That's the training side. So there I would say autoregressive models are more data efficient. However, at the sampling, at sampling time, autoregressive models have to-- don't get across that. They have to autoregressively create one token after the other- Yes... because they're iterative in this na- like, like data direction, right?

  6. 1:15:27

    And diffusion models, you can just go down and distill fifty steps down to four, and by that you can speed that up so much, which is on the autoregressive side, just not possible because you have to autoregressively generate token after token after token. So these are just two, two, um- Well, you, you make it sound like it's just worse then. So, so- Yeah... autoregressive generation is in that respect, it has a limitation, but on the other side, diffusion has in, in the terms of data efficiency or training, has also a limitation.

  7. 1:15:54

    So we, we actually-- I think it's-- I, I like both approaches. We've also been working on autoregressive models like- Mm-hmm... long, long, long back, and both are super exciting, I would say. Yeah. So on, uh, data efficiency, autoregressive wins out though? Yeah. Okay. So scale. Um, I guess one of the other big things that you see, having talked to people that worked on both like ChatGPT and Imagen is how you can add in the loop RL, like agentically.

  8. 1:16:18

    So models can do a-agentically, you know, generate something, reason on it, edit the image and continue that workflow. Yeah, they can do that too. Yeah. And this is, but this is separate than pipelining in diffusion generation, right? It's all the one same model doing it, where as you train that in on a data level, you know, you can scale that. Um, I guess, is there a counter to this in diffusion, or? There's no inherent limitation to-- that you couldn't have a model that is diffusion based- Yeah... do the exact same process that can do all of these aspects.

  9. 1:16:48

    Yeah. That haven't been enabled yet. Yeah. This is like really blowing my... Yeah. Like, I, I had the wrong, basically nano banana happened and I was like, "Holy shit, like autoregressive is a thing," but you're saying it's not. I think, am I wrong that isn't nano banana diffusion? I thought nano banana was diffusion. It's- I think it is diffusion based. I think the auto, like it is- Oh, it was, it was 16 bit?

  10. 1:17:03

    And I, I don't have like insider knowledge here, so I might be wrong, but I'm pretty sure it's, it has the language component to do like autoregressive, but it isn't like, I think it's what, what we'd call like a transfusion model. Mm-hmm. So like it's, uh, you know, it, it can essentially do both. But there's no, yeah, there's no inherent limitation that says you, you, you have to do it one way or another. And it's not like we're...

  11. 1:17:21

    They're, they're explorations. [laughs] Yeah. So yeah, yeah. Like, uh- What I'm getting is that- Yeah... it's not, it's not settled consensus. And I thought it was that auto, there was a wave of people shifting from diffusion to autoregression for Imagen 2. I think there's a lot of, there, there's a lot of exciting research on that end right now going on. Yeah. Like I, I will say there's, there's some very impressive work out there. Obviously, we've seen models work extremely well on that front. The same is going on for, for flow matching is also improving.

  12. 1:17:43

    Like it, uh, I would say the debate is not settled yet. Yeah. Like there's still significant re-research on both fronts- That's exciting... going on right now. Yeah. And I think, I think it's important to, to watch both and be ex- you know, try both, exci- be excited about the, you know- And finally, the same is happening for robotics as well. We have, we have- Absolutely...