The alpha, twice a week
That was 5 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Latent Space · 5 min · Nov 16, 2026
Training should focus compute on informative denoising steps
They explain why training uniformly across flow matching’s noise-to-data trajectory can waste compute, and why some regions deserve more attention.
The episode’s best minutes
Full analysis- 57:12A small dataset unlocked general image editing in hoursPro pick 4 min
- 34:53A video model learned enough 3D to fine-tune in 1,000 steps3 min
- 40:12Training should focus compute on informative denoising stepsPro pick 5 min
- 5 more moments in the full episode
“where should we spend our compute? Which parts of this trajectory from data to noise carry the information that's most important?”
Takeaways Pro pick
- Uniformly sampling the trajectory may spend compute on less useful steps.
- Oversample regions that carry the most important information.
- The aim is to improve performance with the compute already available.
One of Pro's picks, open to you here. Try Pro free for 7 days
One of Pro's picks. The rest of the summary and the 3 takeaways are in Real Alpha Pro; the video, the quote and the transcript stay free.
Read it with Pro, 7 days freeAbout this segment
They explain why training uniformly across flow matching’s noise-to-data trajectory can waste compute, and why some regions deserve more attention. Better allocation can improve learning without simply adding compute.
Black Forest Labs

Shawn Wang (swyx), Host · Co-founder, AI Engineer
Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.
From “Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs”, Nov 16, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
From the transcript Latent Space publishes with the episode.
- 40:17
[laughs] No, this, this paper was really about understanding more in which part of this trajectory from data to noise does the model actually learn what properties and where do we have to spend most of our computation that we put in- Yeah... training to maximize the performance of a model. Yeah. So b-before-- A-and obviously, that was also the transition from classic diffusion to flow matching with a, a slightly different, uh, uh, paradigm basically.
- 40:41
We call it rectified flow. Exactly. Um- Th- this was also the first time we used like diffusion transformers, right? With the, the, the- Yeah. And then, and then, and then we took it a bit further, yeah. So lot, lot, lots of changes like going from a convolutional architecture to a transformer architecture- Yeah... changing the probabilistic model from, from a diffusion, um, to basically flow matching- Mm-hmm...
- 41:03
which is way simpler mathematically, which is super, super nice. I think you should actually define some of those terms for people that haven't heard. [laughs] So latent division, you know, flow matching, what. How did you come up with the concept of these ter- things here, right? Well, he didn't come up with flow matching. Well, I- We, we, exactly. We, we didn't come up with flow matching. We just saw this algorithm as, as something that is in a way more general because it can-- Like for diffusion, you always need one of your two distributions- Yes...
- 41:26
between which you're basically translating needs to be a standard normal. That does not hold for flow matching anymore. And instead of approaching this kind of problem to, to, to, um, to transform one probability distribution into the other with a-- Seeing it as, as a solution of a stochastic differential equation, you're actually seeing it as a d- a solution of a ordinary differential equation, which simplifies everything completely, like mathematically completely.
- 41:55
It makes also fast sampling much more, um, feasible you could say. Because what we always do, obviously, is when we have trained the model, we want to get-- be making as many steps, as little st-steps as possible to basically sample as fast as possible, right? Some people would say it's, it's actually a special case of, of diffusion, flow matching. Other, others might say, okay, it's actually a generalization of flow matching, uh, of, of diffusion, the flow matching algorithm, because you, you, you go from-- you can translate between two distributions no matter how they are shaped- Mm...
- 42:24
and you don't need this, this kind of, um, standard normal basically as, to, to be one of, one of them. But yeah, in, in general, it can be seen as, again a, a, a model that learns basically to a vector field between two d- that, that, that translates between two distributions And once the model has learned that vector field, you can just integrate it numerically, which is then as easy as solving a ODE by means of an Euler forward, uh, step basically.
- 42:49
So it's, it's, it's very similar in, in many respects. Um, but it was just like, I would say mathematically easier, more general from, from the, the, the kinds of probability distributions that it can slate be- can, can translate between, and that's why we basically, uh, went with it. And maybe as a s- a third part I said, it's also easier to derive a fast model from those, uh, flow matching ones basically. It, it was, it was much simpler and more scalable.
- 43:14
Like it was, it was, it was much, much easier to work with and then had this like nice performance benefit as well. So like, I don't know, there were... the, the, the diffusion space got quite complicated. [laughs] Uh, are we satisfied that there's no loss? It's like, or is it, is it lossy in acceptable ways? [sighs] Lossy, you mean like do you, do you mean- No, loss, lossy compared to a full diffusion. No. All right.
- 43:33
Yeah. Okay. Yeah, yeah. So then it's good. It's a strict improvement. Exactly. It's, it's- It's great... it, it is an improvement definitely. It-- Yeah, I mean, it's, it's effectively just drawing a line between like, you know, noise to data really. Yeah. It's, it's, it's like, it, it, it kind of took this like there was this complex space on like how do you map between these two, uh, the, you know, the, like these two distributions as Andy was saying. Yeah.
- 43:51
And it simplified it down to [laughs] like the simplest of just drawing a line. It, it's just like, y- y- because you're trying to do a shortcut, like what if it's a bumpy distribution? Yeah, yeah. So that's the, that's the thing. And like the most important part of the paper or one of the most important like on the, on the, on the probabilistic modeling side, um, was actually looking at the trajectory from noise to image and which parts really matter and which don't. Because- Mm-hmm... if you just like we, we, we train this in kind of steps, right?
- 44:18
We, we, we al- we always on, on this part from data to noise, we only always train on one kind of step, and then we approximate this direction in flow matching, um, with the model. And if you just sample those steps in between data and noise uniformly, you might actually spending a lot of time optimizing for things that actually don't matter. Mm-hmm. So one, one central part of the paper was where should we spend our compute? Which parts of this trajectory from data to noise carry the information that's most important?
- 44:45
Because there-- this is where you should spend the bulk of your compute at training, right? And oversampling these parts during training and then undersampling other parts that are less important. It, it's effectively like a further, like a, a, a much further better exploration of part of what we talked about with SDXL, which was like the refiner. We spent time like training the refiner on the end of this trajectory to learn these high frequency details, and this was like more of an exploration of the entire space of where like you should spend time to make the most like general- Yeah...
- 45:13
good model to learn the data. Yeah. Amazing. Uh, one thing I did pick up with SD3 was text, uh, in images, which you guys- Yes. Mm-hmm... you guys showed off a lot.