The alpha, twice a week
That was 6 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Latent Space · 6 min · Nov 16, 2026
The SDXL refiner came from a different, unreleased model
Dustin Podell reveals that SDXL’s refiner came from a separate architecture with better fine detail but weaker structure.
In this episode
Full analysis- 57:12A small dataset unlocked general image editing in hoursPro pick 4 min
- 34:53A video model learned enough 3D to fine-tune in 1,000 steps3 min
- 40:12Training should focus compute on informative denoising stepsPro pick 5 min
- 1:24:25BFL aims to crack one-shot robot learning by 20278 min
- 1:13:06Autoregression is more data-efficient; diffusion can sample faster5 min
- 27:39The SDXL refiner came from a different, unreleased model6 min
- 47:01BFL was founded to capture value its models had created4 min
- 1:05:17More than ten image references are hard to review5 min
“there was a version of SDXL that was never released.”
Takeaways
- The refiner was a full diffusion model, not an upscaler.
- The main model handled structure; the refiner added fine detail.
- The team used separate models to specialize different jobs.
About this segment
Dustin Podell reveals that SDXL’s refiner came from a separate architecture with better fine detail but weaker structure. Splitting those jobs helped the team improve the released system.
Black Forest Labs

Shawn Wang (swyx), Host · Co-founder, AI Engineer
Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.
From “Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs”, Nov 16, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
From the transcript Latent Space publishes with the episode.
- 27:45
I was gonna say, what were the, what were the things we did in SDXL? [laughs] The refiner. The... Oh, no please, not the refiner. Wait, why, why that reaction? So SDXL was a big piece of work in trying to make the models feel good to use for the first time, if I was to kind of like lay anything out. Not to say like Stable Diffusion 1 and 2 weren't, you know... I mean, they were very exciting, they were very fun to use, but, uh, to anyone that was like in the space at the time between the period of SD, you know, SD1 to SDXL, uh, we saw a ton of work out in the, in the open source community working on how do we fine-tune these models, how do we make these models, you know, more aesthetic, how do we make these models into like, you know, uh, multi, you know, uh...
- 28:24
If anyone remembers like the original Stable Diffusion were only able to do square images. And so SDXL was the first work of trying to actually release like a found, like the, the base foundation model kind of with a lot of these general improvements implicit in the model. So, uh, SDXL involved, you know, the first time we did large scale, uh, multi-aspect training. There was a, an... You know, at the time, uh, this was before, uh, packing.
- 28:48
There was like an efficient way we could do essentially bucket style training. So training, you know, on all sorts of different resolutions of, of images in a, you know, a, a large scale. This was the first time we did multiple actual training stages, so taking it from pre-training, actually developing like post-training sets. So the, there was a lot of data work that was done in SDXL. I'm trying to think like all the other aspects. And then, uh, I'll, I'll give a kinda little, a little bit of a backstory on the refiner and maybe, maybe some things that, uh, didn't quite make it in.
- 29:15
So here, here was the whole process of the final release model is essentially SDXL fundamentally works similar to the original Stable Diffusion. It was a latent diffusion model, took in text. It had... You know, you had a noisy latent, took the context of the text, actually generated the, you know, generated the imagery in latent space and then a VAE decoded it. And then from there, the refiner was actually another full diffusion model that the, uh, the whole point of it was to take essentially what was not a rough image, but SDXL wasn't the best at what we would call high frequency detail, which is this like the fine detail, like skin texture, texture of grass, texture of- Yeah, upscaling.
- 29:52
Up- upscaling. Yeah. It wasn't an upscaler because it didn't actually change the resolution, but it, it, y- yeah, upscaled in terms of- It's a refiner... in terms of the, in terms of... It refined it. The reason I thought it was janky is because I worked, I... Not just me, the, the whole team that worked on this was, was fantastic. I don't want to take all the credit at all because there were, there were a lot of fantastic people that, that worked on this.
- 30:10
We did a lot of work to try to push the detail quality of the model while trying to keep this like generality And we just didn't quite get there. And so then there was this like side track of work of like, "Okay, what are other methods to do this?" And this is where the refiner came into being with actually... The refiner was actually a model that was, it was what was supposed to-- So th- this is a little bit of the secret lore, is there was a version of SDXL that was never released. Uh, it was a different architecture.
- 30:38
It didn't quite work as well. But it was trained pretty far along, even to like the post-training stages. Uh, the refiner was that model. It is a full diffusion model. I actually completely forgot about that. [laughs] Yeah, yeah. It- it-- Like the refiner was a full diffusion model. And the one thing about that model is it had a little bit of better performance on these high-frequency details. It just had very poor structure.
- 30:57
And so this whole idea came of like, what if we essentially like- Different model, different job... different model, different job. We, we took it, we did a lot of work to really push the, you know, the work on how well it could actually refine this high-frequency detail. And then we made this, this, essentially this pipeline to- Yeah, which, which is the XL part... which, which then became, okay, this together is like a more performative model.
- 31:18
You know, or a more performative system altogether. And then that was what we ended up releasing here. Which is actually interesting. It was one of the early pioneering works of like actually-- That looked into, okay, what kind of- Yeah... properties learned in which part of the trajectory of this diffusion kind of, uh, trajectory, right? Because you have these structural things that are learned closer to the noise, if you- Mm-hmm... if you come closer to the noise.
- 31:39
And then you have the fine high resolution, uh, high-frequency stuff that comes further down and, and is closer to a kind of, um, the clean image, basically. And like today there's a bunch of different works that actually analyze this. But you do actually, you, you kind of, you regenerate from low frequency to high frequency if you go from like noise to clean image, right? Along this trajectory. But that was not really known. Like, it was informally maybe a bit known by then. Yeah. And this was, it was one of the first works to basically exploit this- Yep...
- 32:06
and say, "Okay, I, I actually s- I dissect this kind of trajectory, and I, I have two kind of experts, one for the structure and the next for the texture, basically." Yeah. And yeah, right now we have modulation, which is like condition on the time step to actually account for that as, as- Mm-hmm... well, and stuff like that in the architecture. But that was really the first, uh, one of the first works to, to really, um, make this distinction. And it, as you see, it worked, right? [laughs] Yeah. It, it did work quite well.
- 32:30
I guess the reason I, I still hold frustration, because at the time I wanted this one model to be so good at everything. Mm. And I felt- It was a cop-out. Yeah... may- maybe, yeah, maybe it was like a little bit of a cop-out. Now, nowadays I take a, a bit of a different view, 'cause I mean, we see systems of models do incredible work together. But at the time, I remember feeling, and I still just have this like sort of old frustration- Chippery, yeah... you know, just, just, just from back then. Um, but- Well, it's also very fair- Yeah...
- 32:54
because you should always go with the simplest solution that works. Yeah. And if, if you then have a kind of a, you, you need to understand, okay- It's just not, like- At, at which point do I need a- A machine learning-... evolve the refiner?... approach. Yeah. Yeah. It's just like- Exactly. Yeah... when it comes to scaling, whatever the simplest solution is that works- But-... you should go with kind of a bit of Occam's razor, uh, as well in, in statistics, basically. Yeah.