The alpha, twice a week
That was 4 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
The TWIML AI Podcast · 4 min · Nov 10, 2026
Design editing needs benchmarks beyond single-shot generation
Design benchmarks still emphasize first-pass images, while real design work includes editing existing compositions.
In this episode
Full analysis- 5:09Canva's data makes smaller design models more economical3 min
- 16:34Detailed scene descriptions help smaller models create better designs5 min
- 39:35Owning the model stack creates more room to cut inference costs4 min
- 10:58Layered designs let people move elements before regenerating pixels4 min
- 22:51Design editing needs benchmarks beyond single-shot generation4 min
- 27:04Style mismatches can be fixed without retraining from scratch3 min
- 31:14Brush revises designs through tools and visual feedback5 min
- 35:48A simpler agent harness lets the model use familiar code paths4 min
“And you have benchmarks for image editing, which is great, but design editing is different than just image editing.”
Takeaways
- Current benchmarks favor first-generation images over iterative design edits.
- Canva has not found an industry standard for editing existing designs.
- Structured rubrics help VLMs score readability and composition.
About this segment
Design benchmarks still emphasize first-pass images, while real design work includes editing existing compositions. Stefano Corazza uses structured VLM rubrics alongside human evaluation to score readability and layout.
Stefano Corazza is head of AI research at Canva.
Sam Charrington, Host · Host, The TWIML AI Podcast
From “AI Design Still Feels Like a Slot Machine”, Nov 10, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 22:55
about benchmarks for image generation. You know, there are well-established benchmarks, but I always wonder, you know, how well they capture kind of the human perspective and human judgment and taste, as you just referenced. And I'm wondering, you know, someone who works on this constantly, like how do you think about, you know, the applicability of the popular benchmarks, how you supplement them, that kind of thing?
- 23:19
That's a great question. I would say the benchmarks overall are lagging a little bit behind to where the power of AI in design is going to be because they're mostly focused on single shot, the first generation. And you have benchmarks for image editing, which is great, but design editing is different than just image editing. And so as of today, There is no established benchmark for how good a model is editing a design, for example.
- 23:53
There's some public benchmark for generating, let's say, a presentation, like there's a present bench. In one, you know, has like 200 examples of like presentation-related slides, related tasks. And you can test it, but again, it's new generation. And so there's not a lot of benchmarks that really can assess the effectiveness, editing and design, the speed and the cost also to go with that. So I hope there's going to be more.
- 24:23
Of course, we have internally, we have a lot of those, But yeah, we haven't found an industry-established benchmark for the ability to actually edit a design because the industry has focused on, oh, AI is so magic, it's going to single-shot everything, which we know is not the case. And so I think now the understanding is finally landing and I hope we're going to see more benchmarks that really start from like a given set of pre-existing designs and defines rubrics for how those designs can be modified and then does the comparison, the benchmarking on that.
- 25:00
That kind of implies that you can use an LLM or a VLM as a judge to look at an image and assess it visually, you know, from any number of perspectives. Are we there yet? Or, you know, how good are VLMs at kind of articulating the qualities that you might want to optimize an image for? I think they're pretty amazing. And we have also to keep in mind that if you structure the rubrics really well, they are really good at defining, I don't know, is this design readable?
- 25:39
Like, what's the score of readability? What's the score of layout and composition? So we have very well articulated rubrics about design quality. And then we can use VLMs to judge that. And we get quite far along. Of course, we always do also human evaluation and some of the arenas are strongly, you know, leveraging that because it's the ultimate test. But I think, yeah, we came a long way. It forced us also to really define what a good design means, right?
- 26:11
And so if humans are not able to articulate that, then VLMs are not going to be able to score it, right? And so I think we had to really go back and really understand what makes the design really good to be able to leverage VLMs to do it. But I think now we are getting to a place where it's quite reliable.