The alpha, twice a week
That was 2 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
MTS · 2 min · Oct 29, 2026
Fine-tuning can beat prompting on narrow enterprise tasks
Sagar Vaze shares an internal annotation experiment where fine-tuning beat prompt strategies. He says even older models can excel on specialized company tasks after customization.
Sagar Vaze is a research scientist at Mistral, leading the multimodal team.
Theo Jaffee, Host · Co-founder & host, MTS
One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.
Sophia Dew, Host · Host, MTS
About this segment
Sagar Vaze shares an internal annotation experiment where fine-tuning beat prompt strategies. He says even older models can excel on specialized company tasks after customization.
From “Why Closed AI Providers Block Cyber Defense | Sagar Vaze”, Oct 29, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
“doing a small, what we call lower fine tuning, or a small fine tuning of the model on the data which we'd collected for this exact task was way, way better.”
Takeaways
- A small fine-tune beat prompt tuning on an internal classification task.
- Company-specific data and expertise can improve model performance.
- Older models can perform strongly on specialized coding languages.
In this episode
Full episode- 14:05Closed-model refusals can leave cyber defenders without recourse2:38
- 9:52Mistral trained its trillion-parameter model on its own computePro pick 3:10
3:26Fine-tuning can beat prompting on narrow enterprise tasks2:13
- 3 more moments in the full episode
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 3:30
Why is it better for an enterprise to use a fine-tuned Mistral model rather than just using Opus or just using Astra? um so different people have different perspectives on this um i think one argument which i've seen against against our thesis is that as these models become more and more capable um the benefit of fine-tuning is going to wash away and here this is actually not something which i've seen so even with our previous generation of models on and Actual real enterprise use cases, custom tasks, fine tuning still yields huge benefits today.
- 4:06
So actually an example I like to give, it's a slightly more niche, but I think illustrates a point. Internally we have a very strong 30b model. A lot of flops went into it, which we use for sort of data annotation. This is a very simple task. Relatively we want to say, does this data belong to category A or category B? It's the kind of thing that you would imagine even models sort of two or three years ago being able to do out the box.
- 4:31
and a couple of months ago I remember running an experiment where we tried to do various prompt tuning strategies, some sort of harness around the model to boost our internal metrics. And ultimately we found that at the end, doing a small, what we call lower fine tuning, or a small fine tuning of the model on the data which we'd collected for this exact task was way, way better. So it's a very simple task.
- 4:54
Okay, sorry, it's true that the model is smaller than the frontier model of today, but it's also a very, very simple task. It's a simple classification data annotation task. And in practice, enterprise workflows are much, much more complicated. On things like, for example, specific coding languages which might be internal to a company, we're finding that even our Mistral Small from, I think it was Jan 2025, fine-tuned on these specific languages can now perform some of the best closed models.
- 5:23
So in this space, it's always difficult to make predictions many years out. But I see the advantage of fine-tuning and customizing lasting for a long time. Right. So with this niche,