Unsupervised Learning · 3 min · Nov 2, 2026

Open-model inference still has major room to improve

DSDanila Shtan · CTO, Nebius

Danila Shtan argues that open-model inference is far from solved and describes optimizations across serving, kernels and model weights.

The episode’s best minutes

39:13 Chip scarcity gave AI clouds room to competePro pick 5 min
25:05Open-model inference still has major room to improvePro pick 3 min
53:19 Nebius lets engineers use AI but keeps accountability human3 min
5 more moments in the full episode

“95% of our customers do not run vanilla open source models. These are modified, fine-tuned”

DSDanila Shtanat 26:43 · Unsupervised Learning

Takeaways Pro pick

Open models can perform poorly with default inference setups.
Optimization spans serving configuration, kernels and model weights.
95% of Nebius customers run modified rather than vanilla models.

About this segment

Danila Shtan argues that open-model inference is far from solved and describes optimizations across serving, kernels and model weights. He says most customers modify models for narrower tasks, creating room to use smaller ones.

DS

Danila Shtan is CTO of Nebius.

Jacob Effron, Host · AI Investor, Redpoint Ventures
Growth investor in Abridge, Ramp and Physical Intelligence; ex-Flatiron Health.

From “The Window Before AWS Wakes Up: Nebius CTO on Racing the Hyperscalers”, Nov 2, 2026. Watch the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

25:08
as an industry optimized on? Well, first of all, I think as an industry we didn't optimize shit. So like, pardon my French, but in a sense, I strongly believe that running inference, particularly in the variety of open source models that are available today, is not a solved problem. If you take something, just run with it as it's published in the initial support in some of the open source inference servers run with the novel models, your results will probably be underwhelming.
25:44
It's not that fast. It's not that cheap, especially with current prices on the infrastructure it's easier to go and use like fable 5.1 astro 6 etc etc etc but because of the open architecture because like of everything that we know as of today you can optimize the way you process those requests you can optimize the way the inference server is configured you'll put a lot of effort in kernel optimizations, sometimes kernel over-optimizations based, like, on the particular pattern that you see in the traffic, right?
26:25
There is a lot of underexplored potential in model weights optimization. And I'm not just talking about, like, stuff like speculative decoding, for example. I'm talking about, like, model pruning, selective pruning, compression. because I believe that, and we see that within our customer base, 95% of our customers do not run vanilla open source models. These are modified, fine-tuned, et cetera, et cetera, et cetera.
26:55
It means that these are the models that start to address a particular use case that the customer has. Particular use case usually means limited domain, scoped domain. Scoped domain usually means that you need to retain frontier intelligence and reasoning for a particular subset of tasks, and you don't care about the rest. And now we are getting somewhere in terms of maybe you don't need such a large model anymore.
27:27
Maybe you need just like a model that is selectively compressed in a way where it works for a particular task and it doesn't work as good for the rest of it. Now, of course, that doesn't work with like open ended, you know, deep research or broad coding tasks anymore. But I strongly believe that for real AI adoption, when we are talking about serious enterprise business, when we are talking about like practice.
28:00
that actually solves problems, you are getting more and more into this specialized agents, limited domains, where you even stop caring about the costs. You care more about predictability of the outcome, and you care about time, as in like real seconds and minutes till you get that outcome. Yeah. And I think there is a lot of potential there, and we will still see a couple of nice breakthroughs there.