# Open-model inference still has major room to improve

Markdown copy of https://realalpha.app/s/open-model-inference-still-has-major-room-to-improve/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Danila Shtan](https://realalpha.app/people/danila-shtan/) · Danila Shtan is CTO of Nebius.
- Host: [Jacob Effron](https://realalpha.app/people/jacob-effron/) · AI Investor, Redpoint Ventures · Growth investor in Abridge, Ramp and Physical Intelligence; ex-Flatiron Health.
- Show: Unsupervised Learning
- Episode: [The Window Before AWS Wakes Up: Nebius CTO on Racing the Hyperscalers](https://realalpha.app/v/qJkcjURcAe8/), Nov 2, 2026
- Moment: 25:05 to 28:32 of the episode (3 min), playing on https://realalpha.app/s/open-model-inference-still-has-major-room-to-improve/; the episode from there: https://realalpha.app/v/qJkcjURcAe8/?t=1505

## Quote

> “95% of our customers do not run vanilla open source models. These are **modified, fine-tuned**”
> — [Danila Shtan](https://realalpha.app/people/danila-shtan/) at [26:43](https://realalpha.app/v/qJkcjURcAe8/?t=1603) · Unsupervised Learning

## Takeaways

1. Open models can perform poorly with default inference setups.
2. Optimization spans serving configuration, kernels and model weights.
3. 95% of Nebius customers run modified rather than vanilla models.

## About this segment

Danila Shtan argues that open-model inference is far from solved and describes optimizations across serving, kernels and model weights. He says most customers modify models for narrower tasks, creating room to use smaller ones.

From “The Window Before AWS Wakes Up: Nebius CTO on Racing the Hyperscalers”, Nov 2, 2026. [Watch the full episode](https://realalpha.app/v/qJkcjURcAe8/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[25:08](https://realalpha.app/v/qJkcjURcAe8/?t=1508) as an industry optimized on? Well, first of all, I think as an industry we didn't optimize shit. So like, pardon my French, but in a sense, I strongly believe that running inference, particularly in the variety of open source models that are available today, is not a solved problem. If you take something, just run with it as it's published in the initial support in some of the open source inference servers run with the novel models, your results will probably be underwhelming.

[25:44](https://realalpha.app/v/qJkcjURcAe8/?t=1544) It's not that fast. It's not that cheap, especially with current prices on the infrastructure it's easier to go and use like fable 5.1 astro 6 etc etc etc but because of the open architecture because like of everything that we know as of today you can optimize the way you process those requests you can optimize the way the inference server is configured you'll put a lot of effort in kernel optimizations, sometimes kernel over-optimizations based, like, on the particular pattern that you see in the traffic, right?

[26:25](https://realalpha.app/v/qJkcjURcAe8/?t=1585) There is a lot of underexplored potential in model weights optimization. And I'm not just talking about, like, stuff like speculative decoding, for example. I'm talking about, like, model pruning, selective pruning, compression. because I believe that, and we see that within our customer base, 95% of our customers do not run vanilla open source models. These are modified, fine-tuned, et cetera, et cetera, et cetera.

[26:55](https://realalpha.app/v/qJkcjURcAe8/?t=1615) It means that these are the models that start to address a particular use case that the customer has. Particular use case usually means limited domain, scoped domain. Scoped domain usually means that you need to retain frontier intelligence and reasoning for a particular subset of tasks, and you don't care about the rest. And now we are getting somewhere in terms of maybe you don't need such a large model anymore.

[27:27](https://realalpha.app/v/qJkcjURcAe8/?t=1647) Maybe you need just like a model that is selectively compressed in a way where it works for a particular task and it doesn't work as good for the rest of it. Now, of course, that doesn't work with like open ended, you know, deep research or broad coding tasks anymore. But I strongly believe that for real AI adoption, when we are talking about serious enterprise business, when we are talking about like practice.

[28:00](https://realalpha.app/v/qJkcjURcAe8/?t=1680) that actually solves problems, you are getting more and more into this specialized agents, limited domains, where you even stop caring about the costs. You care more about predictability of the outcome, and you care about time, as in like real seconds and minutes till you get that outcome. Yeah. And I think there is a lot of potential there, and we will still see a couple of nice breakthroughs there.