# Inference providers need a broader platform to avoid commoditization

Markdown copy of https://realalpha.app/s/inference-providers-need-a-broader-platform-to-avoid/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Erik Bernhardsson](https://realalpha.app/builders/erik-bernhardsson/) · Erik Bernhardsson is co-founder and CEO of Modal. · Co-founded Modal, the high-performance cloud for AI teams; previously led Spotify's music recommendation and data infrastructure teams.
- Host: [Harry Stebbings](https://realalpha.app/people/harry-stebbings/) · Founder, 20VC · Built a venture firm on his podcast; its third fund closed at $400M.
- Show: 20VC
- Episode: [Modal CEO: Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute](https://realalpha.app/v/1dwyL6Iz_PU/), Nov 2, 2026
- Moment: 13:34 to 18:08 of the episode (5 min), playing on https://realalpha.app/s/inference-providers-need-a-broader-platform-to-avoid/; the episode from there: https://realalpha.app/v/1dwyL6Iz_PU/?t=814

## Quote

> “And if you're just selling tokens for dollars, there's **no switching cost**. People can just switch a different provider. It's just a race to the bottom.”
> — [Erik Bernhardsson](https://realalpha.app/builders/erik-bernhardsson/) at [14:20](https://realalpha.app/v/1dwyL6Iz_PU/?t=860) · 20VC

## Takeaways

1. Token-only inference has little switching cost.
2. Adjacent products can create differentiation and higher margins.
3. Specialized inference needs can fit under one platform.

## About this segment

Erik Bernhardsson argues that selling tokens alone creates a race to the bottom, while adjacent products can add differentiation and margin. He expects inference to split into specialized products that one platform can support.

From “Modal CEO: Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute”, Nov 2, 2026. [Watch the full episode](https://realalpha.app/v/1dwyL6Iz_PU/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[13:46](https://realalpha.app/v/1dwyL6Iz_PU/?t=826) inference as a market has many different speculators commenting on it. When you look at the next five years, how does the inference market play out? First of all, I think inference is going to be an enormously large market. It's like the new cloud, basically. But I don't think inference in itself is maybe the only thing you need. One of the things that I feel very strongly about is it has to come with a lot of other products.

[14:07](https://realalpha.app/v/1dwyL6Iz_PU/?t=847) It has to come with post-training, sandboxes, agentic stuff, all kinds of other types of inference, like low latency inference, batch inference. So building that platform, by the way, we've been working on this for the last five years. I think that to me is not a moat, but it's a very hard product to build, that infrastructure. Why do you need to have the breadth of that product set? Otherwise, you're just in a compression layer.

[14:30](https://realalpha.app/v/1dwyL6Iz_PU/?t=870) Yeah, because otherwise you're just selling commodity product. And if you're just selling tokens for dollars, there's no switching cost. People can just switch a different provider. It's just a race to the bottom. But if you couple it with all these other things around it, sandboxes and post-training and reinforcement learning and all these things, you can actually create a whole set of tools. No one wants to buy a single-jerk pony. People want to buy a portfolio of different things. They want to buy a whole suite of tools that basically lets them build amazing AI applications on top of it. And the infrastructure underneath is a lot of differentiation, a lot of staying power in the markets. It's very hard to copy the infrastructure required to build all of these products.

[15:09](https://realalpha.app/v/1dwyL6Iz_PU/?t=909) What does no one see about the inference market, or you don't see commented on much, that is very obvious to you? One thing that people don't see about the inference market is, I actually think in the long term, there is so much gross margin opportunity in this space, because you can build all these things around it. All the sandboxes, all the agentic stuff, workflows, post-training reinforcement learning.

[15:28](https://realalpha.app/v/1dwyL6Iz_PU/?t=928) If you do all those things, there's actually so much opportunity to make money. Is it a market that has software margins? Because we're looking at margins now in the 25% to 40% range. That's kind of traditional players. I don't know if that's yours, but that's kind of what inference providers are generally. Do they go to 70 as you own more and more of the stuff? I think basically almost no company today has software margins anymore.

[15:50](https://realalpha.app/v/1dwyL6Iz_PU/?t=950) Like what you used to see in like Silicon Valley in the, you know, 10 years ago with this like 90% margins plus, I think that's basically gone. And the reason is actually almost like topological. Like you have this like stack of vendors, right? Like today you have applications, you have models, you have infrastructure, you have cloud, you have NVIDIA. And so when a customer pays a dollar at the top of the stack, it has to flow through a lot of different layers.

[16:12](https://realalpha.app/v/1dwyL6Iz_PU/?t=972) And at every layer, there's gonna be, you know, 20%, 30% gross margins. So I think there's a lot of margin opportunity and there's a lot of opportunity to make a lot of money. I mean, the space is so big, but you're also splitting it across a lot of layers in the stack, so, I mean, you know, like the 90% software margin businesses are gone. I think modal has an opportunity long term to get to 40, 50, 60% margins, but we're never going to get to softer margins.

[16:35](https://realalpha.app/v/1dwyL6Iz_PU/?t=995) What's a wild prediction you have on the future of the inference market that you think many people would disagree with? I think one prediction about the inference market is going to split into a lot of different things because the market got so big in the last. I don't think anyone expected, me included, I don't think anyone expected it to be so big. It's like 10x or 100x bigger than I ever thought.

[16:53](https://realalpha.app/v/1dwyL6Iz_PU/?t=1013) And that to me means there's going to be so much room for specialization, low latency, high latency, like various types of fine tuning and post training. I think there's going to be some wacky new technologies that let you do it with way less resources, new hardware, like all kinds of different things. I'm really sorry. Does that not go against what you just said about the need to own it all and have that breadth of product?

[17:12](https://realalpha.app/v/1dwyL6Iz_PU/?t=1032) If it's so much larger that you need real specialization. Specialization by product. I think it's going to fragment into a lot of different products. Now, we can build a lot of those products and host it all under modal. And that to me is very exciting because there's going to be a lot of different products that companies are going to need. For one product, they're going to need low latency inference.

[17:30](https://realalpha.app/v/1dwyL6Iz_PU/?t=1050) For one product, they're going to need batch inference. For one product, they're going to need sandboxes, which we have. For one product, they're going to do pre-training their own models. Maybe it's certain specialized audio models or whatever. We do a lot of stuff besides LM inference. And so that to me creates a lot of opportunity to basically kind of rethink what a cloud looks like in the era of AI and kind of deconstruct the cloud and rebuild it for this new era of software.

[17:55](https://realalpha.app/v/1dwyL6Iz_PU/?t=1075) Earlier, you said about building capacity with GPUs. Yeah.