# Mixture-of-experts models still need all experts in memory

Markdown copy of https://realalpha.app/s/mixture-of-experts-models-still-need-all-experts-in-memory/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Noam Shazeer](https://realalpha.app/builders/noam-shazeer/) · OpenAI; co-author of the Transformer · Designed multi-head attention in the Transformer, and worked on sparse Mixture of Experts, Mesh-TensorFlow and T5.
- Host: [Dwarkesh Patel](https://realalpha.app/people/dwarkesh-patel/) · Host, Dwarkesh Podcast · Co-wrote The Scaling Era, an oral history of AI; named to TIME100 AI.
- Show: Dwarkesh Podcast
- Episode: [Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI](https://realalpha.app/v/v0gjI__RyCY/), Feb 12, 2025
- Moment: 1:44:49 to 1:49:42 of the episode (5 min), playing on https://realalpha.app/s/mixture-of-experts-models-still-need-all-experts-in-memory/; the episode from there: https://realalpha.app/v/v0gjI__RyCY/?t=6289

## Quote

> “it doesn't mean that you don't have to retrieve that memory because really, in order to be efficient, you're **serving at very large batch sizes**.”
> — [Noam Shazeer](https://realalpha.app/builders/noam-shazeer/) at [1:45:50](https://realalpha.app/v/v0gjI__RyCY/?t=6350) · Dwarkesh Podcast

## Takeaways

1. Unused experts still need their memory retrieved and available.
2. Large batches help keep hardware efficiently occupied.
3. Future models may need experts with widely varying costs.

## About this segment

Noam Shazeer explains why mixture-of-experts models must keep their experts available even when each request uses only a few. At inference, large batches spread requests across experts to keep hardware efficient.

From “Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI”, Feb 12, 2025. [Watch the full episode](https://realalpha.app/v/v0gjI__RyCY/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[1:44:52](https://realalpha.app/v/v0gjI__RyCY/?t=6292) One implication is currently if you have a model that has some tens or hundreds of billions of parameters, you can serve it on like a handful of GPUs. In this system where any one query might only make its way through a small fraction of the total parameters, but you need the whole thing sort of loaded into memory, the specific kind of infrastructure that Google has invested in with these TPUs that exist in pods of hundreds or thousands would be immensely valuable, right?

[1:45:28](https://realalpha.app/v/v0gjI__RyCY/?t=6328) I mean, for any sort of even existing mixtures of experts, you want the whole thing in memory. I mean, basically, if you are, I guess there's kind of this misconception running around with like mixture of experts that, okay, the benefit is that, you know, you don't even have to like go through those weights in the model. You know, if some expert is unused, it doesn't mean that you don't have to retrieve that memory because really, in order to be efficient, you're serving at very large batch sizes.

[1:46:03](https://realalpha.app/v/v0gjI__RyCY/?t=6363) Of independent requests. Of independent requests. So it's not really the case that, okay, at this step, you're either looking at this expert or you're not looking at this expert. Because if that were the case, then when you did look at the expert, you would be running it at batch size one, which is massively inefficient. Like you've got modern hardware. the operational intensities are whatever, hundreds.

[1:46:32](https://realalpha.app/v/v0gjI__RyCY/?t=6392) So that's not what's happening. It's that you are looking at all the experts, but you only have to send a small fraction of the batch through each one. Right, but you still have a smaller batch at each expert that goes through. And in order to get kind of reasonable balance, like one of the things that the current models typically do is they have all the experts be roughly the same compute cost. And then you run roughly the same size batches through them in order to sort of propagate the very large batch you're doing at inference time and have good efficiency.

[1:47:08](https://realalpha.app/v/v0gjI__RyCY/?t=6428) But I think, you know, you often in the future might want experts that vary in computational cost by factors of 100 or 1,000. Yeah. Or maybe paths that go for many layers on one case and a single layer or even a skip connection in the other case. And there, I think you're going to want very large batches still, but you're going to want to kind of push things through the model a little bit asynchronously at inference time, which is a little easier than training time.

[1:47:41](https://realalpha.app/v/v0gjI__RyCY/?t=6461) And that's part of kind of one of the things that Pathways was designed to support is you have these components, and the components can be in variable cost, and you kind of can say for this particular example, I want to go through this subset of the model. And for this example, I want to go through this subset of the model and have them kind of, the system kind of orchestrate that. It also would mean that it would take companies of a certain size and sophistication to be able to, like right now, anybody can train a sufficiently small enough model.

[1:48:18](https://realalpha.app/v/v0gjI__RyCY/?t=6498) But if it ends up being the case that this is the best way to train future models, then you would need a company that can basically have a data center serving a single quote-unquote blob or model. So it would be interesting to change in paradigms in that way as well. You definitely want to have at least enough HBM to put your whole model. So depending on the size of your model, most likely that's how much HBM you'd want to have at a minimum.

[1:48:54](https://realalpha.app/v/v0gjI__RyCY/?t=6534) But it also means, I think, you don't necessarily need to grow your entire model footprint to be the size of a data center. You might want it to be a bit below that. And then have potentially many replicated copies of one particular expert that is being used a lot so that you get better load balancing. Right. Interesting. So like this one's being used a lot because we get a lot of math questions. And this one on, you know, maybe it's an expert on Tahitian dance and it is called on really rarely.

[1:49:25](https://realalpha.app/v/v0gjI__RyCY/?t=6565) That one maybe you even page out to DRAM rather than putting it in HBM. But you want the system to kind of figure all this stuff out based on load characteristics. Right now, language models, obviously, you put in language, you get language out. Obviously, it's multimodal.