Real Alpha
Watch modeSubscribed

The MAD Podcast · 5 min · Oct 22, 2026

Mixture-of-experts models trade training complexity for inference capacity

Pierre Stock

VP of Science, Mistral AI

Mistral's first employee, now VP of Science, who leads model releases including the 1-trillion-parameter Mistral Large 4.

  • Matt TurckHost · General Partner, FirstMark Capital

    Publishes the annual MAD landscape of data and AI; sold TripleHop to Oracle.

About this segment

Pierre explains how Le Chonk activates only part of its trillion parameters for each token, then uses a hospital analogy to describe expert routing. The architecture saves inference compute but demands more data and careful training.

From “Mistral Is SO Back: Inside Le Chonk, the West’s Biggest Open Model”, Oct 22, 2026. Watch the full episode

“The advantage is that at inference time, each token sees 50 billion different parameters, interact with 50 billion different parameters at a time”
Pierre Stockat 28:20 · The MAD Podcast28:20

Takeaways

  1. Each token uses 50 billion active parameters.
  2. MoE models need more training data per active parameter.
  3. Routing must avoid experts that are overloaded or unused.
Watch 4:58 from 27:28

In this episode

Full episode
  1. 46:42RL training depends on calibrated tasks and resilient infrastructure5:40

  2. 27:28Mixture-of-experts models trade training complexity for inference capacity4:58

  3. 1:03:20A model pinged its trainer from inside a sandbox4:01Up next
  4. 5 more moments in the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 27:31

    great context on the uh open source ecosystem great context on you but uh now let's deep dive into the model into the chunk so for everyone listening what is a mixture of experts model which is the core architecture here where chunk as a trillion parameters but only 50 billion work on each token and Why does that beat a normal, dense model? So that's, again, that's something we've been researching at Mistral for years.

  2. 28:07

    We've been watching the first, the very first mixture of experts three years ago that sparked the modern architecture. So really, it's really a beast, like a mixture of experts on this side. There's no other world. That's a beast. Now, what do you gain from it? The advantage is that at inference time, each token sees 50 billion different parameters, interact with 50 billion different parameters at a time, which means that the inference speed, roughly speaking, is the same as a dense 50 B model, right?

  3. 28:44

    But the capacity of the model is like way, way, way larger. In this case, it's 20 times larger. So that's the big advantage. The downside, or actually the thing you have to take care of, the downside is actually that in pre-training, those models are much more data-hungry, so you need to feed them more with data, and with good data. So in other words, you pay a premium at the training side by making the bet that the model is going to be way more used at inference time, which is essentially a good bet to make if you're confident about your models.

  4. 29:20

    So they're super data-hungry, you need a lot of tokens per active parameters. So that's the measure we use to pre-train them. And maybe the second aspect that underlines a bit all the science you have to do, in particular if you're constrained on JQUs as we were, is that it's hard to train them properly. Again, it's a beast, it's a living beast. So you might have problems with dead experts. So part of the model that actually never see are tokens.

  5. 29:48

    You also might have problems with some guidance or information flowing badly in the model, an imbalance on which experts see the most tokens, so it's really something you have to control and develop an entire science around to train them, I guess, very well, but it's worth it in particular on the inference side. And to make sure this is accessible to a broad audience, one analogy I heard about what MOE means in the first place is a hospital and the idea that if you have a patient coming in, you're not going to send the patient to, you know, 50 different doctors.

  6. 30:33

    you're going to send the patient to the doctor that can fix the problem so the orthopedist or or whatever example is that directionally correct it's a great metaphor and i would add that the the routing part so which doctor towards the patient yes so how does how does that work it's actually learned by the model so that's the beauty of it the model learns so we have some routing layers from time to time and these routing layers they actually they have one task is to assign to basically assign each token so each patient to its right specialty and big models like that they have like hundreds of experts and the router has this ability to direct the token to the expert or actually the top two or four experts that it deems the most relevant and the beauty of deep learning is that it actually works

  7. 31:27

    in other words if you want to learn end to end training model is able to run both the routing and both the experts but there are a lot of interactions between that for instance if you need to route tokens to some specialty or patients to some doctors then the doctors they can't learn right because they never see a patient so there's this subtle balance you need to to sustain for training and that's one of the key that's that's a dead expert problem you are alluding to if if if the doctor never sees a patient unassigned doctor or dead experts. And on the contrary, you also don't want to overload a doctor because otherwise the other doctors will be really idled and it will disrupt also the training. So it's a super subtle science as to how to balance the traffic, I would say.

  8. 32:19

    And that's a key part of what kept us busy for the pre-training of ML4.