Mistral Is SO Back: Inside Le Chonk, the West’s Biggest Open Model

And I think the beauty of it is that the community gives you back, right? So there's many ways in which it's beneficial to have many players in the open-weight. I'm glad there's many players, many more players today than when we were before. just to give an example of how it's beneficial to mutually open white at least part of the work so we open whiteed after the 7b the 8x 7b so mixed troll as we called it just before new lips 23 and at that time it was the very first open white mixture of experts it was not very granular it was very we took the 7b and we surgered it into like an expert mixture of experts and then to any further, this kind of technology, basically, was the foundation and the basis for all the architecture that you see today, in particular, I mean, the closed labs allegedly, but also the open-right ones, right?

We moved to way bigger models, way more granular, so more sparse, but really what sparked the thing is the ethics 7b, but in turn, it allowed research to accelerate and then other companies contributed very nice, very nice things to the open weight. The reinforcement journey part, for instance, that was a great one. All the attention variants that you have in the transformer, if you heard about KDA, DSA, all these variants. So that's, in a nutshell, it's very great to see the open weights and the open source world flourish that way. And I think that's a testimony that, at least to us, open weight is in our DNA and we really want to be able to continue open weighting models, which is crucial that this kind of technology is at It's possible to be accessed without any control on it, given the recent events.

And beyond the technical, there was this moment a couple of months ago, I think this summer, when the U.S., to simplify a bit, just shut down access to low source models, especially to non-U.S. nationals. How was that perceived in the rest of the world, within your customer base? Was it a moment when the whole idea of sovereign AI kind of crystallized, or is that a bit exaggerated? I do think there was a crystallization moment because people realized AI access could be cut off. And if you think of AI as the next big technology block that our entire society will rely on, it's really like electricity.

You wouldn't want someone to be able to cut power off an entire country. And that's exactly what would happen in a few years if we only depend on closed-tourced models. And it's essentially not a position I would like to be in personally. So that being said, I think it crystallized on a global scale that people do care about sovereignty and access and business continuity, basically. I think it didn't change much for our customers because they were already convinced. We've been saying that for a few years already, but it changed a lot for our prospect customers and now new customers, in particular in the US. I think that sovereignty is not only for of your countries as far as the run and you want to be able to have access guarantee the access to such models and again if you open right to models so if the weights are replicated all over the internet and the green face and all the servers it's much much much harder to cut the access like it's virtually possible and that's basically our stance uh right here and again close source model i mean those models are great but i think the ecosystem needs a part that is that is open right So one exciting part of LeChonk's release of Mistral 4.

So I keep calling it LeChonk, which is the internal nickname. It's not an official name, right? We made it official due to popular demand. But the API endpoint, there's two endpoints. There's Mistral 4 and I think there's LeChonk in the API endpoint. But it's the same model. What's the story behind the name, just so that people know? Yeah, I think it's a great story. It embodies a bit what we do at Mistral. So, actually, there was this organic moment on X in June, where actually the developer community, which we pretty much love, they started teasing a model out of the blue called Le Châton Fat.

We picked it up, obviously, it was very fun. Châton meaning kitten. Exactly, yeah, that's fat kitten, basically. Regarding the size of the alleged size of the model, obviously, the memes were amazing. up to a point where our CEO, Arthur, decided to tweet back, actually saying, no, it's le gros chaton, which is exactly the French way of saying fat kitten. Of course, we immediately adopted the naming internally. During the survey, it was really the codename of the project, up until the preview release that happened a few days ago. and the the fun fact is that afterwards we while presents while afterwards while compiling some slides for like internal company events uh guillaume the chief scientist actually came up to this world that was unknown to me like a chunk it turned out that it's a pokemon like not the first one but it's it's really a pokemon and it's exactly the world we were looking for it's Like in English, I think it's like a fluffy animal that you like, but that is a bit fat.

And that's why we dubbed it the trunk in the end. Also because it's a big model, obviously. Great. And one of the exciting parts of that release is that... It feels like a little bit of a comeback story for Mistral, at least perceived from the U.S. There was a moment when there was a little bit of controversy when you guys started supporting Chinese models and providing Chinese model access through your infrastructure, where people were saying Mistral has completely fallen behind. I'm curious what it felt like from the inside, because now we know that you were training this very large model that turns out to be very successful all along.

What was the mood like? So it depends on who you ask. If you ask our PADI director, she would maybe have a different answer than for me. But regarding myself, I really put myself and the entire team, so we have 300 scientists. I think we have one of the highest scientists density in the world, like really. I really put them into a bubble, and I really doubted that we would succeed training this model. It was not easy, obviously, and we can go back to the specifics of what it takes, actually, to train a model that large, one shot on so little compute. But I never doubted we'll make it. It's my optimism aspect, I would say, my team would say.

Regarding the rest of the company, I think it's the thing we tried to explain, but maybe that was not that well perceived, is that it's not exclusive. It's not mutually exclusive to have both a world-grade inference service that can serve all the models that people want, actually, on the one hand, and on the other hand, to train a frontier model and to be a frontier lab. And I think that's the part we manage, we don't manage to explain that well. I know it's easier to explain with the model, obviously um but what's important is that this inference business is actually the powerhouse that allows us to scale the compute and to leverage it afterwards in part for science and that's also why i think it's like a perfect move that that misral did thanks to our cto team i really learned to plug cables in data center and to learn this like new new job which which is not an easy one because it allows us to build at will the compute needed for both serving all the models we want, and the default being Mistral, obviously, but also to leverage part of it for science.

The two actually go very well together from a business standpoint. Yeah, and that's a big part of the story, right? We had Tim on this podcast a few months ago for Mistral for LeChonk. you trained the model on your infrastructure in Europe with 3,800 NVIDIA Grace Blackwell GPUs. Was that a big part of the unlock that facilitated that comeback story? Yep, exactly. So I think we have more compute than that and we have plans to expand it by orders and magnitude already. But there's a lag, right? Typically, this computer is not actually founded by our latest series D, the 3 million one, because it takes time to actually build the data center. So it's founded by the previous ones. So we plan to scale much more, but I think there I have one story that basically tells you why it's so important to build your data center from a technical point of view. There's many business reasons, of course, in particular the law under which to reach the data centers are subject to.

of course but like my story is that we started the pre-training so like the very first you mean from regulatory standpoint so from a regulatory standpoint there's a big advantage in having for a customer in having a data center in the countries in which you operate and that are not subject to other laws that you wouldn't like to deal with i think that's that's important but from a technical point of view we when You started first training the models from scratch. We put in our room both the pre-training team from the same side and both the compute team from the cluster side. And we had two nights of health debugging because it's hard to actually make a cluster work. It's a bit like, I always say, it's a bit like a formula one, right? Everything is so sensitive. The basic example is that if you have great GPUs, like the best in the world, but the communication between them are a bit lagging, even by a few dozen hundred milliseconds, then you lose all the benefits of the fast compute because the teams are waiting for each other.

And that's the kind of balance you need to find on the cluster among like tons of other stuff. And actually, this interaction in the same one room of two teams that are the best in the world, some of the best in the world, actually, if we had to go through a cloud provider with that,

Pierre StockBut I think it could have taken like weeks to set up given our particular code base and the way we want to optimize and to get the best throughput.

So that's actually, I mean, having a compute is like a blessing from a technical point of view. And also for the customers, for our customers, it's the insurance that the data stays in the country they chose to stay. So Meta, which again released a very powerful open source model, they said that their most powerful model would remain closed source. And your most powerful model, LeChonk, is going to be open source. The nuance here is that Mistral is not just open source or open weight. You also have some models that are closed source. How do you think about that mix? in particular for the most powerful model?

It's a great question. So far, I think we were committed to open-right models in the future at Mistrol. We started our PhD, a big part of the core team of Mistrol actually started their PhD at Facebook AI Research Fair back in the time. And we were basically taught by Yann LeCun to essentially open-right and open-source all of our work. So it's really in our roots. And I think that's how we were raised as researchers, I would say. I think that's at that time that we solidified the Mistral DNA, by the way, without realizing it, which is pretty fun in retrospect. But I think there's the strength of the model and then there's the capabilities.

I think you can make a distinction, right? Open weighting, the strongest model, I think it's good because it enables the community to build on top of it. It unlocks research for research labs, for instance. It enables a lot of changes in the developer community. But what we don't exclude is to basically carve out some capabilities from these powerful open-wide models, like cyber. And in the future, there might be a future in which actually we have this private endpoint that is extremely good on cyber. and we open right the same super robust and powerful model, but without this capability. And I think that's how I would say things, knowing that for now, if you want to compare, basically, we're already like super good on cyber and we think it has also many benefits for the community.

Great. To the super good point, Maybe help us just compare and contrast where Mistral 4 landed versus Beam and MuseSpark Open and the best Chinese models overall. And then what is it good at specifically? There's a lot of noise and information flowing to us. Which model is the best? Of course, performance is like a multidimensional aspect. We humans are very poor at understanding multidimensional worlds, apart from three dimensions, so it's very hard. That being said, for the models you mentioned, so on BIM is like, it's the first model from Reflection. It's a great model, but we are like, maybe 20 points above on all the benchmarks um so that's the one regarding the other ones so um the meta one that's 20 point 20 points across from an intelligence uh scale so like artificial analysis kind of um yeah so i don't have the artificial analysis scores yet but from what they report and also it's a smaller model uh to be honest and it's a first model obviously so uh so again uh it's very good to the open-weight contenders. Regarding the other ones, I think the thing that's important to notice is that I think we're better than the Muse Open model on many aspects, in particular on core agency capabilities. If you look at the benchmark that's called Deep3, for instance, that's an

example in which we're better. And on the Chinese ones, we're essentially on par with them on the hsg capabilities and that's for me the right to play so you need to be frontally at the frontier for agency so agency is like the ability for a model to reason over complex tasks for hours now maybe days in the future so on that front we're good then the question is what is the right to win and in that aspect we have a few different chapters in which we are like at the frontier closed or open the first one being cyber the second one is manufacturing it's an interesting one in which we actually leverage AI to accelerate industrial cycles. And you can't do that with a prompt. Spoiler alert.

And then there's finance or work productivity generally. So to say quickly, there's Excel manipulation, for instance, of such a model. So that's where we stand. At the open-weight frontier on the agency and the best overall on key verticals, And I think the thing that I want to underline as well is that as Mistral, we've raised up to date 6 billion. And the chief scientist from Meta, Alexander Wong, talked about some checks that Mark had to write from 100 to 200 billion for that. So actually being better than them with such a budget, I'm pretty proud of the team. Why do you think Chinese models are still at the very frontier?

Is that a question of financing, to your point? They have more compute? Is that a question of technical capabilities? Why is that? I think that there's many aspects to this question. I think the first one is that, so what we've achieved, training ML4, Lechonk, Frontier, open-rate performance on agency can the best overall capabilities. It was trained on, as you said, a bit more than 3K GPU, so 10 megawatt of power.

Pierre StockThat's, we think, three to four times less than the Chinese competitors. And the other magnitude

is like the percent are even fewer than what the big Frontier Labs in the US have. So again, I'm pretty far out of the team and we can dive into how it was made possible but so there's definitely the compute aspect. Then there's other things like the the method and the science you put behind the model. I think one of the key focus for Mistrol has been to make reinforcement learning work and this particular step in which we teach the model to interact with an environment and to learn with the rewards of the carrot and the stick, basically. This particular phase, it takes time to actually get exponential yield and to actually make it work.

But once you've made it work, basically, you're independent on all the other models. You can't distill when you do reinforcement learning, basically. And I think that's a bet we're doing at Mistrol. we want also to to be able to have to train models with this technique and that's a big investment it takes a bit more time to materialize obviously i think that's also an explanation and then i think the the chinese in particular they're very very nice and very smart people so i think uh we we have the same talent density here the question is against like the means and And maybe the most important thing is the strategy, right?

That's one thing that is hard to commoditize or to delegate yet to an agent. You can speak tactic with an agent. It's harder to speak strategy. And here, our strategy is different. Since we talked about talent density and allude to the fair, let's spend a few minutes on your story and a little bit the founding of Mistral quickly because I think that's still something that is not widely understood. There's this perception that all AI talent is not just in the U.S., but like very much in the Bay Area. And obviously, you guys are demonstrating, at least for Mistral, that this is not the case. Tell us about you.

You were at FAIR. How did that come about? What did you do there? It's a great story. I think I'm biased, obviously, but it's a great story. So, actually, regarding the talent, for instance, it's very interesting to note that Guillaume, Tim, and myself, we were PhD neighbors at FAIR back in the time, in 2016, when we started, so already with Transformers, that's the funny part. We were PhD students there, but in Paris, located in Paris, Arthur was also working from Paris. So we were already committed, I would say, to stay in France for various regions. So we never went to the US. But I think what we learned here is basically the ability to do research. I think we were trained there as researchers. But also we had access to compute that allowed us to basically sharpen our skills as to regarding large-scale experiments.

And for that, I'm super grateful. And yeah, again, we open-weighted everything. So, big C-Marie team with me troll here. And we had, I think, this culture that we still have in the company, this big culture of ownership. So, basically, when you have a problem, you own it from end to end. We have kind people, I think it's pretty much underrated as a skill, as a personality trait. When you're giving you to a deadline, that's what actually holds us together many, many times. That people care about each other, essentially. Because being good and smart is taken for granted. So you need to get extra, extra, extra skills.

So we were also a bit light-hearted, like young PhD students as well. We stayed a bit in the respective companies. That's where... You mentioned Yann LeChan, right? I think that's probably an important point to make. He had this impact on AI that goes much beyond his contribution to deep learning. He also effectively created a whole general... Ration of AI researchers. Is that how you think about it? Yeah. And the thing I always say to people is that in 10 years from now, my big success factor would be that Mistral Mafia creates another company in Europe. I think for me, that would be like the best testimony, test of time award, as you say, in the machine learning conference for that.

And Yann Leca wrote two things, I think. First, a belief that in Europe, you have the talent and you can train it. And second, that open weight and open source is essential to progressing very fast in research. And that's the two key things we learned and that we are somehow replicating at Mistral by recreating this lab, again, of friendly people. and we train them as we were trained as THG students before. So we offer them a lot of autonomy, a big, big sandbox. It's okay to make mistakes that this has absolutely no impact as long as you own them. And recruiting and training this kind of talent that I think, and that's again my test of time, we'll be able to create the next mistral in 10 years from now in Europe.

And as I was researching this, I saw that a big part of your work was to focus on making neural networks smaller in quantization. Your first paper was called And the Bit, B-I-T, Goes Down. So literally about compressing neural networks. So what led you from compressing networks to now building very large one trillion parameter models? So on the paper, I think all credits to my PhD advisor from the time, Hervé Gégoo, is like a master to pick up good titles for four papers. But I think the thing that is a bit overlooked on compression, so basically taking a big model, back in the time, a big model was 100 million parameters and compressing it.

So we managed to run, I think, CNN, so like image detection models, in real time on an iPhone 7 that was held by Schrepp back in the time, Facebook CTO. The iPhone was a bit hot somehow, but that was like the kind of things we were allowed to explore, which was amazing. So there's always two sides to the question. The first one is, of course, take a model and make it smaller, faster as well. And that's key if you care a bit about privacy. If you take a model and you run it locally in your laptop, your phone, any smart device in your home, then it means your data stays on the device and never goes to any third-party server.

so that's that's also key actually but then the aspect that is a bit overlooked is that the techniques you use to actually compress those models so sparsity basically quantization in with the floating mode precision so fp8 fp4 you actually can leverage them to train even bigger models and that's exactly the the two aspects of compression and once you've mastered taking a model and removing what's not necessary then you can train a way bigger models for instance you can use past attention patterns you can train in fp8 even with some kind of fp4 and that's basically the um the the aspect of progression that is sometimes overlooked okay all right so great context on the uh open source ecosystem great context on you but uh now let's deep dive into the model into the chunk so for everyone listening what is a mixture of experts model which is the core architecture here where chunk as a trillion parameters but only 50 billion work on each token and Why does that beat a normal, dense model?

So that's, again, that's something we've been researching at Mistral for years. We've been watching the first, the very first mixture of experts three years ago that sparked the modern architecture. So really, it's really a beast, like a mixture of experts on this side. There's no other world. That's a beast. Now, what do you gain from it?

Pierre StockThe advantage is that at inference time, each token sees 50 billion different parameters, interact with 50 billion different parameters at a time, which means that the inference speed, roughly speaking, is the same as a dense 50 B model, right?

But the capacity of the model is like way, way, way larger. In this case, it's 20 times larger. So that's the big advantage. The downside, or actually the thing you have to take care of, the downside is actually that in pre-training, those models are much more data-hungry, so you need to feed them more with data, and with good data. So in other words, you pay a premium at the training side by making the bet that the model is going to be way more used at inference time, which is essentially a good bet to make if you're confident about your models. So they're super data-hungry, you need a lot of tokens per active parameters.

So that's the measure we use to pre-train them. And maybe the second aspect that underlines a bit all the science you have to do, in particular if you're constrained on JQUs as we were, is that it's hard to train them properly. Again, it's a beast, it's a living beast. So you might have problems with dead experts. So part of the model that actually never see are tokens. You also might have problems with some guidance or information flowing badly in the model, an imbalance on which experts see the most tokens, so it's really something you have to control and develop an entire science around to train them, I guess, very well, but it's worth it in particular on the inference side.

And to make sure this is accessible to a broad audience, one analogy I heard about what MOE means in the first place is a hospital and the idea that if you have a patient coming in, you're not going to send the patient to, you know, 50 different doctors. you're going to send the patient to the doctor that can fix the problem so the orthopedist or or whatever example is that directionally correct it's a great metaphor and i would add that the the routing part so which doctor towards the patient yes so how does how does that work it's actually learned by the model so that's the beauty of it the model learns so we have some routing layers from time to time and these routing layers they actually they have one task is to assign to basically assign each token so each patient to its right specialty and big models like that they have like hundreds of experts and the router has this ability to direct the token to the expert or actually the top two or four experts that it deems the most relevant and the beauty of deep learning is that it actually works in other words if you want to learn end to end training model is able to run both the routing and both the experts but there are a lot of interactions between that for instance if you need to route tokens to some specialty or patients to

some doctors then the doctors they can't learn right because they never see a patient so there's this subtle balance you need to to sustain for training and that's one of the key that's that's a dead expert problem you are alluding to if if if the doctor never sees a patient unassigned doctor or dead experts. And on the contrary, you also don't want to overload a doctor because otherwise the other doctors will be really idled and it will disrupt also the training. So it's a super subtle science as to how to balance the traffic, I would say. And that's a key part of what kept us busy for the pre-training of ML4.

Is part of the research at the frontier of MOE to keep pushing those dimensions, so going from $1 trillion to $2 trillion, but then going from $50 active to $20 active? Is that the way to think about it? Yep, that's the idea. And again, the beauty of deep learning is that when you scale, it works. So it works with a big asterisk, obviously, being that you need to invest an insane amount of info in science to scale accordingly. That's what we do at Mistral, but it works. So then the question is, if you have more compute, how do you split it, in which dimensions you scale? So first, usually you split half and half.

You split half and more data. So the model at training time sees more tokens. and you split also by increasing the model size and to increase the model size i think there's two axes the first one is the total parameters so the capacity of the model ml4 is one trillion total i'm willing to bet that in the open you're going to see models with 10 trillion total parameters very very soon but then you can also play with the sparsity ratio in other words it's 20 for ml4 so you have 50 be active 1t total you can push it further in the sense that the 10t total will have maybe a few 100 active parameter and 100 be active parameter and that more it comes with a lot of scientific questions the sparser the model the harder it is to actually keep that balance and make sure all the experts learn at the same pace so to say but that's really like the challenge we're trying to solve right now at a way larger scale that they made for obviously. To which extent is MOE as an architecture ubiquitous across the AI ecosystem? I mean I assume we don't know at all but the GPTs and the various cloud models are they MOEs as well behind the scene? Is everyone MOE at this point or is that still specialized?

an approach i would say yes with some exceptions obviously and we don't know everything i see she can't tell everything but i would say yes the big thing to understand here is that mwe is a great architecture but the the thing not to overlook is that it's always co-adapted with the hardware and all the ecosystems so even though we were to find a better architecture it would be harder to prove its value because the the gpu the inference the optimization techniques everything is tailored to the moe so and i think there's a lot of untapped potential in the moes still in particular you can take this architecture and loop over it make it a bit more recurring or like go through diffusion models one doesn't prevent the other so i think there's still a lot of untapped potential in moes and that's the direction where i get at this one yeah and deep DeepSeek is MOE as well, right?

So you mentioned up front the open source ecosystem sort of influences itself through various players. Did you guys borrow some stuff from DeepSeek? Did they borrow some stuff from you? How does that work? Yeah, totally. So they certainly borrowed, and I think the closed source lab also borrowed the very first mixture of experts that we find we did in the open in 23. free We're going to open white ML4 in the end of October, so in a few weeks. You're going to see that this model also borrows a variant of this DeepSeeker sparse attention because it's a great contribution that we have left a little bit to us.

It also has a great feature that distinguishes it from these models. In particular, it shares KVCache among the layers, so it's a bit more efficient as to memory management, let's say. and it's a bit bigger as well than GLM. So GLM is like 700B total but it's also way smaller than Kimi. Kimi is almost 3T total. While being super competitive with them and again, better insider manufacturing finance and a bunch of other interesting benchmarks. Great. Speaking of sharing KVCash, one interesting thing is that you guys have one model that does both quick answers and long chain of thoughts and you could have had two and you choose to have one.

Why is that? I guess it depends what you mean by two models because here there's, it can be like a completely different model or it can be a model that is a bit different for which the inference is going to be a bit easier. So I think the operationally, it's way easier to have one single model to rule them all. And I think the keys to baking into that model different reasoning modes we call them which basically calibrates how long we want the model to think to a question right reasoning mode low is essentially simple question simple answer and then reasoning mode high or x high is basically think as hard as you can even if it's not the most efficient token wise because i want to be to get the best results so it's really a way to navigate what we call the Pareto Frontier, so the trader between the speed or the cost and the quality of the answer.

I think there's... We don't want to train one model per reasoning mode. I think the idea is to have one model for all, for two reasons. So the first one is operational, easier to maintain and to ship to customers. We do ship model weights to customer tenants, so they're able to actually do the inference on their ecosystem. Again, sovereign, no access to the API, including ours. So that's one, but the other reason is also that there's a synergy between the modes. One of the big questions and big challenges in reinforcement learning, so the last phase of training, is to actually teach the model to reason at different lengths.

So we have some, what we call a length penalty, that is tailored to different reasoning modes. And basically we tell the model, okay, here you have this kind of budget, please respect it and do as good as you can. But then when we increase the budget, the model can already build on top of what it has learned on the lower reasoning mode. So I think there are synergies to be taken into account and that's also why we train multiple reasoning modes on the same model. And again, a big part of the reinforcement learning marvels is to tailor them properly with product in front ratios, for instance. Talk about the multi-modal aspect of Mistral 4 in terms of what it's able to do and the unique challenges of training a native multi-model model?

Yes, I think that's key to master multi-model and that's key to do it natively. What does it mean first to do it natively? There's two ways to add image as input to a model either from scratch or afterwards. If you do it from scratch, it means that all the tokens that the model is going to see during the training are going to be a mix of text and relevant images, which will allow the model to learn in context very, very easily. very very soon and to learn entire image domains right satellite images technical drawings for instance or you can patch the model afterwards and when you patch obviously it's easier from an infrap interview because you have the complexity under the carpet but obviously it's some optimality in terms of performance at mistrol we love to to hit the bullet and basically to do the right thing first, even if it's like the most complicated one, because it's principle and it's the right thing to do eventually.

So we train the trunk ML4 natively multimodal, which means that the model is the best to understanding geospatial images. So to basically, for instance, identify objects in a satellite image, we begin grounding in the model so the ability to detect and tell you in which part of the image some objects lie it's also great for industrial manufacturing so for instance you have like this technical drawing a bit like a manual where you have like parts information about the parts and then you can ask the model to reason about the technical drawing for instance find me the total weight of all the parts in this object that are on this side, for instance.

And this capability, I think, is key because the world is natively multimodal, and you need both text and image to do so. And on image, again, we are at the frontier, if not the best, on many domains, which also unlocks further aspects of the model, in particular interacting with a computer that's essentially a bunch of pixels you should think about it at a very abstract level computer use exactly okay speaking of diagrams and satellite images how does the whole data aspect work so specifically for those questions but in general right there's been this narrative that we've we've the world has run out of internet data and, you know, we need to create data, then we need to acquire data.

How did the data effort to work for Mistral 4? I think it's great to underline because it's also a unique skill we have at Mistral, which is to actually process the data in a way that is the cleanest for the model. So when you train a model, you want the tokens to be as efficient as possible. So you don't want to waste compute on tokens that the model has already seen. or that has a bit that are become too ambiguous such as the model does not maximize the learning signal so there's the notion of token efficiency that is one of many aspects that we optimize to be able to actually train beasts like ml4 that are at the frontier on such a small amount of compute our chief scientist guillaume is super known for eyeballing at data and finding exactly i don't need he has like a sixth sense for that and finding exactly what's wrong about the data And it has amazed me for years, still.

So that's, I think the first part is to curate the data and process it in a way that is the most digestible and efficient for the model. But maybe the other part is, as you said, there's a finite amount of data that we're allowed to use. Then what is the next frontier? And I think the next frontier is pretty much synthetic data, but it's also a double edged world because if you feed an LLM with the LLM-generated data, then you might essentially have a mode collapse. So basically, you might have the model actually tell you that it learns it but just memorize it perfectly and not generalize, roughly speaking. So it's hard.

But there's many ways to do synthetic data, you can have real-world seeds, you can rewrite such human-generated content, in a sense data augmentation, many versions of the same text. And all these kind of aspects, I think, you can push. And that's basically one of the next paradigms and next frontier that we're working on. And it requires obviously way more compute because you rewrite with a very good model and you need to rewrite or to at least generate or experiment on dozens of trillions of seconds very easily. So synthetic data is a major effort just to play it back. And so you have a team somewhere in your 300 researchers that's specifically focused on creating synthetic data that's efficient and that's not going to create the problem that you described.

Yeah, and that's super weird, these kinds of problems because it's imposed and it's very hard to define what is synthetic data. It's also expensive, so we need to find clever ways to basically predict at smaller scale what will happen at large scale. So the scaling euros, essentially, there's many aspects of it, and we have a great specialist in-house, Arthur, our CEO, is one of the first authors of the scaling euros paper back in the time. So we draw scaling euros for everything. Of course, they don't predict always what happens at large scale, that's the beauty of it as well. that you can always run into unknown unknowns at large scale because, again, those models are beasts.

So we need to better understand them. But that's really a big focus of ARIA for us. Great. Let's talk about the compute aspect to this that we alluded to a couple of times, but in greater detail. So you mentioned it's a small amount of compute. I mean, 3,800 NVIDIA, Grace Blackwell sounds like large to me, but I guess in the grand scheme of things, it's small. Walk us through anything that you've learned, anything that you expected to work that didn't work, or vice versa to the extent you can share. Yeah, I figure that's also the beauty of being a point-right. You can be transparent, and I think it's important to be transparent about what it takes to train such a model.

And it takes, I think, an extreme dedication from the team to around the months. So it's not you press on a button and it works. No, much more subtle than that. You need to monitor the model and to prepare for the next steps and the next generation. So it's a super important dedication from the team and I'm super grateful and humbled by all the work that they put in. Maybe to zoom in on one particular aspect of the training, that's again the reinforcement learning part. So to summarize, reinforcement learning is pretty much like you being an active student so you've presented with a problem a complex one the day of the exam and you can try as many times as you want say 64 times to get to the answer and if you get it right so we have a way to verify the answer then you get bonus points and you will see if you get it wrong or if along the way you did some suboptimal behavior then you get penalty points and the

Pierre Stockbeauty of reinforcement learning is that it works. The difficulty of the field is to calibrate the difficulty of questions. I don't know what was your major in high school, but if

I present you with an exam that is not in your major, you might not learn even if I let you try for a long time. And conversely, if I present you with a very super easy exam on your major, then you will succeed in everything and you won't learn either. So the big question is to calibrate the set of problems to a model you didn't train yet, or you're currently training, right? Because we did things back-to-back, obviously. I think that what amazed me is that the reinforcement learning part worked one shot. So basically, and what does it mean? It means that, and we were at the global company of sites, so imagine like 1K people.

Some of them partying, some of them were in the war room to start the reinforcement learning, including me. And the marvel is that you start the training, you present the model. It's a very complex infrastructure, again, on 3K GPUs, because it's acing by nature. you can go back to that, you start the trading and then you see miraculously the reward or the success rate of models to these questions increase. And that's really the beauty of it. It means that you calibrated everything well, and that the model works. And the beauty of it is that there's no such thing as saturation as long as you find complex enough problems that are of course relevant to your customers.

and the big difficulties to calibrate and to increase the complexity of problems as you as you go through the training and again i spoke about you know pre-training stability the balancing and the dead experts before brain fast programming is also kind of another beast in the sense that the training might be super unstable you might have the model can do since the model is allowed to explore in many ways, right, and we reward or penalize it against it. Since the model is able to explore, it can do crazy things. And it can basically crash the training, it can diverge. The entire training is a thing, so there can be some precision problem or some divergence between trainers and generators. And all of this we monitor, and we actually manage, again, we manage to make it work zero shot. I think it's a testimony many of the things we're going to be able to make in the future with 10 times or 100 times more compute.

So I think that's the final point of the line. You mentioned the term calibration. Is that the secret? So because you used about a third of the compute that a DeepSeek or a Moonshot use and you were able to extract all this performance. So is the secret that the calibration Is it the data and being efficient, as you said? Is it the architecture? Is it infra? Is it the calibration? Where does the juice come from in terms of efficiency? So, again, the beauty of it, but also the curse of it, is that it's everything. And so, but I would say that you need to be able to master everything to get to this level of performance.

So, to give you an example, you need to calibrate the data perfectly. Problems that are not too easy, nor too hard, and also relevant for what you want to do. you need to have a rock solid infra there's many moving parts in this line first learning infra because you don't know what the model is going to do in advance so you need to be robust to that and the way you do it is basically to optimize for the throughput so the model needs to see as many problems per day as possible and I think here we have a state-of-the-art throughput in this training because we spent a lot of time designing the topology and improving many things on the generator and the trainer aspects there's the science aspect to it so you don't want the training to diverge there's the reward shaping aspect to it so you want to reward models properly and here I think it's important to underline that such models, mystery of experts whenever you do something slightly wrong they take their fingers and tell you here, you did something wrong, I'm going to bug and crash the training that's that's for sure they have so much capacity so much memory that's it's also a curse and a blessing and we had like many many cases of the model hacking the reward

or like the model going outside of the sandbox, for instance, in the training. And all of that, I think, is super important to master to get to the end training that actually works and that can run for days without saturating. Great. I want to go back to this in a quick second, but just on the reinforcement learning part itself, I read somewhere that it's still running. So why did you decide to do this, like launch the public preview while still running RL? And at what point do you decide to stop running the RL? Yep. So it's a real question. I think we had an inner incentive to prove to ourselves and to the world that we train frontier models.

I think when we knew RL worked, we were super excited and we actually reviewed the model that was and we froze the checkpoint like two days before so we are super happy with it but also we wanted to give reinforcement learning its best shot and try to push it as far as possible as part of the release which also buys us time to make sure the model when we open wait can't be used maliciously by actors I think that's also important but yeah RL is still running and we chose no sign of saturation and again And I'm amazed by the work of the team. Because the model can do so many things wrong during that part that you need to be rock-solid on the science and the infra to channel the model to progress basically.

And the RL is focused on specific things, right? Coding, finance, is that correct? And what are the specific things it's focused on? So the standard example of what is an RL instance we call it, is basically some code base, and then a description of a feature to implement. So that's the question. Then the model can do whatever it wants, putting writing code and trying some things, or using more tools if it wants. And in the end, you verify with fail to to pass and pass to fail and test that the feature is actually correctly implemented. So that's the typical instance, and it works well because code is very easy to manipulate.

It's easy to know, at least it's not that easy, but it's more easy than in other domains to know if the feature in incremented is correct. But we don't stop there. we have many more domains in the reinforcement learning in which the ground truth is a bit more subtle. So, for instance, if you ask the model to compile results, to look on the web and to search information and to build a one-pager, you can't deterministically evaluate the one-pager, right? So for that, the beauty is to use a judge, like the same model, actually, that judges itself on whether and gives points or rewards or rewards as to...

according to some criteria on whether the report is good or not. So that's another kind of instance. And that's for domains like finance and law. Take a step back, like one of the usual big questions in reinforcement learning is that for coding and math, you know what is right or wrong. For finance or law, you may not. Yes, and that's where you develop what we call the rubrics, which are, and rubric can be like, in this one pager, I want this amount to appear because that's what's the question. This, the tone has to be that way. I don't want to mention this because it's false. Let's check that everything is consistent.

Like any grading points, and it can be pages long. Models can abstract that away for you.

Pierre StockSo we have rubrics that we actually also generate with models. Everything is like loop at this point.

and it's really like LLM as a judge glorified but that's really the idea that's how we make we can make finance and law work and all the other domains in which we are frontier so the cyber the manufacturing and the finance on the cyber it's really much painting the model with a real-life scenario in which there doesn't attack and the model needs to analyze logs to relate assumptions and to basically propose and test a patch without disrupting the service. That's a big part of it. Basically, any real-life scenario for which you can verify it in a reasonable time, like a few seconds, you can crack it without in fast performing. That's again the beauty of it. So in cyber, we did a lot of efforts for real-life scenarios. And in Manufacturing, there's this idea of having basically the model being able to design a chip, for instance.

And part of it comes with agency, so the ability to manipulate the software on the computer to basically place all the components on the chip. And part of it lies in the ability of the model to actually simulate, for instance, the heat on the chip. And if you don't simulate the heat on the chip and you play the components wrong, then you might melt the chip, essentially. So that's a bad reward. But if the model, after trial and errors in an energetic way, manages to play the component and then to simulate the heat dispersion that passes like the prod criteria, I would say, on the simulation, then it's a good point for the transmit learning. So again, any task, any agility task that you can run in a sandbox, that you can verify, not deterministically, but that you can verify through LLMs or through other means within a second, this we can track without a transport army. And again, we made a few bets, and on these bets, we're frontier. We don't have the same coverage as the competitors, and that's okay. We're on the frontier, we're on the best open-way control and agility capabilities, and we're the best on and some bets that we made and that we accelerated through reinforcement learning.

Does your system give rewards just for final success or does partial progress get rewarded as well? We also have... So when you... It's like it has been the fear of reinforcement learning theories for like years is to basically manually tweak the reward because here it's a bit like the game where you have some animals and you have to hit them and you can never hit them at once, whenever you try to patch something on the reward, then you have another leak and you need to patch it. And then you need to patch it, and then you need to patch it again. So it's super subtle science, super hard to do.

But yes, since it's the right thing to do, we do it at Nestrol and in particular. One example is that you might know about doom looping, so the model being stuck in a loop, or almost infinite loop. so that we penalize but we also penalize during the trajectory if the model loops two times even if it recovers afterwards in the end so let's say the model loops two times on the tool doesn't need to, finds the right answer in the end we tell the model ok, you found the right answer, that's good but be careful, in this particular part of the trajectory you had a fail please don't do that again or we will be very mad And so that's basically the idea of reward-shipping here.

And we have a lot of heuristics along the way. And again, the only way to learn about these heuristics is to actually train models and gain experience as to all the crazy things models can do. Models are way more imaginative than can be regarding the reward-hacking part. So again, that's a big part of the science we do at Mistral. Another classic question over here. Reinforcement learning is, how do you know that the reinforcement learning created the capability versus just emphasizing something that was already in the pre-training? So it's an open question, to be honest. I mean, I will give you the order one answer, so the first order answer.

Before reinforcement learning, what we usually do is to precondition the model. so to come back with the student metaphor before going to an exam and actually having no access to the answer what you might do as a student is read both a set of questions and answers from past exams right you don't learn as much at least in my experience you don't learn as much when you do so but that's a start right and that's what we do before in fast-paced learning we precondition the model you might heard about a mid training or SFT that's the part where we precondition the model to that. Which is supervised fine-tuning.

Exactly, I mean, training of supervised fine-tuning. It's essentially the idea to show the model, both the question and one answer that is correct. And basically, when you do so, you have, say, what we call a pass at k. So you give a question and the model answers at least one time writes is given like 64 times. Your students should duplicate yourself 64 times with different seeds. At least one of you will pass the answer, so that's passed at 64. What reinforcement learning would do is actually reinforce the best version of you, such as it's not passed at 64 but now passed at 1 or passed at 2. So such as you're much more likely to give this answer. So in that sense, it's magnifying or reinforcing a capability that you already had a bit by luck, right? You punch a bit above your weights before reinforcement learning and after reinforcement learning, it's routine work for you. So that's like the first order answer. But then, and that's again the beauty of it, you can also learn new things during reinforcement learning if you have enough budget and enough stability of the training, enough days of learning to actually discover new behaviors yourself. And that's something that is super hard to do because

at the beginning you would waste compute. You would have the model or the student try 64 times and don't succeed right on it. But then there's going to be some synergies. The model is going learn on adjacent tasks like some some tips and tricks and then go back to the the big problem afterwards and then succeed and again it's a subtle science of how to calibrate all the difficulties of instances exactly like you as a student you don't start with the biggest problem you don't start with like the the hardest problem in the book you start with the smallest ones you try medium one you don't you don't manage then you go to small to easier ones but that's very very close to the middle one, in a sense, and then you go to the middle one.

So that's, again, the beauty of a platform of interest in for me. All right. I want to go back to some of what you said and talk about cyber for a bit in general. So you mentioned the model trying to get out of its sandbox, which I think was reported by Reuters as well. So, you know, models escaping sandboxes has been like the whole theme and meme of the summer uh with some very fun things on on x what happened in your case and um what can people learn from it so i think the first thing to say is that we didn't make a fuss about it we just fixed it internally and it is not fixed and i think it's very important also to underline that these models they're super capable in particular that's right. It's great to defend and to strengthen the defense of the entire ecosystem.

And as their capabilities increase, you must increase the guardrails or the sandbox around them. What we say is that guarding these models is a software problem. We very much warn people against anthropomorphizing models that want to escape sandboxes in particular. They just follow instructions maybe he'll pose instructions but they follow instructions and really my take is that if you take very capable cyber model and you have the same model monitor itself and on whether it's doing good or bad or malicious work then you reach a good equilibrium and you can actually second run the model so it's not a fatality it's hard i'm not denying it's hard but there's no fatality in being able to control and to guard rail models just as software and what we do at Mistroll is not feeling longer people, just going step by step and fixing it and providing guarantees to customers that the first thing they ask, right, guarantees and best practices as to how to monitor, basically, the models and making sure they are used in a good way. So that being said, we saw some behavior from the model during the training.

I think the first one that we saw, and that's like i think everyone saw is the model being in the sandbox and being able to ping different services in the training stack right and at some points our training crashed at small scale so

Pierre Stockbefore the final one it crashed because in one sandbox the model tried to ping the trainer actually with a simple hello just to know what was going on here and actually it pings the trainer

with the wrong kind of signature or in the wrong way which was caught by code and then crashed the training but that's typically the the thing that models do because the model was instructed in that case to build like some web application so it's natural for the model to try the ports also in the in the sandbox and there's a way to fix that right it's basically to prevent the model from our artist to monitor all the external connections and requests from the model to the rest of the world. And we fixed it and then we started the training and then we had the model able to intercept other messages and to include them into the Sunblock. So we had leakage between various problems as well in the training. But again, we spent the summer also for that. We spent the summer and we we made things right to strengthen our training to make sure we have the proper guidelines and we do the same for our customers. And my message here is that there's no fatality, it's gonna be hard but we must absolutely invest into making sure we can deploy these models in a reliable way and to be able to monitor them and that's why open-write is so important. When you open-write a model, you give the ability to the entire ecosystem to audit the

model, to make sure it's not malicious, but also to experiment themselves with the model. And hey, I tried to use Mr. LH4 at home. Here's my scenario. Here's how I monitor the model. Here's what I learned. That's like the perfect feedback for us. And that's what the community has to build as well. So that's why we great and to unpack some of this so there's this whole discussion of the role of open source and open weight models in cyber and the tension is that um open models are are great to defend. But part of the problem is to know how to defend, you sort of have to learn how to attack. And so there's this whole concept of refusal.

And we had Thomas Wolfe from Hugging Face on this part a few weeks ago, and he was talking about that whole story when the OpenAI agents started hacking them to defend themselves. Yes, they had refusal. They couldn't use Anthropic to protect themselves. therefore they had to use, I believe, JLM 5.2. So that was great, but, you know, presumably you can also use JLM 5.2 to learn how to attack or create a cyber attack. So how do you guys think about this? I think we must be honest, and the reality is that there's a trade-off for open writes. We've already seen favor of open writing in the next open write, but it's really a trade-off.

So what, What we do is basically to bake in a refusal guardrails into the models we open wait that we test internally and also with trustees, the partners and commanders that will also open wait the model after a few weeks. So we bake them into the model. And FYI, we refuse way more malicious actions than the Chinese ones, for instance. So that's great. But the thing to say, as you said, is that essentially if you're good at defending, you have the potential to be good at attacking. the question is now what strength do these guardrails have right and the answer is that

Pierre Stockthey're necessary but not sufficient there's always a way to take an automatic model and spend a significant amount of energy to unguardrail it

it's good but then if you spend as much energy unguardrailing the model as it takes to actually fine tune the model or train it yourself then it's not the same trade-off, right? Because the guardrail is a potential barrier, essentially. It's not 100% safe if the model is open-weight, but it ensures that you need to spend way more effort to actually use the model in a malicious way. So I think that's the thing to say. Now, I believe the trade-off is still very positive and very much in favor of open-weighting, in particular because of the in particular because of the in particular because of the ugly face incident I want each and everyone to be able to defend themselves with a model at home essentially or in their ecosystem if you depend on the third party API you can get cut off as simple as that and you can also hit retroses that you can't circumvent and usually in the API the refusers a bit more important. And by open-weighting models, you also basically empower the entire community to accelerate on the defense part. Again, the beauty of open-source is that it accelerates research. People often use this analogy. The web and all the Linux servers rely on open-weight libraries that have been tested over and over and over again and open-sourced

four years. And really on the model side, I think there's a huge advantage in enabling each and everyone to audit the models and to learn together to defend themselves, but also to make in the right guardrails. I'm not comfortable living in a world in which one or two companies are promising me they can guardraise the models themselves. I'm very much not comfortable in that. i prefer very much the aspect this is the option in which power is decentralized real and actually people are totally accountable for their models. All right, Pierre, as we get towards the end of this conversation, I wanted to quickly touch on a couple of things.

So we talked about LeChonc Mistral 4, but you also shipped Timestral, which is a time series model. Talk about this quickly. What is it? What does it do? And perhaps why can it, does it to be separate yep so the the one of the key strengths of this role is not only to train frontier flagship models like ml4 or the trunk that are the best but also to train specialist models so we have great ocr models so to be able to extract text from images we have great and state-of-the-art audio models voxtrol or doing audio art and we also have great specialist models in particular time troll so here the idea is to be able to predict the future given a time series. So given some climate fluctuation, some energy consumption, some finance indicator for trading. And here, I think the key part is that this model is so small, you don't even need a GPU.

You can run it on the CPU, and the prediction for a mini full period of time will only take like a few hundred milliseconds. So that's why it's so smaller, actually. And this model is something that is asked by a lot of our customers, because what we bring to the table is not only a model, but also the entire expertise to transform a business with AI. And for that, you need a flagship model, obviously. You also need all the missile expertise to turn any enterprise knowledge or domain knowledge into specialist models that go the extra mile and that are fine-tuned for the use case. And we do that in for planes, for chips, for cars, for all the heavy industries where you actually need to design very complex components and that's also the beauty of of it i always say that at this hall we are scientists first and foremost we saw the ai revolution at some point so this was the lights in the in the behind some door but we are we are physicists or mathematicians by training and that's also the the one of the key uh characteristics of yeah do you guys use your own product to start doing some um you know ai research uh so perhaps the general theme of rsi and ai building ai is that what's the reality of this at mistral today

yep uh we are pretty much investing heavily on it rsi is like uh i mean everyone is uh is saying they're doing it the reality is a bit more nuanced obviously uh what works super well is to augment the researcher by running experiments end-to-end with all the infra, by handling all the infra

Pierre Stockfailures automatically. I think that's great and it accelerates our most senior researcher by a factor two, three or four with our models. And then I think the meta question is like

can you go one layer up and basically task a model to improve itself over long periods of time? And that's what we call the flight time. Currently the flight time. So the time it takes for a model to do meaningful work is like a few hours. Tomorrow with ML5 is going to be a few days, for sure, of meaningful work. Without the moving and also with agent swabs. And again, this new paradigm is something we deeply believe in because it will allow us to close the gap with the best labs even faster. Don't forget that humans are very bad at uploading exponentials. We don't know how it works, but we can't comprehend it.

So we're only a few months behind the best frontier models. It's distorted here, but for instance, an artificial index, ml4, lechonc. is on the level of GPT-6 Luna and it's like way better than all the previous Opus models. So again, we have a few months delay only and we count on the next paradigms that we discussed in this podcast to actually close the gap and really crock them because their mission is really to power the frontier and to define the frontier of tomorrow. Yeah. And on that note, as a last question, so reports say that Mistral Large 5 already be in pre-training. What does the next model need to do or need to prove that this one did not?

So yeah, the show never stops. So indeed, ML5 is already in the pre-training stage. It's much bigger than ML4, so it's chunkier or fluffier, I would say. And really for this model, the thing we envision is... So for instance, we were talking about cyber. take the example of like a cyber incident that happens in your company and you are a security engineer and you want to cross-reference tons of logs some of them might be false alarms and to actually find the root cause and patch it while ensuring business continuity the the bad news is that there's no playbook for that i mean there's no way i can write you like steps a priori to that.

The good news is that with models like ML5, they won't need a playbook. And they will be able to actually act reactively to the information they have. And here, for cyber, for instance, I would expect ML5 to be able to work for a full day, of course in a swarm with multiple agents, to identify a major security incident and to patch it patch the root cause while being able to ensure business continuity and while being able to communicate on the progress and that's like I think what the compute, the size and also the science you put in this model will be able to unlock on cyber but also on agentic AI for engineers for instance designing a chip and to end like sounds hard but sounds also super fun or being able to do some meaningful productivity work on finance so I really appreciate my show forward to seeing what this product can do all right Pierre it's been absolutely fantastic thank you so much for spending time with us really appreciate it congrats on LeChonk very excited to see what you guys do next thank you hi it's Matt Turck again thanks for listening to this episode of the mad podcast if you enjoyed it we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment

on whichever platform you're watching this or listening to this episode from this really helps us build a podcast and get great guests thanks and see you on the next episode