Mistral trains domain skills with tests, rubrics and simulations

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 53:39

   And the RL is focused on specific things, right? Coding, finance, is that correct? And what are the specific things it's focused on? So the standard example of what is an RL instance we call it, is basically some code base, and then a description of a feature to implement. So that's the question. Then the model can do whatever it wants, putting writing code and trying some things, or using more tools if it wants. And in the end, you verify with fail to to pass and pass to fail and test that the feature is actually correctly implemented.

2. 54:22

   So that's the typical instance, and it works well because code is very easy to manipulate. It's easy to know, at least it's not that easy, but it's more easy than in other domains to know if the feature in incremented is correct. But we don't stop there. we have many more domains in the reinforcement learning in which the ground truth is a bit more subtle. So, for instance, if you ask the model to compile results, to look on the web and to search information and to build a one-pager, you can't deterministically evaluate the one-pager, right?

3. 54:59

   So for that, the beauty is to use a judge, like the same model, actually, that judges itself on whether and gives points or rewards or rewards as to... according to some criteria on whether the report is good or not. So that's another kind of instance. And that's for domains like finance and law. Take a step back, like one of the usual big questions in reinforcement learning is that for coding and math, you know what is right or wrong.

4. 55:25

   For finance or law, you may not. Yes, and that's where you develop what we call the rubrics, which are, and rubric can be like, in this one pager, I want this amount to appear because that's what's the question. This, the tone has to be that way. I don't want to mention this because it's false. Let's check that everything is consistent. Like any grading points, and it can be pages long. Models can abstract that away for you.

5. 55:52

   So we have rubrics that we actually also generate with models. Everything is like loop at this point. and it's really like LLM as a judge glorified but that's really the idea that's how we make we can make finance and law work and all the other domains in which we are frontier so the cyber the manufacturing and the finance on the cyber it's really much painting the model with a real-life scenario in which there doesn't attack and the model needs to analyze logs to relate assumptions and to basically propose and test a patch without disrupting the service. That's a big part of it. Basically, any real-life scenario for which you can verify it in a reasonable time, like a few seconds, you can crack it without in fast performing. That's again the beauty of it. So in cyber, we did a lot of efforts for real-life scenarios. And in

6. 56:48

   Manufacturing, there's this idea of having basically the model being able to design a chip, for instance. And part of it comes with agency, so the ability to manipulate the software on the computer to basically place all the components on the chip. And part of it lies in the ability of the model to actually simulate, for instance, the heat on the chip. And if you don't simulate the heat on the chip and you play the components wrong, then you might melt the chip, essentially. So that's a bad reward.

7. 57:23

   But if the model, after trial and errors in an energetic way, manages to play the component and then to simulate the heat dispersion that passes like the prod criteria, I would say, on the simulation, then it's a good point for the transmit learning. So again, any task, any agility task that you can run in a sandbox, that you can verify, not deterministically, but that you can verify through LLMs or through other means within a second, this we can track without a transport army. And again, we made a few bets, and on these bets, we're frontier. We don't have the same coverage as the competitors, and that's okay. We're on the frontier, we're on the best open-way control and agility capabilities, and we're the best on and some bets that we made and that we accelerated through reinforcement learning.

8. 58:13

   Does your system give rewards just for final success