RL training depends on calibrated tasks and resilient infrastructure

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 46:45

   the reinforcement learning part. So to summarize, reinforcement learning is pretty much like you being an active student so you've presented with a problem a complex one the day of the exam and you can try as many times as you want say 64 times to get to the answer and if you get it right so we have a way to verify the answer then you get bonus points and you will see if you get it wrong or if along the way you did some suboptimal behavior then you get penalty points and the beauty of reinforcement learning is that it works. The difficulty of the field is to calibrate the difficulty of questions. I don't know what was your major in high school, but if I present you with an exam that is not in your major, you might not learn even if I let you try for a long time. And conversely, if I present you with a very super easy exam

2. 47:44

   on your major, then you will succeed in everything and you won't learn either. So the big question is to calibrate the set of problems to a model you didn't train yet, or you're currently training, right? Because we did things back-to-back, obviously. I think that what amazed me is that the reinforcement learning part worked one shot. So basically, and what does it mean? It means that, and we were at the global company of sites, so imagine like 1K people.

3. 48:13

   Some of them partying, some of them were in the war room to start the reinforcement learning, including me. And the marvel is that you start the training, you present the model. It's a very complex infrastructure, again, on 3K GPUs, because it's acing by nature. you can go back to that, you start the trading and then you see miraculously the reward or the success rate of models to these questions increase.

4. 48:39

   And that's really the beauty of it. It means that you calibrated everything well, and that the model works. And the beauty of it is that there's no such thing as saturation as long as you find complex enough problems that are of course relevant to your customers. and the big difficulties to calibrate and to increase the complexity of problems as you as you go through the training and again i spoke about you know pre-training stability the balancing and the dead experts before brain fast programming is also kind of another beast in the sense that the training might be super unstable you might have the model can do since the model is allowed to explore in many ways, right, and we reward or penalize it against it. Since the model is able to explore, it can do crazy things. And it can basically

5. 49:35

   crash the training, it can diverge. The entire training is a thing, so there can be some precision problem or some divergence between trainers and generators. And all of this we monitor, and we actually manage, again, we manage to make it work zero shot. I think it's a testimony many of the things we're going to be able to make in the future with 10 times or 100 times more compute. So I think that's the final point of the line.

6. 49:59

   You mentioned the term calibration. Is that the secret? So because you used about a third of the compute that a DeepSeq or a Moonshot use and you were able to extract all this performance. So is the secret that the calibration Is it the data and being efficient, as you said? Is it the architecture? Is it infra? Is it the calibration? Where does the juice come from in terms of efficiency? So, again, the beauty of it, but also the curse of it, is that it's everything.

7. 50:35

   And so, but I would say that you need to be able to master everything to get to this level of performance. So, to give you an example, you need to calibrate the data perfectly. Problems that are not too easy, nor too hard, and also relevant for what you want to do. you need to have a rock solid infra there's many moving parts in this line first learning infra because you don't know what the model is going to do in advance so you need to be robust to that and the way you do it is basically to optimize for the throughput so the model needs to see as many problems per day as possible and I think here we have a state-of-the-art throughput in this training because we spent a lot of time designing the topology and improving many things on the generator and the trainer aspects there's the science aspect to it

8. 51:26

   so you don't want the training to diverge there's the reward shaping aspect to it so you want to reward models properly and here I think it's important to underline that such models, mystery of experts whenever you do something slightly wrong they take their fingers and tell you here, you did something wrong, I'm going to bug and crash the training that's that's for sure they have so much capacity so much memory that's it's also a curse and a blessing and we had like many many cases of the model hacking the reward or like the model going outside of the sandbox, for instance, in the training.

9. 52:07

   And all of that, I think, is super important to master to get to the end training that actually works and that can run for days without saturating. Great. I want to go back to this in a quick second,