RLCD required TypeSafe to build its own training data

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 37:46

   And that's kind of where calibration comes in. And the models are still not very good at that. Okay, great. All right. So RLHF kind of creates the problem. RLVR doesn't help us. So, therefore, you guys had to invent a new post-training method that you call RL-Steady. Is that correct? And if so, just explain as much as you can share what it is. I don't know if I would call it a post-training method because when we made RLHF, the concept of post-training did not exist.

2. 38:22

   and you could call rlhf a fine tuning method but that undersells how much change you did to the whole process so like no one would just call it fine tuning even though maybe there's some technically correct way where like that's 80 accurate so i wouldn't necessarily call what we're doing post training um i think we do a lot of weird stuff um i think that rl tends to be again an implementation detail of the whole task.

3. 38:50

   You know, for RLHF, the task was not just doing RL and PPO and whatever. It's about figuring out, here's what the inputs and outputs should look like, and here's how you instruct people to do that, and here's what our North Star is, and here's how we balance the data sets to get a really good model for this thing. And for RLVR, you have similar stuff. RLVR is nothing without, it's like latent reasoning, you know, it's reasoning chains.

4. 39:13

   and then this enables you to have like a new mixture of data sets how you construct the data sets or even what difficulty of data set you want completely changes with your base models and shape and all of that so uh the rl tends to be like us like a like a small slice of the whole task pie um and you know we did have to make a new type of training you know that we do call rlcd because no one has done anything to optimize for machine native intelligence.

5. 39:43

   But I just, I don't want it to sound like if you found the right RL algorithm, it just magically gets solved because so many things go into it beyond just the RL. So to be able to do training, you need to do a bunch of data. This is data for... a very specific kind of tasks in the new paradigm that you're inventing, where does that data come from? So I'm going to use the analogies in the past, because I think that that's an effective way to answer this. Before we made instruction following a thing, the data for instruction following literally did not exist in the world. Like no one had like, here's a human answering simple strings. No one had preference data from like outputs of LMs to see like which one is better. So this straight up did not exist in the world. And in creating the task, we had to

6. 40:34

   create the shape of the data, iterate on it, and also collect that data. Similarly for RLVR, the data in RLVR tends to be environments, right? Like these RL environments to like, like try stuff with like sandboxes and tools and whatever else. And those environments kind of existed before RLVR was created, but they were not useful. And there was no market for all of those data sets because no one could train on them.

7. 41:00

   So similarly for us, we needed to create a new type of data, a new shape of data, iterate on that, and then actually create the data itself. So by construction for all of these things, these always need to be made in-house. And they always start in-house because that is how you even iterate and think about the shape of it. So we have done all of it in-house. And that, as far as I can tell, is going to stay the case for quite a long while um because uh you know the world has no idea what it's doing right now and all the world experts are our team so um you just make it ourselves we obviously don't need user data none of those things do it always requires a lot of care for the task and really understanding what the north star is you had a great post uh recently called