Outcome-based RL spreads noisy credit across a whole trace

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 40:58

   Conceptually, how should we think about the way that humans are able to build a rich world model just from interacting with our environment and in ways that seems almost irrespective of the final reward at the end of the episode? If somebody's starting to start a business and at the end of 10 years she finds out whether the business succeeded or failed. We say that she's earned a bunch of wisdom and experience.

2. 41:21

   But it's not because like the log probs of every single thing that happened over the last 10 years are up-weighted or down-weighted. It's something much more deliberate and rich is happening. What is the ML analogy and how does that compare to what we're doing with LMS right now? Yeah, maybe the way I would put it is humans don't use reinforcement learning is maybe what I've said. I think they do something different where it's just, yeah, you experience.

3. 41:41

   So reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before it is much worse. Because previously, we were just imitating people, so it has all these issues. So in reinforcement learning, say you're working with, you're solving a math problem. This is very simple. You're given a math problem, and you're trying to find a solution.

4. 42:07

   Now, in reinforcement learning, you will try lots of things in parallel first. So you're given a problem, you try hundreds of different attempts. And these attempts can be complex, right? They can be like, oh, let me try this, let me try that, this didn't work, that didn't work, et cetera. And then maybe you get an answer. And now you check the back of the book and you see, okay, the correct answer is this.

5. 42:27

   And then you can see that, okay, this one, this one, and that one got the correct answer, but these other 97 of them didn't. So literally what reinforcement learning does is it goes to the ones that worked really well. And every single thing you did along the way, every single token gets up-weighted of like, do more of this. The problem with that is, I mean, people will say that your estimator has high variance, but I mean, it's just noisy.

6. 42:47

   It's noisy. So basically, it kind of almost assumes that every single little piece of the solution that you made, that write the right answer was the correct thing to do, which is not true. Like you may have gone down the wrong alleys until you write the right solution. Every single one of those incorrect things you did, as long as you got to the correct solution, will be up-weighted as do more of this.

7. 43:04

   It's terrible. It's noise. You've done all this work only to find a single, at the end, you get a single number of like, oh, you did correct. And based on that, you weigh that entire trajectory as like up-weight or down-weight. And so the way I like to put it is you're sucking supervision through a straw because you've done all this work that could be a minute to roll out and you're like sucking the bits of supervision of the final reward signal through a straw and you're like putting it, you're like basically like, yeah, you're broadcasting that across the entire trajectory and using that to up-weight or down-weight that trajectory.

8. 43:37

   It's crazy. A human would never do this. Number one, a human would never do hundreds of roll-outs. Number two, when a person sort of finds a solution, they will have a pretty complicated process of review of like, okay, I think these parts that I did well, these parts I did not do that well. I should probably do this or that. And they think through things. There's nothing in current LLMs that does this.

9. 43:56

   There's no equivalent of it. But I do see papers popping out that are trying to do this because it's obvious to everyone in the field. So I kind of see as like, the first imitation learning actually, by the way, was extremely surprising and miraculous and amazing that we can fine tune by imitation on humans. And that was incredible. Because in the beginning, all we had was base models. base models are autocomplete. And it wasn't obvious to me at the time, and I had to learn this, and the paper that blew my mind was InstructGPT, because it pointed out that, hey, you can take the pre-trained model, which is autocomplete. And if you just fine-tune it on text that looks like conversations, the model will very rapidly adapt to become very conversational, and it keeps all the knowledge from pre-training. And this blew my mind, because I didn't

10. 44:37

    understand that it's just like stylistically can adjust so quickly and become an assistant to a user, through just a few loops of fine-tuning on that kind of data. It was very miraculous to me that that worked. So incredible. And that was like two years, three years of work. And now came RL. And RL allows you to do a bit better than just imitation learning, right? Because you can't have these reward functions and you can hill climb on the reward functions.

11. 45:01

    And so some problems have just correct answers. You can hill climb on that without getting expert trajectories to imitate. So that's amazing. And the model can also discover solutions that a human might never come up with. So this is incredible. And yet it's so stupid. So I think we need more. And so I saw a paper from Google yesterday that tried to have this reflect and review idea in mind. What was the memory bank paper or something?

12. 45:27

    I don't know. I've actually seen a few papers along these lines. So I expect there to be some kind of a major update to how we do algorithms for LLMs coming in that realm. And then I think we need three or four or five more. Something like that. But you're still going to come up with the evocative evocative phrases sucking supervision through a straw is like so good um why hasn't so you're saying like your problem with outcome-based reward is that you have this huge trajectory and then at the end you're trying to learn every single possible thing about what you should do and what you should learn about the world from that one final bit um why hasn't given the fact that this is obvious why hasn't process-based supervision as an alternative been a successful way to make models more capable?

13. 46:13

    What has been preventing us from using this alternative paradigm? So process-based supervision just refers to the fact that we're not going to have a reward function only at the very end of, after you've made 10 minutes of work, I'm not going to tell you you did well or not well. I'm going to tell you at every single step of the way how well you're doing. And this is basically the reason we don't have that. It's not tricky how you do that properly.

14. 46:31

    Because you have partial solutions and you don't know how to assign credit. So when you get the right answer, it's just an equality match to the answer. Very simple to implement. If you're doing basically process supervision, how do you assign in an automatable way partial credit assignment? It's not obvious how you do it. Lots of labs, I think, are trying to do it with these LLM judges. So basically, you get LLMs to try to do it. So you prompt an LLM, hey, look at a partial solution of a student. How well do you think they're doing if the answer is this? And they try to tune the prompt. The reason that I think this is kind of tricky is quite subtle. And it's the fact that anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters, and they're gameable.

15. 47:09

    And if you're reinforcement learning with respect to them, You will find adversarial examples for your LLM judges, almost guaranteed. You can't do this for too long. You do maybe 10 steps or 20 steps. Maybe it will work. But you can't do 100 or 1,000 because it's not obvious. Because I understand it's not obvious. But basically, the model will find little cracks. It will find all these spurious things in the nooks and crannies of the giant model and find a way to cheat it.

16. 47:34

    So one example that's prominently in my mind is, I think this was probably public. But basically, if you're using an LM judge for a reward, so you just give it a solution from a student and ask it if the student did well or not. We were training with reinforcement learning against that reward function. And it worked really well. And then suddenly, the reward became extremely large. Like, it was a massive jump, and it did perfect.

17. 47:56

    And you're looking at it like, wow, this means the student is perfect in all these problems. It's fully solved math. But actually, what's happening is that when you look at the completions that you're getting from the model, they are complete nonsense. They start out okay, and then they change to the, the, the, the, the, the, the. So it's just like, oh, okay, let's take two plus three and we do this and this and then the, the, the, the, the, the, the, the.

18. 48:15

    And you're looking at it and it's like, this is crazy. How is it getting a reward of one or 100%? And you look at the LLM judge and it turns out the, the, the, the, the, the is an adversarial example. It's for the model and it assigns 100% probability to it. And it's just because this is an out-of-sample example to the LLM. It's never seen you during training and you're in pure generalization land.

19. 48:33

    It's never seen you during training. And in the pure generalization land, you can find these examples that break it. You're basically training DLLM to be a prompt injection model. Not even that. Prompt injection is way too fancy. You're finding adversarial examples. These are nonsensical solutions that are obviously wrong, but the model things are amazing.