RLHF made models pleasing, but raised automation expectations

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 27:17

   why language models are not able to do this and why a whole new family needs to be invented to do this. And so my understanding is that part of the root of the problem is RLHF, which you're one of the co-inventors of, which served its purpose incredibly well for what it was supposed to do. However, it comes with a set of side problems, which in particular are bad for the mission at end. So maybe let's start again in an effort to make this approachable.

2. 28:00

   By a bunch of people, what is RLHF at the simplest level? Ooh, I would describe as RLHF as making models chat and making models pleasing for chat. Because so models are pre-trained, so they have the whole, all the knowledge in the world. However, turning them into something that can converse, you know, requires them expressing certain type of preference and certain ways of interacting with humans, right?

3. 28:31

   And lots of stylistic stuff. Actually, I think a lot of the optimization for RLHF, in hindsight, looks extremely stylistic rather than is it about, you know, like being better at like tapping into its facts deep into its network. RLHF stands for Enforcement Learning from Human Feedback. So how were humans involved initially when you guys did that? So actually, there's many different parts of RLHF. I think I started working on it perhaps end of 2020. But there was like predecessor work, to be clear, that I would want to call out. So I think that we built on top of the shoulders of giants and many of those predecessor works could be called RLHF as well. I'm happy to cite them now, but I'm not going to do it right now. I've cited them in other spots.

4. 29:22

   um the the thing that made rlhf rlhf in my opinion is about the task the task about optimizing for human preference and the shape of it as being text in text out that i think is kind of un-arbitrary instructions uh that to me is the root of what originally rlhf was at the time we were still trying to figure out this shape i think it's very obvious in hindsight but there was a lot of uncertainty we were kind of like the black sheep of open ai um and it is like no one likes putting expensive humans in loop because it's very hard to iterate on it and we were really trying to figure out like hey will this work because when we were playing whack-a-mole with the issues like things kept coming up and we kept having to solve them until we made it like really general and pleasing and good i believe that the hardest part of creating a new direction in ai

5. 30:15

   is even knowing the task so if we knew in hindsight the task should be shaped this way it's possible it would have taken us uh like it would have been way way way faster at this but because of that uncertainty it was a little a little bit harder to figure out like yes the way to do this is data make this diverse data and also like even the specification for labelers like here is how to label these things because this is the thing we want out these are things that we didn't know at the time, which everyone who copied us didn't have to figure out because we had done all of the hard work of that iteration and then published it in a paper. Not to have a stab at them, but I do want to call out that a lot of the hard part is the discovery here. And I think that that's a lot of our hard part with type size. So we are, to our best proxy, optimizing for

6. 31:06

   human preferences and that is why you never get outputs that obviously look bad even though they might be equivalently bad to one that is just wrong like you never have models where you never have outputs these days even though the models are wrong all the time but you can just look at it and be like this is clearly crap um because if that's punished so hard and it's like really warps the probability space for these models yeah so uh and we're talking about all of this because again And that's sort of the heart of the of the problem.

7. 31:34

   So because the models were trained to please humans and sound confident as part of that process of our LHF, then that's why you end up with issues like sycophancy over refusal that you alluded to. And also like mode dropping, like the models, the LHF models are very stylistic. You know, they're all repetitive in different ways, but you can like kind of get a sense if you use a model a lot, like, wow, it keeps saying this thing, you know, and that is a side effect of RLHF, because RLHF heavily encourages the models to stay on their stylistic lane, because that's the safe path to not get punished. And this results in like a lack of creativity that pre-trained models didn't have.

8. 32:27

   Pre-chain models were wild and creative and weird, and they have all of the styles in the internet inside of them, and they sometimes just go completely off the rails, which is beautiful, and they didn't have this particular problem that I think people, from what I've seen, kind of dislike in LLMs. Great. So, again, RLHF, great for all of us to discover the power of generative AI through a chat experience, but bad for the problem of classification because you have over refusal, you have sycophancy.

9. 32:56

   Then you also have the latency and overhead problem that we alluded to that because the model was trained to do text to text, then it needs to autoregressively generate prose line by line and sound very confident doing it, which is not helpful if you're trying to say customer angry, not angry or problem yes or no. I agree with all of that, though I will emphasize that I do believe RLHF makes the models slightly dumber at decisions than the pre-trained models.

10. 33:33

    The root of the problem to me is that RLHF actually greatly increased expectations. so if you imagine like how good the models are from human perception versus like reality um are like just really warp that because all of a sudden people were like holy smokes these models are really really smart we can do all sorts of stuff with them and that decoupling is i think quite harmful to the world and that to me is was my biggest worry here so uh like it you know no one you know like the expectations for pre-trained models were very different but because rlhf models are so pleasing to humans and they seem so smart they it makes people think like holy smokes a lot of stuff can be automated and that empirically did not happen so that's rlhf which