Pairwise human feedback can train models on subjective taste

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 15:48

   Yeah. So it's funny. Very early on, in the early RLHF days, the only thing we trained models on were essentially taste. There was reinforcement learning through human feedback. And the human feedback was always a pairwise signal. It was like given two options, which one's better. And so the early days of reinforcement learning were built on the notion of training on human taste. And then obviously things like verifiable rewards and environments and, you know, we built a lot of machinery to do other types of rewards and other types of RL. And so people kind of forgot that a lot, the early days of reinforcement learning were all around human taste. And so how do you train on human taste? It's, I think the technical answer is quite simple. I think the like practical answer obviously is more complicated. The practical answer is you just train a reward

2. 16:32

   model, right? You just train a reward model. You get a bunch of humans to look at pairs of things and say which one is better. And this still works. So, like, fun fact, like, we did this for slides, right? Like, I mean, how do you go about making these models better at generating presentations? Well, we just get a bunch of humans to look at pairs of slides and say which one looks better. And humans are actually really, really good at aesthetic judgment, right?

3. 16:53

   Actually, much better than models is something that we discovered. It's like a human can look at two slides and within seconds say, this one looks better. And if you have a model, like a grading model, try to go about doing that. It's going to spend a whole bunch of reasoning tokens, this pixel is this way, this thing's that. and then maybe come to the wrong conclusion. And so humans are still really valuable in making these models better.

4. 17:12

   And so we will rely a lot on human preference to imbue things like taste. So that's one. The last thing I'll kind of add is there's precedent for this, which makes us confident that this strategy will work. And that precedent, I would say, is coding. The models have gotten, I would say coding is not fully solved, but let's say we're 80% of the way there and we're making progress on the last 20%. Coding is a combination of verifiable rewards and human taste.

5. 17:36

   And so the verifiable rewards are, does a code compile? Are there syntax errors? Is the output of the function expected? And so there's these verifiable rewards that are in place. But a lot of good software engineering is, is this clean system design? Did you build a system that's maintainable over time? Is it easy to unit test? Is it following paradigms that the vast majority of the community or the industry thinks is the right paradigms?

6. 17:58

   And if you look at it from that lens, a lot of financial work looks very similar. It's the, you know, I have an Excel model. does it sum up? Are there reference errors? There's a whole bunch of verifiable things that we can check to make sure that the work is passing this first pass sanity check. But there's also a lot of, is this the way I would have built this model? Something I learned about as I started working in finance is there's this notion of the feel of the model, which is you perturb the inputs and you look at the outputs. And an experienced financial professional can perturb the inputs, look at the outputs, and understand does the data flow through this model in a way that feels good?

7. 18:31

   And that is a human preference. That looks a lot like, does this code look clean? Is it well written. And I think our success in coding gives me confidence that we can solve a lot of these problems in finance as well. Yeah, we can shout out