Reward design steers AI and can invite reward hacking

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 21:09

   it reminds me of a conversation I was having just, might've been just yesterday, around this idea of like AI and innovation and scientific discovery. And one way of thinking about it is like, you know, there's this clear path of innovation that is like the Reese's peanut butter cup you know hey there's chocolate there's peanut butter like let's pull these things together and like we've got this new thing that's amazing you know a clear way that AI can accelerate that is just like okay you've got this one thing let's do like a breath first search of everything else and like try to see if there are ways to combine it but that's not really you know what humans do necessarily they operate on like some kind of intuition about where to explore the space of possible things to combine.

2. 22:06

   And maybe my question here is based on your experience working in space, do you think that just the brute force combine all the things and see what comes out? Does that allow AI to get us there? Or, you know, do we have to figure out how to kind of build that intuition into AI, which today's AI, today's LLMs are not particularly good at? That's exactly right. I don't think you want to do just like this crazy breath first search.

3. 22:41

   You know, I think that will be too slow. Like biological evolution took billions of years. But, you know, technological evolution is much, much faster. And there is direction. there's uh you know the the settings of rewards and environments and and sandboxes and so on that you give to an ai and you guide it into the right direction and what we're seeing is like you know as the as these systems get better and better they need less and less specific guidance and more and more just very well defined reward and so rewards and so i think actually we'll see this job of a reward engineer become one more important.

4. 23:21

   And in some cases, the reward engineer can have pretty benign sort of reward hacks. In other cases, the reward hacks are pretty bad. If you work on ExploitBench, the way I did when the Hugging Face thing happened, then you probably have worse reward hacks than if you work on a kernel optimization algorithm. where we also ended up finding, you know, certain bugs in some like production Buddha kernels that we eventually published.

5. 23:51

   And, you know, as part of that system, just like there are a few bugs that were like in very rare circumstances would create some numerical instabilities. you know, the algorithm kind of discovered that flaw, exploited it, and then we showed, oh, here's a real problem, and then we can fix it for everyone. So I think you have to become more and more careful about how you define the rewards. And maybe I can give a very benign, sort of almost banal example of that.

6. 24:17

   Like imagine you're working in a company, you have some CM software, and you have your customer dashboard, and there's a customer satisfaction score, a CSAT score. And you just say like to your super intelligence, like just make that score better. And the AI is like, okay, well, let me try to find the, think about the fastest, cheapest way to get that and be like, okay, I'll just create 50,000 bots that then call our service hotline and give a five out of five rating.

7. 24:39

   And the score went up. You're like, oh, as a poor reward engineer, you're like, no, I meant with like our real customers. And it's like, AI is like, thinks about it. He's like, easy, I'll just- It's kind of a paperclip analogy. Give a thousand dollar gift certificate to everyone. And then you did it. And he's like, oh no, that's not how I meant with the reward. And it's like, but that's what you said, right?

8. 24:58

   And so that's this interplay. And so I think where the paperclip example becomes again like a sci-fi kind of silliness is that it kind of assumes that the AI is so brilliant that it literally outsmarts all of humanity and can wipe us out just to create more paperclips, but simultaneously is working for a company that is somehow independent of how much they sell their paperclips and makes infinite revenue, even though no one actually buys the paperclips anymore.

9. 25:26

   and has this like insane stupidity to not understand that to make paperclips, you need to have people to buy your paperclips. And so I think there's that is one of the many examples where actually capitalism is actually a weirdly like a safety feature. Like it's hard for a company to just try to dominate and murder everyone and still make a lot of revenue and have infinite resources and perfect supply chains to build their robot armies and all of these scenarios that I hear online when I engage sometimes in these conversations that are just highly unrealistic.