Reinforcement learning can train agents for specific jobs

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 35:45

   wrong? Yeah, I'll give you a take. Sam Altman had this post 10 years ago called Bubble Theory. He was basically saying, this was in his YC days, and he said, you know, everyone here says that all these companies are overvalued and we're in a bubble. And it's obviously fun to talk about bubbles and all, but I don't believe that and I'm going to give you a bet. And he said, it looks hilarious, by the way, when you read it in retrospect, which is like, here are the top, you know yc companies today and it's literally like uber and airbnb and like you know whatever and it's like today these are worth this much i predict that five years from now they're going to be worth you know 3x more you know and then it's like here are these kind of like emerging you know mid mid-level companies and then that list was like you know stripe and some of these

2. 36:22

   others and it's like today these are worth this much i predict these will be up at least 3x you know and he basically said everyone says we're in a bubble if you'd like to go and bet on that and put your own money down, I am happy to put, you know, money on this. I have to say in AI, I really feel this. I feel this, I think now more strongly than ever, which is over the last two years, people kind of think of it as this Gen AI wave.

3. 36:45

   Again, like we're saying, like lots of incremental jumps, but the truth is RL is the master, like the big story I'd say over the last year or two in capabilities. And I think people have really underappreciated how much is possible with RL. What do people not see with RL that they should see? because I kind of, I don't. Yeah, we've had a few years of, I would call imitation learning before RL, which is, that's what went into GPT-3 and 3.5 and so on, which is basically you take the entire internet and you read the whole internet and you get a model that sounds like somebody on Reddit.

4. 37:15

   And it was crazy. I mean, it was obviously ChatGPT like shocked the world and it was a huge moment for everybody. But yeah, you could kind of see why, well, you know, talking like the average, it is not necessarily the thing that you need to do to go build a great, you know, medicine specialist or a great, you know, software engineer or any of these things. And I actually got a lot of the way there, to be fair, more than people might have guessed.

5. 37:34

   But I think that the next big thing with RL is, as we said, you can take any benchmark and solve it. It's crazy to say, but I think the next step of that is honestly an open question for everyone and everyone. re-application, which is, hey, what is the benchmark? If you're an accountant, you know, the benchmark, I mean, in real world terms, the benchmark is roughly like, all right, you submit returns on behalf of the client. And, you know, if you get audited by the IRS and something comes up, then that was a fail for the RL. And if you don't, then you did a good job. But obviously, you know, you want to be able to constrain it to shorter feedback cycles than that. And think about what are the ways that you can determine if an agent's work was a success or a failure.

6. 38:11

   But once you have that, it turns out that you can just train agents to do all sorts of things. And that's what we've seen, by the way. I mean, we've seen like folks getting gold medals and these international math competitions and things like that with AI. And the truth is that this is really hard stuff, right? And obviously, you have to go and apply it to every vertical, but it's going to happen. So I guess on that point, I guess I was just going to say, it's like, you think about the companies today, there's the foundation layer companies, right? There's the application layer, and there's a lot of the tooling, I would happily bet, take all the foundation labs today, right? It's the private ones, right?