Inference is cheap enough to justify much more thinking

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 19:38

   How high do you guys think the ceiling is for continuing on this like test time compute vector? I mean, obviously you've, you know, I think very publicly like Ilya's come out and said there's this entirely, you know, new direction that's needed to really advance AI to the next level. Do you agree with that? Yes, I pretty much agree because LLMs are just too cheap. Like, you know, like if you, what, like operations cost, you know, under 10 to the negative $18 these days.

2. 20:09

   So, like, you know, if you can infer relatively efficiently, even on a very large model, you're, you know, you're getting like over a million tokens per dollar. You know, if, you know, I guess you can just check the prices on Gemini or anybody else. Yeah, so you're getting millions of tokens per dollar. That's orders of magnitude below the cost of most other things you can think of. If you think of like a really cheap pastime, like go like buy a paperback book and read it or something, you're paying like 10,000 tokens per dollar.

3. 20:46

   So we're like a couple of orders of magnitude cheaper than, you know, than like reading a book and, you know, probably, you know, four orders of magnitude cheaper than, you know, paying anybody to do anything. and, you know, whatever, six-to-eight orders, magnitude cheaper than paying a software engineer. So there's a huge, huge margin of difference there to apply more compute and make the thing smarter and, you know, if the value is there, which I think it is.

4. 21:27

   Like, you know, would you pay, you know, five cents an hour for the bad engineer or ten cents an hour for the good engineer? You know, like it's there's so there's a huge amount of sort of unexploited flops in there for to use if we can find ways to use them. And one way to use them, straightforwardly, is, OK, just train a bigger, better model. We're already doing that. But still, model training costs tend to go up quadratically with the size of the models.

5. 22:16

   So then you still end up with a relatively cheap inference, if you do it right. So then, of course, what everyone is doing now is just apply more compute at inference time through this chain of thought thinking or any other brilliant algorithms we can come up with. And so I think we're just going to start seeing a scaling curve there as well as we're seeing in a lot of places. Right. But does that scale us all the way to the AGI future people envision?

6. 22:52

   or is there some, you know, some kind of completely adjacent thing that's required to, you know, I guess where does an ascent tip? Yeah, I guess that's where we'll see whether, you know, whether it's the humans that invent the next breakthrough or the AI. But I'm not, I've given up on like organizing my garage and stuff like that because I'll just wait for the robots. Yeah, you think they're coming?

7. 23:16

   I guess you guys had a big robotics release yesterday. Yeah, that was awesome. Yeah. Is it ready to clean your garage, though? Not that I know of. Yeah, I think the question of how much more is there to give in this test time compute paradigm? Is it all the way to AGI? I don't think it's all the way to AGI. I think we already kind of established there's other components, being able to act in a complex environment.

8. 23:38

   Acting is very important. Research into acting agents, that's a definite investment. And there's many other aspects. But are we seeing kind of test time compute asymptoting? I think like the kind of Ramanujan example is always in my mind here where it's like I don't want test time compute to just for any given problem like think longer and then eventually arrive at solution. But we also want it to be able to think like very deeply and actually like create maybe useful knowledge that it's going to like actually incorporate to then solve further tasks in its thoughts and thus improve dramatically improve data efficiency.

9. 24:17

   if you can just have like one math textbook and you spend most of your time really just thinking and playing around with the ideas and then you can become a world class mathematician that would be the kind of thing that I feel like you know should be and we should strive to achieve with like a very deep thinking model are we there yet no like do we have a path there. I think there is. There are many directions of gradient towards such a model.

10. 24:43

    We've been already seeing, and this is one thing I think people don't really talk about too much with test time compute, but we're already seeing amazing improvement in data efficiency by training the model to think deeply with reinforcement learning when it's solving the task. So even if we're going to have a bunch of RL data, we're not going to add any more, even just like going, the test time compute paradigm is allowing us to learn a lot more from that data. And I think we could probably push that much more so. There's kind of many particular research angles we're interested in and how this model could think not just in one