Pre-training still has room to scale

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:00:11

   So I think the way I would describe at a high level the last two years is, you know, we've put so much resourcing into reasoning, into understanding this primitive and making it work. And it really has worked. And I do think one byproduct of that is you lose a little bit of muscle on your other functions like pre-training and post-training. In the last six months, Jakob and I have done a lot of work to build that muscle back up.

2. 1:00:41

   I think pre-training is really a muscle that you exercise. You need to make sure all the info is fresh. You need to make sure people are working on optimization at the frontier, are working on numerics at the frontier. And I think you also have to make sure the mindshare is there. That's kind of one of the recent things I've been focusing a lot on, just kind of directing and shaping what people talk about at the company.

3. 1:01:06

   And very much today, that is pre-training. We think there's a lot of room in pre-training. A lot of people say scaling is dead. We don't think so at all. In some sense, all the focus on RL, I think it's a little bit of an alpha for us because we think there's so much room left in pre-training. And I think as a result of these efforts, we've been training much stronger models, and that also gives us a lot of confidence carrying into Gemini 3 and other releases coming this end of the year.

4. 1:01:37

   The way I picture it in my head sometimes is that you guys have been on this, you've just been running so fast. The whole field has been running so fast. And so we're at a moment where it's like, okay, we've gathered up this vast volume of information from the internet, we've we've thrown it onto this supercomputer and and that you know chat gpd pops out and then we're just on this this like this incredible race that's going on and so like when i hear you guys i'm just trying to think about this in like a it's a level set maybe for people who don't follow this as closely um so you you know in that initial moment you just had so much data you're throwing it at this machine you try to shape that data a bit initially and what now we're just learning like more efficient ways to shape that just it's not yeah yeah not always clear on what the mistakes

5. 1:02:33

   were um so i do think um yeah you touch on something i've been thinking about a lot right um when you think about pre-training right you're taking human written data and you're teaching the model how to essentially emulate it, right? It understands human patterns of writing. And in some sense, that also bottlenecks and puts a ceiling on the capability that you're able to achieve, right? You can't really surpass what humans have written when you're imitating what humans have written. And so, you know, you work on things like RL. There, you know, you can really provide, steer towards the hardest tasks that humans can come up with and have the model basically think outside of the box, outside of what it's learned from imitating humans and achieve higher levels of capability. But there is this kind of interesting problem now of how do you

6. 1:03:27

   go beyond what humans are able to do today? And I do find a serious measurement problem there too. Even in the sense of like, can humans judge superhuman performance in the sciences, right? How would we know that like this superhuman mathematician is better than that superhuman mathematician? And we really do need to kind of come up with better evaluations for what it means to make progress in this world, right?

7. 1:03:56

   We've been lucky up to this point, right? There have been contests like the IMO. IOI, really just like gauging who's the top one mathematician in the world, right? But when the model capabilities go beyond humans, there are no more tests. Right, okay.