Models bring different intuitions to research problems

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 24:02

   Contests have helped me realize lately is when you put the models and deploy them towards solving these contest problems, which they are quite good at these days, they work in a very different way from humans. We typically think of these machines as they're very good at pattern recognition. You can take any problem, if it maps to a previous problem, it's probably going to be able to solve it. But what I've noticed is in some of the previous IOIs, this problem like message is very ad hoc.

2. 24:32

   I didn't think the models would solve it at all, but actually one of the easier problems for the AI. So, yeah, I mean, this is giving me the sense that AIs plus humans in frontier research, it's going to do something amazing just because the AI has a different intuition for what's easy and what's not. So, okay. Is it vaguely, you know, when DeepMind did the whole AlphaGo thing, you know, there was that moment where it was doing things human, it was playing in ways humans hadn't played before.

3. 25:00

   So kind of like vaguely similar to that? I think so. I think so. I think really with GPT-5 Pro, right, there's been an inflection point in frontier research. And one of the best anecdotes I have for this is, you know, I think three days after the launch, I met up with a friend who was a physicist and he had been playing around with the models. uh felt like you know they they were cute but not super useful and i challenged him with the pro model just try something ambitious and you know he put in his latest paper um it thought for 30 minutes and just got it and i would say that that reaction in that moment um it was kind of like seeing lisa at all during that you know move 37 38 um and i just think that is just going to keep happening more and more for frontier mathematics, for science, for biology,

4. 25:59

   material science. The models have really gotten to that point. I was going to ask you this question, which is not very original because I think we've been doing this ever since Big Blue and all the chess stuff. But yeah, just as somebody who had followed all these competitions, if, I don't know, there's a sadness when you start seeing these models solving these things that were like the, this height of these achievement for these very unique human minds?

5. 26:25

   Well, yes and no. I mean, I was good at competitive programming. I was never at the absolute top. And maybe this is a way to get revenge. But I do think, no, there's certainly a moment for myself, right? We tracked coding conscious performance while we were developing reasoning models for a while. and you know at at the start of the program you know they were not super great you know uh at the level of any average competitor going uh going to the contest and yeah over time they just started creeping up and up in terms of capability and you still remember that moment when you walk into the meeting and they have where your performance is and then the models exceeded that um man that was also a shock to me. It's just like, wow, we've automated to this level of capability so fast.

6. 27:17

   And of course, you know, Jakob was there still a bit smug, but within like one or two months, it was also surpassing him. So yeah, no, the models are at the frontier today, right? It's so clear by, even through the results we've done this summer at Coder, right? Top optimization competitive programmers in the world. I think it achieved second place there. And so really it's jumped from you know 100th place last year to top five this year and like do you think we'll still