AI math has advanced from school problems to open research

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 20:21

   If I may, I would rewind history just a bit here to say some of the, say the trajectory we've been on. Because once you step back, it's wild. So, up until fall of 2024, so we're talking two years ago, grade school math was still a challenge for these LLM-based AI systems, the kind of AI we're talking about here. and all and the benchmark there was literally a benchmark of 8 000 you know word problems so and so you know has so many apples that sell for this gsm 8k gsm 8k that's the one and gpt4 which was 2023 had like done pretty well on this so people started looking at so like adding into the mix a benchmark just called math like four capital letters uh that um started pulling from some high school math competitions.

2. 21:25

   And there was like slow progress on this, but not a lot. And then at the end of fall of 2024, OpenAI comes out with the O1 preview model, full O1 by the end of 2024. And this showed a huge spike in math competition capabilities. But we're still just talking like medium hard high school math competitions, like something, you know, thousands of math nerd, nerdy math, 16 year olds were into across the U.S., and this became, you know, the new territory. My company, EPOC, around that time released a benchmark called FrontierMath. At the time, that's all we called it. We now call it tiers one through four to distinguish from later versions I'll get to in a bit. But the original FrontierMath consisted of problems that I would say go from advanced undergraduate to advanced grad student. So sort of early career research warmups, almost. These are problems that are no

3. 22:25

   longer thousands of high schoolers would tackle them. These require deep background knowledge in various areas, niche areas. We're talking a problem from elliptic curves that you'd give to to a grad student like shortly before they're going to embark on their novel thesis research just to get them familiar with the details of the of the niche that they're going to be trying to work in and make an original contribution to over the same period we also saw high like so over 2025 we also saw high school math contests get completely aced we got the gold medal on the international math olympiad by the um even by the beginning of of 2026, we saw most of these frontier math problems solved, though not all of them had in fact been solved by an AI system at that point. And so just like, even just here, this is a very

4. 23:24

   rapid trajectory from over the course of just a little over a year going from, you know, like, hobbyist high schooler to graduate student like very competent kind of you know uh up there graduate student in terms of the problems they could solve really rapid ai progress at the same time we started to see people test ai systems on unsolved math problems uh problem like the the frontier was no longer something this is what i mean we've gone from exam to on the job, where it wasn't like, here's your qualifying exam as a grad student anymore.

5. 24:07

   It became more interesting to say, well, here's a small problem that maybe a mathematician, a professional, thought about for an hour, didn't solve AI systems. Can you do it? And at that level, we started, this was all just this year, like 2026. At the beginning of the year, there were starting to be claims of, oh, maybe here's a problem that was posed by a famous mathematician. This is maybe the Erdos problems.

6. 24:30

   There's this, just for background, there's this very prolific mathematician, Paul Erdos, who proposed over the course of his life, maybe like well over a thousand, maybe a couple thousand problems that people still like haven't fully cataloged all the questions he asked. Some of those became central to whole fields of math. Some of them were, you know, no one really paid much attention to, including Erdos himself.

7. 24:52

   And AI started to notch a few solutions to some of the easier ones of those. Fast forward to May of this year, and an internal model, probably something like the Astra model that we now have publicly available, was able to solve a big one of those. This was sort of a historic moment, the so-called unit distance problem, which was a problem about how many points you can fit in the plane that are just one unit, distance of one apart from each other.

8. 25:26

   sort of like very cool that you could frame it. It's such a simple problem, but there hadn't been progress, much progress on this problem, despite a lot of attempts since Erdos had posed it, you know, decades and decades before. And it was a serious problem. It wasn't one of these, no one had thought about it. It was definitely had received a lot of attention and an AI system solved it. And it was really this, this was like a aha moment of, oh my goodness, like AI can solve problems that humans have cared about just independently. And we've seen a lot more of that since. And the, you know, with a sort of the recent wow moment being the solution of the Millennium Prize problem. This was, you know, the Navier-Stokes problem. This was a collection of this, these Millennium Prize problems, a collection of seven problems that mathematicians

9. 26:14

   sort of, or at least one math institute set as like good goals for the new century. Like they They're much older than the year 2000, but they were sort of canonized in the year 2000 as, hey, these are worthy targets of entire subfields of mathematics, and it would be a big deal. And so one of them was solved. So it's been a hell of a ride. The historical context is super interesting. You know, we're all living through this trajectory, but to step back and see, you know, how much has happened in the past couple of years is pretty, pretty nuts.

10. 26:49

    Yeah. You have to, as one of my colleagues said, you have to like not get frog boiled by it. Like the temperature is rising and it's easy to, I don't know. Anyway. So there is this interesting