# Automatic checks make unsolved math problems usable as benchmarks

Markdown copy of https://realalpha.app/s/automatic-checks-make-unsolved-math-problems-usable-as/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Greg Burnham](https://realalpha.app/people/greg-burnham/) · Greg Burnham is a leads capabilities research at Epoch AI.
- Host: [Sam Charrington](https://realalpha.app/people/sam-charrington/) · Host, The TWIML AI Podcast
- Show: The TWIML AI Podcast
- Episode: [AI Progress Explained: From Math Olympiads to Navier-Stokes](https://realalpha.app/v/lz9J-Dbj2zw/), Sep 29, 2026
- Moment: 44:12 to 48:58 of the episode (5 min), playing on https://realalpha.app/s/automatic-checks-make-unsolved-math-problems-usable-as/; the episode from there: https://realalpha.app/v/lz9J-Dbj2zw/?t=2652

## Quote

> “This is maybe the big piece of work on our side, was to make it easy to **verify automatically** whether an AI system has solved this problem.”
> — [Greg Burnham](https://realalpha.app/people/greg-burnham/) at [45:35](https://realalpha.app/v/lz9J-Dbj2zw/?t=2735) · The TWIML AI Podcast

## Takeaways

1. Problems range from moderately interesting to major breakthroughs.
2. Automatic verification makes unknown-answer problems measurable.
3. A collection of problems can provide a smoother signal than one result.

## About this segment

Epoch curates unsolved problems that mathematicians care about and builds ways to check answers automatically. This lets researchers track progress even when the solutions are not known in advance.

From “AI Progress Explained: From Math Olympiads to Navier-Stokes”, Sep 29, 2026. [Watch the full episode](https://realalpha.app/v/lz9J-Dbj2zw/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[44:16](https://realalpha.app/v/lz9J-Dbj2zw/?t=2656) system, whatever. Okay. So now, these are, I'll lump them together, because it really is the, this is our main tool. So we have, you know, our latest iteration on frontier math consists of a bunch of problems that humans have tried and failed to solve. They're curated to be of a special interest to mathematicians, where, and some of them we've characterized even on a bit of a scale from, like, this would be moderately interesting, or this would be, like, maybe what that means is two teams of mathematicians have tried this problem. It's at least 10 years old, but it's not central to any research program. It's just something that people would be happy to get an answer to all the way up to like a major breakthrough of four tiers here breakthrough is the highest uh none of the breakthroughs have been solved yet but don't get me wrong if navier

[45:10](https://realalpha.app/v/lz9J-Dbj2zw/?t=2710) stokes had been in this problem set it would surely be a breakthrough there um regardless of the fact that humans had like done a lot of work and gotten maybe within spitting distance of the of the solution before ai finished it off um but yeah we really view this as just a way of finding AI, a way of finding cases where AI might have had a creative idea, because we can tell easily whether it solved these problems. This is maybe the big piece of work on our side, was to make it easy to verify automatically whether an AI system has solved this problem. It's not obvious when you think about it that no human knows the answer. So how do you tell without a human looking if AI has found the answer? But there's a couple techniques you can use for this, and we employ different techniques for this um it can go into that but it's something of a technical detail

[46:02](https://realalpha.app/v/lz9J-Dbj2zw/?t=2762) and so when we when a new model comes out astra came out we hit go and it's like oh it solved three new three new problems from our list okay let's go look at the solutions let's share them with mathematicians let's see what's going on here um and and then we we analyze that we with the help of mathematicians analyze that for like what kind of qualitatively what kind of solution was this what does it mean when the models are so good that we have to shift from you know these benchmarks that we understand to like this collects these collections of problems that we just don't have the answers to it's definitely a phase transition in the the science of ai capabilities measurement uh which which is you know what what i work on but but it doesn't change the um it doesn't change the game all that much it just means we have to look for harder problems

[46:51](https://realalpha.app/v/lz9J-Dbj2zw/?t=2811) I think I joke sometimes that even certain forms of optimization problems, like allocating scarce resources efficiently, that's a benchmark of sorts that will survive the singularity. Like even if AI is like off going crazy, running rampant in the galaxy, it'll still have to, you know, it'll still have to decide. how to, you know, optimize. So optimization is always going to be a benchmark, even if we're superhuman. Basically, it used to just be, we knew the answers ahead of time, now we don't.

[47:21](https://realalpha.app/v/lz9J-Dbj2zw/?t=2841) So it takes this extra trick of, okay, how do we tell when AI has gotten something right? But this isn't insurmountable, especially like I was saying with, you know, numbers in a scientific context. It's intuitive that, okay, the best humans were able to do was, you know, X, but we just say to AI, hey, try to do Y greater than X. And, you know, if it does, that's, you know, those numbers go on and on. So it's no problem. But yeah, you're right to sort of note the phase shift and to be somewhat surprised by it. Yeah. And is there a methodology, a concrete methodology for assessing and awarding partial credit? Is that part of the way you think about benchmarks in this, you know, post phase shift? Yeah, it's a good question. It depends on the benchmark so sometimes it's something like well you just want to make this number go up and the

[48:13](https://realalpha.app/v/lz9J-Dbj2zw/?t=2893) higher the better so partial credit is very uh continuous and very very natural in those cases for math problems sometimes it's just like well what we care about is is you know if x does y always hold like is this is this always true and then it's kind of sometimes humans will find and publish partial results for the ai systems we usually just say you know all or nothing but we have a large enough collection of such conjectures that, you know, we hope to have some, you know, smooth gradient of, well, okay, this one got two more out of 100 or something. And so it looks like a pretty smooth signal. And that's typically how we approach these things. Yeah, across benchmarks.