# Humanity’s Last Exam uses problems from active researchers

Markdown copy of https://realalpha.app/s/humanitys-last-exam-uses-problems-from-active-researchers/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Alexandr Wang](https://realalpha.app/builders/alexandr-wang/) · Alexandr Wang is CEO of Scale AI. · Dropped out of MIT to co-found Scale AI in 2016; Meta invested $14.3B in it in 2025 and hired him to lead Superintelligence Labs.
- Host: [Garry Tan](https://realalpha.app/people/garry-tan/) · President & CEO, Y Combinator · Co-founded Posterous and Initialized Capital, an early Coinbase backer.
- Host: [Harj Taggar](https://realalpha.app/people/harj-taggar/) · Managing partner, Y Combinator · Co-founded Auctomatic with the Collison brothers and later Triplebyte.
- Host: [Diana Hu](https://realalpha.app/people/diana-hu/) · Managing partner, Y Combinator · CTO of Escher Reality; after Niantic bought it, ran AR for Pokémon GO.
- Host: [Jared Friedman](https://realalpha.app/people/jared-friedman/) · Managing partner, Y Combinator · Co-founded Scribd, which YC funded in 2006; has advised 20+ YC unicorns.
- Show: Lightcone
- Episode: [Alexandr Wang: Building Scale AI, Transforming Work With Agents & Competing With China](https://realalpha.app/v/5noIKN8t69U/), Jun 18, 2025
- Moment: 42:00 to 47:48 of the episode (6 min), playing on https://realalpha.app/s/humanitys-last-exam-uses-problems-from-active-researchers/; the episode from there: https://realalpha.app/v/5noIKN8t69U/?t=2520

## Quote

> “the best models were scoring like 7%, 8% on it. Now the best models **score north of 20**%. It's moved really, really quickly.”
> — [Alexandr Wang](https://realalpha.app/builders/alexandr-wang/) at [43:58](https://realalpha.app/v/5noIKN8t69U/?t=2638) · Lightcone

## Takeaways

1. Researchers contributed new problems from their recent work.
2. Top model scores rose from 7–8% to above 20%.
3. Future evals will test real-world tasks, not just fixed exams.

## About this segment

Scale gathered novel, difficult problems from researchers; top model scores rose from 7–8% to above 20%. Alexandr Wang expects benchmarks to shift toward fuzzy real-world tasks.

From “Alexandr Wang: Building Scale AI, Transforming Work With Agents & Competing With China”, Jun 18, 2025. [Watch the full episode](https://realalpha.app/v/5noIKN8t69U/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[42:04](https://realalpha.app/v/5noIKN8t69U/?t=2524) One of the things that we built in partnership with the Center for Eye Safety is Humanity's Last Exam. It was a funny name. I think, unfortunately, there will be yet another exam beyond it. But, you know, the idea was how, like, let's effectively work with, you know, the smartest scientists in the field. And, you know, we worked with many very brilliant professors, but also very many, like, individual researchers who were, like, quite brilliant.

[42:30](https://realalpha.app/v/5noIKN8t69U/?t=2550) And we just collated and aggregated this data set of what the smartest researchers in the world would say the hardest scientific problems they've worked on recently are. They solved them or they sort of like came to the right, you know, they were able to solve the problems, but they're sort of like the hardest problems that they're aware of and know of. I was curious how you came up with these problems.

[42:49](https://realalpha.app/v/5noIKN8t69U/?t=2569) So each of the professors contributed new problems. So these are not these are problems that have never appeared in any textbook or any exam ever. They just like came out of their brains and they like typed up like a new problem, like from scratch. Am I saying this right? Yeah, and the general guidance was like, you know, what has come up recently in your research that you think is a particularly hard problem, right?

[43:10](https://realalpha.app/v/5noIKN8t69U/?t=2590) The problems are stupidly hard, incidentally. They're like insane. I don't know if you guys have looked at these problems. They're totally crazy. Yeah, it's totally crazy. And by the way- They cannot be searched on the internet. It's like you need to have a lot of expertise and actually think about them for quite a long time. Yeah, they require a lot of reasoning. I'm recently like right now. So we have a time limit where the models can only think for I think it's 15 minutes or 30 minutes.

[43:36](https://realalpha.app/v/5noIKN8t69U/?t=2616) And one of the most recent requests from one of the labs is like, can you please increase that time limit to like a day so that the model has like up to a day to think about the to think about the problems. But, yeah, no, they're they're deviously hard problems. Unless you have expertise in the specific problem, you probably don't have a chance of getting it right. But even this evaluation, like I think when we first launched it...

[44:00](https://realalpha.app/v/5noIKN8t69U/?t=2640) You know, and this was earlier this year, the best models were scoring like 7%, 8% on it. Now the best models score north of 20%. It's moved really, really quickly. And I think, you know, I think... Do you think we're going to get a benchmark saturation for this one as well? I think eventually, yeah, it'll be saturated. And then we have to move on to new evaluations. I mean, I think the like, the saving grace for the naming was that it is the last exam.

[44:26](https://realalpha.app/v/5noIKN8t69U/?t=2666) The new evals will be sort of like real-world tasks, real-world activities, which are sort of like fundamentally fuzzier and more complicated. Have you solved any of the problems yourself, Alex? I know you were a competitive math person for a long time. Yeah, yeah. I mean, the math problems require a lot of – they're like very deep in the fields. I think I managed to get a handful, but like most of them are like hopeless.

[44:48](https://realalpha.app/v/5noIKN8t69U/?t=2688) Yeah. I looked at the ones that the models can solve. So that was that was one of the evals. And we've produced a number of other evaluations. But but yeah, I think that like the the in the AI industry really, I think, continues to suffer from a lack of very hard evals and very hard tests that show really like the frontier of of model capabilities. and these evals when you get when you build an eval that sort of like becomes popular in the industry it has this like deeper effect which is that that's all of a sudden the like north star in the yardstick that in that researchers are trying to um optimize for and so it's it's actually this like very gratifying activity you know we built humanities last exam um you know most of the like all the model providers um you know will always report their their their their results

[45:40](https://realalpha.app/v/5noIKN8t69U/?t=2740) there's like tons of researchers who are really motivated by by doing a good job i mean it's it's And the models are going to get, you know, deviously good at like, you know, frontier research problems. I guess Sam's starting to talk about, you know, that sort of stage four innovators of AGI is coming. And, you know, that's the prognostication for the next year. Do you think that's, you know, correct?

[46:00](https://realalpha.app/v/5noIKN8t69U/?t=2760) The next 12 to 24 months is like really the moment that literally new scientific breakthrough is coming from the operation of reasoning and these models. I mean, I think it's super plausible. Well, you know, in fields like biology, and this is probably one of the ones that comes up the most, but there's like there's probably intuitions that the models have about biology that humans don't even have because it's just, you know, they have this like different form of intelligence.

[46:25](https://realalpha.app/v/5noIKN8t69U/?t=2785) Right. And so you'd expect there to be some areas in some fields where the models have some fundamental deep advantage versus humans. And so I think it's like very realistic to expect in those fields. Biology, I think, is sort of like the clearest one for me. It kind of already happened for chemistry. Last year, the Nobel Prize went to the Google team, Demes and John Jumper with AlphaFold. Yeah, exactly.

[46:50](https://realalpha.app/v/5noIKN8t69U/?t=2810) That was like a huge jump. Before that, there was like this competition where they were trying to get more printing fold structures that were going to get solved. And it was like abysmal and AlphaFold destroyed it. It's a strange time to be a scientist, but an exciting time for science. there's this um uh short story it talks about this future where like you know there's uh effectively AIs that are like that are conducting all the frontier of R&D research and um and scientists you know what scientists do is they just sort of like look at the discoveries that the AIs make and sort of like try to understand them yeah I mean I think that like very exciting time to to see the how the frontier of human knowledge expands and then I mean I think that be great because in areas like um in biology will fuel breakthroughs in medicine and healthcare and

[47:41](https://realalpha.app/v/5noIKN8t69U/?t=2861) and all these other and all these other things and then the majority of the economy will will chug