Real Alpha
Try Pro free

MTS · 3 min · Nov 7, 2026

Realistic AI evals require extensive data work

HBHerbie Bradley · AI Researcher & Writer

Herbie Bradley says GDPX gets closer to real-world work through detailed, data-rich tasks, while remaining imperfect.

In this episode

Full analysis
  1. 0:53Fast takeoff remains a low-probability scenario5 min
  2. 17:27Enterprise AI gains depend on changing workflows4 min
  3. 21:35Anthropic’s RSI threshold is not a full acceleration loopPro pick 4 min
  4. 5:26Realistic AI evals require extensive data work3 min
  5. 11:38Benchmark gains can diverge from real-world utility3 min
  6. 14:00AI demand could grow sharply through diffusion alone2 min
“They put in months of effort, and still there's flaws in this benchmark, undoubtedly, but it's one of the closest that I've seen to good effort.”
HBHerbie Bradleyat 6:33 · MTS
6:33

Takeaways

  1. Good evals represent real work and resist cheating.
  2. GDPX uses detailed tasks for less-verifiable work like finance.
  3. Even months of work can leave benchmark flaws.

About this segment

Herbie Bradley says GDPX gets closer to real-world work through detailed, data-rich tasks, while remaining imperfect. The example shows how much effort useful evaluation requires.

HB

Herbie Bradley

Herbie Bradley is an AI researcher, writer, and stealth startup founder.

  • Theo Jaffee, Host · Co-founder & host, MTS

    One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.

  • Sophia Dew, Host · Host, MTS

From “Anthropic Sees Imminent RSI Acceleration | Herbie Bradley”, Nov 7, 2026. Watch the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 5:31

    you know, TerminalBench4 is interesting. You know, I've actually delved into quite a lot of the coding benchmarks, and I still find that when they do publish agent traces for Astro or Fable or whatever, if you examine them enough, you can often find that Astro or Fable has, in fact, just hacked the benchmark in the published traits. And so a lot of these benchmarks still have huge weaknesses. And it's very, very hard to even go up with something which kind of represents the full suite of skills that humans apply in their jobs in the real world.

  2. 5:59

    I think Semi-Analysis had this benchmark published yesterday, GDPVal, which sort of, I think, approaches closer to that. GDPX, yeah. GDPX, yeah. We just went over it on stream. It actually seems fantastic. Yeah, yeah. I think particularly the comparison in a blog post between the older GDPVal and GDPX on tasks like finance where there's less verifiable things, and the approach that GDPX took to really construct this very detailed, realistic set of tasks that's more representative of the real world.

  3. 6:28

    I mean, this also represents the sheer amount of effort that is required to even do this. They would need to collaborate with a large variety of data companies to source the required data for this. They put in months of effort, and still there's flaws in this benchmark, undoubtedly, but it's one of the closest that I've seen to good effort. so what exactly makes an eval good fundamentally um i mean partly it's just how representative it is of real world work and real world tasks but also you know how easily can the ai cheat and that's definitely a big one that people are putting increasing efforts into i think it's very hard to in some domains find tasks that are actually representative of real world work and where the ai can't figure out some way to cheat if it was capable enough um you know

  4. 7:19

    there's even the idea of uh in mathematics you could have a benchmark which is like uh publish a novel mathematics paper that wins a fields medal or something um but even that can in theory be reward hacked uh or good-hearted because the ai could figure out how to sort of fool the human judges uh in such a way that like the paper you fool the human judges in math well i mean without lean in theory you could fool the human judges by you know publishing some proof that is very plausible but i mean it doesn't quite hold up but the human judges can't figure it out this is a bigger issue i think it will be in ai r&d itself where it could be that a model is coming up with a proposed architectural improvement that fools human overseas at the labs, but still doesn't actually hold in generality.

  5. 8:09

    Hmm. Yeah.