# Benchmarks can punish capable models for unfair reasons

Markdown copy of https://realalpha.app/s/benchmarks-can-punish-capable-models-for-unfair-reasons/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Edward Hu](https://realalpha.app/builders/edward-hu/) · Edward Hu is Head of AI Modeling at Mercor and former OpenAI researcher. · Leads model training and research at Mercor; invented LoRA and μTransfer at Microsoft.
- Host: [Theo Jaffee](https://realalpha.app/people/theo-jaffee/) · Co-founder & host, MTS · One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.
- Host: [Sophia Dew](https://realalpha.app/people/sophia-dew/) · Host, MTS
- Show: MTS
- Episode: [AI Outperforms Junior Accountants on Real Work | Edward Hu](https://realalpha.app/v/vAANNpTXD9E/), Oct 28, 2026
- Moment: 9:22 to 14:57 of the episode (6 min), playing on https://realalpha.app/s/benchmarks-can-punish-capable-models-for-unfair-reasons/; the episode from there: https://realalpha.app/v/vAANNpTXD9E/?t=562

## Quote

> “meaning we're asking the ai to do something that's **actually impossible** like a human might not even be able to do it”
> — [Edward Hu](https://realalpha.app/builders/edward-hu/) at [11:00](https://realalpha.app/v/vAANNpTXD9E/?t=660) · MTS

## Takeaways

1. Benchmarks need to evolve as models improve.
2. A low score can reflect an unfair task, not weak capability.
3. Rubrics can reward models for guessing the right answer.

## About this segment

Edward Hu explains how unrealistic tasks and narrow failure cases can distort benchmark scores. Fair, diverse evaluations are needed to measure real economic value.

From “AI Outperforms Junior Accountants on Real Work | Edward Hu”, Oct 28, 2026. [Watch the full episode](https://realalpha.app/v/vAANNpTXD9E/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[9:26](https://realalpha.app/v/vAANNpTXD9E/?t=566) Because you have a benchmark, you have a specific task, a lot of people are racing to actually be the best at that task or the best model that can accomplish that specific goal. However, sometimes when you have benchmarks, you have the opposite of almost a good heart's law, where what you're measuring becomes not that great of a metric anymore because you're optimizing for that specific metric. So how are you guys approaching that issue?

[9:52](https://realalpha.app/v/vAANNpTXD9E/?t=592) First of all, benchmarks are ever-evolving. Benchmarks that we looked at a couple years ago whenever starting to do research, where we're looking at machine translation, natural language inference, and those benchmarks, they serve the purpose for where the field was back then. And the benchmarks that we need today is drastically different. And the benchmarks we have today are not going to be good benchmarks in just a year or two or even less.

[10:24](https://realalpha.app/v/vAANNpTXD9E/?t=624) So it's ever-evolving. And as researchers, our goal is to always be at the frontier and design benchmarks in a way that captures just the boundary of what the model is capable of doing and then ensuring that the benchmark is number one fair. And this is something we are increasingly discovering, whether it's in our own work or in other people's work, as the models become more capable it's increasingly hard to fail the model and there is this there is this tendency to fail the model at all costs often at the expense of fairness or benchmark diversity meaning we're asking the ai to do something that's actually impossible like a human might not even be able to do it or we find one way the model fails and we replicate it so many times So we do have a very low score from the model, but it's all because there's one thing that the model is missing.

[11:20](https://realalpha.app/v/vAANNpTXD9E/?t=680) And the challenge there is, well, if it's unfair, of course, we're not actually measuring the model's economic impact. So for us, we are laser focused on aligning our benchmark performance to how model actually able to produce economic output. And the second thing, the diversity issue will cause the model to be brittle. one extreme would be if the entire benchmark is testing just one failure model of the model the benchmark score will swing from zero to near 100% very quickly so as researchers it's important to be very thoughtful about these design decisions and it's easy for someone looking at a benchmark to look at the number and draw conclusions right away but as researchers We really need to be thoughtful about the underlying design of the benchmark so that it's not just a number people can cite, but a reliable instrument for the field.

[12:10](https://realalpha.app/v/vAANNpTXD9E/?t=730) When you say fair, what do you mean by that? Often that includes, are we actually testing the model for something we're asking for? I'll give you a couple examples. One example would be, let's say we are asking the model to quench some numbers, build a financial model and in the rubrics we have the task which describes what we want the model to do, the environment which contains all the information, could be a complicated file system with apps that contains everything the model needs to know to solve the task, and a grader which often involves a language model as a judge with a lot of different rubric items.

[12:51](https://realalpha.app/v/vAANNpTXD9E/?t=771) a rubric item might be the final answer needs to be $3 million. And let's say, imagine that rubric item actually specifies that it needs to be $300 million with two decimal points. And the model just said $3 million. And even though it's numerically correct, but because the rubric is already specific, then we're failing the model in an unfair way. Another example, this is an interesting one. We released a benchmark called Apex Agents earlier this year.

[13:27](https://realalpha.app/v/vAANNpTXD9E/?t=807) And Apex Agents is mirroring how a professional would actually work in front of a computer. So the agent will have access to a file system with dozens of files spread out in the folder structure. And we have different apps, mails, calendars, and then professionals, real professionals come up with legal or consulting tasks. for the model to do over the course of 50 to 100 steps, interleaving reasoning and tool use.

[13:54](https://realalpha.app/v/vAANNpTXD9E/?t=834) And one thing we noticed is that some models that are not in the top model family in terms of parameter size is doing much better than we expected. And we inspected the trajectories. And it turned out the model was doing what we call scattergunning. And essentially, let's say we're looking for the answer 3 million, Like in the previous example, the model, for some reason, maybe didn't look at the right file.

[14:23](https://realalpha.app/v/vAANNpTXD9E/?t=863) It doesn't have the right domain knowledge. It's not sure about the answer. But it would say, well, if A is true, it's 2 million. If B is true, it's 3 million. If C is true, and it goes on and on, you might do it 10 times. And because the way we design the rubrics initially, as long as the model mentions 3 million, the LLM judge would say, awesome, you got it right. Wow. And that's another way where we have an unfair task.

[14:47](https://realalpha.app/v/vAANNpTXD9E/?t=887) where a model that is not necessarily the best was being rewarded on the leaderboard. That's so cool.