# Realistic AI evaluations must reflect how work happens

Markdown copy of https://realalpha.app/s/realistic-ai-evaluations-must-reflect-how-work-happens/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Edward Hu](https://realalpha.app/builders/edward-hu/) · Edward Hu is Head of AI Modeling at Mercor and former OpenAI researcher. · Leads model training and research at Mercor; invented LoRA and μTransfer at Microsoft.
- Host: [Theo Jaffee](https://realalpha.app/people/theo-jaffee/) · Co-founder & host, MTS · One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.
- Host: [Sophia Dew](https://realalpha.app/people/sophia-dew/) · Host, MTS
- Show: MTS
- Episode: [AI Outperforms Junior Accountants on Real Work | Edward Hu](https://realalpha.app/v/vAANNpTXD9E/), Oct 28, 2026
- Moment: 14:52 to 17:33 of the episode (3 min), playing on https://realalpha.app/s/realistic-ai-evaluations-must-reflect-how-work-happens/; the episode from there: https://realalpha.app/v/vAANNpTXD9E/?t=892

## Takeaways

1. Frontier models can handle complex tasks when given the needed information.
2. Real work spreads information across files, apps, and conversations.
3. Evaluations may need to measure ongoing work, not isolated tasks.

## About this segment

Edward Hu says models handle complex work when all relevant information is provided, but real workplaces scatter it across tools and people. Mercor is exploring evaluations that capture that reality.

From “AI Outperforms Junior Accountants on Real Work | Edward Hu”, Oct 28, 2026. [Watch the full episode](https://realalpha.app/v/vAANNpTXD9E/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[14:56](https://realalpha.app/v/vAANNpTXD9E/?t=896) When you've been actually designing these benchmarks, your whole team has, what's been the most surprising thing that you guys discovered? It's becoming increasingly hard to actually induce failures in these models. One thing that I was pretty surprised by, and my team as well, was actually today, if we give all there is for the model to know about a task whether it's accounting or banking or recruiting let's say we take all the relevant information we put them all in the prompt and we let the model do the work it could be quite complicated the model might need to modify files might to build a financial model you might need to call a bunch of tools but the model can for the most part do it the frontier models can and this is actually part of part of the reason behind the result that aiden shared the models are so capable of crunching these numbers now when when

[15:53](https://realalpha.app/v/vAANNpTXD9E/?t=953) the files are all present however that is not how we do work often when we perform work Everything is not nearly written together in an instruction. It's not like that exam style. So for, and actually the very first benchmarks in the APEX series, the agent economic index series was designed that way with just a big prompt with all the information the agent needs to know. And in our next iteration, APEX agents, we made it more realistic by actually scattering that information in this realistic environment where you have a file system, you have a chat server, email server.

[16:33](https://realalpha.app/v/vAANNpTXD9E/?t=993) And for our next step, we're making that even more realistic by adding more apps and by grounding our tasks in real enterprise data. And increasingly, we are purchasing company data, real company data, and grounding our tasks in it. And that's another way of scattering that information. But increasingly on the team, we're realizing that a task is not a job. So instead of giving AI a task, an environment, and just let it go do it, why don't we actually measure AI the way we measure a human?

[17:08](https://realalpha.app/v/vAANNpTXD9E/?t=1028) When I join a company, I'm giving a work chart, a place in it, and many people, some of them are coming to me to ask for things, and there are people I need to chase down to get approvals, and maybe there's someone I need to just get to know to build a relationship for future information. That's the way we work, and maybe that's the way we should be evaluating AI, and we have some exciting work coming up in the next month.