# AI Outperforms Junior Accountants on Real Work | Edward Hu

Markdown copy of https://realalpha.app/v/vAANNpTXD9E/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

Edward Hu says AI now beats many junior accountants on realistic, knowledge-heavy tasks, but benchmarks must measure economic work fairly.

- Show: MTS
- Hosts: [Sophia Dew](https://realalpha.app/people/sophia-dew/)
- Guest: [Edward Hu](https://realalpha.app/builders/edward-hu/) is Head of AI Modeling at Mercor and former OpenAI researcher.
- Published: Oct 28, 2026
- Length: 22 min

## The best minutes

- [9:22](https://realalpha.app/v/vAANNpTXD9E/?t=562) [Benchmarks can punish capable models for unfair reasons](https://realalpha.app/s/benchmarks-can-punish-capable-models-for-unfair-reasons/) (6 min)
- [14:52](https://realalpha.app/v/vAANNpTXD9E/?t=892) [Realistic AI evaluations must reflect how work happens](https://realalpha.app/s/realistic-ai-evaluations-must-reflect-how-work-happens/) (3 min)

## Also worth watching

- [5:29](https://realalpha.app/v/vAANNpTXD9E/?t=329) [Mercor uses training runs to validate its data](https://realalpha.app/s/mercor-uses-training-runs-to-validate-its-data/) (3 min)
- [17:29](https://realalpha.app/v/vAANNpTXD9E/?t=1049) [Mercor published the steps behind its 400-billion-parameter run](https://realalpha.app/s/mercor-published-the-steps-behind-its-400-billion-parameter/) (2 min)

Skip the company background and accounting benchmark discussion before Hu's modeling-role answer, plus the post-interview sponsor reads.

## Full analysis: Edward Hu says AI beat many accountants with under five years’ experience

- On realistic, knowledge-heavy accounting tasks, AI outperformed many platform-sourced junior accountants with less than five years’ experience.
- But Hu gave no sample size, score margin, or comparison protocol, so the result cannot size productivity gains.
- Use it as a task-level signal, then measure context gathering and coordination in your own workflow.

Its 6 sections, what it means for you and what to watch are on [the page](https://realalpha.app/v/vAANNpTXD9E/#analysis), free with an account:

- [3:38](https://realalpha.app/v/vAANNpTXD9E/?t=218) AI beat many junior accountants on a narrow task set
- [16:55](https://realalpha.app/v/vAANNpTXD9E/?t=1015) A benchmark task does not capture an entire job
- [14:34](https://realalpha.app/v/vAANNpTXD9E/?t=874) A flawed grader can misjudge capable models
- [11:34](https://realalpha.app/v/vAANNpTXD9E/?t=694) A single failure mode makes benchmark scores brittle
- [15:16](https://realalpha.app/v/vAANNpTXD9E/?t=916) Workplace evaluations must test scattered context
- [6:42](https://realalpha.app/v/vAANNpTXD9E/?t=402) Mercor links data delivery to model training

## Summary

Edward Hu leads AI modeling at Mercor and previously researched at OpenAI. At Mercor, he works on model training, benchmarks that measure AI's economic impact, and open-source research.

Hu says AI outperformed many junior accountants on a narrow set of realistic, knowledge-heavy accounting tasks, while cautioning that a task is not a job. He describes Mercor's shift from data delivery toward economic research, benchmark development, and model training. He argues that benchmarks need to be fair and diverse, since flawed rubrics or scoring can reward the wrong behavior. Hu says future evaluations should reflect how work happens across real environments, and previews open model research and benchmarks for more of the economy.

### Minute by minute

- [0:28](https://realalpha.app/v/vAANNpTXD9E/?t=28) **Mercor is expanding into research.** Hu says Mercor began as a data provider to frontier and newer AI labs, then expanded to enterprise customers. As the need for data changes, the company is building environments to measure AI's economic value and adding research in data, model training, and the economy.
- [2:19](https://realalpha.app/v/vAANNpTXD9E/?t=139) **AI beats junior accountants on some tasks.** Hu describes Apex accounting, a benchmark Mercor co-created with RAMP. On realistic, knowledge-heavy and retrieval-heavy tasks, AI outperformed many accountants with less than five years of experience, and in some cases beat AI paired with a human. Hu stresses that a task is not a job.
- [4:11](https://realalpha.app/v/vAANNpTXD9E/?t=251) **Mercor targets the research gap.** Hu says Mercor is positioned to measure AI's economic impact and give model developers signals about what improves performance on real work. He says the company has recruited authors of benchmarks including PsyCode, Critical Point, Tau2, Tauvoice and Post-Train Bench to develop public, neutral evaluations.
- [5:29](https://realalpha.app/v/vAANNpTXD9E/?t=329) **Training connects Mercor's research.** Hu says he joined to train models, which helps Mercor assess data quality and serve model-training customers. The team aims to pair each data delivery with a training recipe and results using an open training stack. He says the research teams work together, and most of Mercor's model-training work will be open source.
- [9:13](https://realalpha.app/v/vAANNpTXD9E/?t=553) **Benchmarks need fair, diverse tests.** Hu says benchmarks must evolve with model capabilities and test the boundary of what models can do. If researchers make tasks impossible for humans or repeat one failure mode, scores may misrepresent economic value and swing sharply as models improve.
- [12:07](https://realalpha.app/v/vAANNpTXD9E/?t=727) **Rubrics can distort model scores.** Hu says a rubric can unfairly fail a model when it conflicts with the task: one example specifies $3 million as the answer but requires $300 million with two decimal points. In Apex Agents, a model listed many possible answers, including the right one, and the language-model judge rewarded it.
- [14:52](https://realalpha.app/v/vAANNpTXD9E/?t=892) **Real work scatters information.** Hu says frontier models can handle complicated tasks when all relevant information is in the prompt, but actual work spreads information across files, chat and email. Mercor has made its Apex evaluations more realistic and is grounding tasks in company data. Hu says future evaluations could also reflect approvals, relationships and the work of coordinating with people.
- [17:29](https://realalpha.app/v/vAANNpTXD9E/?t=1049) **Mercor plans more open research.** Hu describes a September release: a 400-billion-parameter Quinn model trained on a post-trained checkpoint using Apex Agents. Mercor open-sourced the work except for the training data, including its traces and steps, and Hu says similar work is coming. He also previews benchmarks for economic value in organizations and science, and research to map more of the economy.

## Transcript

MTS, hosted by Sophia Dew, with Edward Hu.

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[0:00](https://realalpha.app/v/vAANNpTXD9E/?t=0) A surprising finding is for the slice of tasks we cared about, which was very knowledge-heavy, retrieval-heavy, quite realistic, and it often takes quite a lot of time to complete for those tasks, actually AI was able to outperform many junior accountants. And those junior accountants were sourced on our platform, was defined as people with less than five years of experience. AI model 12 months ago could not have completed these tasks. But the models today are doing better than human, and even better in some cases than AI plus human. Welcome back to MTS. Today, I am joined live with Edward Hu. He is the head of AI modeling at Mercor, and he was a former OpenAI researcher. So welcome to MTS.

[0:43](https://realalpha.app/v/vAANNpTXD9E/?t=43) Thank you for having me, Sophia. Thank you for joining, Edward. I know right before we officially went live, we're talking about all the exciting updates with Mercor and how you guys are expanding a lot. You recently announced earlier this month, the new research team. Can you tell me a little bit about some of these updates? Absolutely. Mercure started out, and just for a quick recap, Mercure started out as a data provider to Frontier Labs and Neo Labs and increasingly more and more of enterprises as well. And we started out really focused on delivering high quality data. But over time, we realized that the need for data is shifting drastically. We are moving to complicated our environment and we're really laser focused on measuring the economic value of AI really building environments that mimic how people do actual work and how agents will actually work in practice for enterprises and as we push the frontier we realize that increasingly we need research at a company all levels whether when it comes to the way we produce data the way we measure how AI is progressing or the way we are getting into model building as a means of improving our data and serving our customers' needs. And also, as AI is being

[1:58](https://realalpha.app/v/vAANNpTXD9E/?t=118) deployed into the economy, many of our customers, especially enterprises, are wondering, what is the ROI of AI? What is the impact of AI on the future of work? And as a result, we're building an econ research team as well. So it's a very exciting development. And I think Mercor is becoming more and more of a data research company and eventually an AI research company. This is pretty fascinating. We had on Eric Aiden on the show on MTS a few weeks ago. who is also sharing about some of those updates. When you share about the economic impacts of AI, and that's direct data that you guys are researching and studying and putting out, high level, what can you share about that?

[2:35](https://realalpha.app/v/vAANNpTXD9E/?t=155) Yeah, so we are very interested in, number one, making sure that the tasks we produce, the benchmarks we produce, really reflect how people actually do work. At number core, we're laser focused on closing the gap between what frontier models are capable of and all economic evaluable work. And part of that is really mapping out what are the areas in the economy that we should be covering with our benchmarks, with our tasks. And also a part of it is understanding where AI actually is at doing this economic evaluable work. So Aiden's work was on Apex accounting, a benchmark we co-created with RAMP.

[3:16](https://realalpha.app/v/vAANNpTXD9E/?t=196) And there the goal is to really measure what are some of the realistic accounting tasks that an accountant would leverage AI to do. And there, a surprising finding is for the slice of tasks we cared about, which was very knowledge-heavy, retrieval-heavy, quite realistic, and they are often quite, takes quite a lot of time to complete for those tasks. Actually, AI was able to outperform many junior accountants. And those junior accountants were sourced on our platform, was defined as people with less than five years of experience. and that doesn't mean that the work of accountants is not important anymore and it's a very important distinction we're making in the blog that a task is not a job but for this very narrow A slice of the work that accountants do, we're seeing tremendous progression. AI model 12 months ago could not have completed these tasks, but the models today are doing better than human, and even better in some cases than AI plus human.

[4:14](https://realalpha.app/v/vAANNpTXD9E/?t=254) So tell me specifically about the research that you guys are doing that Frontier Labs aren't doing. So, you know, paint the picture. If McCore wasn't here, you guys didn't exist. What is that gap? Yeah, Frontier Labs today, first of all, I think they are working on all kinds of things and I'm sure they are overlapped. But the research that we are uniquely positioned to do are ones involving how do we actually best measure the economic impact of AI. And we're in a unique position where we don't compete with the Frontier Labs or enterprises. Our goal is, hey, let's really figure out how is AI doing?

[4:53](https://realalpha.app/v/vAANNpTXD9E/?t=293) How can we build instruments that directly measure the economic impact of AI and provide signals to those who are improving models what it means for AI to be better at the work we are doing, what it takes for it to provide even more economic value? So which is why we have built this research team with some of the top benchmark authors in the world, like folks behind Psycode, Critical Point, Tau2, Tauvoice, Post-Trim Bench. And those folks come to Mercor because they believe this is a place for them to do their best work to build the next generation of frontier benchmarks and do it in a neutral and public way.

[5:33](https://realalpha.app/v/vAANNpTXD9E/?t=333) Yeah, that's something I wanted to talk about with you as well. I know your team is expanding. Lots of new folks are joining. You yourself used to be a researcher at OpenAI and you've joined McCore. You're now leading all of their AI modeling. Tell me a little bit about your role during McCore. You say you came in doing one thing and now your role has expanded a ton and the type of research you're doing has expanded a ton. Yeah, we were just talking right before going live. Initially, when I joined the team, my role was let's start training some models. And training model from our core is important in many ways.

[6:11](https://realalpha.app/v/vAANNpTXD9E/?t=371) First of all, as we serve model training companies, as we serve post-training researchers, it's so important to know what it takes to train these models. And that directly affects the quality of our data. And increasingly, the best way to know if a piece of data is valuable or not, if a data delivery for a particular benchmark actually helps with that benchmark, with the domain, is to actually train a model on it. And increasingly, we're doing that for a lot of our customers.

[6:42](https://realalpha.app/v/vAANNpTXD9E/?t=402) **Edward Hu:** And our goal is actually to, with every data delivery, we actually attach a training recipe and some training results based on an open training stack.

[6:52](https://realalpha.app/v/vAANNpTXD9E/?t=412) So that's initially what I set out to do at Mercore. and very quickly we realized the research opportunity at Mercor is enormous. There are so many things to tackle, especially things that we're uniquely positioned to tackle, like what we talked about, building these frontier benchmarks. And I'm working very closely with the benchmark authors I just mentioned on building that next iteration of the frontier, also working with folks, pushing the frontier of model training. And I'm very proud to say most of our work for model training is going to be open source because we want to support the open model ecosystem.

[7:29](https://realalpha.app/v/vAANNpTXD9E/?t=449) And like what Aiden talked about here on the show two weeks ago, we're also working on econ research. And to me, it's important for all these research directions to be integrally connected. It's really important that we are not just a team with some people training models and some other people building benchmarks. The culture we're cultivating is that everyone, on the research team, first of all, they are part of one team and they all work closely together. One example is the APEX accounting benchmark is a frontier benchmark we built on the research team in collaboration with RAMP. And that became the vehicle for our economic research team to do this never-been-done research that became quite impactful.

[8:18](https://realalpha.app/v/vAANNpTXD9E/?t=498) And we also have our upcoming benchmarks actually measuring another way that we believe AI should be economically valuable, but it's undercover by the benchmarks today. And that's actually a collaboration between the Econ Research team and one of our frontier benchmark authors. And I can't wait to share more about it next month. I want to shout out our sponsor, Lovable. You know the app you've been meaning to build? The internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable code base you can inspect and change, plus two-way GitHub sync.

[8:56](https://realalpha.app/v/vAANNpTXD9E/?t=536) Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable.dev. Now, on to the episode. It's pretty fascinating because these benchmarks end up informing how a lot of frontier models develop. Because you have a benchmark, you have a specific task, a lot of people are racing to actually be the best at that task or the best model that can accomplish that specific goal.

[9:35](https://realalpha.app/v/vAANNpTXD9E/?t=575) However, sometimes when you have benchmarks, you have the opposite of almost a good heart's law, where what you're measuring becomes not that great of a metric anymore because you're optimizing for that specific metric. So how are you guys approaching that issue? First of all, benchmarks are ever-evolving. Benchmarks that we looked at a couple years ago whenever starting to do research, where we're looking at machine translation, natural language inference, and those benchmarks, they serve the purpose for where the field was back then. And the benchmarks that we need today is drastically different.

[10:17](https://realalpha.app/v/vAANNpTXD9E/?t=617) And the benchmarks we have today are not going to be good benchmarks in just a year or two or even less. So it's ever-evolving. And as researchers, our goal is to always be at the frontier and design benchmarks in a way that captures just the boundary of what the model is capable of doing and then ensuring that the benchmark is number one fair. And this is something we are increasingly discovering, whether it's in our own work or in other people's work, as the models become more capable it's increasingly hard to fail the model and there is this there is this tendency to fail the model at all costs often at the expense of fairness or benchmark

[11:03](https://realalpha.app/v/vAANNpTXD9E/?t=663) **Edward Hu:** diversity meaning we're asking the ai to do something that's actually impossible like a human might not even be able to do it or we find one way the model fails and we replicate it so many times

[11:14](https://realalpha.app/v/vAANNpTXD9E/?t=674) So we do have a very low score from the model, but it's all because there's one thing that the model is missing. And the challenge there is, well, if it's unfair, of course, we're not actually measuring the model's economic impact. So for us, we are laser focused on aligning our benchmark performance to how model actually able to produce economic output. And the second thing, the diversity issue will cause the model to be brittle. one extreme would be if the entire benchmark is testing just one failure model of the model the benchmark score will swing from zero to near 100% very quickly so as researchers it's important to be very thoughtful about these design decisions and it's easy for someone looking at a benchmark to look at the number and draw conclusions right away but as researchers We really need to be thoughtful about the underlying design of the benchmark so that it's not just a number people can cite, but a reliable instrument for the field.

[12:10](https://realalpha.app/v/vAANNpTXD9E/?t=730) When you say fair, what do you mean by that? Often that includes, are we actually testing the model for something we're asking for? I'll give you a couple examples. One example would be, let's say we are asking the model to quench some numbers, build a financial model and in the rubrics we have the task which describes what we want the model to do, the environment which contains all the information, could be a complicated file system with apps that contains everything the model needs to know to solve the task, and a grader which often involves a language model as a judge with a lot of different rubric items.

[12:51](https://realalpha.app/v/vAANNpTXD9E/?t=771) a rubric item might be the final answer needs to be $3 million. And let's say, imagine that rubric item actually specifies that it needs to be $300 million with two decimal points. And the model just said $3 million. And even though it's numerically correct, but because the rubric is already specific, then we're failing the model in an unfair way. Another example, this is an interesting one. We released a benchmark called Apex Agents earlier this year. And Apex Agents is mirroring how a professional would actually work in front of a computer. So the agent will have access to a file system with dozens of files spread out in the folder structure.

[13:39](https://realalpha.app/v/vAANNpTXD9E/?t=819) And we have different apps, mails, calendars, and then professionals, real professionals come up with legal or consulting tasks. for the model to do over the course of 50 to 100 steps, interleaving reasoning and tool use. And one thing we noticed is that some models that are not in the top model family in terms of parameter size is doing much better than we expected. And we inspected the trajectories. And it turned out the model was doing what we call scattergunning. And essentially, let's say we're looking for the answer 3 million, Like in the previous example, the model, for some reason, maybe didn't look at the right file.

[14:23](https://realalpha.app/v/vAANNpTXD9E/?t=863) It doesn't have the right domain knowledge. It's not sure about the answer. But it would say, well, if A is true, it's 2 million. If B is true, it's 3 million. If C is true, and it goes on and on, you might do it 10 times. And because the way we design the rubrics initially, as long as the model mentions 3 million, the LLM judge would say, awesome, you got it right. Wow. And that's another way where we have an unfair task. where a model that is not necessarily the best was being rewarded on the leaderboard. That's so cool. When you've been actually designing these benchmarks, your whole team has, what's been the most surprising thing that you guys discovered?

[15:04](https://realalpha.app/v/vAANNpTXD9E/?t=904) It's becoming increasingly hard to actually induce failures in these models. One thing that I was pretty surprised by, and my team as well, was actually today, if we give all there is for the model to know about a task whether it's accounting or banking or recruiting let's say we take all the relevant information we put them all in the prompt and we let the model do the work it could be quite complicated the model might need to modify files might to build a financial model you might need to call a bunch of tools but the model can for the most part do it the frontier models can and this is actually part of part of the reason behind the result that aiden shared the models are so capable of crunching these numbers now when when the files are all present however that is not how we do work often when we perform work Everything is not nearly written together in an instruction.

[16:04](https://realalpha.app/v/vAANNpTXD9E/?t=964) It's not like that exam style. So for, and actually the very first benchmarks in the APEX series, the agent economic index series was designed that way with just a big prompt with all the information the agent needs to know. And in our next iteration, APEX agents, we made it more realistic by actually scattering that information in this realistic environment where you have a file system, you have a chat server, email server. And for our next step, we're making that even more realistic by adding more apps and by grounding our tasks in real enterprise data. And increasingly, we are purchasing company data, real company data, and grounding our tasks in it.

[16:50](https://realalpha.app/v/vAANNpTXD9E/?t=1010) And that's another way of scattering that information. But increasingly on the team, we're realizing that a task is not a job. So instead of giving AI a task, an environment, and just let it go do it, why don't we actually measure AI the way we measure a human? When I join a company, I'm giving a work chart, a place in it, and many people, some of them are coming to me to ask for things, and there are people I need to chase down to get approvals, and maybe there's someone I need to just get to know to build a relationship for future information. That's the way we work, and maybe that's the way we should be evaluating AI, and we have some exciting work coming up in the next month.

[17:32](https://realalpha.app/v/vAANNpTXD9E/?t=1052) Fascinating. That was what I want to ask you for our last question, is what's next for you guys at Mercore? What can people expect to see over the next few months? One of the proudest work among many that I've done on the team is the open model release we did back in September, where we trained a 400-billion-parameter Quinn model on top of a post-trained checkpoint on Apex agents, and we demonstrated a significant improvement, but also we open-sourced everything except for the training data.

[18:08](https://realalpha.app/v/vAANNpTXD9E/?t=1088) **Edward Hu:** We open-sourced the ways, the traces, and most importantly, the actual steps it took for us to get there.

[18:14](https://realalpha.app/v/vAANNpTXD9E/?t=1094) And it turns out the big run was only a small part. We have many de-risking steps on the systems level, on the data level and we did many ablation studies and trial runs before we did the final RL run and we believe open source research like that can really help people demystify what it takes to train a model at scale and you can expect another work in that vein in the next month or so and we also have many benchmarks like the one I just talked about and also not just the frontier of AI's economic value in organizations, in work environments, but also for the frontier of science. And we also have a very exciting research in the economic space.

[19:01](https://realalpha.app/v/vAANNpTXD9E/?t=1141) For example, how do we map the entire economy into benchmarks? And it turns out we haven't mapped most of the economy. This is so fascinating. I would argue that McCoy is one of the best places to see what's actually happening at the frontier of AI. how are these models, how is this intelligence going to be diffused into society? What is their actual impact on the tasks that we're using them for? And what could be their impact in society? So super fascinating stuff. Edward, I really wanted to thank you for joining us today on MTS and sharing about the latest updates at Mercor. Thank you so much for joining.

[19:38](https://realalpha.app/v/vAANNpTXD9E/?t=1178) Thank you so much for having me, Sophia. modality. 11labs.io slash mts. Scale your startup on Neon. Millions of developers and startups have already chosen Neon for their backend. Start on the free plan or get up to $100,000 in credits for your startup. at neon.com slash mts. Special thanks to our sponsor Kong, the AI connectivity platform. Connect APIs, LLMs, agents, and systems with serious security and governance. KongHQ.com. Finally prove what your AI investment is worth with Lairdyn. Head to Lairdyn.com today to start measuring your AI's bottom line. MongoDB, trusted to run critical applications by 75% of the Fortune 100, the intelligent data platform for the era. Apply today for access to up to $15,000 in Atlas credits, dedicated technical support, and exclusive events and go-to-market opportunities.

[20:37](https://realalpha.app/v/vAANNpTXD9E/?t=1237) mongodb.com. Chargebee, billing and monetization infrastructure for the AI era. Chargebee.com. Blitzy, autonomous software development for enterprise code bases. Ship 5x faster. Blitzy.com. adquick make your brand a billboard out of home advertising as easy to scale as digital adquick.com arena measuring ai performance in the real world arena.ai support for the show comes from vcx the public ticker for private tech the u.s stock market started history's greatest wave of wealth creation from factory workers in detroit to farmers in omaha anyone could own a piece of the great american companies but today our most innovative companies are staying private longer, which means everyday Americans are missing out. Until now. Introducing VCX, a public ticker for private tech. Visit getvcx.com for more info. That's getvcx.com.

[21:35](https://realalpha.app/v/vAANNpTXD9E/?t=1295) Carefully consider the investment material before investing, including objectives, risks, charges, and expenses. This and other information can be found in the Innovations Fund prospectus at getvcx.com. This is a paid sponsorship. You