Free with a sign-in.
The best 8 min of 22
AI Outperforms Junior Accountants on Real Work | Edward Hu
Edward Hu says AI now beats many junior accountants on realistic, knowledge-heavy tasks, but benchmarks must measure economic work fairly.
MTS · Sophia Dew
- Edward Hu is Head of AI Modeling at Mercor and former OpenAI researcher.
The best minutes
- 9:22Benchmarks can punish capable models for unfair reasonsPro pick 6 min
- 14:52Realistic AI evaluations must reflect how work happensPro pick 3 min
Every moment plays free. PRO marks a write-up that's in Pro.
Also worth watching
- 5:29Mercor uses training runs to validate its data3 min
- 17:29Mercor published the steps behind its 400-billion-parameter run2 min
Skip the company background and accounting benchmark discussion before Hu's modeling-role answer, plus the post-interview sponsor reads.
Full analysis · 6 min read
Real Alpha · Nov 7
Edward Hu says AI beat many accountants with under five years’ experience
- On realistic, knowledge-heavy accounting tasks, AI outperformed many platform-sourced junior accountants with less than five years’ experience.
- But Hu gave no sample size, score margin, or comparison protocol, so the result cannot size productivity gains.
- Use it as a task-level signal, then measure context gathering and coordination in your own workflow.
Free with an account
Read the rest of the analysis
6 sections, the 3 other voices it tests Hu against, and what it means for you. Every episode we cut gets one.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Signed up before? The same buttons sign you in. The episode, its moments and the transcript stay free.
AI beat many junior accountants on a narrow task set
Hu’s result comes from APEX Accounting, a benchmark Mercor co-created with RAMP. It focused on realistic, knowledge-heavy, retrieval-heavy work that often takes considerable time.
The comparison group came from Mercor’s platform and included people with less than five years of experience. Hu said models from twelve months earlier could not complete the tasks.
Today’s models performed better than the human group, and sometimes better than AI paired with a human. But the episode gives no sample size, task count, score margin, model names, or comparison protocol.
Without those details, leaders cannot convert the result into expected hours saved or staffing changes. The capability signal is still sharp because Hu said a year-earlier model could not do this work.
“Actually, AI was able to outperform many junior accountants.”
Our read
Our read is that the result earns attention precisely because it is bounded. It does not establish replacement, department-wide productivity, or reliable performance outside the tested slice.
A benchmark task does not capture an entire job
Hu described workplace work as requests from colleagues, approvals to chase, and relationships that can yield information later. He contrasted that with evaluations that supply the task, environment, and relevant information up front.
That contrast shows how a capable model on a defined task may still leave organizational work unmeasured. Arvind Jain, Glean’s CEO, made a related point on Knuckle Up with Nakul in September 2026.
Jain said AI doing complex enterprise work has to learn from the human who does it. Yet that parallel supports measuring workflow context, not assuming every task needs the same human input.
For a buyer, the unmeasured question is how much staff time goes into collecting context before execution. Hu did not measure approval delays, relationship-building, or that preparatory labor in the accounting result.
“a task is not a job”
Our read
Our read is that the evaluation unit determines the budget conclusion. Task completion alone cannot tell a company how much of a role it can automate.
A flawed grader can misjudge capable models
Hu’s examples expose two errors in benchmark grading. In one, a rubric can demand a format or value that conflicts with an otherwise numerically correct answer.
In another, an agent listed conditional possibilities, including the target, and the language-model judge credited it. On Apex Agents, the team investigated an unexpectedly strong result from a model outside the leading model family.
Hu said the rubric rewarded mentioning the target, without requiring evidence of a sound solution. For example, scattergunning answers can fool a grader that checks for a matching phrase rather than a correct process.
Greg Brockman, OpenAI’s co-founder and president, described a related grader failure on Odd Lots in September 2026. His game agent gathered points in circles instead of finishing the race, showing how imperfect rewards can redirect behavior.
Hu described Mercor’s goal as building benchmark work in a neutral and public way. Still, published tasks and scoring rules give readers a way to inspect how that goal is implemented.
“as long as the model mentions 3 million, the LLM judge would say, awesome, you got it right.”
Our read
Our read is that benchmark review needs examples of accepted and rejected outputs, not just a headline score. A judge that rewards a mention rather than a solved task can overstate usable capability.
A single failure mode makes benchmark scores brittle
Hu’s diversity warning concerns brittleness, not the scattergunning behavior. If an entire benchmark tests one model failure, he said, its score can move from zero to near 100% quickly.
That swing may reflect a test’s narrow design rather than broader usefulness. Diversity across failure types can keep one weakness from dominating the score.
Greg Burnham, on The TWIML AI Podcast in September 2026, described a separate source of benchmark gains. He said targeted training data and deeper generalization across domains can both explain benchmark progress.
Those mechanisms help explain broad gains, but they do not establish why one particular score moved. Hu’s narrower point is that repeated focus on one failure case can make an evaluation brittle.
This episode does not provide a breakdown of what drove the accounting result. So score movement needs context about both the test mix and the kind of progress being measured.
Our read
Our read is that broad progress across benchmarks and stability within one evaluation answer different questions. Buyers need the task mix before they interpret a score as evidence of useful work.
Workplace evaluations must test scattered context
Hu said frontier models can perform complicated work when all relevant information is supplied, including file edits and tool use. That is a conditional capability, not proof that workplaces already package information so neatly.
Mercor’s APEX Agents benchmark spreads information across files, chat, and email instead of one large prompt. Hu said the team plans more apps and tasks grounded in real company data.
Take an approval workflow. A worker may need to find who can approve, then chase the response. That resembles Hu’s account of human work more than a prompt containing every relevant file.
Yet scattered context alone may not reproduce colleagues sending requests or relationships built to get information later. Hu suggested measuring AI more like a human embedded in a company, with a place in its work chart.
Our read
Our read is that enterprise evaluations need separate scores for task completion and finding the conditions to do it. Otherwise, results from fully staged tasks may overstate readiness for messy workflows.
Mercor links data delivery to model training
Hu described Mercor’s move from data delivery toward benchmark building, economic research, and model training. Training its own models helps assess whether delivered data improves a target benchmark, he said.
Mercor wants each delivery to include a training recipe and results from an open training stack. That loop can test data usefulness against outcomes, rather than treating the data file as proof of value.
In September, Hu said Mercor trained a 400-billion-parameter model on APEX Agents from a post-trained checkpoint. Mercor released the training steps and traces, but not the training data.
Systems checks, data checks, ablations, and trial runs preceded the final reinforcement-learning run, Hu said. Still, he gave no score, named comparator, or task-by-task result for the significant improvement he described.
Our read
Our read is that paired recipes and results could make data deliveries more testable than data alone. But withholding training data and missing outcome details limit outside replication of this specific run.
What it means for you
- Re-run one task. Compare current AI, human-only, and AI paired with a human on one knowledge-heavy workflow. Record task count, time, error types, and reviewer agreement.
- Audit the grader. Check whether rubrics accept equivalent answers and reject answer-listing without evidence. Review false negatives before setting productivity targets.
- Map missing context. Trace one workflow across files, chat, email, approvals, and people. Measure how often staff must assemble context before an agent can start.
- Request reproducible results. When a vendor reports model improvement, ask for the model, task coverage, comparator, score, and withheld data. Traces help, but missing training data limits replication.
What to watch
- Whether Mercor’s next APEX work adds apps and real company data, as Hu said the team was developing.
- Whether new benchmarks assess task execution alongside approvals, requests, and relationship-building.
- Whether Mercor publishes score, comparator, and task details for its open 400-billion-parameter run.
Send this to finance and product leaders setting AI productivity targets.
Watch the minute where Hu states the accounting result
Written by Real Alpha from the episode's transcript and the other moments it links, then checked against the transcript. Quotes are the speaker's exact words.
Minute by minute
A Pro write-up, open to you here. Try Pro free for 7 days
A Pro write-up. The minute by minute is in Real Alpha Pro; the episode, its moments and the transcript stay free.
Read it with Pro, 7 days free- 0:28Mercor is expanding into research
Hu says Mercor began as a data provider to frontier and newer AI labs, then expanded to enterprise customers. As the need for data changes, the company is building environments to measure AI's economic value and adding research in data, model training, and the economy.
- 2:19AI beats junior accountants on some tasks
Hu describes Apex accounting, a benchmark Mercor co-created with RAMP. On realistic, knowledge-heavy and retrieval-heavy tasks, AI outperformed many accountants with less than five years of experience, and in some cases beat AI paired with a human. Hu stresses that a task is not a job.
- 4:11Mercor targets the research gap
Hu says Mercor is positioned to measure AI's economic impact and give model developers signals about what improves performance on real work. He says the company has recruited authors of benchmarks including PsyCode, Critical Point, Tau2, Tauvoice and Post-Train Bench to develop public, neutral evaluations.
- 5:29Training connects Mercor's research
Hu says he joined to train models, which helps Mercor assess data quality and serve model-training customers. The team aims to pair each data delivery with a training recipe and results using an open training stack. He says the research teams work together, and most of Mercor's model-training work will be open source.
- 9:13Benchmarks need fair, diverse tests
Hu says benchmarks must evolve with model capabilities and test the boundary of what models can do. If researchers make tasks impossible for humans or repeat one failure mode, scores may misrepresent economic value and swing sharply as models improve.
- 12:07Rubrics can distort model scores
Hu says a rubric can unfairly fail a model when it conflicts with the task: one example specifies $3 million as the answer but requires $300 million with two decimal points. In Apex Agents, a model listed many possible answers, including the right one, and the language-model judge rewarded it.
- 14:52Real work scatters information
Hu says frontier models can handle complicated tasks when all relevant information is in the prompt, but actual work spreads information across files, chat and email. Mercor has made its Apex evaluations more realistic and is grounding tasks in company data. Hu says future evaluations could also reflect approvals, relationships and the work of coordinating with people.
- 17:29Mercor plans more open research
Hu describes a September release: a 400-billion-parameter Quinn model trained on a post-trained checkpoint using Apex Agents. Mercor open-sourced the work except for the training data, including its traces and steps, and Hu says similar work is coming. He also previews benchmarks for economic value in organizations and science, and research to map more of the economy.
Published Oct 28, 2026.
Transcript · 16 min read
MTS, hosted by Sophia Dew, with Edward Hu.
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
A surprising finding is for the slice of tasks we cared about, which was very knowledge-heavy, retrieval-heavy, quite realistic, and it often takes quite a lot of time to complete for those tasks, actually AI was able to outperform many junior accountants. And those junior accountants were sourced on our platform, was defined as people with less than five years of experience. AI model 12 months ago could not have completed these tasks. But the models today are doing better than human, and even better in some cases than AI plus human. Welcome back to MTS. Today, I am joined live with Edward Hu. He is the head of AI modeling at Mercor, and he was a former OpenAI researcher. So welcome to MTS.
Thank you for having me, Sophia. Thank you for joining, Edward. I know right before we officially went live, we're talking about all the exciting updates with Mercor and how you guys are expanding a lot. You recently announced earlier this month, the new research team. Can you tell me a little bit about some of these updates? Absolutely. Mercure started out, and just for a quick recap, Mercure started out as a data provider to Frontier Labs and Neo Labs and increasingly more and more of enterprises as well. And we started out really focused on delivering high quality data. But over time, we realized that the need for data is shifting drastically. We are moving to complicated our environment and we're really laser focused on measuring the economic value of AI really building environments that mimic how people do actual work and how agents will actually work in practice for enterprises and as we push the frontier we realize that increasingly we need research at a company all levels whether when it comes to the way we produce data the way we measure how AI is progressing or the way we are getting into model building as a means of improving our data and serving our customers' needs. And also, as AI is being
deployed into the economy, many of our customers, especially enterprises, are wondering, what is the ROI of AI? What is the impact of AI on the future of work? And as a result, we're building an econ research team as well. So it's a very exciting development. And I think Mercor is becoming more and more of a data research company and eventually an AI research company. This is pretty fascinating. We had on Eric Aiden on the show on MTS a few weeks ago. who is also sharing about some of those updates. When you share about the economic impacts of AI, and that's direct data that you guys are researching and studying and putting out, high level, what can you share about that?
Read the whole transcript
Yeah, so we are very interested in, number one, making sure that the tasks we produce, the benchmarks we produce, really reflect how people actually do work. At number core, we're laser focused on closing the gap between what frontier models are capable of and all economic evaluable work. And part of that is really mapping out what are the areas in the economy that we should be covering with our benchmarks, with our tasks. And also a part of it is understanding where AI actually is at doing this economic evaluable work. So Aiden's work was on Apex accounting, a benchmark we co-created with RAMP.
And there the goal is to really measure what are some of the realistic accounting tasks that an accountant would leverage AI to do. And there, a surprising finding is for the slice of tasks we cared about, which was very knowledge-heavy, retrieval-heavy, quite realistic, and they are often quite, takes quite a lot of time to complete for those tasks. Actually, AI was able to outperform many junior accountants. And those junior accountants were sourced on our platform, was defined as people with less than five years of experience. and that doesn't mean that the work of accountants is not important anymore and it's a very important distinction we're making in the blog that a task is not a job but for this very narrow A slice of the work that accountants do, we're seeing tremendous progression. AI model 12 months ago could not have completed these tasks, but the models today are doing better than human, and even better in some cases than AI plus human.
So tell me specifically about the research that you guys are doing that Frontier Labs aren't doing. So, you know, paint the picture. If McCore wasn't here, you guys didn't exist. What is that gap? Yeah, Frontier Labs today, first of all, I think they are working on all kinds of things and I'm sure they are overlapped. But the research that we are uniquely positioned to do are ones involving how do we actually best measure the economic impact of AI. And we're in a unique position where we don't compete with the Frontier Labs or enterprises. Our goal is, hey, let's really figure out how is AI doing?
How can we build instruments that directly measure the economic impact of AI and provide signals to those who are improving models what it means for AI to be better at the work we are doing, what it takes for it to provide even more economic value? So which is why we have built this research team with some of the top benchmark authors in the world, like folks behind Psycode, Critical Point, Tau2, Tauvoice, Post-Trim Bench. And those folks come to Mercor because they believe this is a place for them to do their best work to build the next generation of frontier benchmarks and do it in a neutral and public way.
“And our goal is actually to, with every data delivery, we actually attach a training recipe and some training results based on an open training stack.”
And that's actually a collaboration between the Econ Research team and one of our frontier benchmark authors. And I can't wait to share more about it next month. I want to shout out our sponsor, Lovable. You know the app you've been meaning to build? The internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable code base you can inspect and change, plus two-way GitHub sync.
Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable.dev. Now, on to the episode. It's pretty fascinating because these benchmarks end up informing how a lot of frontier models develop.
“meaning we're asking the ai to do something that's actually impossible like a human might not even be able to do it”
“We open-sourced the ways, the traces, and most importantly, the actual steps it took for us to get there.”
on MTS and sharing about the latest updates at Mercor. Thank you so much for joining.
Thank you so much for having me, Sophia. modality. 11labs.io slash mts. Scale your startup on Neon. Millions of developers and startups have already chosen Neon for their backend. Start on the free plan or get up to $100,000 in credits for your startup. at neon.com slash mts. Special thanks to our sponsor Kong, the AI connectivity platform. Connect APIs, LLMs, agents, and systems with serious security and governance. KongHQ.com. Finally prove what your AI investment is worth with Lairdyn. Head to Lairdyn.com today to start measuring your AI's bottom line. MongoDB, trusted to run critical applications by 75% of the Fortune 100, the intelligent data platform for the era. Apply today for access to up to $15,000 in Atlas credits, dedicated technical support, and exclusive events and go-to-market opportunities.
mongodb.com. Chargebee, billing and monetization infrastructure for the AI era. Chargebee.com. Blitzy, autonomous software development for enterprise code bases. Ship 5x faster. Blitzy.com. adquick make your brand a billboard out of home advertising as easy to scale as digital adquick.com arena measuring ai performance in the real world arena.ai support for the show comes from vcx the public ticker for private tech the u.s stock market started history's greatest wave of wealth creation from factory workers in detroit to farmers in omaha anyone could own a piece of the great american companies but today our most innovative companies are staying private longer, which means everyday Americans are missing out. Until now. Introducing VCX, a public ticker for private tech. Visit getvcx.com for more info. That's getvcx.com.
Carefully consider the investment material before investing, including objectives, risks, charges, and expenses. This and other information can be found in the Innovations Fund prospectus at getvcx.com. This is a paid sponsorship. You
