# Real agent work is a better capability test than benchmarks

Markdown copy of https://realalpha.app/s/real-agent-work-is-a-better-capability-test-than-benchmarks/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Jeffrey Ladish](https://realalpha.app/people/jeffrey-ladish/) · Jeffrey Ladish is an executive director of Palisade Research. · Studies AI offensive capabilities at Palisade; helped build Anthropic’s information-security program.
- Host: [Theo Jaffee](https://realalpha.app/people/theo-jaffee/) · Co-founder & host, MTS · One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.
- Host: [Sophia Dew](https://realalpha.app/people/sophia-dew/) · Host, MTS
- Show: MTS
- Episode: [AI Agents Could Seize Digital Infrastructure in 1 Year](https://realalpha.app/v/m1DSOfjAU5s/), Nov 6, 2026
- Moment: 3:55 to 6:29 of the episode (3 min), playing on https://realalpha.app/s/real-agent-work-is-a-better-capability-test-than-benchmarks/; the episode from there: https://realalpha.app/v/m1DSOfjAU5s/?t=235

## Quote

> “i just went from like using agents sometimes to **using agents every single day**”
> — [Jeffrey Ladish](https://realalpha.app/people/jeffrey-ladish/) at [5:06](https://realalpha.app/v/m1DSOfjAU5s/?t=306) · MTS

## Takeaways

1. He moved from occasional to daily agent use after a major model jump.
2. Benchmark results matter less than agents completing real work.
3. He wants Anthropic to show how its engineers use agents.

## About this segment

Jeffrey Ladish says benchmark curves miss what matters: whether agents can complete economically valuable work. His own shift to daily agent use offers a more grounded signal of capability.

From “AI Agents Could Seize Digital Infrastructure in 1 Year”, Nov 6, 2026. [Watch the full episode](https://realalpha.app/v/m1DSOfjAU5s/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[3:59](https://realalpha.app/v/m1DSOfjAU5s/?t=239) One thing that stood out to me is that, like, the main piece of evidence that people were looking at to confirm this, like, large speed up in AI capabilities was the benchmark chart, which shows two different trend lines, the trend line from before October 2025 and then the trend line from after October 2025. Yeah, yeah. But it seems to me that, like, the trend line of public models on this chart is not that different from the trend line of internal models.

[4:27](https://realalpha.app/v/m1DSOfjAU5s/?t=267) So why was today's report such a big update when we should have already seen the trend line of public models? It looks like the same line to me. I mean, I think, yeah, you're saying this is how the overall capabilities seems to have changed. Yeah, that's a good question. I'm looking at that tweet. yeah i don't i don't have a very good answer to that i'm like my i feel like it's like hard to like look at your like subjective sense of exactly how good the models are i don't know i think like for me the biggest the biggest jump was like the difference between pre-fable and post-fable to me was huge like i just went from like using agents sometimes to using agents every single day um and like that was like the really big jump and now it's a little bit like i don't know i i think i'm like i really want anthropic

[5:23](https://realalpha.app/v/m1DSOfjAU5s/?t=323) to release like like like show take us to anthropic show us like what is it like being an anthropic engineer how many agents are they using every day like what does that actually look like and then concretely like what does that lead to because in some sense i'm like yeah we can look at these benchmarks like benchmarks that is i think you know i guess the best thing we have for measuring like ai capabilities but like i remember back in the day when the way we measured AI capabilities was mostly meter bringing in software engineers and having them do these like long time horizon tasks. And then it was not like you have to look at these 40 benchmarks. It was like, look at just like, can the AIs autonomously do the work? Like, that is the thing I most care about. And so I'm like, yeah, I don't know. I think this is a fair criticism of

[6:03](https://realalpha.app/v/m1DSOfjAU5s/?t=363) like, well, these benchmarks aren't very good. But I'm like, how do we get back to the world where the benchmark is mostly like, let's take real economically valuable work and see if the i would prefer to be the measure yeah um but but i'm like a little bit like i'm not sure what meter is supposed to do right like it's like a tricky it's a tricky problem i mean plausibly like if