# Agent traces can be scored against team-specific failure modes

Markdown copy of https://realalpha.app/s/agent-traces-can-be-scored-against-team-specific-failure/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Zach Lloyd](https://realalpha.app/builders/zach-lloyd/) · Zach Lloyd is founder and CEO of Warp.dev. · Founded Warp and led its April 2026 open-sourcing of its agentic development environment, built around cloud agents.
- Host: [Shawn Wang (swyx)](https://realalpha.app/people/swyx/) · Co-founder, AI Engineer · Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.
- Show: Latent Space
- Episode: [⚡️ The 2026 Software Factory Infra Stack — Zach Lloyd, Warp.dev](https://realalpha.app/v/LL0YmIyBKSU/), Nov 3, 2026
- Moment: 6:33 to 12:09 of the episode (6 min), playing on https://realalpha.app/s/agent-traces-can-be-scored-against-team-specific-failure/; the episode from there: https://realalpha.app/v/LL0YmIyBKSU/?t=393

## Quote

> “sometimes they write a bunch of **tests that you don't need**. They write it in the course of like figuring out how to like implement something”
> — [Zach Lloyd](https://realalpha.app/builders/zach-lloyd/) at [10:50](https://realalpha.app/v/LL0YmIyBKSU/?t=650) · Latent Space

## Takeaways

1. Track automation, cost and PR throughput across the team.
2. Use LLM judges to grade stored agent runs.
3. Add custom scorers for failures such as redundant tests.

## About this segment

Warp centralizes agent traces and grades stored runs for issues such as redundant tests. Teams can use those scores to find failure modes and tune workflows instead of relying on anecdotes.

From “⚡️ The 2026 Software Factory Infra Stack — Zach Lloyd, Warp.dev”, Nov 3, 2026. [Watch the full episode](https://realalpha.app/v/LL0YmIyBKSU/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[6:40](https://realalpha.app/v/LL0YmIyBKSU/?t=400) And once you have that working, you basically get a whole bunch of benefits that flow out of it. And so one of the big benefits is that wherever these agents are running and you're sort of seeing a real time view across Warp's internal team, how are we using our own factories? you get a full trace of what everyone is doing with coding agents across any product that has a factory for it and this is cool not just for like oh i want to see what people are doing like i want to see what sofic is doing which is cool but it also starts centralizing all of the agent conversations and agent artifacts and all of the work that is being done by coding agents across your team. And so when I talk about like having a closed loop system, these agent conversations, these agent traces are like the key aspect of how this system improves. And so for instance,

[7:34](https://realalpha.app/v/LL0YmIyBKSU/?t=454) just to look at one of these agent traces, Safik is working on our mobile app here. Safik was doing this in Slack, so I could pull this up in Slack if I wanted to. But like the thing is actually generating, it has a full record of the conversation, has a full record of all of the, these are computer use screenshots that are coming out of building this thing. You can see a full record of exactly what the different pieces of the orchestration agent were doing.

[8:02](https://realalpha.app/v/LL0YmIyBKSU/?t=482) And so you have this, but you have it not just for one thing, you have it for every single thing. So on your team today, when I talk to companies, for the most part, they don't have something like this because most of their engineers are like individually using cloud code in their terminal or codex the codex app or something like that in their own local setups but they want to get to a world where they have this centralization when you have the centralization you can basically get all this interesting data on how your agentic coding is working across your company and you could think of this data along different dimensions so you could think of this data as like okay what's our team's output in the world of using coding agents. And so that's things like, you know, traditional Dora metrics, how long does

[8:50](https://realalpha.app/v/LL0YmIyBKSU/?t=530) it take to ship something? It's things like how automated are you? And so we have a sort of metric around like, for every PR that gets merged, how many times did a human have to do something with it? What's your throughput? It's metrics around cost. And so we're using this like constantly. And when we talk to companies, this cost thing is very, very important where you want to be able to tune your factory with the right models, the right skills to basically bring this cost per PR down by time. You want to be able to slice the cost by person. You want to be able to slice it by automation, by where in the SDLC are you spending money. The final thing that's really cool, once you have this sort of closed loop system going, is that you can actually look underneath the hood at how well your coding agents are doing along different dimensions. And so you can think

[9:42](https://realalpha.app/v/LL0YmIyBKSU/?t=582) of this as like online evals or we call it scorers and these can go from very general things so like are your agents being too verbose are they being efficient is the code quality good we have work you do on this do i as a user define this or do you provide this as a platform yeah so both so we provide out of the box a set of ones that we think are like generically useful but then you know as we work with companies, you know, they may have specific failure modes for their agents that they're interested in tracking. They may be like, hey, is the agent making a specific type of thing that is like change that is like breaking our build? They may want something, you know, very specific to their workflow. So again, this is all defined in code and all customizable. And what you do is like when you set up one of these what's actually happening behind the scenes is that

[10:36](https://realalpha.app/v/LL0YmIyBKSU/?t=636) there's some set of runs that you you know when i say runs i mean like historical agent conversations that have been stored that an agent then goes and like retroactively grades and so this is a good example of one where it's grading for redundant tests which is like this is a common failure mode if you're using coding agents, like sometimes they write a bunch of tests that you don't need. They write it in the course of like figuring out how to like implement something and then they leave it in the code.

[11:04](https://realalpha.app/v/LL0YmIyBKSU/?t=664) And so this like bloats your code base, it wastes tokens. And so, you know, say you wanted to check for that, you know, you basically create one of these evals. And just to show you like a little more about what the eval looks like, the eval is, again, think of it as LLM as a judge. So you're picking a judge model. You're then seeing, okay, like where do these run? What's the prompt for judging it?

[11:26](https://realalpha.app/v/LL0YmIyBKSU/?t=686) and then a sampling rate because you might not like these are agents that are running you might you got to be wary of the cost of them but when you have this set up think of it then as like you have all these smart agents that are looking across your team is how at how all your your coding sessions are going and they're saying okay sometimes there are agents that are writing too many tests right and so you get this record and this understanding of where you should look to sort of improve the way that your team is using coding agents.

[11:59](https://realalpha.app/v/LL0YmIyBKSU/?t=719) So far, so good. Because I want to show a couple other things just to make it more specific what you can do with this.