Agents can improve the factory that runs them

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 13:06

   it's on warp factories or not is like you want to get to a spot where everything is sort of stored everything is observable everything is measurable because that's when you can actually start to like apply engineering rigor to how you make coding agents better you know you probably you see this on twitter all the time everyone is always tweeting about like you should check out this new model you should use this new harness what's better is it pi or cloud code or you know cursor or whatever and so what our opinion on this is and what the world sort of needs is like out of the the realm of vibes for understanding these coding agents and actually into the world of measurement and so we think that this is the direction that companies are going to want to go they're going to want to like put in place a system that's like warp factories whether they actually use warp factories

2. 14:00

   or they build their own which collects all this information and provides all this way to see how your coding agents are doing i guess one other thing i like kind of take for granted that's part of this system is like you can create all these different automations around it so like i mentioned people work in slack but people are also setting up a different trigger base ways of putting work into the factory once you have this view that i showed earlier of like scoring how your factory is doing, you can also set up another kind of loop of agents and everything is kind of loops these days where those agents are looking over, they're looking for failures in your scores. Like they're looking for things that aren't going well and trying to suggest changes to your system to make it go better. And so like a really simple example here, we call this self-improvement is

3. 14:56

   like, okay, our self-improvement agents were looking at how well our triage step was doing. So we have a triage step and they're saying, okay, we have 54 failure cases of this particular form where it would have been better if our triager had been able to spawn a sub-agent to do its investigation. And so what they're able to do, and like, you could go look at like why it thinks this if you want. But the cool thing is like, because you have your whole factory defined as code and these are coding agents they can essentially make a pull request against the factory definition to try to improve the factory itself and so this is like a like a full learning loop that's built into the system to try to actually self-optimize and like we don't have this yet but the next thing that we want to do with this is like actually for any suggested change

4. 15:49

   like this you could go back replay the runs and make sure that this actually would have improved them so the possibilities of like how you can engineer this stuff get pretty cool once you have the right primitives in place and i'll show one last thing and then we could sort of maybe talk more generally about the whole approach here but the other thing that is um you know i think companies should strive for that we find very useful internally is like the ability to actually do what we call benchmarking. And so benchmarking in this context, in the factory context means essentially the ability to like take any coding agent that's running your system, you can sort of add it to a benchmark suite, which means like, you know, you might curate a set of coding agent runs that were related to like doing front end coding or something like that.

5. 16:36

   And then once you have that, you have the ability to sort of replay those same tasks against a matrix of different configurations. And so, you know, I'm a big believer in not just allowing you to vary the model, but also to vary the harness. And so, you know, in Warp, we have our own harness, but we actually will let you benchmark against Cloud Code or Codex or Pi. you can pick the model and when you do something like this what you end up getting is a result that looks very similar to like the published results for sweet bench or terminal bench but on your company's own real runs for instance we're looking at like a sort of cost versus correctness trade-off for a certain set of things here and like you want to be in the upper left hand corner So you want like lowest cost, highest correctness.

6. 17:30

   These are these models are now slightly older because the world's changing so fast. But the idea is basically, you know, we could see, OK, GPT-5-6-SOL here was like 95% correct on this task set, but it costs six bucks on average. And GLM-5-3-Flash was 93% and cost two bucks. And then the cool thing about this is you can take this information and feed it into a model routing strategy or harness routing strategy, honestly, that's defined specifically for your own workflows in code base. And so the, you know, if I had, again, to zoom out, the really big picture thing that I think is so cool and powerful here is like, you're getting out of the realm of guessing and you're getting into what we call factory engineering, where it's like you are taking a real engineering mindset. You're like, how do I, we have this amazing new

7. 18:19

   primitive of coding agents? How do I know as a team, as a company that we're actually deploying them in the right way? And so, you know, we're trying to build this infrastructure and tools for companies to do this. And like this approach is really, really resonating with a lot of companies that have been trying to build, you know, their own versions of this. So that's like a quick tour of like, what do I mean when I talk about factory? I can sort of stop talking for a that. I do think that the power move, I guess, is to sort of thaw lead so hard