Factory tuning cut Warp’s cost per PR by about 75%

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 23:57

   The real power of this whole system is the ability to improve it. And so just to give some stats at Warp, when we first started using the factory approach, it was costing us about $80 per PR because we were using like Opus where we shouldn't have. Our triage agent was using like a really high powered model. you know we were doing a bunch of redundant work with our the shape of agents like we were doing too many sub agents relative to just having fewer agents and so we basically did a whole bunch of internal tuning and testing using these built-in features and we were able to get the cost per pr down to like 20 bucks or 15 bucks something like that well we can measurably improve the throughput of our prs over the last month it's gone up by 40 percent we measure like the revert rate and it hasn't gone up. And so I really feel like a lot of confidence in this general approach of like

2. 24:52

   observe, measure, tune. Finally, you need some sort of like orchestrator. Sometimes people talk about the orchestrator like it's the whole platform. I think it's a piece of it. What I mean by orchestrator is like you need some system that will take the factory definition code and then actually move work along the different stages, calling out to the right models, loading the right skills, etc. You need something that will fire the right automations. And then finally, you need a bunch of access points. And again, our philosophy with this, which you can, I could see people have different philosophies, our philosophy is to sort of bring the factory into a company's existing knowledge work tools. So think of like, you know, the most obvious one is Slack. And like, we're doing like 60% of our work is starting in Slack. But sometimes it makes sense to start in the task

3. 25:41

   tracker, sometimes it makes sense for a developer who is more comfortable using like cloud code in their terminal to do a bunch of work there and then submit a spec into the factory. And so, you know, our philosophy is to try to make this whole thing API driven and MCP driven so that the factory itself is like a piece of kind of headless infrastructure where it has all these different entry points. So I think this stack based view of it when I've shown it, I don't know, we talked to companies seems to land very well it's like okay yes this this makes sense this is a new thing in the like developer infrastructure stack it kind of sits one level back from like where ci cd is but i think most companies will have something like this for software development yeah that's a lot of layers actually i've never actually seen it spelled out like like you have