Useful model routing needs to understand the agent harness

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 14:00

   One of the things that harnesses also need to do, I imagine, is model routing, depending on request. How much value is in the model routing layer? Model routing is so interesting. I think it's one of the hottest and maybe most divisive topics right now. So a few thoughts. First, there's a question of where the model routing should even be. Should it be at the harness level or should it be in the gateway level?

2. 14:21

   And this is like getting a little bit into the technical weeds. What is the difference? I'm sorry. The difference is in the harness level, it's in kind of like the code of how it runs. And in the gateway, it's just an API call. So the benefits of putting it in the gateway level, because it's just an API call, you can drop it into any harness. You could put this routing that's happening at the gateway level into cloud code, into any harness without having to change the code.

3. 14:44

   The downside and why I think we're more bullish on routing happening in the harness, as are I think Factory and Cognition have also written things about this, is that in the harness you just have more knowledge of what's going on. You have more awareness of the state. When you have an API, it's generally assuming that it's kind of like an open AI chat completions API and so it only takes in a certain amount of state.

4. 15:03

   you have more state in the harness to know, hey, what's going on with the prompt cache? You want to try to keep the prompt cache warm and available. You can also change the prompt more easily. So a lot of these models have different prompting strategies. And so if you do the routing at the harness layer, you can more easily change the prompt. And so I think a lot of what we've seen is one, this debate between gateway and harness and where does routing go.

5. 15:28

   I think we probably believe it belongs in the harness. And just so we understand, the open routers and all the other providers, which there are several, they are in the gateway. They are in the gateway. A lot of, obviously, open router, incredible company, incredible usage. Had them on the show. Love Alex. Alex is great. I think a lot of why people use open router and Vercel and other gateways is largely for the model access.

6. 15:51

   So I think there's also a difference between a router and a gateway. And so I would probably say most of open routers' usage, and Alex obviously knows the numbers a lot better than I do, but if I had to guess, I would guess a lot of it just comes from being a gateway, not a router. So people specify the model they want up front, and Open Router just provides a single API with a single API key for accessing that. In terms of smart routing, I think that's actually being done way less than people think. And I think when it's done, it's built into the harnesses by folks like Factory. I think Factory does a tremendous job about this. They've written a number of technical blogs that are very well done on terms of how they do routing. But I think right now, I pretty strongly believe that routing, you have to be harness aware. You have to know what the task is.

7. 16:33

   You have to know what the harness looks like. So I don't think you'll have routing that works for law and works for finance and works for coding and have it all be the same. It's going to be very domain and very harness specific. I'm sorry, when I listen to you, it seems so unwaveringly obvious that routing should be at the harness layer, given the context and memory that's stored there. When I have such an obvious response, I like to understand why am I stupid? What is the bull case for it being at the gateway layer?

8. 16:58

   I think when people talk about it at the gateway layer, I think there's maybe like two things that they're talking about. So one, it makes it way easier to port. So you can take cloud code. People love cloud code, right? If you tell people, hey, you can't use cloud code, you have to use this other harness. There are other good harnesses out there, but let's pretend for a second that you love cloud code and you want to let them keep on using that, but you want to do some semblance of kind of like routing to maybe like a deep seek or an open source model or something like that. Then the only way you can do that is at the gateway layer because you don't control the harness.

9. 17:26

   Side note, this is another argument for open source harnesses that you can control and modify. And then the second thing is when people talk about routing at the gateway layer, I think they're oftentimes... implicitly assuming that there's a domain. So there's kind of like two general domains that you could maybe argue that you can build general routers for. And I actually think Open Router has done this to some extent.

10. 17:49

    You could argue coding. Coding is the most popular domain. That's where all the usage is going. Let's build a router that works really good for coding. Maybe we won't be harness-aware, but we'll be use case-aware. So we know different models are good for different things in coding, and we can put it at the gateway. The other thing you could say is, hey, we've got this general chatbot. We're not trying to be best in the class at law or best in the class at finance, but we're just going to be a general chatbot across the company. And we can know well enough that if you say, hi, how are you doing? You don't need Opus for that. You can use a cheap model.

11. 18:19

    And so I think when people talk about routing at the gateway level, they're implicitly kind of assuming that you're doing either general purpose chat or coding, and you're not doing a domain-specific thing. As soon as you get into a domain-specific thing, I really think it has to be at the harness layer. You're not going to use an off-the-shelf gateway router for that. So in two to three years' time, you think the majority is within the harness, given the benefits that come from that context. But for very horizontal tasks, it will sit in gateway.

12. 18:47

    I think that's exactly right. If you're building a vertical agent, I think routing needs to be in the harness. If you're thinking about a general purpose, horizontal task, like just general chat or coding, it could maybe be in the gateway. Is it a commodity? Does everyone just build it? I don't think routing is a commodity. I do think people will build it, but I think building it is quite hard even today. And I think part of the reason for that is the models just change so rapidly. So you can argue that certain models are better at different things, but then a new Opus or new OpenAI model comes out and it's just better at everything. And so then the value of