Vertical agents should start with context and harness, not custom models

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 9:02

   If value is derived through verification, I think the people who win are the domain experts, the people who are able to quantify what that value means, which is very hard. Building evals is a very hard job. And so I think a lot of the value will accrue to whoever can write those verifiers, capture that domain knowledge. I'm massively bullish on vertical agents for different domains for this reason, because I think the domain experts who are best suited to write those verifiers or build those evals, as we kind of talk about it, they will be able to build harnesses and models. And it won't be just the harness. It won't be just the model. It will be some combination of them. So very practically, if you're getting started building a vertical agent, I would highly suggest you start at the harness layer. You do context engineering. You kind of

2. 9:47

   tweak the harness. And then when you've got users, then you start tweaking the model in post-training. I think it's much higher ROI to start at the context or harness layer first. And so I think we'll have all these different verticals. They'll be able to quantify what good looks to them in a very domain-specific way. And that'll let them build not only vertical agents, but also vertical agent experiences.

3. 10:06

   Different agents might have different experiences. And so I think a lot of the value will be massively bullish on vertical agents. And so we want to enable a lot of that. You said if I was building that, I would start at the harness layer. We're seeing a lot, whether it's Harvey or I think it's Ramp, build their own models. To what extent are they actually building their own models? Mike Cannon, Bruce, at Atlassian said on a show we released today that it's largely kind of respectfully BS, and they're fine-tuning open source Chinese models.

4. 11:02

   Is that true, and how do you think about that? I think a lot of it is they are post-training open weight models, absolutely. I still think that's a valuable thing to do. I don't think there's kind of like anything wrong with that. I think there's different dimensions that you can go in terms of owning your intelligence. I think the cheapest way is you just control the context. You use an off-the-shelf harness, off-the-shelf model.

5. 11:23

   The next level down is you start tweaking the harness. And that gives you more control than just the context, but it's a little bit more expensive to do. The next level down is you post-train an open weight model. And so you might take an existing open weight Chinese model and post-train it for your domain. And then the very bottom layer is they actually train a model from scratch. I don't think many folks are training a model from scratch.

6. 11:46

   I think most are post-training. I think if you're training a model from scratch, you're probably at Neolab. I think those are one of the few that are doing that. But yeah, the RAMPs and the Harveys of the world, they have a ton of talent. They have a ton of data. They have a bunch of usage and are trying to think about really optimizing it. And so they've gotten all the juice they can out of the context, all the juice they can out of the harness, and now they've arrived at basically post-training of these models.

7. 12:05

   Can I ask you, the jury is kind of out on the efficacy of these specialized models in the post-training realm, like we mentioned there. Some reports I've seen say they're massively more efficient and better, and some less so. So how do you think this will play out? I think what we've seen in regards to post-training is that a lot of the post-training that practically makes sense right now is probably for like simpler tasks that make up a larger harness.

8. 12:39

   So what I mean by that is like when you have an agent harness that's doing interesting work, it's actually very rarely just one model running the loop. That might be the core of it, but there's a bunch of other model calls that happen either as like guardrails or routing or sub-agents or sub-workflows or things like that. And so a lot of the places where we see post-training being efficient now are for simpler tasks like classification or extraction or summarization that might be a tool the agent uses or a guardrail or things like that.

9. 13:06

   I think those types of post-training right now make a ton of sense and will continue going forward. I think the question that is more unanswered is for this core model that's driving the loop because the main way an agent works is there's an LLM running in a loop calling tools. And so the question is, for this model that's driving that loop, will those be kind of like post-trained models or will those be kind of like frontier models?

10. 13:27

    And I think this is where we see more of like a back and forth where like every time a new frontier model comes out, the better they are at just running this kind of like core loop. We do see people like Harvey and Ramp and others. There's kind of like experimenting with post-training models for this core loop. And so I think Harvey recently in the past month or so announced something exactly in that vein.

11. 13:46

    I think that's an open question. I think if you were building your own vertical application, would you post-train your own or not? Not to start. Absolutely not to start. But like if we're talking like five years in the future, assuming that I'm getting a ton of usage, I think it could be a realistic possibility.