A simpler agent harness lets the model use familiar code paths

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 35:52

   to think about it. Craft is a model and brush is kind of an agent. And so in creating brush Like I'm imagining like part of the task here is like a custom harness. And I'm wondering what all that entailed. How did you approach that harness development? This is such a great question because I think people are like, usually they are all focused on the model and not the harness. But I think the importance of the harness cannot be overstated.

2. 36:24

   And for us, this probably was the hardest part and the ones, the part where we get the most iterations. In fact, we already like our third generation of hardness for brush. And really like the high level goal is basically to give as much freedom to the LLM to really run free of constraint and really be able to like tap into the tools and have as much insight on what is able to generate. And so we have really focused on making the hardness are thinner and smaller and more agile and more simple, right?

3. 37:02

   And then at the end, you know, a brush can write code to operate tools and make design operations. And so we were at the same time trying to leverage how good these LLMs are at writing code and at the same time simplifying the harness in the most basic possible way to really unleash the full power of the LLM. And so that's where we did a lot of iteration. And actually, it was more about like removing the human, you know, knowledge and just letting the LLM like run free to some extent.

4. 37:36

   And when you talk about the code generation, what kind of code are we talking about? I can imagine a lot of what Brush might want to do is running a Python code, but I could also imagine you creating some design DSL that more natively represents the things that you care about. Yeah, it can be that, or it can be just writing code also to operate API calls and layout operations and that kind of stuff.

5. 38:05

   So is it primarily code or a little bit of both? It's definitely both, yeah. Like the capabilities are quite broad. What's an example of something that a user would do in brush that uses, you know, either code or this kind of some kind of DSL or abstraction? I mean, a lot of the editing operation that you can do on an editor, they can be expressed as like editing through an editing API, right? And the problem is there is there's like infinite ways of describing that design operation.

6. 38:39

   and let's say I take an object, I move it to the left and I rotate and I make it bigger, right? Like these three operations can be encoded as a format change. You know, you can have a format that describes the whole state or you can describe them as diffs and they can be achieved through like an editing API. And so that is the complexity that we had to navigate trying to understand like what was more natural for an LLM, which actually has to do with what is the most available data out there that the LLMs are trained on, right?

7. 39:18

   So this way, you kind of leverage a groove that has been already created by an enormous body of data instead of trying to teach something new or like that is underrepresented in the training set or is out of distribution in the training set of an LLM. And so I think that's where, you know, to align with the wind right and and the capabilities as opposed to trying to like fight it earlier i mentioned like kind of these two north stars you had quality and efficiency