The harness fills model gaps until training catches up

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 36:43

   know you didn't have to keep your laptop open can you tell me a bit on how in the past how did you improve codex because i remember when i first used codex this was one of the early versions you know like you could talk to it it did stuff but for example i said like all right make this change and it did that change and i had unit tests and it didn't run it And then later, a few months later, I don't know exactly when, it just started to run it automatically.

2. 37:07

   Were these things, did you improve the script that runs, the instructions? I'm not sure how exactly you call the bootstrapping script or whatever that is. Is it improving the model? As a dev, how can I imagine you making each version better between the harness and then between the model? And what's the connection between the two? Yeah, this is a good question. So the harness, in a sense, is always a little bit ahead of the model.

3. 37:35

   Oh, really? How so? What I mean by that is that you have the model, it's capable of certain things, but then you set it up with a couple of crutches so that it can actually do the thing to a level of reliability and in a way that is efficient and also with the behavior that you expect as a user and sort of like that's the role of the harness right is like you know provide guardrails like safety make it more efficient make it more like steerable controllable and then the harness usually is also responsible for you know what we call like the the developer um message uh which is sort of like infected in the context uh at the start of each turn and so that affects obviously like the the purpose of that to affect like the behavior of the agent throughout uh throughout the turn a lot of what you have is like the result of

4. 38:34

   of the harness and the model initially like maybe you're like oh it doesn't run tests so you know you have to remind it to run tests and then you know we train a battle model that is uh just you know capable of like better reflecting on what is it that you really want when you ask for something and then, you know, you don't actually have to tell it anymore. So over time, what we see is like the system, the developer message shrinks and then the hardness also shrinks.

5. 39:01

   Inside of the codex team, do you have specific goals? Do you say like, all right, right now the codex as a hardness and model combined is not very good at this or it's kind of doing silly mistakes right here? Or how can I imagine now how, as the engineering team, how you're working on the next, you know, version of Codex. Because the thing that I don't really get as a dev is like, okay, there's a model, which to me is this magical thing, which will get better.

6. 39:27

   Of course, I'm sure you have some feedback channels, but you also have the hardness, which is the tools that you're building. Like that's probably what the team is responsible for. How do you set even your goals, right? Like in traditional software, you'd be like, we will build this feature and you build that feature because you know how to do it. But it feels a bit more fuzzy to me, this development process.

7. 39:42

   Yeah, it is. And it's why we co-design, you know, most things. And it's a process where it's a collaboration between research and the engineering team, like primarily building the core agent harness. It's always a question of like, OK, we see today that, you know, we are very good at this, but we're not very good at this. And, you know, we have a desire to do like another thing because it would be a very cool product feature.

8. 40:09

   And then, you know, we always look at it. It's like, OK, should this be like a harness change or should this be a model change? and if it's a model change like how soon can we have it can we have it in a month can we have it in you know three months six months and we sort of like work through that and then depending on you know how soon we can just fix it in the model at which level of training then we might decide to not even do something in the harness at all and not and just wait for for for the model to to solve it you know it's it's agents all the way right so we use agents to analyze like a lot of the feedback to like you know come up with themes you know to just help us have these conversations and decide on priorities, but we analyze it across all of coding.

9. 40:48

   We analyze it across all of the other domains, like finance, comps, marketing, all the things where our users are using these agents nowadays. And there's subcategories within those. And then we roughly know, You know, how well we perform, and then we're always pushing the frontier. And there's a thing that is interesting is, like, as we make, you know, as our pre-training model gets better, as we make the overall model better, like, the whole thing lifts up.

10. 41:13

    But then there are sometimes things that we pay a little bit more attention to. You mentioned, you know, you analyze agents all the way.