Add prompt instructions only after repeated failures

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 7:16

   Let's talk a bit about how then you build this new prompt when there's a new model release, like for everyone in the room, everyone will want to try Opus 5 and they're going to press delete on their system prompt. How do they go about rebuilding the system prompt? How do you set up? your environment? So you do it kind of piece by piece. So the first step is you delete. The next step is you use it. And you don't want to guess what's the instruction that the model needs because you might not predict it correctly. The thing that you want to do is you want to run it. And if it's like a custom agentic product that you're building, you want to kind of run the product. You want to see where it fails with the model. You want to see what it does well.

2. 8:00

   if you're using quad code, you want to see where it does well with your code base or maybe where it stumbles over the architecture or stumbles over something else. And only when you see it repeatedly stumble on the same thing, that's when you add it back. But you don't want to do it too early. Because remember, the model is going to read this instruction every single time you use it. So you really want to make sure that the model needs this instruction. I think this is sort of the crazy thing about building on models. It's just so different than all the engineering that I've ever done. Like in the past when you built on systems, you build these like big beautiful systems and you really think about the system design up front. You have like a big suite of unit tests. You think about everything and you know like a re-architecture is a big project

3. 8:43

   and sometimes it takes months. I've worked on re-architecture products at you know big companies that take years and the motto is not like that. It's the way to think about it is almost like a like a living creature like as something more organic it's a thing where every model generation it behaves differently it has a slightly different personality and you have to take the time to get to know it and then adjust the harness based on that and I think it's just very much like an empirical one kind of scientific thing you have to take a very scientific mindset to it where you try something you see the result and then you iterate based on that if you're building in this world right now what then becomes uh stable are evals something that you keep from the previous models and keep using them in each new model release um we do until we max out the eval

4. 9:35

   so that's sort of the tip for everyone so code and system prompt you have if you want to build at the bleeding edge and have the most capability for models you've got to delete those but evals are constant and keep appending to them basically yeah you keep you keep appending what happens is um you know i actually wouldn't even go this far to be honest i think evals they outlive the harness a little bit but not by that much like an evil might live for maybe one two three model generations but nowadays the you know we're on the exponential the model is improving so quickly very often we just saturate the evil and then we have to throw it away and we have to come up with a new eval. And this is just part of the process. And again, it's about being empirical. You have to use the product, you have to use the model, you have to see where it struggles. And then based on

5. 10:23

   that, that's the eval set that you should build. I think one term I heard you describe how to build