Verify hard tasks instead of over-specifying them

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 19:41

   How do people get better at that? And effectively, how do people get better at prompt engineering? Do people still need to do a lot of prompt engineering? Or is that changing as well? Tell us about where this is going. Yeah, I remember like a year ago, one of the most popular job openings was prompt engineer. And then it kind of changed. And then I think it became like context engineer. So, there's these kind of waves of it.

2. 20:06

   I think these will kind of, like, come and go. I think the skill nowadays is less about prompt engineering and more about figuring out how do you give Claude a hard task that seems a little bit too hard. And then how do you make it possible for Claude to verify its work along the way? And the verification, I think, is probably the single most important thing that people do not get right, largely. um one example of this is people were uh you know we have this desktop app for cloud and it's built using electron we've made it quite fast so now it's like a pretty awesome experience six months ago it was like sluggish and it wasn't very reliable now it's pretty awesome and you know it's the thing that most of the team uses as an experiment though i wanted to see like what would it feel like if it was native? And so what I did is I started a quad tag session and quad tag is

3. 21:00

   just you know it's a it's a new product we have it's just quad running in Slack. My first question was hey tag do you have access to a macOS runner on GitHub? And it said no and then I hooked up a runner so it was able to start a Mac virtual machine using GitHub. And then my second question is I created this like empty code base that was a quad desktop app rewritten in Swift and I asked can you access this code base? It said no. And then I gave it access and I was like, okay, great.

4. 21:27

   Now I have access. And then I was like, okay, now I want, what I want you to do is I want you to rewrite the Electron app in Swift. I want you to run the Electron app in the Mac virtual machine, screenshot it, and then look pixel by pixel, compare it to the Swift version. Don't stop until you're done. And that was your prompt, basically. That was my prompt. And how long did this take to run? It's still running. When did you start it? It's been a little over two weeks. So it's like 14 days, 15 days. Yeah. So I don't know if anyone in the audience has gotten clocked to run a task for more than two weeks. I don't know. Raise your hand. Anyone in the audience?

5. 22:21

   This is like one of these, this is about elicitation. So this is really one of those examples where the model can do it today, you just have to let it do it, and you don't need the fancy stuff. You don't need slash go, you don't need slash loop. These help, but really all you need is give the model the task, give it a way to verify the output of its work so it doesn't get stuck and it'll just go. And actually in this case, Quad also decided to live blog it.

6. 22:47

   So what it did is it created a Slack channel internally and it started just posting screenshots every few minutes of its progress. Wow. So the prompt sound is so simple. I mean, everyone here could do it. And I guess, what is separating the people here that can become the top 1% Clocker users? How can people learn to use clock code like Boris? Maybe, like, don't listen to the LinkedIn influencers.

7. 23:17

   Don't listen to it. Don't read Twitter. This is the thing about the model is I think everyone's looking for, like, the one weird trick to do it. There's just, like, that doesn't exist. There's nothing like that. The way the model works is you have to approach it. directly. You have to give it a task that's too hard. You have to give it the tools to verify the work, like you would yourself, like you would if you were doing the task. You have to see where it struggles, and then you have to fix that, either with better prompting or with a skill, or if the model's missing context, give it an MCP so it can pull in the context that it needs. That's kind of it. It sounds very simple. I think people tend to overthink it a little bit. I think people tend to over-engineer. Because I think in a lot of ways, when we built systems in the past, that's the way

8. 24:04

   you had to do it. So when I look at engineers that have been coding for a long time, for years or for decades, this is a really, really common failure mode, is trying to over-specify. And it's trying to be overly specific. And get the model to do the task exactly the way that you would have done it. And that's just not the way the model works. But I think a lot of people are kind of un-learning this and it's a journey to un-learning it and it's a journey to kind of figure out how do you treat this thing like you would a co-worker. I think that's the level of intelligence that it's at now. And as part of this, let's go deeper into this task that's still running two weeks since you launched