Agents work best when they can test and debug their own output

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 57:27

   three other things it crashes or you give it the good thing how to be effective with coding agent is always like you have to close the loop it needs to be able to debug and test itself that's the big secret um that's also something i i think that's part of why i got so much more effective um but But yeah, with Cloud Code, I often had to go back and fix up the stuff, or it just takes a lot of iterations.

2. 57:55

   So in the end, it's not that much faster, it's just more interactive. And these days with Codex, it just almost always gets it right. My general strategy is always I build a feature, and of course, you always let it write tests, and you make sure that it runs its own systems. It runs them, yes. And so even when I write a Mac app, I don't know, like I just, yesterday I debugged this feature where the Mac app couldn't find a remote gateway, but like the same code in TypeScript could, but Mac app is kind of annoying to debug because like it builds it, you have to start it, you have to look at it, you have to like, no, this is not working.

3. 58:39

   So now I just tell it like, you know, you're going to build a CLI just for debugging that invokes all the same code paths that you can call yourself and then you just iterate and you fix it yourself and then it will just cook and it just cooked for an hour and it was done and it told me like there was a race condition here and here and like a misconfiguration blah blah blah I was like yeah it sounds sensible I don't need to I don't need to see that code it's like but you don't need to see it because you set up the validation loops and you trust that because it ran it I mean this is I guess it's not too dissimilar to like sometimes when you work on a large project in a large company when all the tests pass I mean it doesn't mean that 100% is there but it's it's a pretty good and and all the new new code has test as well you know someone thought about

4. 59:19

   it and tested it and all that but even even on my on the very little project we always had bugs for like anti-gravity has like a certain a certain weirdness with how it takes tool calls in the loop in the format so you have to like some filtering yeah and i broke a bunch and it actually took me way too long to realize like what am i doing here i just need to automate this so i was just going to codex it's like design live tests that spin up a docker container install the whole thing spin up a loop use my api keys from this and this file and then you tell the model to read an image create an image before and then look into the image and see what it sees so i don't not just tell the loop i'll still tell tool calling make it work and then it solved itself it took forever but it it tested all my api keys like from from entropic over setai over glm like

5. 1:00:18

   everything and it fixed all those little intricacies where where sometimes the tool calling didn't work or the ordering was wrong because i closed the loop and and that's that's And closing the loop, you mean just have a way to have the agent be able to validate its work? That's why, that's the whole reason why those models that we currently have are so good at coding, but like sometimes mediocre good at writing creative, because there's no easy way to validate, right?

6. 1:00:46

   But code, I can compile, I can lint, I can execute, I can verify the output. If you design it the right way, you have a perfect loop. like even now for websites I built the core in a way that can be run via a CLI so I have this perfect execution loop because the browser loop is insanely slow you want something that loops fast So it sounds like one thing that is not really changing from like before is we had this before like backend or business logic heavy thing could easily be or more easily be verified that it's correct Surprise! Actually using authentic coding makes you a better coder because you have to You have to think harder about your architecture so that it's easy and verifiable because verifying is the way how to make things good.

7. 1:01:31

   Well, and remember back even before AI, for complex systems, like once you got someone who built these things before, what they started with, how do we make it testable, right? Like you need to design interfaces, classes, testable. You need to think about like, am I going to fake things? Will I use mocks? Will I use end-to-end testing, which will be long, et cetera. But these are like really hard architectural decisions.

8. 1:01:53

   And once you make them, they're, I guess, harder to change. And in your word, you know, like the model would cook a lot longer if you ask it to make a massive refactor. And, you know, if you have tests, it'll get it right. But, you know, we still have these trade-offs. It's still software. I would say I write better code now that I don't write code myself anymore. And I wrote really good code. But, like, even back at the company, sometimes testing was so tedious.

9. 1:02:21

   and you come up with all those edge cases and the branching. I mean, outside of Ken DeBeck, who I deeply respect, and he was on the podcast and we talk, like he still writes tests first and he tells me, he's not mad at me for not writing it, but if you want to write like, you know, poor quality code, it's on you. But I don't know many developers, myself included, I never liked writing tests. And even when I pretended that I did, I just never did.

10. 1:02:48

    It's a little bit like writing documentation and writing tests to me it was never a creative expression it is so good now like i i would say for my last project i have really good documentation and i didn't write a single line myself like no i don't write the test i don't know documentation i explain the model the trade-off so like why we did something like this and then tell it like like write that write the entrance section beginner friendly and then more technical detail at the end and it is so good i never had a project with that good documentation just by every time we design a feature this is a part of this is a part of the process and also like testing i'm almost like okay we built this how are we going to test this yeah we could do this and this what if we build it this way and oh yeah then we can test it better

11. 1:03:32

    it's like this is now part of my thinking because i i always think like how do i close the loop how do i the model always needs to be able to verify the work itself um which automatically