Feedback loops help AI produce better code

Transcript

From the transcript Odd Lots publishes with the episode.

1. 18:29

   Why do the models when you ask them to produce some code, like often they'll produce code and they'll be a bug in it, and then you ask it to debug itself and it does it and I never understand, like it knows the answer, But the first iteration is wrong. What exactly is going on here? At a technical level? I guess that you know, the first thing is a bit wonky, but then it fixes itself in the next iteration.

2. 18:53

   Yeah, I mean, like think about how you do a math problem or you know, like how you do a piece of writing. Like usually, like when I do a piece of writing, I don't get it perfectly right the first time. I do like a first draft right, and then maybe I'll edit it like a few times, and then at the end it becomes something good and sometimes it doesn't. But you know, it's kind of the same thing for us, like the creative process never goes directly to the right answer.

3. 19:16

   Models are not even for. Code, which I think of as like a very structured thing.

4. 19:21

   You think about a structure, but you know, like to me as an engineer, like I've been writing code for a long time. To me, when I write code, it's like writing poetry or something. It's a it's a creative act. There's many ways to write code. There's some ways that are beautiful and there's some ways that are ugly, and there's just there's a big spectrum. It's not just black or white like this.

5. 19:36

   I'm glad you asked it, because this is another question and I have no idea what the answer is. If you look at code, like we all know about the writing ticks that all AI models have. It's not actus. It's why the m Dashers, et cetera. And it's weirdly an area where we haven't really seen much. It's funny because I use I use now. I'm actually like switching to parenthetical is more just because I'm self conscious about it. I'm just curious, like, as someone who like knows code, is there other equivalents in the code world that you see where like, I'm just curious.

6. 20:12

   I wouldn't even know how to ask this question, but these sort of formulating ticks in the actual production of code that would be the equivalent of writing and language.

7. 20:20

   You know, I think six months ago I could have given you a big list. Nowadays, the code the model rights is almost every time better than the code I would have written. Really, and this is new. This is since I think Opus four point seven maybe four point eight definitely fable. That's where it got to this point.

8. 20:37

   When we see like, okay, you give it a prompt and you know people have to show on like Twitter or whatever, like one shot of this. I asked it to build like an app, and it did it in one prompt, et cetera. How much of this, when you say it's better, is because it produces code that's better or because of that iterative process, and I mean the whole thing with coding, and we should get into this that's different than creative writing, et cetera. Is it like could try things and it doesn't work. That it tries things, it doesn't work. It tries thing, it doesn't work until it gets at the right answer. And you could see, like very clearly when you're using Claude code when it runs into a dead end, how much is it about like it could produce a better code or versus is just very efficient at these iterations until it arrives at quote,

9. 21:25

   you know, the right outcome. It's definitely both of these. The way I like to think about it is imagine that you're a sculptor and you let's say you're just like the best sculpture in the world, but you know, this time you're making a sculpture and you got to wear a blindfold. You can't see it, and you also can't feel it. You can sculpt, but you can't see it. It's going to look okay, but it's not going to be your best work.

10. 21:43

    I bet you know. If you're the best sculptor, but if you can maybe feel the sculpture, or if you can kind of peek at it with one eye, maybe the sculpture will come out a little bit better. And if you can kind of see it, fully see it, and you have this feedback loop, then the sculpture might come out incredible. And it's the same thing with the model. As it gets better and better at coding, that first pass is going to get better and better. So it's like the sculpture is going to look nicer and nicer, but without that feedback loop, Like, if quad can't test the website it's building in a browser, if it can't open the iOS up it's building in an iOS emulator, if it can't open up the distributed system that it's writing and actually we're on the service end to end and use it, is this not going to be as good as it

11. 22:21

    could have been. And so it's kind of the same thing. If it can loop a few times and it can check the output of its work, it can iterate, then it's just gonna be much better.