A single prompt is a poor test of AI coding

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:03:43

   steers me to better architecture so why do you think there's you know a bunch of like experienced devs who are still pushing quite a bit back on on just like the idea that ai can do a lot of this that was a week ago i i stumbled over a blog post by McGallagher, Cook of His Love, that I deeply respect and learned a lot from. And this blog post was just was a dissing of the current way how models work.

2. 1:04:14

   And what he did was he tested like five or six models, including some that make no sense, like the OpenAI 120 billion open source one. It's not good enough to write good code. And he He just, he wrote a prompt, as far as I understand it, there was not a lot of information on the episode, but to me it sounded like he wrote a prompt, he put it on CloudWeb, and he pressed send. And then he took the output and ran it and it didn't compile.

3. 1:04:47

   And then he was disappointed. But he's like, of course it will not work. Do you think I can write bug-free code on the first attempt? and those little those models are ghosts of our collective human knowledge they work very similar in many ways of course you don't get it right the first time like there will be mistakes that's why you have to close the feedback loop and also you don't just send a prompt to the model you start a conversation hey this is what i want to build it's like he complained that it used old api yeah you didn't specify the macOS version so it's so it made an assumption to default to like old api because that information was missing and it is it is trained on a lot of data not just the last two years and there's just more old data than new data so this is like the more you understand how

4. 1:05:37

   those little beasts think the the better you get at prompting um and then and then and then he He spent maybe, I don't know, a day on playing with it and then just decided that this technology is not really good. But to be effective, you have to spend significantly more time. You know, it's like you know how to play guitar and I put you on the piano and you try it a bit. It's like, oh, this sucks.

5. 1:06:05

   I'll go back to my guitar. No, no, it's like it's a different way of building. It's a different way of thinking. you have no idea how often i screamed at like 3 a.m to cloud code because it did something silly i slowly started to understand why those things do what they do with like exactly the way i tell it to do things and sometimes you can literally ask you can even even last year like i for this project i the last project like cloudboard i feel like a human merge button because the community is like blowing off and all i do is like reviewing prs i i have very little time to actually write code myself anymore and in the beginning it would often like just cherry pick things and would close the pr and i was like so annoyed so i'm like why are you doing this yeah when you say this and this

6. 1:06:51

   i interpret this this and this it was like ah like i i learned the language of the machine a little bit more i tweaked my my prompting and now i get exactly what i want because it's it's a skill like any other skill yeah and this is like simon willison has been saying the same thing even though we've been using it for years and i think everyone i think once i start to use it i also realize like i'm not i'm okay at it but i i could do better what if we put this to a real test because