Give models harder tasks to reveal hidden capabilities

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 14:42

   So there's a couple of things that I will think about. One is you should give the model slightly harder tasks than what you think you can do. I think a really common mistake that I see is people are using quad code, they're using quad, and they just give it, like, way overly specific instructions. They're like, I want you to do this, but I want you to do it in this way, this way, this way. You must do, like, one, then two, then three, then four.

2. 15:10

   And for modern models, that's actually really not the way to do it. You want to go a little bit higher level. You want to describe the task. You want to describe the guardrails. You want to describe like the exit criteria and then just go with the model cook and come back in a little bit. And I think it'll surprise you. And again, like this is just not something that would have worked six months ago, but it does work today.

3. 15:31

   Can you give some examples of these challenging? tasks or capabilities that people should explore that it can do now that it couldn't six months ago? Yeah. So, okay. One example is the model can now rewrite essentially any code base from one language to a different language. It's just sort of crazy. Like, it's this work that would have taken just like a very long time as an engineer, and now the model's like quite fast at it. So one example of this is Cloud Code is built on the BUN JavaScript runtime. It's an open source JavaScript runtime. It's an alternative to Node.js. It's kind of a faster node. BUN was written in ZIG. ZIG is a systems programming language. It's kind of like C. It's very well level. One of the problems with ZIG is you have to manually manage memory. And so it's quite easy

4. 16:23

   to run into situations where there's memory leaks and other memory management issues. And so, one thing that the BUN team was doing is they were having Claude fuzz the code base and try to simulate and trigger memory leaks, and they were doing this for a long period of time. They were able to find a lot of memory leaks. It was sort of like a case at a time. And that was kind of the capability of the model at the time was doing this fuzzing.

5. 16:46

   And then at some point, Jared on the team was like, okay, let's just rewrite write it. Maybe the model can do this. And I think this is like one of these test problems that he kind of threw at the model with every new model generation. And starting with Fable, the model started to be able to do it. And so I think Opus 5 could do it as well. And so what he did was essentially he defined a test suite. The nice thing about Bun is it's very, very well tested.

6. 17:15

   There's a big test suite in Bun. There's a big test suite in Node.js. So it's easy to know if you did the right thing and he had the model rewrite it from zig to rust it was one prompt it was a dynamic workflow and a dynamic workflows are a feature in quad code that essentially let you orchestrate you know dozens hundred thousands of agents to do work productively and it ran for 11 days and it rewrote the entire code base and this was one shot it was one shot with well no it wasn't one shot but there was steering there was steering um but previous models just couldn't do this even even with the steering it just wouldn't have been possible just 11 days oh my god this would have taken in the past even with the best engineers multiple months years definitely over a year yeah yeah over a year this is like over 100 000 like javascript runtime is

7. 18:03

   really complicated there's there's a lot of stuff in there um and yeah it like it works this is in production now this is what quad code uses now when when you're running it so this is kind of one example. I would give a second example, so a product overhang. And so this is like a practical use case where like there's a problem you're solving. It's like a business problem, an engineering problem, a product problem. And you should just keep throwing the latest model at it to see if it'll just do it. Because even if a previous model didn't, the new one might. I think the second way to think about it is experiment. And just give yourself like freedom to play with the model and do creative things often it'll surprise you so something that's actually been really popular at a internally that's been kind of viral within anthropic the last couple weeks

8. 18:47

   is someone figured out that you can give opus 5 open cv and you can have a draw and so something you can do is you can ask opus like hey use open cv to like draw this image and it's actually quite good it can do like portraits it can draw like animals it can do like landscapes and we didn't train the model to draw. It's just like the solicitation gap. If you ask it to do it the right way, it can just do it. And we discovered this kind of accidentally just by playing around and trying creative things that didn't have direct commercial applications. But it's just kind of interesting. And my hypothesis is there's probably dozens, hundreds of opportunities like this with the models of today that no one has yet realized. And the big area of research for this