Claude Code gave models write access to whole files

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 10:30

   the best agentic products on top of a plot is this concept of an unhobbling plot. And tell us more about what that means. Yeah. So hobbling is this idea in research that the model is doing something and you're just getting in the way. There's this kind of like way of thinking about it that I really like. It's very useful when you're building product. And it's called product overhang. And the idea is the model is able to do all sorts of things with today's models, not a future model, but today's model, that we have not yet realized.

2. 11:13

   And there are so many capabilities the model has like this that people are not aware of. And this is like the ability to, you know, like maybe use a particular tool, use a particular language, solve a particular kind of problem, do things a particular kind of way that we thought was kind of beyond the model's capability. And there's this overhang because the model can do this at every given model generation.

3. 11:41

   But there is often not a product that lets the model do this and lets it express this kind of ability to do this. And on the flip side, often what happens is the product gets in the way. And this getting in the way, we call this hobbling, and then not eliciting the correct behavior from the model, we call this product overhang. So it's kind of like two sides of the same thing. One example of this was the original plot code.

4. 12:06

   When I first started working on it, this was, you know, like a year and a half, two years ago, something like that. This was like SANA 3.5. At the time, that was an incredible coding model. That was like the best coding model that exists. Nowadays, it's, you know, a pretty terrible coding model by modern standards. But I think that was like the first great coding model that we built as Anthropic. And at the time, if you looked at the coding products at the time, what were they doing?

5. 12:33

   They were doing single-line autocomplete. They were doing sometimes multi-line autocomplete. That was sort of a new idea. They were doing chat. So you can talk to the agent, but it wasn't write access. You could only read. You could ask about the code base. And so the feeling was that there wasn't really a product that was fully eliciting the model's capability to write entire functions at a time, entire files at a time.

6. 12:59

   At the time, it wasn't entire features. We weren't there yet, but probably entire files. That was the level of capability at the time. And so the idea with quad code was, all right, we think the model can probably do this. What if we get rid of all the scaffolding and just give the model the simplest possible harness so it can write an entire file at a time and build an entire feature? And that was kind of it.

7. 13:23

   Like, that was the product overhang at the time. The model was capable of doing something, and everything was just kind of getting in the way. I think that nowadays, with modern models, there is so much product overhang that I'm not seeing startups capture. And I think there's people thinking about these problems, but there's just a huge amount of opportunity to elicit these behaviors from the model that are just amazing and interesting and commercially valuable.

8. 13:50

   I think this is such a special insight for everyone here in the room. Basically, all of you could create the next CloudCode if you figure out how to unhubble the models because that's effectively the birth story of CloudCode. You unhubble Sonnet 3.5 because all the previous iterations were still getting the model very rigid in IDEs. And CloudCode was one of the first instances that gave it just a full terminal access.

9. 14:21

   Yes.