Give engineers tokens before optimizing costs

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 24:02

   different. I hear you have these very specific principles that you've codified for your team, that when people join you, you kind of walk them through them. I believe one of them is what's better than doing something, having Claude do it. And it feels like that's exactly what you described with this memory leak is just like, you almost forgot that principle of like, okay, let me see if Claude can solve this for me.

2. 24:21

   There's this interesting thing that happens also when you underfund everything a little bit, because then people are kind of forced to clodify. And this is something that we see. So, you know, for work where sometimes we just put like one engineer on a project and the way that they're able to ship really quickly because they want to ship quickly this is like an intrinsic motivation that comes from within it's just wanting to do a good job one if you have a good idea you just really want to get it out there no one has to force you to do that that comes from you um and and so if you have claude you can just use that to automate a lot of work uh and that's kind of what we see over and over so i think that's kind of like one principle is underfunding things a little bit i think another principle is just encouraging people to

3. 25:02

   go faster. So if you can do something today, you should just do it today. And this is something we really, really encourage on the team. Early on, it was really important because it was just me. And so our only advantage was speed. That's the only way that we could ship a product that would compete in this very crowded coding market. But nowadays, it's still very much a principle we have on the team. And if you want to go faster, a really good way to do that is to just have Claude do more stuff. So it just very much encourages that. This idea of underfunding, it's so interesting because in general, there's this feeling like AI is going to allow you to not have as many employees, not have as many engineers. And so it's not only you can be more productive, what you're saying is that you will actually do better if you underfund. It's not just that AI

4. 25:48

   can make you faster, it's you will get more out of the AI tooling if you have fewer people working on something. Yeah. If you hire great engineers, they'll figure out how to do it. And especially if you empower them to do it. This is something I actually talk a lot about with CTOs and all sorts of companies. My advice generally is don't try to optimize. Don't try to cost cut at the beginning. Start by just giving engineers as many tokens as possible. And now you're starting to see companies like, you know, at Anthropic, we have, you know, everyone can use a lot of tokens.

5. 26:19

   We're starting to see this come up as like a perk at some companies, right? If you join, you get unlimited tokens. This is a thing I very much encourage because it makes people free to try these ideas that would have been too crazy. And then if there's an idea that works, then you can figure out how to scale it. And that's the point to kind of optimize and to cost cut, figure out like, you know, maybe you can do it with Haiku or with Sonnet instead of Opus or whatever.

6. 26:43

   But at the beginning, you just want to throw a lot of tokens at it and see if the idea works and give engineers the freedom to do that. So the advice here is just be loose with tokens with the cost on using these models. People hearing this may be like, of course, he works at Anthropic. You want us to use as many tokens as possible. But what you're saying here is that the most interesting innovative ideas will come out of someone just kind of taking it to the max and seeing what's possible. Yeah. And I think the reality is at small scale, you're not going to get a giant bill or anything like this. If it's an individual engineer experimenting, the token cost is still probably relatively low relative to their salary or other costs of running the business.

7. 27:24

   So it's actually not a huge cost. As the thing scales up, so let's say they build something awesome and then it takes a huge amount of tokens and then the cost becomes pretty big, that's the point at which you want to optimize it. But don't do that too early. Have you seen companies where their token cost is higher than their salary? Is that a trend you think we're going to find and see? You know, at Anthropic, we're starting to see some engineers that are spending hundreds of thousands a month in tokens.

8. 27:49

   So we're starting to see this a little bit. There's some companies that we're starting to see similar things. Yeah.