How Anthropic ships the #1 LLM in the World — with Katelyn Lesse and Angela Jiang

And it's notable that basically in Q2 of twenty twenty-five, Anthropic was basically losing half a dollar for every dollar of revenue, and now it's making two dollars for every one dollar in rev-- in cost. And I think that's an interesting, uh, analog-analogy of the, like, how enormously huge this inflection in Anthropic's revenues and business outlook has been over the past one year. Like, this year was extremely critical for Anthropic to execute well, and it has done that. And I think we're g-- basically gonna drive deeper into this and tell that story as much as we can from the podcast external point of view.

Um, I-- the one consequence of this entire increase is that Anthropic has just reported its first profitable quarter, like, not adjusted profit, which is profit after we take out all the things that cost us money. No, this is generally accepted accounting principles GAAP profit, uh, of nineteen percent, and that is enormous, again, completely flipping from one year ago. Um, more than flipping. Like, one year ago, it was losing a hundred and twenty-nine percent, uh, of every revenue, uh, dollar they came in. So the primary attribution, I think, that explains this extraordinary efficiency, as you know, Claude loves loops, Boris loves loops, Dario loves loops, and there's two loops going on in Anthropic.

The first is the improvement loop. Basically, there is the cycle of you have Claude. Claude leads to better models. Mo-- better models lead to more use cases. More use cases generate more feedback, and that leads to a better Claude. And then there's also the investment loop. So revenue from, uh, expanding adoption leads to higher growth. Higher growth leads to more investment, and more investment leads to more capabilities. That leads to a better Claude. So these two loops basically spin together and form an infinite loop that is driving the increase in efficiency until the end of time, according to the Anthropic S1. So I think that is our transition from the overall efficiency story to how they execute it.

And today we're gonna focus a lot on the Claude platform because that is primarily, uh, one of the three things that they call the diffusion of Claude in the Anthropic S1. So they call this Claude platform our comprehensive agentic development environment. They provide customers with direct access to the primitives to build, deploy, scale their own applications that leverage Claude's capabilities within their own products and workflows. I think one thing that comes up a lot in the discussion in this podcast is the concept of ad fooding, where, uh, Claude is bu-- a lot of Claude's products is actually built with the same Claude platform that is available to you.

So, um, something that's also interesting in the Claude platform that, uh, was disclosed is over fifty percent of their customers with a hundred thousand more-- dollar or more revenue on a trailing 12 month basis use Claude Platform. And so that is an increase in an adoption and emergence of Anthropic not only as a model lab, but as al-also a hyperscaler and a cloud platform, which I think maybe is in the name inherent in a Claude platform, in a cloud platform, but is going from theory to actual practice, and this is something that we're focusing on in the podcast. The other thing that I also really try to emphasize, which I talked about before, is Claude Code, Claude Cowork, they all use the same primitives that Claude Platform, um, exposes, so I think that's really interesting.

Ultimately, it looks like Anthropic is really positioning their IPO to talk about going from chatbots to copilots to virtual collaborators. They use this term instead of agents, maybe it's more friendly, but it's a lot of the same things that you look for from agents, from long-running autonomous work to judgment, self-correction, and iteration and tool access. So owning a goal and working alongside of you instead of working inside of one app or task. So that's ultimately the context I want to provide for this-- today's podcast. This was recorded when Opus 5.5 was launched, and I would say overwhelmingly among all the high-taste testers that we talked to, Opus 5.5 is the best model in the world right now.

And Anthropic is even just sitting on Fable 5.5 and probably will release it in the next two weeks. I don't have, have insider information, but it makes a lot of sense for the IPO. So let's look forward to that and learn a little bit more about how they build Opus 5.5. Yeah. So maybe a distinction that I would make is in a lot of the incidents that people are hearing about and reading about, these are incidents that are happening to models, not just our... Like, everybody's models, right, that are in training, where the way that you're running these models is you, like, put them in a box and you say, like, "Solve impossible task. I'm gonna stress you out.

Ready, set, go," right? And you, like, can only do what you can do within your environment, and the agent is basically incentivized to try to get out of his box and go solve the problem. When people are running production workloads on our API or on anybody else's API, you're running an environment where, like, there are safeguards that are running that are, like, not necessarily running in training where, like, the whole point is to actually train the model, right? But in production workloads, first of all, you have those online safeguards, which is really great. Second of all, you have more of this architecture and infrastructure around what's happening.

Like, the agent isn't just in a box trying to get out, right? Like, the harness is over here, the hands are over there, and, um, it's just, like, not even a problem that you would have to contemplate. All right. We're here in the studio the day after Opus 5.5 with Kaitlyn and Angela. Welcome. Thanks for having us. Thanks for having us. [laughs] Uh, congrats on launching the top model in the world right now. Fair. I would say depends who, depends who you ask. Yeah. [laughs] Depends who you ask. Um, what is the main takeaway personally for you guys? And, you know, you have, you have your official blog posts.

The, the pricing cuts are, are really cool. What do you take away when you launch a new model like this? We're super proud of this one. Um, the team has put in a ton of different work to just go make this one happen, and I think we've, like, really heard the feedback on Opus 5's writing is, like, kinda convoluted. We really tried to clean that up, and I'm really glad to see that. Like, it seems to be resonating with people. Internally, when we, like, tried out the model, we felt the same way, so it's just really g- great to get that validation. To your point around the pricing cuts, there was, uh, a lot there, and I think increasingly people are looking for good intelligence per dollar.

Um, so making sure we get the per dollar part right, um, has been really important. There's some stuff on the edges that aren't really about, like, you know, maybe stuff that we haven't talked about in the blog post, which is that we did do a lot of work on trying to make our API harder to distill from. And so there has been some changes that we have made to go and make that a little bit more concrete, and these are things about, um, you know, like, how do we do, like, prefix lock thinking or how you can replay thinking tokens across different types of models. Um, and there's been a lot of, like, work and effort on our part around making sure that we try to adjust things for very valid use cases for where you might want to go and do this, like potentially in your compaction strategies and things like that, versus in scenarios where, um, the use case of that is actually from, like, a bad actor effectively.

So there has been a lot of threading the needle around those fronts, and I'm glad to see a lot of that co-opt as well, um, in addition to just, like, shipping a model that's, uh, good intelligence, uh, for, for your dollar and also, like, just communicating a little bit clearer. Yeah. Uh, specifically you're talking about, uh, this paper, Stealing Reasoning Traces from Proprietary LLMs. Yes, that paper. [laughs] That one. Which is real as far as- Yes. Right? They- Oh, yeah... I mean, obviously they did, like, responsible disclosure with you guys. They did super responsible disclosure, and we're super thankful for, for them coming to us and just like- But, like, could be, uh, for, for those who haven't heard about it, it's... this is really cool- Mm-hmm...

that you could do it. I mean, it's patched, but, like- [laughs]... like, what, what was it? Like, did it cause a freak-out in, in, in Anthropic? Probably wasn't that big a deal. I don't know.

While we're on the topic of it, anything on the other side of distillation, so, uh, you know, the other half? [laughs] Yeah, no, we, we have seen that it is harder to distill, like through the measures that we are able to, to take a look at.

It has been significantly harder 'cause this is, uh, and, you know, credit to the, the researchers here, they, they did disclose it, uh, very responsibly. Yeah. Um, and so closing it down was, was great. Um, and they did publish the results, uh, very publicly, and I know it, it did kinda trend quite a bit on Twitter. Um, so many people could, could use it. Uh, but yes, we have seen that actually from a, the areas that we are able to measure, um, it does seem to have- Mm... uh, reduced the effect, um, the negative effects that we don't want. And yeah, the bit I would just add to what Caitlin just said is I think, uh, at least I see a lot on Twitter or X or whatever you wanna call it, um, so it's a commercialization problem or something like that.

You know, if you can steal a model and you can take this, then it's, you can run it for cheap and, and so on and so forth. While true, um, it's actually not something we really care about very much. We care much, much more about this from like a safety point of view, um, more than anything else. And so that's why we, we tend to think about it from that angle more than just like commercially. Realistically, if you distill from us, we, you actually pay us, so commercially we like make the money. Um, so it's not really that kind of an impact. Um, it's just more on like making sure that our model's like we wanna cla- we wanna ensure Claude is like good, and Claude could be derived, and if it ends up negative, that would be unfortunate.

Just generally al- also on Phi-5 'cause you, you guys are- Mm-hmm... just fresh off this, one thing that's really fun to establish the dyna- dynamic between you two because you also do a lot of appearances together, is you're running, uh, you're head of products, you're head of engineering. How does this play between the, the two of you? [laughs] Uh, especially for a model launch. Like just take an example. So for model launches, so our team works together very closely, um, like between product and, and eng. On my side, a lot more of it is about like what is the balance of trade that we need to make between like launching something that has the right like intelligence bar, but also the, the cost to preservation.

Um, like a lot of like our, our updated cash costs and things like that are indications of us being like, we want to like encourage you to take integration changes that are beneficial for actually everybody, and just like pass those savings on. Um, but a lot of these kinds of decisions around that and then how do we balance, uh, some pieces around our safety motion as well as making sure that it's like an awesome model that people can, can take advantage of. And so those kinds of traits around like I would say maybe the qualities and surrounding components around how we launch the model are more on product.

And then from an integration experience standpoint, a lot of that is from like the engineering team directly. So we work, obviously if you're like talking about the experience of an API that's like very, very, very tightly integrated, and so it's actually been really awesome. We have a lot of engineers who actually are deeply product-minded, and we actually like love that. And similarly on my side, um, there's a lot of like product managers who are much more technical, and we want this like blended kinda interdisciplinary type of people because then they can kind of analyze it from all the different angles to try to figure out how, how, uh, to do that best.

But I tend to lean a little bit more on like commercial impact versus I think I'd actually frame it as more like the engineers are going to care much more about the integration experience. Um, so they almost take more of a UX angle to it, um, where the UX is obviously out more about the API, um, and the developer tools. Yeah. It's fun to be a product engineer who works on developer products and- Yeah, developer platform... and platforms. Yeah. Mm-hmm. So I would echo that. I think we have a lot of engineers who love thinking about the product, who love thinking about the user experience, who love thinking about the developer experience.

I think it's a dynamic that we have between ourselves too, where, um, I have a lot of opinions and [laughs] am annoying with our product team on how we think about these things, and our product team is also really technical and they can get in the weeds of decisions that we're making for scale or for whatever it might be, um, which is a fun dynamic that our team has. Yeah. Yeah, something, uh, Caitlin and I have worked together for a long time, um, and- Stripe and- At Stripe, and then, uh, I guess we did not work together at OpenAI. Um, but- Separate labs... just separate labs. [laughs] Um, and then I came over.

But, uh, I think some, one interesting property of both of us has been Caitlin frequently actually gets mistaken for a product manager, and I frequently get mistaken for like someone in engineering- Mm... um, even though like those are not actually our job functions. Um, and so I think that pairing tends to like help out a lot with just making decisions that are consistent- Yeah... across both sides. So I, I think I come across a lot of companies with a similar like- Hmm... product leader, engineering leader set up. As software engineering gets solved, maybe those two roles merge? I kinda think so. Yeah.

I think everyone tends to lean towards- But if you guys are, you know, separate still, then we, we're not there yet because- [laughs] We're definitely not there yet. Um- So, so like why, right? Like what's left? I think that like software engineering getting solved is, I would, I don't know if I would ever use language quite that strong. I think there's, the way that I tend to think about these things, like ambitions will continue to elevate, right? And maybe a good example, like a lot of what our team is spending time on right now is, of course, obviously all the things that we have to do to launch our model safely is actually the most important thing that we're doing because of all the work that we have to do on security with agents, with offline safeguards and online safeguards, all these things that are happening.

But the other half of what we're doing is trying to scale. [laughs] Um, and part of that is because when I joined Anthropic a little over a year ago, we had the messages API, it was a Python monolith, we were running it in a, you know, single GCP region, and everything was fine, and none of my friends and family, like, knew what Claude was when I was going to take this job- [laughs]... right? And now we're in an extremely different place. And so the metrics are really wild in terms of how much volume we've seen and have had to support on the API. CPU is really constrained, uh, which has been a really interesting one for us.

Um, and so we've done a lot of work on how do we scale these systems, and that takes a lot of obviously just, like, distributed systems, expertise, and certain language and frameworks expertise that the models have, but when it comes to operating and evolving a system at scale like this, we're not quite- we're not nearly in a place where it's like, "Good luck, Claude. Have fun, uh, and just go and kind of make all the changes you need to make to the system, roll them out safely, and operate this thing at scale." Um, there's still just a lot of work for humans to have to do that I imagine will continue for a while. I'm curious how that's changed.

So there's a lot of complex things that are starting to happen around model launches that, again, we just never used to have to think about. Oh, that's good. Yeah, and it's, it's scaling up with capability. Like, as our capabilities get w- uh, more and more advanced and they start to spike, you know, in specific areas, that's where we need to pay, like, extra attention.

And so a lot of that evaluation comes just from, like, those very particular cap- capabilities that we might either see as, like, offense-dominant or defense-dominant and therefore might take different strategies for all of that. And as we bake that all the way through to eventually the model launch, there will be components, kind of like Caitlin mentioned, where we kind of don't exactly know how people might use those capabilities. There are some more obvious ones, like between coding, right? Like, great. Go make awesome products. Go ship more things. Um, it can be really awesome for you. And then other areas like bio and cyber, it's like it could be dual use, and there could be negative effects, but obviously, of course, you could be a really valid cybersecurity researcher.

You are trying to find vulnerabilities. You are trying to be a good actor. Similarly with bio, you might be researching and stuff like that. And so a lot of these programs, like the one that you, you have up here, are to basically provide access to that dual use. You do have to specify your use case and things like that. Um, but we do want to enable the community, uh, to be able to take advantage of these capabilities and see what amazing things that they can do with it as well. Well, we'll put up the links, and maybe we'll dive into them later. There's also one for bio. Mm-hmm. Also, I just think at a high level, you know, it's, it's really important that you guys actually think about this stuff. People forget, like, Fable was rolled back and- Indeed...

there's all this distillation, and people are like, "Ah, zero data retention is not a thing on Fable. Crazy." But you have to remember, like, you know, it got pulled back. Uh, there's distillation attacks. Like, it's good you're overthinking and doing it in the right way while releasing it, right? Yeah, and a lot of that is, like, building the right framework, um, just so, again, kind of talking about the end side of it, building the right framework in our system to be able to do that. So I talked a little bit about this, like, this idea of, like, authorization and trust grants that we've built throughout our system now. [laughs] Never say never, right? Not, not as, not as precedent. Yeah.

Uh, but also it's, like, a very common thing in platform engineering that you, like, uh, you have one incident and then you, like, overprepare for that incident, and then the incident never happens again. Yeah. [laughs] Oh, yeah. It's just like [laughs] it's like- It's true... what's your regular- It's true. Like, you know, now you have the button. Well, it's true, and it's funny you say that because it was a, like, with everything going on, we then had this line item in our, you know, the work that our team was doing that was, "We're gonna put this system in place that makes it so that it could be a single button to pull back for everybody."

Yeah. But, um, and so it's funny 'cause it's like- Do we justify that we're doing this work? 'Cause we don't, we hope that something like that never happens again. But I think in this case it's really just a laddering up of those same primitives that we wanted to put in place everywhere to help us manage these verification programs and access to models like Mythos and otherwise, to just say like, just sit that same thing below, like everybody's model access, so that in case we ever needed to, we had the right controls in place. Yeah. And- Decisions like this have actually proven to be... Like, we'll see about this one in particular, but examples like this have been consistently the way that we've had to actually transform the way we develop our roadmap because, you know, in traditional product management, I guess you can be like, okay, like these are the set of things I hypothesize to happen.

They will have like a s- certain sequence, a certain ordering, but you are picking bets in a way that are much more concrete versus, uh, like, you know, the de-deployment of Fable, for example. I don't think any one of us could have guessed that that would've been like a valid use case that would've happened, um, by the government and the way we just like had to go do it. But clearly the probability is non-zero. And so from this form of thinking, it's become deeply probabilistic actually, um, from like a product management standpoint of like, so should we prioritize this? What are the chances of it ever happening again?

When we look at things like that, we're like, okay, if we are to imagine the most crazy set of possibilities, we probably actually have very little certainty as to where things will go. And so we actually just kind of need to like hedge and build a portfolio of bets. And so more often than not, we will take hedging scenarios of like, okay, we don't think it would happen again, but in the event that it does, we would actually want that hedge or, uh, have the ability to be able to do that very well. And so we will prioritize things like that. And then we have increasingly found that more often than not by taking that strategy of product development, most of the time we actually find ourselves being like, we actually do use the thing.

Is there an- a another example that's a separate one, just so I can understand? For example, this, this access program. This access program is a really good one. This, uh, it's internally it's, it has a bunch of granting and all these kinds of things. We developed it specifically for reasons about like being able to actually give programs like this out to customers, like very, very specifically. Do you wanna give a high level what the access program specifically is? Sure. So if you want dual use, if you want... So for, let's, let's take a model. So you take a model, uh, call it like Opus, um, and you want to be able to use advanced cyber capabilities on, on Opus for research or for whatever you wanna go do, like you're, you're a defensive user.

Um, you can apply to this program to be able to, with your use case, and if approved, what we'll do is actually diminish the safeguards on your version of the model so that you can then go ahead and try to take advantage of those like cyber activities, um, for, for your use case. Now obviously that requires like per use case understanding, it requires scoping down to a particular user, 'cause we wouldn't do that generally, you'd have to like apply. And even things like this, like this kind of program, we designed a lot of the primitives that Caitlin was talking about to be able to handle things like this.

Now, it was surprising to us, or maybe like, you know, in the portfolio approach, we found that we were actually able to take this and make it ex- we were like, when we're designing it, we're like, what kind of should it be? How generic should it be? Should it not be too generic? Should it be overly focused on making this verification, you know, style portal plus granting of, of very specific capabilities of the model? And we tended to opt for like more genericism because we were like, we don't know. Like maybe we can use this for other things, maybe not. And then it ended up being massively useful for things like, um, you know, uh, access to the model writ large for managing, uh, any of our trusted access programs for handling de-deployments of models because we were actually very, very generic about how we handled every single primitive.

From a normal product standpoint, I would nor- normally probably be like, "That probably doesn't make a ton of sense. We should focus on the use cases that make sense." But here, increasingly, because we choose the more generic approach, we find ourselves reaching for that solution more often than not, and it pays like increasing dividends for that reason. It's something that you needed to have anyway and you just had a forcing function- Mm-hmm... to have it. On a technical basis, what, uh, so I'm, I'm from the AWS world, right? Yeah. I AMs- Yeah... OC, and then people like, uh, who get really intense about this talk about Zanzibar and all these, all these other technical architectures. I'm not familiar with the word token grants or, or like, uh, uh, trust grants.

Trust grants, yeah. Trust grants. Is there a difference between normal OC and, and what trust grants are? I mean, not really. It's, it is, it's a, it's a form of authorization. Yeah, it's IM, but am- anthropics IM. Yeah, kind of. I think, so really we have like, um, there's like authorization of the form of like, okay, you're, you made a request to an API that like under the hood is gonna access some data, and we're making sure it's like your data that you're accessing, right? Like really, really vanilla authorization. This is just like we put a data model in place that's like a very clean... Like a trust grant means like you have been given access to some specific model or you've been given access to some specific program or, you know, like there's a, a bunch of different permutations of this.

We actually find ourselves reaching for it all of the time. Mm-hmm. Um, another good one was when we first rolled out, um, some of this preserved thinking in, uh, or for the, for Fable 5.1, we kind of grandfathered in some people and gave a bit of a, you know, just like a time- Grace period. Yeah. Yeah. A grace period to go and, uh, make the integration changes. And as part of grandfathering in, we said, okay, we have to like track who is like an existing customer who needs the extra time, right? And then we did a trust grant, and so then we're able to say for anything, we're just running this service that is kind of on the hot path of our request to make sure that, um, you know, we just like flow through and, and check whether you have the grant for the thing that you're trying to do.

Um, which is, it's really not any different than like vanilla authorization really, but like we just had to spin up a system that was specific to our system and like where we needed to be able to insert these checks. Yeah. Correct. I mean, I, I think, uh, these are all like the engineering things that people kind of take for granted. Don't- Mm-hmm... don't really think about. What is another example, I guess, of the platform stuff that had to happen, you know, maybe serving cloud code, uh, maybe serving, you know, obviously I talk a lot with cognition- Yeah... but other customers just in terms of a model launch, right? Like, we'll talk more about the, uh, the, the, the safety work that is, that's being happened.

I just wanna sort of go at a high level because it's fresh off your memory, like what goes into a model launch? [laughs] It's become a bit of a, a machine these days. Uh- Minus the safety stuff, which I think is, is the more complicated part. But outside of those pieces, there is a lot of getting feedback from our customers. So, you know, we make the model, there's a lot of safety testing. Um, there's also evaluation of where we think it needs to be. We make the required trade-offs and decisions as necessary, and then we have to go through testing. Uh, we'll cut various versions of the model, we'll evaluate it.

We'll do a lot of internal, um, ant fooding, uh, is- which is what I call it, and that actually always really helps. Like, we catch like really early things like, uh, this is, you know, maybe not the best, and we obviously use Claude a lot for our work internally, so we're able to actually see early signals around stuff. Um, and it helps us detect like, you know, which, which variants might be better than others. And then when we feel really confident with like a certain set of, of variants, uh, we will cut a version for our, for our customers. Um, sometimes that will go through our first-party products, uh, like through Cloud Code or Cowork or things like that.

Um, in other cases it will go through our API customers, um, and we'll engage in a kind of like a, a early access program effectively with those customers. They'll need to make changes into their harness, and we've engaged with, uh, a lot of customers like, like Cognition, like, like Hers, like these folks, and we have a bit of a process on how we handle that, um, so that we all kind of get... They always get the feedback that we, we need. Uh, does it perform well in your evals? Do you like it? What don't you like? Some things are, are vibey, and we like to hear those things too. Um, and some things are just like straight up evals, like, did it do well on the things that you care about? Yeah.

I think it's probably the... I- if I, if I were in your shoes- Mm... like one of the hardest things about PMing for a model, which is, uh, let's call it alignment versus personalization. There's a default personality, a default behavior that you want for all consumers, and then you want to be able to respond very quickly to someone's Claude MD saying, "No, no, no," like, "Please be more proactive than default." Uh, and then like, uh, so as to my knowledge, you know, and I was involved in Frontier Code, uh, we were involved in the Fable launch, uh, which put Frontier Code on the map, uh, and now Opus 5.5, we, we have it back on, on, on the, the, the model benchmark.

That is a very big part of Frontier Code, right? The, the reason that the model curve looks weird for Frontier Code is because it pen- penalizes too much activity- Mm... or too much, uh, files, uh, which is good. Like, that was a whole design purpose of Frontier Code, but like, I could easily just be like, in my Claude MD just say like, "Don't do that," and then it won't do it. Mm-hmm. Mm-hmm. And so, like ev- am I penalizing Claude Opus 5.5 unfairly because my benchmark doesn't have that personalization element, which is subjective? So I don't know what the question is, but like, how do you deal with like- [laughs]... like personalization, right?

Like, like it's, it like clearly- A model can do so many different things, and it's so depe- everyone has their own Claude MD. Uh, how do you eval that? [laughs] Yeah. We don't do a great job of it. [laughs] There's like the... There's... No, seriously, it's, it's really hard. It's super hard, and they're, they're competing and contrasting. Yeah, look, look at chat. It's, it's horrible. Yeah. Yeah. Um- I hear, I thought, I thought there's the Claude Constitution. I think- That's old. I, I... Yeah, yeah, yeah. There, there is a constitution, but I'll, I'll let you guys answer it, you know. The constitution's helpful, but, but for things like this on how it personalizes to instructions, uh, which are different, uh, and preferences which are very, very different across these things, um, is very hard for us to do.

We generally try to solve this by having more parameters that then allow us to tweak it per customer base, and if we don't control your customer base and you're an API customer, then we try to give you recommendations, and then you try to do that tweaking. So there are things like, you know, around... It doesn't exact- It's not exactly at the same level, like personalization to the Claude that, you know, various Claude MDs or now Agents MDs or whatever, but we do try to give things around the layer of like level of effort to like how much effort you should put in, like what are, uh, the, like the level, like maybe the size of the model that you might actually like try to care about, things like that.

But it's... I think it's, um, it's not quite there for what we would actually need for like that level of personalization. Um, so it's a very hard problem, and right now, a lot of our way to solve that is beyond the parameter, like the giving multiple parameters to something, is try to exhibit conflicting taste, honestly, a- across like all of the feedback that we get and try to ch- try to make like a judgment call on some of these things. I think over time they actually... and as you scale up more customers and more users, that's like not a very scalable way to go do something. Um, and we will need to develop ways for Claude to kind of like take in your personalization, um, and maybe that's like a signal you can provide or something you could tune on, or things like that.

Um, but I think we'll have to like take a route that's much more oriented, uh, towards that direction. Uh, but for now, yeah, no, we, we actually just try to make judgment calls on, on a lot of the stuff, and it's, it's pretty tough. [laughs] It is what it is. Uh- A small, a small note there. I guess the default effort reasoning- Mm... changed. It's now medium. This is one of those things that probably affects the average customer that doesn't use this toggle. Yeah, it's like a Claude code setting, right? It's a Claude code setting. They default it to medium. Yeah. They just- Yeah. You can turn it on, right? [laughs] And, and this, and this thing varies. This thing does...

Like, this is an, this is a perfect example of a parameter that we use to kind of like personalize. I, I would say that's maybe a little too aggressive of a term, but we kind of like customize maybe for different sub-segments. So like, uh, the default setting will adjust for, uh, whether you're a free customer, a prosumer customer, Claude code versus, uh, co-work versus enterprise, like things like that, versus API and stuff like that. Um, knowing that you can obviously always adjust it, but those are kind of like light ways where we try to give that. One thing that we are hesitant to do is we don't want people to feel like they can't control their customization of Claude.

Uh, so we could take a heavier hand on these things, for example, but we want to give users the ability to experience the intelligence levels of Claude as they kind of see fit, especially if you want to go towards like the higher end, or you want to experiment with maybe sometimes in the, in the lower end if that actually is just like faster for your use case, which is a hard product trade-off I think. Um, and so we, we tend to lean towards gentle defaults but give you options to, to adjust, which I think works okay-ish for now. But I imagine us, you know, like learning how to do that better over time for customers.

We're not like super, super happy with any of this yet, but we know as we launch a model, like y- you asked like what does it take to launch a model? One of the big things obviously that we have to do is like, okay, Claude like got good at a thing, [chuckles] and like sometimes that thing is something we should expose as a parameter- Mm... on the API to let you control. But I think as this has evolved over time, we've ended up with like this funny mishmash of like there's effort, there's budgets, there's all these things that you can set on the API, and it's like how could anyone like possibly figure out what is the right combination of things for any given task for what you should go and set?

And it's something that this is us just very much like figuring it out as we go and figuring it out with customers and figuring out like which of those parameters do we deprecate over time? Which of those parameters do we just like default better? Which of those parameters should be like a one to 100 scale versus a low, medium, high scale, right? Mm. Like, because actually we've trained Claude to get really good at understanding like a whole spectrum of options on some of those sorts of things. But like do we expose the like entirety of those millions and billions of permutations of options? Like, you wouldn't really get very far, and that's one of the things actually that, um, is part of the value prop with something like Claude Managed Agents, which is like still programmatic access to Claude in an API, but you get a harness out of the box where we do every time we're launching the model a lot of the work to go figure out like what are the new parameters, what are the breaking changes, how do we get the best out of Claude within our harness, so that for the people who are using it, um, you just kind of get that out of the box.

You don't have to think about it. Just for people who are, uh, attract- drawn to that- Yeah... and which there is a significant and growing population, particularly we actually did a whole CMA workshop at, at IAE, which I thought, uh, was really well done. Is there a ch- separate feed for people to subscribe to like updates on CMA? Feed like on- Like a Twitter feed, newsletter? I, I don't know. Like change log. No, I think actually our, our Claude Dev channel, we are- That's it? Okay. Yeah. We're, we're pretty loud about it, um, because it is- Yeah. That's, that's, it's Claude Dev... like an interesting option for people.

It's Claude Dev. Okay. Yeah. Yeah. I just want to give people like- Yeah... if that's your, if that's your way of consuming Claude, that's the, this is the path. Um- I think it's also pretty underrated, um, for those that aren't consuming Claude this way, what is Claude Managed Agents? Why should people use it? Oh, yeah, we should probably do a- [laughs]... do a little intro. Yeah. [laughs] I don't know. They- That's Ami's house. Yeah. [laughs] Ami will talk about it, but- A little, little, little two, two-sentence thing. Yeah. I think a lot of people glanced over it when it came out, right? Everyone was in this phase of I need to do harness engineering, I need to control this, and like you hear all that, and now it's like, do you need to do all that, or do you just use Managed Agents and you have a lot of infra you can skip over setting up?

But give, give the quick three. Yeah. It, it depends on your use case, but, um, so Claude Managed Agents is our API today. We have the Messages API, which is essentially raw token access. Then we have Claude Managed Agents, which is a suite of APIs, but it lets you define an agent, start a session with that agent, um, and it's essentially just kind of a higher order abstraction that lets you skip setting up all the infrastructure, setting up your harness, and doing a lot of that type of work. It's a much more out-of-the-box integration for people who want to programmatically run agents. Does CMA, uh, share infrastructure with Cloud Tag?

Because the sequencing of it seemed very like, oh, like that's the [laughs] Yeah. Yes. Cloud Tag. Um, a lot of it. Um, a lot of it. So one of the big things that we do is, as the platform team, we run our APIs. We also kind of run all of our foundational product infrastructure and product services. So, like, a really good example is, like, our team runs our sandboxing fleet that backs both Claude Managed Agents as well as products like Cloud Tag and otherwise. But at the actual harness layer for those two products, like, there's just differences in what we might wanna do within the harness so that they don't, they don't actually share a harness.

Um, but they share a lot of the infrastructure that powers the execution and, um, powers the APIs. I think that's, that's really... Like, people should use this more. Uh, I think everyone basically had copycat clones of Managed Agents after you guys launched it- [laughs]... which, eh, you know, imitation is a sincere form of flattery. Since we're on this topic, we'll briefly mention, uh, all the memory stuff that, that has been going on. I feel like it was a big part of Anthropic's work in the first half of the year. Has it stalled out? You know, is, is it still, still ongoing? Like, what's, what's the emphasis here now? Managed Agents specifically? Uh- Memory, dreaming. Dream... Yeah. Oh. It is a big part of what we're doing.

I think maybe the way to think about it is we launched Managed Agents as a platform for people who, like, didn't wanna do all these things themselves, right? Or just wanted to, like, offload, like I'm getting the best out of Claude because our team is literally sitting here, like, eval-ing all these things and making sure that the harness is gonna be really powerful. Memory and dreaming is all about just, like, how do you make sure your agent can continuously improve over time? And all of that is still an important focus for our team. But what I would actually say is, to the point of what we talked about earlier, as launching models has become harder and harder from a safety perspective, I think most of our team's energy is going less into like, "Here's a fun feature to, like, hill climb on this agent or whatever," and much more into, "How do we bring safety into all these things?"

Because the most important thing that we can do is get the power of the model out there and get it out there safely. And so I would say a lot of our energy is take all these same concepts and give you an agent system that's gonna be really powerful, but also that's gonna be really safe, and so that's where we're putting our energy. I think compared to maybe earlier this year, we have slowed down on, like, skills, dreaming, memory, like these kind of, like, moment in time injections. So you dropped your skills? No, no, no. We love skills. We love skills. [laughs] Um, we, we are making, we are making updates. Um, I'm not saying that we're not doing anything, but, um, we, we love skills as well.

But yeah, so I think we've slowed down a little bit comparatively, um, for some of the other efforts. But, uh, we are very bullish on, uh, people kind of doing some level of hill climbing at the harness layer, uh, to, you know, some of the previous conversation here. We do a lot of this ourselves, but the harness that we provide is very simple in a way. It's, it's intentionally simple. Uh, we don't want to be actually overly opinionated. We wanna make sure the best practices get merged in, and there's some kind of undifferentiated pieces of the harness, uh, agentic loop that, yeah, are, uh, hard to string together and also, like, you can kind of just boilerplate do it.

Now, the interesting part on top of that, though, is, like, once you get, get that base layer, there are these, like, hill-climbing strategies that we construct, and, like, dreaming is actually one of them. So dreaming under the hood is a multi-agent system that we hill climbed on top of CMA, and it's actually ag- CMA on top of itself, basically. CMA on top of itself. It's, it, like, spins up another- Yeah. Yeah. Uh- Okay. Uh-... the way dreaming works is that we spin up another managed agent that then can expect the previous managed agent and then write, uh, memories to that managed agent. So it's managed agents all the way down.

And this is a bit of a thing by design because we envisioned that you could probably put together a lot of these modular components for something interesting, and dreaming's just, like, an example, one of the more interesting things that we were going after. Uh, but we kind of have been looking at this as being like there's actually a ton of other stuff that you can do. You can do, like, best of N as a type of, like, multi-agent strategy. You can do a verifier agent, uh, that kind of like goal thing or outcome thing. Um, there's a bunch of other stuff in there, and we've seen people do some pretty funky crazy stuff.

Like, there's been a harness that, like, shares, like, crazy memory across everything, and they spin up, like, tons of parallel agents. And we're like, "We should actually just probably enable you to do that." Uh, in the same way that we did, we built dreaming that's, like, managed agents all the way down. There's a update that we'll make relatively soon, um, where we're updating the underlying primitive. We'll expose it, uh, to allow users to kind of, like, construct harness strategies maybe, or however you wanna call this stuff. Like multi-agent strategies. Multi-agent strategies- Yeah... that allow you to kind of build your own version of this, and we'll give you, like, you know, templates of, like, this is how we started with dreaming.

Maybe there's, like, an update to this that you would wanna make as a developer, and you would find some more value on top of. Yeah, I think that's a really good framing. It's less that we're slowing down on this sort of stuff and more saying, like, we've got the primitives in place. We can continue to let people just, like, take and run with those things while giving them the tools to say, "Here's some, like, cool multi-agent strategies and, like, things that we've put into the, into Claude Managed Agents to make it really powerful." Now you can, like, let Claude do work to take advantage of those primitives and make it even more powerful yourself, like create new multi-agent strategies and whatever it might be, while a lot of our team's attention turns to, okay, great, like, community, run with that, right?

And, and you do you while we continue to focus on what are all the things that we need to do to make sure that you get the access to the models within that system safely. One thing you brought up was you run a lot of evals on this stuff. Mm-hmm. Pretty hot topic right now. Any advice, best practices? For eval specifically? Evals on harnesses, agents. There's a lot of different types of evals, right? Like, not all evals are made equal for different, you know, like, situations that you're in and things that you're trying to test for. You obviously have just, like, static deterministic stuff, and then you have LLM as a judge. You have all these different strategies.

But I guess I would just say, like, what we found from most people is, like, you probably need more rather than less. [laughs] Um, it's just, uh, there's not, like, one perfect, like, couple sets of evals that you could put together that are gonna get you all the answers that you need to make your system better. We found that people end up needing, like, a relatively large suite of these things, and then it's like you can have your agents work with you to evolve your eval suite over time as well. Um, but I think it's just, like, n- underinvesting is maybe, like, the biggest mistake that we've seen from people on that front. Yeah. And we start simple.

I think some people, like, go a little crazy, and they're like, you know, they, like, look at all the, the menu of options- Yeah... and you kinda list, and you're like, "Wow, this is so crazy. Let me try this and that, and what's the best one?" And, like, while they do all those things, like, the number of evals they actually have are, like, zero. Yeah. So you might as well, like, do something simple. It will help. Um- Yeah, I think some, some part of it, people are inexperienced, they're not trained to do this, um, and they don't know what good looks like. Totally. So I think for you guys publishing more examples of that is, is really helpful. Yeah. Uh, something I, I, I do tell people a lot, uh, that I'm doing with more Fable and Astro class- Mm...

type models is we're treating engineering like research, and when you treat your engineering like research, you naturally do more evals, you do more labeling, you do more scaling of runs, or, like, prototyping and scaling of runs. And you're not training a model, you're building the harness, the product, whatever it is, but it... you're approaching it like research. You're defining upfront what your, uh, goal is, trying to formalize it as an eval, and then, uh, try to identify, okay, why, why is it failing? And sometimes your eval is wrong. Totally. Actually, a lot of the times your eval is wrong. Yes. Sometimes your environment's wrong, and then sometimes the model's wrong. Yeah.

[laughs] And, and, uh, actually that is, like, a lot of part what I think AI engineering is becoming- Yeah... which is doing all this stuff. Completely agree. Yeah. Um, and we'll continue to launch more features within Claude Managed Agents to give people more tooling to do this sort of stuff out of the box. Yeah. And you'll probably find from us that you'll get something that's like, "Here's a product that will help you create evals for your managed agents," or whatever it might be, and it'll feel very much like working with Claude [laughs] in order to do that. Yes. 'Cause, like- Right... that is, that is the best way to do it. Claude's actually really good at creating evals to, um- And eval UIs... evaluate agents. Yeah, exactly. Right.

Actually, because, like, people, like, blur- blank out when they look at, like, JSONLs, like just- Yeah. [laughs] Right? But- Yeah... a pretty eval UI where you're, like, left, right, left, right, left, right- Yeah... and you're, you're sort of, like, doing what the human grader, data people, labelers- Yes... would do anyway. Yes, yes, yes. Yeah. Uh, you can do that yourself. Yeah. You can. Just, like, throw Fable at it. Yeah. And it'll materialize, but also write comments, because it just, like, outputs all these ratings, right? It'll give you an ELO, give you whatever, but, uh, Claude can just consume that and then work on itself, right?

Yeah. That's the feedback it needs, and people just don't know they can prompt for an eval. [laughs] Yeah. [laughs] Yeah. I love that you said that. I think it's, like, a very, like, AGI-pilled way to, like, do it, uh, because it also allows you to update and edit that system, like, very systematically. And so as new models improve, as things change, you can really- Swap out. Yeah... inspect where your system is, and it'll just, like, evolve with you. Detune every part. Exactly. Uh, yeah. So ba- ba- basically, I, I think it's, it, it's, you know, how we talked about how maybe product and engineering- Mm... are, are merging.

It's also there's, like, application layer engineering and, and sort of model research are sort of merging in, in those techniques that they are being shared, right? Like- That's very true... model people are very, very used to doing evals. Application people know, like, the on- [laughs] the only thing you care about is does it fi- fit the PRD, and like [laughs] Yeah, it's, like, it's, like, a whole different- It's a whole new skill set, I think... whole fish. Yeah. Um, for sure. I think you guys put out a pretty cool blog post on this today. It's somewhat different. It's about making the UI a lot faster. But to your point of treating it like research, uh, basically define a goal of we want stuff to load faster, and then define and, you know, question things.

What can we do instead of wall clock timing? Uh, there's fun quotes in here. I think it's nice to see how the actual researchers or, uh, engineers prompted Claude. There was stuff like, you know, we had, we had 12 things. We said, "You have two weeks." It did 9 out of 12 of them in, like, three days or something. But fun blog post for people to research, for people to read through. Yeah. I love it when we share our actual prompts, and you- Yeah... and, like, sometimes people are like, "Oh, God, they must have crazy, unique ways of prompting." And it's just like, "Try harder, Claude." [laughs] Thank you. Uh, though my only feedback is it should link to the workbench, which is- Mm... is no longer a thing, but I used to- That is, that is a good- Yeah... that's a good idea...

I used to love the workbench. Okay. Good feedback. No, yeah, make everything replayable. Feedback heard. Love it. Yeah. Uh- Yeah, but, uh, you were gonna share more details behind the engineering story, uh, or the- Yeah... the rush work that you guys did. Um, yeah. So I think I talked a little bit earlier about, like, scale is a problem for us. Um, it's a problem for everybody. So it's scale and safety, right? Like- Yeah, scale and safety. Those are, like, the two biggest things we're spending our time on. Um, and I mean, it's really interesting because most people are used to talking about scale or just, like, I don't know, compute at the, like, GPU accelerator layer, and much less so at the CPU, um, you know, like, old school layer, and that's, uh, it's, it's just, like, as much of a problem today as the accelerator layer, which is really interesting.

And so- I think you cannot get enough VMs. You cannot get enough sandboxes. Yeah, exactly. That's primarily- That's correct... just everything being able to spin up, spin down. Yeah. With, with all the resources that you have, you, you- Yeah... like, really? You got a lot of customers too. [laughs] Yeah. A lot of users. Which is crazy, yeah. And, and think about it, because we're using CPU across obviously just, like, running our product services, running the API, running all of our apps. We're using CPU on sandboxing, all the sandboxing that we're doing, and then just, like, all the things every... Like CI, you know?

Like, you'd be surprised how much more code is getting written, therefore there's so many more CI runs, therefore you need more CPU to handle that. And so on the whole, and this is not just us, this is everybody that I talk to- I would add training. Training is probably using a lot of the sandbox. For sure, yeah. So there's- That's a different bucket, right? Like, I'm sure they don't commingle that much. Well, they don't commingle, although, like, CPU for Anthropic as a whole is, is, is, like, a zero-sum game. [laughs] You know? Like, we get our... we negotiate our deals with, you know, the CSPs, and we get our CPU, and we have to allocate within Anthropic how much of that goes to all the things I just mentioned, including, um, like, research sandboxing versus product sandboxing and otherwise.

And so we really are allocating across absolutely everything, and it's just everyone in the industry right now is, uh, you used- Short on everything. Yeah. You used to be able to just, like, you know, like, scale up on demand into CPU capacity if, as long as your budget allowed you to, right? And, like, now it becomes something that you have to, um, capacity plan for in advance, which is, you know, like, kind of punching everyone in the face. It's brand new, and people are starting to do it. And so- Dealing with that means a few things. Like, one is obviously just go try to get more CPU. Cool, we're doing that as much as we can.

But two is we wanna be able to, at the infrastructure layer, run it more efficiently, right? Like, figure out how we can make sure that we're running our sandboxes as, as efficiently as we can without compromising on things like tenant isolation or properties that we, like, you know, care about for security or otherwise. But then at the application layer, how can we, how can we run our applications and our APIs more efficiently?

There's always like, "What if we add this new thing that didn't exist before in the rewrites? Since we're rewriting anyway, right?" Yeah. Uh, but then, like, that adds scope. Yeah. Yeah. So do you just, like, avoid that? Is it, is it a pure rewrite? So we actually, we did. It was a pure... It was, um, maybe the thing that I would call it is, like, a port. What we did was we, we did exactly that. We, like, resisted the urge- [laughs]... on all these things. Then, a- and because what the team already had to keep up with was little, like, SWAT team that we put together to go and do this was doing this while the whole rest of our team was, like, building features into the API, right?

And they did all the, like, static analysis stuff. They would yell at you because you built a feature over here and you didn't consider Rust over there, right? And, like, they dealt with all of that. But they resisted the urge to add features themselves or, like, make the system different. Now, their next goal is to go and do some sort of re-architecture type work, including stuff that I think will be more user-facing. Like, we're starting to roll out some things that are like, you know, what if we were able to not have you have to, like, pass back and forth the entire, like, conversation state, right?

And, um, if you didn't have to do that on the messages API because you were open to, like, you know, a storage of that state for some period of time, right, then, like, especially for long conversations where the context really builds up, like, you get a lot of really great latency and efficiency wins there, too. Like, these are the sorts of things that the team that was doing that port was like, "Ooh, like-" Could do. "... what if we do this?" Right? And we're like, "Not yet." Does the new caching API have that? So we have prompt caching, yeah. Yeah. But you still have to, like, pass back the state of the conversation. Okay. Yeah. Very, very niche small subsets, but I guess this is the levels of the- No, this is, this is- Yeah... this is important. Uh- Yeah, yeah, exactly. Exactly... yeah, yeah. There's a lot of savings to be done. Yeah.

Well, so from- But it also turns them into a cloud. Yeah. [laughs] And then from, from type of, from number of users point of view, it's small. Yes. But if you look at the actual token volume on those, those types of workloads- Yeah, yeah... it's, like, going to be the majority. Well, I'm very, like- Yeah... like, what Cloudflare did with Durable Objects, I, I, I feel like every cloud provider should clone. Do, do you know what I mean? Yeah. Like, right? Is, isn't that the right shape for a messages API that needs the continuation? Yeah, I mean, it is really interesting 'cause I think what we're... We are experimenting with various, like, what is the- Yeah...

right shape of, like, how we should run our compute- Yeah... in order to power all these sorts of things. Yeah. Um, but it's also interesting because we can't necessarily rely on, like, because we get our compute across. We run on all three clouds. We're running all over the place. Like, we don't necessarily have, like, one shape of here's exactly, like, the compute, the types of machines that we know we're gonna run on and the way that some certain compute works that we can, like, rely on across the board, if that makes sense, other than, like, the most vanilla versions of these things. But I think these things are all gonna, I don't know, continue to innovate and evolve as time goes on, and so we'll be able to take more advantage of those things. Yeah.

Uh, I'll just mention if people want, uh, open source Cloudflare durable objects, uh, Rivet is the open source equivalent. Mm. Um, there are 50 people listening that will kill me if I don't ask this question. You want CPU, they have CPU to sell, who do I sell to? Who do I email? [laughs] [laughs] Absolutely. Uh, and what do you, what, uh, you know, and, like- I'm surprised... what, what is a, what is a good... Help them help you, right? Like, what, what is- Yeah, yeah, yeah... what does good look like? Well, I guess, like, uh, email me and [laughs]... Are you sure? [laughs] I don't know. Yeah. Yeah.

Well, that- I'm gonna get a lot of emails. But, like, you're, you're looking, like, minimum fleet size, SOC 2 obviously, enterprise. Like, yeah- Yeah... like, all, give me all the privacy guarantees. And for what it's worth, like, so we have a really awesome team that secures our accelerator compute- Yeah... um, that is now spending a lot more time with us on CPU- Okay... and, like, where and how we can procure it, um, to make sure that we can just continue to run our systems at the scale that we need to. And so it's not actually, like, my job thankfully is not to go and, and run around and try to, um, procure compute, but we do have teams that are doing that, which is really great and really important.

But obviously, like, there's a quality bar that matters to us. There's, like, a scale bar that matters to us. We're not gonna, like- Yeah... go and integrate and run on certain machines if, like, we can't get enough of it. And so all the properties that you might expect of a system at this sort of scale is all what matters to us.

which is products Yeah Is that- Products? Is it the rough shape of what's going on? Uh, yep. That seems right.

Yeah, I, I f- I feel like, uh, it's weird because obviously you're famous, or CloudCode is famous for being the super unsafe [laughs] dangerously allow permissions thing on, on your machine, but now you're like finally evolving to where you probably should've started in the first place. Basically, I think, like, all these sandbox leaks, right, are they, uh, a fundamental issue with our s- our understanding of sandboxing or go to container tech, or are we just bad at this? Like, are we... Are we just, like, not disabling, like, not doing the allow list, right? Like, it sounds like we're just not doing the allow list.

Yeah, so maybe a distinction that I would make is in a lot of the incidents that people are hearing about and reading about, these are incidents that are happening to models, not just our... Like, everybody's models, right, that are in training, where the way that you're running these models is you, like, put them in a box and you say, like, "Solve impossible task. I'm gonna stress you out. Ready, set, go," right? And you, like, can only do what you can do within your environment. And the agent is basically incentivized to try to get out of his box and go solve the problem. When people are running production workloads on our API or on anybody else's API, you're running an environment where, like, there are safeguards that are running, that are, like, not necessarily running in training, where, like, the whole point is to actually train the model, right?

But in production workloads, first of all, you have those online safeguards, which is really great. Second of all, you have more of this architecture and infrastructure around what's happening. Like, the agent isn't just in a box trying to get out, right? Like, the harness is over here, the hands are over there, and, um, it's just, like, not even a problem that you would have to contemplate. And so I think that some of this is just, like, people having the right understanding that what's happening, some of these incidents that are happening in model training and otherwise are just things that, like, wouldn't happen in production workloads, where a lot of work has been done to put the right guardrails in place so that those things, like, never really could happen.

The second thing I would say is just, like, everybody's figuring all of this out, like, relatively newly because, you know, long-running agents is a relatively nascent thing that everybody is building and trying to deal with. And so everyone's figuring out some of these things on the fly and just racing to put the right, you know, like, controls in place and, you know, some of that is just happening as we go. So I would say it's not, you know, we're- people are bad at this, right? I think it's just more it's a nascent field, f- people are figuring it out, and I think when we talk about things like pacing the frontier, the way that we think about it is really specifically we're doing all the work that we're talking about right here for all of our agents internally and making sure that all these things are accessible to all of our customers so that, like, by default, the right guardrails are just in place so that you don't end up having agents doing things that you don't want them to be doing.

Fair enough. Um, uh, m- well, f- 'cause you mentioned pacing, we gotta go there. We also talked about this the last, last time. It is, uh, superficially contradictory to advocate for pacing and then also release a new frontier model 11 days later. To me, it's just, it just means that it just passed your safety checks, right? Yeah. Uh- And, and that, yeah. And- Everything we talked about earlier on, like, how we launch the models- Yeah... um, it's like we are continuing to evolve more and more and more safety properties in place, again, across these, like... I kind of am starting to think about this in these three dimensions.

It's like the actual, like, architecture and infrastructure security, then it's the online safeguards that are running when a model is doing its work, and then it's the offline safeguards classification that's happening on data at rest. So across all those three dimensions, we're putting all the right work in place to make sure that we feel really good that we can put a model out there and it's going to be safe, and really, that's fundamentally what we mean. And so for Opus 5.5, it's an excellent model. It's also really, really great on all the dimensions that we care about across those three things, and therefore we feel good about releasing that model.

So we went for what we thought, what we were hoping is the best possible user experience within this while we continue to hill climb on making those false positives better, but it's obviously, like, it's still a frustrating experience for people when you're trying to work with a model and then you fall back to a different model, and so it's just something we have to keep working on. Yeah. I think people also understand it in the safety sense. Yeah. Like, you want more false positives, right? Like, it's a safety issue. We clearly see the whole invasive s- incident, and you're still giving access.

Yeah. Yeah. To, to, to me, this is, like, a, a, a close cousin of the general model routing problem of do you want a smart router or a dumb router? Does that make sense? Yes. Like, uh, because as far as I can tell, the, the, the Anthropic fallback router or, uh, detection thing is a separate model that's online. So- Uh, it's, it- once it hits a classifier, there's just logic to where you just need to drop to. Exactly, right. Like, so it is not the full fable, right? You're not, you're not s- Mm-hmm... you're not, you're not running full fable at it, and that, that, that's gonna cause classification issues 'cause then it's not, it's clearly not as smart as fable. Like, clearly.

Um, so then there's the question of, like, well, then, like, when I have a classification issue, that's, that's, like, an unjustified fallback. Like, you never, you never need to fall back. I'm just doing my bio homework, but I mentioned bio one time, and then it can fall, fall back. Uh, so, like, there's the question... Th- then this is very similar to the routing problem because, like- It's very similar, yeah... you're, you're routing models and, like, how can this, the, the- a dumb model route to a smarter model if it doesn't even recognize this? That's, that's a smart model problem. Mm-hmm. Right. Um, how do we solve this?

[laughs] It's a hard problem. [laughs] I, I think, I think in some super alignment sense, models are good at classifying stuff, but not answering them, right? Uh, uh, not the smartest model can tell this is a dangerous query, even if it can't make any impact on that, so... I think the super simplifying thing to say would be, like, basically you have the smarter model figure it out over the one that, um, it needs to detect. I, I think people also- It's slower, X cost. There, there's trade-offs there There's trade-offs. I'm not saying it's, like, the right user experience, but it would probably be a slightly more effective route.

I, I think one thing interesting is you also don't cut off immediately. I've seen cases where it will start answering, start reasoning, and then get hit by a safety filter. So w- my first example with Fable was throwing the Fable safety card- Mm-hmm... and it started re- I was like, "Just break down anything interesting in here." Starts breaking it down five minutes in- Exactly... and then it gets into cybersecurity, and it's like- Yeah... "Oh, we're cutting. We're rolling back to Opus." But it did a lot, and there's a lot of context, and- I see it. I, it got to a point where it's all about cybersecurity, safety- Mm-hmm... and then it rolled back, and it's like it wasn't rolling back my query, it was the combination of reasoning and the whole chain, right? Yeah.

Um- Do you have a idea of what's better? Yeah, let's brainstorm. What, what's better? I think the user experience and how we resolve that user experience is probably, you know, a separate question. Like, maybe it's a new super smart model to handle it or, or whatever. Um, but I think that the more appropriate UX that I would imagine for a user is to actually have a more consistent, unified experience with Claude, where instead of getting you falling back and then you can double fall back, there are, like, logical points, um, in our tree that, that do do that for, again, like, safety reasons and whatever.

But in the UX, it's just, like, really convoluted for a user. It's like you almost... You tr- you're trying to do something, and I, I would probably take the case of, like, a good user, and you're a good user, you're just trying to do something, so you bio homework, like assess the cybersecurity, like, safety card. Totally reasonable, and you get blocked, and you fall back. And I think instead it would just be like you are... Our model should have gotten, or classifiers or however you want to do this, would be smart enough to detect that that is a situation that happens, and just write down the me- the answer for you more concretely while dropping down the nature of the, like, uh, negative use of that particular capability.

So going back to that bio homework scenario, while we may have classifiers that have detected that it is possibly bio- I'll also say, like, the, on the API, we went with a spectrum of solutions for this for customers. So first of all, like, if you've built something on the API and you run into a block, like, you don't have to, like, continue the conversation on a fallback. That's what we've done in our apps, and we've made that a feature of the API, and you can do it, like, server side within our API. You can choose to, like, set, "I want to do the fallbacks," right? And, like, we'll do that. We also have it within our SDKs on the client side. There's helpers for if you want to, like, manage fallbacks in various ways.

Or you could choose to, like, re-prompt differently in some way, right, and, and continue with the same model. [laughs] Um, but I do think that there's some in-between interesting solutions that we'll continue to experiment with to figure out how we unblock people and, and do so safely, but with a good user experience. Yeah. And then you brought up the, the bit how it's, like, very similar to routers. I think, like, I totally agree with that. I think it has a similar UX problem as well. Like, with the router you, you go through, it's like, let's say you want to do a cost router, right, for the task complexity that you have. Makes sense. Uh, very hard to do.

Uh, you don't know everything upfront about your task, so you can't, like... Even with perfect, you just don't actually have perfect information. Even if you could actually understand each model perfectly, which you don't, but even if you did, you also don't have perfect information about your task. And so there's, like, a lot of error associated with your, your routing logic that you would have to create, whether no matter how good a classifier you produce. And yeah, maybe it's, like, a super big brain model type of problem, but regardless, it's I think a very difficult one. And in each one of those, like, scenarios, if, um, to find a way to, like, route that properly, it's very, very convoluted.

You could expose all that logic to customers like, "Hey, you know, your task was to center a div, so I'm gonna route you to something super cheap." Uh, and like, then this other one's, like, detect vulnerabilities, so I'm gonna route you to something, like, really expensive. Um, if you expose all of that, I don't think that's, like, a great user experience ultimately either. And for that similar class of problem, I think the more idealized UX, uh, would be something along the lines of, like, dynamic, like, effort or dynamic, like, understanding of the level of complexity of your task, which you might actually evolve over time as you learn more and more information about the task.

But in that world, if you're kind of, like, moving between a lot of different models, it actually becomes very, very disjointed. And so, like, maybe a more unified approach would be something like it's a consistent model, but, um, it's, like, very, very low effort level or something like that. It's actually very, very cheap and very, very affordable, but it can scale itself up very, very continuously, but you have, like, a very, very consistent persona towards that model. And obviously speaking about this in the context of, like, you know, like our family of, of models. If you try to even do that across other family of models, other model providers, like the personality, the disjointedness, like, starts becoming very, very complicated.

I don't know how you solve it in that layer, but at least within, like, our family of stuff, like, I'm more bullish on things that are just more consistent, and those might be ways to solve it, uh, more robustly for customers. Yeah. You're, you're not responsible for anyone else. But within Claude, yes. Yeah. I mean- Yeah... this is your, your territory. [laughs] Yeah. It's nice to see the way you guys think of these problems, right? Intent in and of itself is a hard problem. Like, I feel like the models have gotten better, but sometimes people don't do a good job at classifying what they want, or they just slide her all the way to one extreme, or, you know, the model doesn't really understand.

You're asking a simple question, and it's like, "Oh, I have a whole HTML landing page now." I just expected a few line output, you know? [laughs] So there's always a, a mess around there. Yeah. I think we, we... In this conversation where we talked a lot about safety, we talked about, like, launching the models, like, all this kind of stuff, but it's obviously in, in pursuit of helping power the workloads for people. And like we discussed kind of at the earlier part of this conversation, we are seeing workloads, uh, change quite a bit. I think, like, in some workloads, intelligence does seem to be increasingly saturated, which is, like, a, a new, uh, situation.

Um, and it's, it's getting to the place where I think there are some class of tasks that are still very, very unsaturated. I actually think there are more than maybe people anticipate, but they're increasingly going into the direction of you get that alpha, um, or that return by doing extremely long-running agentic tasks on things that historically you were not really able to brute force, but maybe you can with a lot more intelligence behind it. Um, and those workloads are really, really fascinating to us, and we see a lot of value in that kind of area, both from businesses getting that value from themselves as well as, you know, like new companies or startups or however you want to frame it, um, who are interested in building kind of like agentic products on top of it.

I think if you look at the class of things today, a lot of it tends to be... So obviously a lot of coding stuff. There's obviously a lot of, like, enterprise, uh, innovation, acceleration kind of stuff, um, but they tend to take the shape of automating a very distinct and specific process. And I think in some areas, that's actually really starting to work, which is awesome, and some areas it still isn't, so there's still a lot of, like, room to, to kind of evolve and push the frontier for those models. And then there's, like, this new class of workload, like the, the bio stuff that we talked about or some of the cyber stuff. Like, that's a...

It's just a very, very complicated set of, like, I would say, hard engineering or hard science problem. Um, and those workloads are increasingly showing up as well for us as, like, a really interesting space to just deliver value to customers. You know, you, you are at- One of the few spots in the world where you're discovering all these workloads and we're automating it. It's like- Yeah, it's pretty crazy. [laughs] Uh, I mean, you know, that's, um, very inspiring. Uh, you guys are having so much impact and, and I'm really, uh, grateful for that. We're gonna go into a few closing questions- Sure...

I think, do you remember what he said to me? Uh, Tariq, Tariq tweeted this. The right way to use model capabilities is not to ship 10X more features into prod. Uh, spend more time understanding your users, trying experiments, building whatnot. But basically, you know, you could very easily just keep shipping random slop, but you, you should actually think about it. You should spend more time kicking off that first prompt. I know now today he's like, "Kick off plan mode. We don't need it anymore."

[laughs] But, uh, outside of plan mode, you should think about what you're doing, right? It's very easy to put out a lot, and you can get product slop. Yeah, I mean, I, I tend to kind of go on this rant about there's a lot of just, like, fundamentals about how software gets, like, and products get planned and shipped and otherwise that, like, doesn't need to change just because agents are, like, helping speed up certain parts of the process, but there's this inherent, like, almost laziness that's starting to come out from a lot of people where it's like, "Oh, I don't have to, like, think about some part of the process at all because the agents are doing it and helping me or whatever."

But, like, those are still, like, incredibly foundational parts of the process that, like, should be things that you're spending your time and energy on, and I think that's kind of the spirit of, um, what he's getting at here. It's just, like, as a founder or cr- creator, let's call it as well, um, it just feels like because the cost of experimentation or exploration is so low now that I should be doing more of that, and he's basically saying to not do that, to, to, like, use it in service of a, a very intentional goal that I start from the start. Which, like, uh, I obviously I'm coming around to, it's just hard. [laughs] It's like you have this toy that does everything. It's just like, well, just throw it at the wall. [laughs] Yeah, there is scope too.

So, like, the blog post of today was two weeks to make Claude AI faster. There were 12 things. Uh, we hit-- there were 13 things. We hit 12 of the 13 targets in three days, right? Well, you could just stay, okay, it's a two-week sprint, we'll do the 13, but instead there's 20 things you push bound. This is not product slop. It's not new products. It's refactoring. But still, you guys do a very good job at outputting a lot, but keeping high bar quality. Yeah. Nice. Okay, part two of three. Uh, this is gonna make Angela very uncomfortable. [laughs] This is Angela three-corner. Uh, I will tell- [laughs] Nice. Oh, no. [laughs] I will, I will, uh, I will tell people nonstop to- Ugh. Uh, I, and I, I, I'm pretty sure you've deleted stuff[laughs] Because I'm like, there's, there's stuff, there's stuff in your deletions that I'm like- That you like?... still... Yeah, I know. That like really like altered my DNA. Oh, I appreciate that. Well, uh- Wow.

What do you refer people- High compliment. What do you think about the most? Uh, what do you refer people to the most that, that you wanna sort of bring up? And I, I can, I can pull it up for you if you want, but... Um, I actually, I don't think I have a favorite, honestly. Um- Oh, come on. No, I really don- uh, okay. I guess maybe I like the one that you say, where's half doing something is expensive way of not doing it. Yeah. Maybe that's my favorite. That's, that's very on topic for us today. Okay, nice. [laughs] [laughs] Nice. Love it. It's, uh, yeah. But I, I guess, uh, y- you know, you, you obviously you're in a different job now- Yeah...

and you don't do this anymore, but like, uh, that, that... I think this is a power tool. This is like a N- Naval Ravikant. [laughs] It's Naval knows you- Yeah... that do this. Uh-huh. And like, what mode- I've, I've stopped it, but yeah... what mode of what, like, what framework of mind, frame of mind are you in? Frame of mind? That's a good question. Six years ago. [laughs] Yeah, no. So this was a very long time ago. Um, I went through a period of my life where I was like, reading constantly, actually. I found myself like, very bored, um, of the stuff that I was doing, and I felt like I was just like, not being like, intellectually challenged.

And so I actually went on as a bit of a bender where I, I think I read something like 200 or so books in like a year. Um, and I was just curious about like every... I just wanted to like read everything. And I was missing a little bit of like the, the kind of like intellectual, like, I don't know, like muscle building you get when you're like in school where they like, they're like, "This is a new topic. You gotta learn." It's like tough. It's like, not super fun, actually, but you are really like, stretching yourself. And I felt like after just like working for a couple years, I just like stopped doing that.

Like, yeah, the job's like interesting, and yes, it's like challenging, but it doesn't have like that like intellectual bit to it. So anyway, went on this bit of a, a bender, uh, a reading bender, uh, and I like, read like 200 books in a year, and it was great. I had like a really long commute, so it was like, it was a- [laughs]... good use of my time, very long, like Cal train ride. But, uh, I ended up coming out of that and being like, I was just like, thinking a lot, um, as a result of that. And at the time, it really wasn't to build like a Twitter follower- [laughs]... or anything. I was just like tweeting into the, like abyss. There was like... I have like, no followers.

These were like tweets for myself, and I was like practicing a little bit of like, how would I try to compress the idea that I have in as few words as possible. It was kind of like the personal test I had for myself. And then, I guess people just started liking it, and so I just, I just kinda just kept going at it. Yeah. And then I got too busy with work and I, I stopped doing it, so. I gotta ask, after all the reading, any book, book recommendations- Yes... blogs, long articles? My favorite one still is probably The Black Swan by- Nassim Taleb... uh, Nassim Taleb. Um, that was like one of those like, I was like ki- to me at least, it was like a bit of a, like a life-changing philosophical book where I was like, that's a completely different way to look at the world.

And the concept- You know, I was thinking of Black Swan when the Fable- Oh, really? Yeah... Fable thing happened, right? It's a thing that you didn't know could happen. Very much so. Now that it happens, like it's a, it permanently changed your framework of- Exactly... how things work. And, uh, the idea's like, so simple, but I think he did a great job of just really hitting it home and articulating it and how it's like a common property of actually of so much of the world. So I really love that book, and that's probably my favorite. Uh, and for listeners, if you're bored with that one, you do Fool by Randomness, Randomness, also by Taleb. Did you like, uh- Before he got-... Antifragile?

Less so. That's so much. Less so. Yeah. The, the more famous he got, the more intolerable he became. [laughs] Uh, but al- also question for Cailin, mi- if, if you do have one. Do you have a similar like, this is a book I recommend to everybody? No. Okay. That's a good question. Yeah. I need to think about it. It is funny 'cause I do think we have very different styles on this front. Yeah. Like, she's no longer like, in fortune cookie tweeting mode- [laughs] I stopped, yeah... but I do think that like... You stopped that, but I do think a, a big part of how you work and how you work with other people is like philosopher mode sometimes, you know? Do you drop fortune cookies in Slack? [laughs] Oh, I, I do actually. Oh.

Not intentionally. I think it's more just me somewhere trying to articulate myself. But I think that's the core success- All the time, yeah... of succeeding in topic, is like you have to be very good on Slack. Yeah. Oh, you do have to be very good on Slack. That's actually true. Right? Um, it is, it's a pretty intense place, but I think I'm a little... So I'm kind of psycho about like inbox zero on both email and Slack. It's easier on email these days than Slack. Um, but I like pretty aggressively prune like here are the channels that I'm like actually in and paying attention to, versus the ones that are like muted below the line sort of thing, and I'm like always making sure that I'm on top of the ones that I need to be on top of.

But it is, it's basically, it feels like a full-time job. Like there are days where I'm like, I could cancel all my meetings and write no documents, do no work, right? And literally just sit there like on Slack the entire day, and I would have done what felt like an entire day's worth of work just by like engaging on Slack with all the things. Yeah. It's pretty wild. No, but that's horrible. Like, what, what do you do? So that's exactly how I feel. I feel like you do all this e- you put in all this effort, and yes, like you, you met the mark on like responding to people and getting certain things over the line, but you didn't actually produce the output that you wanted to.

And like that's like a, that's like personally a very hard thing for me to deal with. Yeah. So I try to be like, I need to like dedicate time to like actually thinking and actually like trying to produce something in order to feel like I'm like actually being productive. And ignore s- and drop stuff, right? You, you're gonna drop stuff. Um, yes. Right? I have gotten a lot of feedback. I'm a little hard to reach on Slack. Um- Mm. Oh, yes Yeah, pretty terrible thing that happens now- I've been exchanged. Exchanged... is I get a lot of Slacks saying, "Can you get Angela to respond to this?" [laughs] Oh, no. And I'm like, I'm like really trying to write a doc here. Yeah.

Wait, wait, so do you guys don't have like a personal cloud, like m- Assistant something that I don't know is going to help you There, there's got to be a way, Pam, but there's obviously- So I had no one, no one written in, like a personal EA for each of you... in non-code mode. Yeah. Well, so we do have, we have a human one also. Yeah. Yeah. Yeah. We also have, we have plenty of Claude Help in Slack, for sure. Yeah. Um, but I do think actually the, the types of Slacks that I'm like when I'm pruning through and getting are often just like our team is like blocked on something 'cause there's some hard decision that needs to happen or whatever it is, right?

And it's like, it is just like a form of, whether it's in Slack or some other way, right? Like it's a form of getting the team unblocked, like executing in high quality, that is my job anyway, and it like happens to happen a lot in Slack. But I do agree that it ends up feeling like, what did I even produce, when you get through a whole day worth of literally just like going back and forth with people and trying to solve problems and- You, you produced a, a functioning team. Yeah. [laughs] Exactly. You- that, that's unblocked. [laughs] So I will give you props for that. As a, as a former manager, that is- Yeah...

yeah, yeah, yeah, you don't feel like you c- you produced anything, but like you produced a team. Yeah. Yeah. Uh, speaking of your team, uh, I was gonna go to you next, uh, Caitlin, uh, Devrel Corner. Oh, nice. Uh, we are- [laughs] I- I know you're really hiring Devrels. Yeah. I'm gonna make a call out for, for developer relations. Nice. Uh, you hired Adi Osmani recently. Yeah. Uh, great get, uh, Lance, uh, you know, I don't- Yes... but you have, you have, you still build out a team. Uh, what i- what do you guys want to hire for? You know, this is, this is it to talk about it. Yeah. So- And I'm, I'm happy to trade notes. Yeah. Well, props to you because I will say when- I- I'm the outsource Anthropic recruiting, uh, Ca- uh, Caitlin recruiting team. You, literally. Literally. [laughs] Um- It's fun...

and it's not e- it's actually not even just recruiting. Like, I'm gonna, um, toot your horn a little bit. When, so I, when I joined Anthropic, we, you know, we're a small team. We're like, all things considered, we're an extremely small team, and so we run the platforms, the API business, and, and all of our product infrastructure. And um, we, we had like a very nascent devrel function- Yeah... within Anthropic that was, you know, like not necessarily within the product and engineering org. And when I joined, um, we were like, "Okay, we like need to grow up a little bit and need to figure out how to like engage with the community, engage with the world, and, um, we need to like really invest in this devrel function."

And um, we ended up spinning up a devrel team within platform and a devrel team within Claude Code, in part because it's like two pretty different, like developer ecosystems that we're trying to engage with across the API business and agentic coding, right? And so we ended up doing it as like two separate teams, but kind of like dotted line to each other and spend a ton of time together. Um, but it was, it was something that I was going to be responsible for, like devrel, which I've spent plenty of time with devrel before at Stripe or otherwise, but it was never something that I was responsible for.

And I was like, "I have no idea what I'm, what I'm doing." And you were like, "I got you. Like, here are the three pillars of devrel. Like, here's how you think about it," which is really great, across like product, content, community, and then now I can like rattle that off to people and they're like, "Wow, you really understand devrel." I'm like, "No. Someone stopped me." [laughs] Um- You're like, "I copied that." Yeah. [laughs] Yeah. That, that was not original, Pam. It's apparently, that post is apparently like the default reading for Google when, when they onboard new devrels. Yes. So like really funny. Yeah. Nice. Uh- Um, it's...

it really, like this, this really helped me [laughs] because I had no idea what I was doing. Um, but then on top of that, we, you know, like I've, I don't have a manager for the devrel team within platform right now, and so that's one of the things that, um, I'm hoping to hire, and our devrel ICs, um, report to me directly, in part because I think it's a really, really, really hard problem right now, but also in part because I find it really fun. [laughs] And I'm having a lot of fun working with the team. But really specifically, like what's hard on platform is platform in some- sometimes in our like public narrative, like our platform business is a really large business, but it can kinda get lost in between research and policy and all this work that we're obviously doing that's like our most frontier important work, and then our applications like Claude Code and Cowork and all the things happening over here.

When you think about Anthropic and you think about our public narrative, those are some of the things that often come to mind, and less so that we're running this like massive developer platform business that's really important and so many people are building on top of. And so a big goal that we have with our devrel team within platform is figuring out not only like the really classic like how do we better engage with like, you know, product, like what people are building and how we get feedback back into the product and these sorts of things, but a lot of it's just like finding our voice- Mm...

and finding what our brand needs to be and what our narrative needs to be and like who we even are gonna be in the community and not kind of get lost in all of the things that are happening. And so that I think has been interesting 'cause I think the brand that we've started to spin up for our devrel team, in part because actually Lance joining our team was you connecting me with Lance, but Lance is highly technical in the way that he goes about devrel is very much like teaching, right? Like he writes a lot about memory and managing costs and these sorts of things, and he goes really deep. He builds evals.

He like has a lot of really technical content that he puts together, and I really like that idea, like that idea of our, our devrel team being kind of like thought leaders for people- Yes... within what is a really nascent, really hard, really evolving space. Educators, thought leader, explainers of things. Yeah. Um, and there's another version of this I really like called the master builder, uh, which is I will build the things with the platform to show you what's possible, and then you'll be like, "Oh, I didn't know that was possible- Mm... with, with the Claude platform." Yeah. And then you, you just go. Yeah. Uh, and this is a thing stolen from Lego. Um- Oh, nice. Yeah, because Lego, how do they sell Legos? They have master builders. Yeah.

They build things with Legos and you're like, "I wanna do- Well, that's the real cool... I wanna do that. Yeah. [laughs] I wanna be like this person." And then they, they, they do that. Nice. And so- That's awesome. Yeah, and- Yeah... um, the, uh, CJ, the other person on our team who's really great as well, I think he also does a lot more of that sort of like- Yeah... um, you know, writing cookbooks and like putting out examples and really engaging with the community- Yeah... on it. So it's- Cookbooks is funny, right? Like when APIs launched in 2023, let's call it, for LLMs- Yes... very important. Now- Not so much... I don't know. [laughs] Yeah. Well, it's interesting 'cause you're less in the spot where you'd like copy-paste the code and like use it, right?

[laughs] Um, but I do think it's like- It, it shows the art of the possible for people and just like what even is the use case, right? And like how can you go about accomplishing it. But I hear you that like, you know, the code itself is maybe less... Like we for the longest time had this like there's, there's an engineer on our team who wrote our cookbook on, uh, computer use, and it like haunts him to this day because it was like massively complex and like, you know, kind of a beast of a thing that, um, I think, you know, a lot of people got use out of at a time when it's like, "I'm gonna build a computer use implementation, and I'm gonna deal with all the, the screen shotting and all the things," right, that are happening between the messages API.

Um- Yeah, we were, we were there for the hackathon that you guys launched at Office. Yeah. Yeah. Um, and like tho- that was the era when you like needed to have that sort of thing, and now, um, you know, it's much more about we have a skill, like the Claude API skill, and it's like really, really excellent at helping you either within Claude Code or any other coding agents, um, to build a really high quality integration. And now like these sorts of things like, "I'm gonna build a computer use integration, like how do I go about doing that?" Like very few people I, I think/hope are reaching for the big gnarly cookbook, and more so they're loading up the skill on a coding agent and working through that. Yeah.

Or they're just putting it into a reference architecture. Yeah. It actually do- does much better. Like increasingly we're seeing that for a lot of the things that you might want to have different implementations on, uh, typically at the harness layer, but there could be other variants. You just point Claude at it, and you say, "This is the reference architecture I want." Yeah. And then you build it. Fun fact, the Claude API skill that helps people build against the API is owned by our DevRel team. Um, so that was- As they should... built by Lance. As they should. Right. Exactly. Like that's the sort of thing.

This is the most, uh, high leverage form of DevRel is to like distill everything that is best practices that's gonna go into those agents. Yeah. One of the reasons it's weird because like, uh, I'm regarded as this like DevRel person or authority, but I... Like it's this, like there's some part of me that just, just wants to tell you you don't need DevRel. [laughs] Um- You get so good you don't need it. He... Right. Uh, or right, you turn the engineers, the, the, the leaders, um, into the public faces. Yeah. 'Cause all you really are trying to do is public faces, and like, "Here's how we do things here," and then, and that, that's kind of it. Like you- Yeah Some- sometimes it's natural... you're philanthropic.

You don't need like, you know, to... People will hear about you. So like there... I think to me when Boris came, burst onto the scene, like it seemed like a very, like, "Hey, I'm Boris. I'm the creator of Claude Code," blah, blah, blah. "This is how we do things," blah, blah, blah. And you, you, you're like, yes, he has the day job, but he's also doing that func- functionality. Yeah. So like why, why doesn't everyone just do that? [laughs] Yeah, I mean, it's a really good question. And there are teams that have done this really well. Um, like I think the Cloudflare team is a really great example of like- They're all... Yeah. They're all- All pumping out stuff... super awesome, right? Nonstop. Yeah.

[laughs] And to me that's a actually, by the way, is, is a very vi- visible sign of a good manager, is that they're all happy with their jobs. They're all, they're all like s- like publicly identifying. They're not quitting. Yeah. None of them are quitting. Totally. It's interesting because, uh, I think this is actually something we are trying to do more of with the platform team and with our team and, and folks who, you know, um, can kind of just get out there and like be part of the community and, and be speaking and be on X and be wherever they are, right? I think the problem for us is just like having time [laughs] and kind of knowing the right way to do that stuff at high quality.

So I th- I see the DevRel team in some ways as just being like the experts in doing this sort of stuff at quality. They can do it all day, every day, and they can help enable the rest of the team to think about how to do this with the small amount of time that they are able to spend on this sort of stuff to do it at high quality too. Yeah. So like, yeah, so the... I call this like kind of the unbundling of DevRel, uh, into a front office and a back office. Yeah. So like a, a typical person who does this job does the whole thing, start to finish. They're kind of autonomous. Mm-hmm. But actually you can sort of build out this back office that lets your product leads and engineering leads, um, actually take the spotlight with a, a simple contract of like, "Okay, you sit down with me for 30 to 60 minutes, and we'll do the brain dump, and then you're done.

You go back to work." Yeah. Uh, and then we'll like, we'll like ghost, ghostwrite for you, whatever. Uh, but like that team of like production people, writing people, whatever, is, uh, sort of provides the service to the rest of- Yeah... rest of the people. And, uh, I, I think that to me, that's like kind of the future of, of th- of this stuff, to be honest. Huh. Uh, because people, like, don't actually really wanna hear from DevRel. They wanna hear from Lance. Uh, and, and like- Absolutely... you know, you know. And you have like lots of... You have Tariq at Target. Yeah. Uh, y- you know, and Lydia- Why Chris and not at DevRel? You got Lydia through Bud, you know? Yeah.

E- e- exactly. Yeah. But like he, he like functionally- He remains... he does the job... top of mind. He does the job. Mm-hmm. Um, and so like that, you want more of those- Yeah... 'cause people, people wanna hear from people in the product. Yeah. Not, not like who are like paid spo- paid shills- Mm-hmm... like paid, paid spokesperson. Yeah. And that's [laughs] Wh- which I get a lot of, and like that's fine. That's, that's a job. Somebody has to do it. Yeah. Like something like reasonably you can expect people who are, are working on engineering to not be good at it. Yeah. So like you should pay someone who specializes in it, like specialty is a, is a skill. But I, I think there is a back office/front office separation that, that can e- Hm... emerge. Yeah. That is really cool. Yeah.

And, and my analogy would be like everyone hires, but you have an HR department. HR department enables- That's true... you to hire. Indeed. You hire. Like- Yep... yeah. [laughs] It makes a ton of sense. Uh, cool.