Real Alpha
Try Pro free

Latent Space · 3 min · Oct 27, 2026

Claude helped cut Messages API CPU use by up to 50%

Anthropic ported its Python Messages API monolith to Rust without Rust developers on the starting team, using Claude to write the code. Shadow traffic verified parity; CPU use fell 40–50% and time to first token improved 10–20%.

Katelyn Lesse

Head of Engineering, Anthropic

Leads engineering for Claude’s APIs, developer tooling, and product infrastructure.

  • Shawn Wang (swyx), Host · Co-founder, AI Engineer

    Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.

About this segment

Anthropic ported its Python Messages API monolith to Rust without Rust developers on the starting team, using Claude to write the code. Shadow traffic verified parity; CPU use fell 40–50% and time to first token improved 10–20%.

From “How Anthropic ships the #1 LLM in the World — with Katelyn Lesse and Angela Jiang”, Oct 27, 2026. Watch the full episode

Answers “Is AI paying off for companies?” See all 32 answers

“we literally built, like, an entire fork of the entire API in Rust, and then started running shadow traffic through it”
Katelyn Lesseat 57:26 · Latent Space57:26

Takeaways

  1. The team ported the whole API before gradually routing traffic to Rust.
  2. Shadow traffic let them compare Rust and Python outputs before rollout.
  3. They resisted adding features so the rewrite stayed focused.

In this episode

Full episode

  1. 56:02Claude helped cut Messages API CPU use by up to 50%3:19

  2. 1:03:18Separating the agent from its sandbox limits security risksPro pick 4:56Up next
  3. 5 more moments in the full episode

Transcript

From the transcript Latent Space publishes with the episode.

  1. 56:05

    The thing that I'm, like, most excited about, as of last week, we fully rolled out the messages API, um, written in Rust, which is one of those, like, insane feats of, like, what you're able to do in the agentic coding era that used to be... Like, we started talking about this, like, earlier this year. We were like, "We're gonna hit a CPU wall. There's stuff we need to do." And, and we've done it, right? Like, we've gone multi-region.

  2. 56:27

    We've done all these things we need to do. But we were like, "We're running this Python monolith, and it's like [laughs] it's, like, very CPU inefficient." Um- [laughs] And so what are we gonna do to make the system more efficient? Fork Python. Yeah. [laughs] So we chose Rust, which was a crazy decision at the time because we had a team of people... Like, on paper it's obvious, but our, our team was like, "We don't have a single-" Not Rust people. Got it. Yeah. Yeah, we don't have a single Rust developer in the room. Yeah. Who cares? Just go for it. Um, exactly.

  3. 56:53

    [laughs] Literally. Or not- And, and not only that, but... Yes, Claude is gonna write the code. The humans, like, design the system, right? Wow. So, like, the humans are very involved in making sure at the end of the day, it's just like a dis- it's a distributed systems architecture, right? Yeah. Like, and Claude can help you both with actually writing the code, but also helping those engineers, like, you know, bounce around ideas and learn what they need to learn to design that system.

  4. 57:16

    But what's most interesting is the two things that mattered most were the plan to do the thing, and then the verification of the thing. 'Cause the actual implementation was, like, Claude writing code. Yeah. And so what we ended up doing was we literally built, like, an entire fork of the entire API in Rust, and then started running shadow traffic through it and verifying that the outputs that we were getting from the Python system and the Rust system- This is the same strategy- Feature flags is all you need.

  5. 57:42

    Yeah. Literally. [laughs] Um, and it's just- Load balancing feature flags. Yeah... it's just, I don't know. I never in my career would have had a team say like- Having to be in Rust... "Yeah, yeah, cool. We're just gonna, like, port the entire-" But do, do you still have Rust people or no, no Rust people? Uh, well, they would probably call themselves Rust people now. They're Rust people now. [laughs] Yeah.

  6. 57:57

    I mean, they're feeling they've stepped to Rust. Yeah, exactly. Um, and along the way, we have hired a couple of people who have more experience with Rust. Sanity check. But yeah. Oh. But it, it still is just, like, you never would've signed up for that. It would've taken years and years and years. Like, you wouldn't do it all in one shot. You would've carved out pieces of the system and say, like, "Migrate this, migrate that, migrate that." Yeah, easily a multi-year effort. But like... Yeah.

  7. 58:18

    But you would never, like, kind of one, in one swoop, right, like, port the whole thing over and then just, like, slowly turn it on, but that is what we ended up doing. Um, and it worked. And so not only did we actually save a lot of CPU, but we, uh, our latency has, uh, seen a nice improvement, which is pretty cool, too. Um, and you know, it's just... it's, it's a crazy era we live in 'cause you just never could have done that sort of thing previously. Uh, order of magnitude of savings on the, on the CPU side? Um- Ju- just for people considering this. Yeah, yeah.

  8. 58:48

    I'm, uh... Okay, order of magnitude, that's really interesting. So- Like ballpark. Yeah. Yeah, ballpark. I guess I would say, like, we've s- somewhere, like, 40 to 50%. Something, like, really crazy. Yeah. And then latency, I think we saw somewhere between, like, 10 and 20%, um, improvement on, like, time to first token specifically. So a lot of just really cool improvements, uh, across the board.

  9. 59:11

    It's higher on, like, uh, longer running, like, workloads, but- So it'll, it'll roll out- Yeah... um, soon. Yeah. But it'll get faster. Do you fight the, the urge... You know, I've been involved in some major rewrites.