# Early-access model testing depends on customers with robust evals

Markdown copy of https://realalpha.app/s/early-access-model-testing-depends-on-customers-with-robust/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Katelyn Lesse](https://realalpha.app/builders/katelyn-lesse/) · Head of Engineering, Anthropic · Leads engineering for Claude’s APIs, developer tooling, and product infrastructure.
- Host: [Shawn Wang (swyx)](https://realalpha.app/people/swyx/) · Co-founder, AI Engineer · Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.
- Show: Latent Space
- Episode: [How Anthropic ships the #1 LLM in the World — with Katelyn Lesse and Angela Jiang](https://realalpha.app/v/MRQTlwcW-aQ/), Oct 27, 2026
- Moment: 31:56 to 35:37 of the episode (4 min), playing on https://realalpha.app/s/early-access-model-testing-depends-on-customers-with-robust/; the episode from there: https://realalpha.app/v/MRQTlwcW-aQ/?t=1916

## Quote

> “the only customers really that are part of our early access programs are customers that have **really robust evals**.”
> — [Katelyn Lesse](https://realalpha.app/builders/katelyn-lesse/) at [32:22](https://realalpha.app/v/MRQTlwcW-aQ/?t=1942) · Latent Space

## Takeaways

1. Choose early-access customers that can return actionable eval results.
2. Use both quantitative tests and qualitative reactions.
3. Model tuning has to balance proactivity against restraint.

## About this segment

Anthropic keeps early-access testing windows tight and works with customers that have strong eval suites. Negative feedback helps tune behaviors such as laziness versus excessive action.

From “How Anthropic ships the #1 LLM in the World — with Katelyn Lesse and Angela Jiang”, Oct 27, 2026. [Watch the full episode](https://realalpha.app/v/MRQTlwcW-aQ/)

## Transcript

From the transcript Latent Space publishes with the episode.

[31:59](https://realalpha.app/v/MRQTlwcW-aQ/?t=1919) And we take that all together and once we've all feel good about like all those different pieces, we launch the model. The state of model feedback is- feels very unstructured still. Mm-hmm. Basically, it's always a Slack channel between companies of like, "Hey, this thing seems weird." And then people are like, "Okay, I'll fix it, I guess. Give me an ID or something." Yeah. It's very- [laughs] I mean, it's, it's the nature of like a product- Can we do better? [laughs]... like non-deterministic, right?

[32:23](https://realalpha.app/v/MRQTlwcW-aQ/?t=1943) Like, but I think to your point, the only customers really that are part of our early access programs are customers that have really robust evals. Like, I do think a lot of this is vibes-based, and I think a lot of the conversations tend to be kind of like vibes-based and, and on the fly, and part of that is because we're, like, racing, right? It's like we wanna minimize as much as possible the window between we have a version of the model that we've cut for testing and we give everyone access to that model, and so it is just like this really tight window, but the customers we tend to do it with are the ones who, like, we know they're gonna go like kick off their whole eval suite and be able to come back to us with like really actionable feedback as quickly as possible alongside the vibes, um- Yeah, yeah...

[33:04](https://realalpha.app/v/MRQTlwcW-aQ/?t=1984) the vibes-based feedback. So, um, of course there's like definitely like processy things that we could do to make these things better over time, but I mean, you'd be surprised. I mean, you probably wouldn't be surprised, but I think some people would be surprised about how few, you know, like products that exist in the world that are agentic that, like, don't have good evals [laughs] to be able- Oh, we had-...

[33:24](https://realalpha.app/v/MRQTlwcW-aQ/?t=2004) to help test these models. We had Boris on the pod two years ago saying we don't have evals for Cloud Code. Yeah. [laughs] I mean, that's changed, but- That is true. [laughs] That's- that's changed. But you know, like, uh, uh, yeah, a lot of things like, you know, your evals are your users, and you don't have time to formalize them, or you don't have the experience or whatever- Yeah... and that's fine.

[33:39](https://realalpha.app/v/MRQTlwcW-aQ/?t=2019) At some point when you get serious, you- you'll have evals. Yeah, totally. Exactly. And I think the good thing is you guys do a lot of amp fooding, as you say, so it's commonly said that you find a lot of this testing it yourself before it comes out, right? Yeah. Yeah. Yeah. The APIs, uh, customers in particular- Yeah... are very useful because they'll put Cloud into a different harness. Um, and we want Cloud to be very, very general.

[33:58](https://realalpha.app/v/MRQTlwcW-aQ/?t=2038) Uh, so its ability to work in other harnesses is, is quite important to us, and so that style of feedback tends to be exceptionally useful for us because it's, it's novel, um, from what we would've gotten from ant fooding. What kind of feedback helps? Is it like regressions? Mm-hmm. We have regressions here. We notice good attributes here. Yeah. Uh, the good stuff is always, it's always very validating.

[34:18](https://realalpha.app/v/MRQTlwcW-aQ/?t=2058) Uh, I think we're very frequently not surprised, but we like to look to be validated on some of those things that we might hypothesis. Honestly, the most useful feedback is negative. Um, like, you know, it didn't do well at this, it didn't do well at that. We love that feedback, and then we're like, "Great, how do we fix it?" Yeah, and it tends to be some of these things that are like... It's really hard because not everyone actually wants, like, Claude to land at the exact same spot on every spectrum. Mm-hmm. And, and maybe a good example is like, you know, laziness versus proactivity and taking action.

[34:46](https://realalpha.app/v/MRQTlwcW-aQ/?t=2086) And so there's sometimes when, like, we have a model and, and people are like, "Ugh, like, just finish my task." Like, "Stop being lazy," like, "Do better." And other times when it's like, "Hey, we were like just kind of feeling this out and planning, and then you like went and wrote a bunch of code, or you went and like took- Yeah... a whole bunch of action that I wasn't ready for," and getting that exactly right is literally impossible because everybody actually wants something different in reality.

[35:08](https://realalpha.app/v/MRQTlwcW-aQ/?t=2108) But I do think we've had situations where we've cut an early version of the model, we found that where we saw too much laziness, right? And like, and we like put it back in the oven for a bit- [laughs]... right? And, and tried to train that back out. And so those are the sorts of things that we may not have caught if all we're doing is internally ant fooding within our products and within our API. Mm-hmm.

[35:28](https://realalpha.app/v/MRQTlwcW-aQ/?t=2128) It's way, way more important for us to get that out there in the API and get our, all of our other customers running with it and let us know what we need to change. Yeah, it's a core question.