# Most regressions were caught before pre-release

Markdown copy of https://realalpha.app/s/most-regressions-were-caught-before-pre-release/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Stephen Toub](https://realalpha.app/people/stephen-toub/) · Stephen Toub is a Distinguished Engineer at Microsoft.
- Host: [Ryan Peterman](https://realalpha.app/people/ryan-peterman/) · Host, The Peterman Pod · Went from new grad to staff engineer at Instagram in three years.
- Show: The Peterman Pod
- Episode: [Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub](https://realalpha.app/v/PLMOVlZC4eU/), Nov 2, 2026
- Moment: 42:15 to 45:26 of the episode (3 min), playing on https://realalpha.app/s/most-regressions-were-caught-before-pre-release/; the episode from there: https://realalpha.app/v/PLMOVlZC4eU/?t=2535

## Quote

> “I would say probably two-thirds of these things that I'm calling regressions were caught **very soon after merging** to main and before we shipped any pre-release.”
> — [Stephen Toub](https://realalpha.app/people/stephen-toub/) at [42:43](https://realalpha.app/v/PLMOVlZC4eU/?t=2563) · The Peterman Pod

## Takeaways

1. Two-thirds were caught after merge but before a pre-release.
2. Pre-release CLI builds reached 5–10% of customers.
3. Some corner cases remained unnoticed in stable releases for weeks.

## About this segment

About two-thirds of regressions were caught on main before a release; some reached pre-release builds sent to 5–10% of customers. Corner cases could remain unnoticed in stable releases.

From “Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub”, Nov 2, 2026. [Watch the full episode](https://realalpha.app/v/PLMOVlZC4eU/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[42:22](https://realalpha.app/v/PLMOVlZC4eU/?t=2542) And I appreciate them for sponsoring my show. We talked a bit about regressions. And you mentioned that a lot of them were caught before they went out to prod, it sounds like. And I think a lot of people might hear regressions and they think, oh, you're taking down the app or, you know, things are exploding all the time or their incidents. Could you explain maybe the further protections after end-to-end testing that made it so these aren't bad regressions? I would say probably two-thirds of these things that I'm calling regressions were caught very soon after merging to main and before we shipped any pre-release.

[43:02](https://realalpha.app/v/PLMOVlZC4eU/?t=2582) Even though we're shipping, at this point, we're shipping, I think, three pre-releases a day. So Then there was, there were some that made it into a pre-release and, um, you know, our form of ring deployments with the CLI is basically pre-release and stable where pre-release on any given day is probably hitting five to 10% of the, um, the audience, um, of the customer base. And then, you know, 90 or 95% is only unstable.

[43:31](https://realalpha.app/v/PLMOVlZC4eU/?t=2611) And so a, a, a reasonable number of these issues, uh, I don't know exactly what percentage 10 20 percent um found their way into a pre-release and then pretty quickly we'd get an issue opened or someone saying hey like i'm seeing this difference in behavior was this by design or not and so a lot of these were things that were different not necessarily bad but different enough that someone noticed and then asked us like was this on purpose um You And some of those cases were ones where I'd failed to prevent the agent from fixing bugs as part of the port.

[44:08](https://realalpha.app/v/PLMOVlZC4eU/?t=2648) And so some of them are actually desirable changes, but we hadn't intended to, you know, they weren't purposeful in that regard. And then there were a number that made it into stable releases. And the ones that made it into stable releases, because we hadn't noticed them in either main or pre-release they often ended up staying in and being unnoticed in stable releases for quite a while um and this is mostly the trickle that we have been seeing over the last you know well we don't really see much anymore but a month ago would have been the trickle that we were seeing you know over a few weeks uh where some super corner case thing just behaved slightly differently and then we get an issue open like hey this this isn't working the way i expected you know is this on purpose or not and oftentimes they weren't and we'd fix them as soon as we heard about them

[44:59](https://realalpha.app/v/PLMOVlZC4eU/?t=2699) um there were also ones that we would see from various telemetry systems you know uh monitoring for exceptions crashes uh things like that and obviously those are unintentional when we fix those in the post you you shared this graphic of which models you used
