Try Pro free

The Peterman Pod · 3 min · Nov 2, 2026

Most regressions were caught before pre-release

About two-thirds of regressions were caught on main before a release; some reached pre-release builds sent to 5–10% of customers.

In this episode

Full analysis
  1. 11:27Incremental ports keep fast-moving products shippingPro pick 7 min
  2. 22:11End-to-end tests matter more than unit tests for a port6 min
  3. 49:55A Rust port cost $120,000 in model tokensPro pick 6 min
  4. 5:36Rust won after performance and embedding tests tied4 min
  5. 32:02Strict one-to-one ports limited agent scope4 min
  6. 42:15Most regressions were caught before pre-release3 min
  7. 1:00:01The port grew to three times its estimated size3 min
  8. 1:03:24A rewrite needs a measurable business return3 min
“I would say probably two-thirds of these things that I'm calling regressions were caught very soon after merging to main and before we shipped any pre-release.”
STStephen Toubat 42:43 · The Peterman Pod
42:43

Takeaways

  1. Two-thirds were caught after merge but before a pre-release.
  2. Pre-release CLI builds reached 5–10% of customers.
  3. Some corner cases remained unnoticed in stable releases for weeks.

About this segment

About two-thirds of regressions were caught on main before a release; some reached pre-release builds sent to 5–10% of customers. Corner cases could remain unnoticed in stable releases.

ST

Stephen Toub

Stephen Toub is a Distinguished Engineer at Microsoft.

  • Ryan Peterman, Host · Host, The Peterman Pod

    Went from new grad to staff engineer at Instagram in three years.

From “Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub”, Nov 2, 2026. Watch the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 42:22

    And I appreciate them for sponsoring my show. We talked a bit about regressions. And you mentioned that a lot of them were caught before they went out to prod, it sounds like. And I think a lot of people might hear regressions and they think, oh, you're taking down the app or, you know, things are exploding all the time or their incidents. Could you explain maybe the further protections after end-to-end testing that made it so these aren't bad regressions? I would say probably two-thirds of these things that I'm calling regressions were caught very soon after merging to main and before we shipped any pre-release.

  2. 43:02

    Even though we're shipping, at this point, we're shipping, I think, three pre-releases a day. So Then there was, there were some that made it into a pre-release and, um, you know, our form of ring deployments with the CLI is basically pre-release and stable where pre-release on any given day is probably hitting five to 10% of the, um, the audience, um, of the customer base. And then, you know, 90 or 95% is only unstable.

  3. 43:31

    And so a, a, a reasonable number of these issues, uh, I don't know exactly what percentage 10 20 percent um found their way into a pre-release and then pretty quickly we'd get an issue opened or someone saying hey like i'm seeing this difference in behavior was this by design or not and so a lot of these were things that were different not necessarily bad but different enough that someone noticed and then asked us like was this on purpose um You And some of those cases were ones where I'd failed to prevent the agent from fixing bugs as part of the port.

  4. 44:08

    And so some of them are actually desirable changes, but we hadn't intended to, you know, they weren't purposeful in that regard. And then there were a number that made it into stable releases. And the ones that made it into stable releases, because we hadn't noticed them in either main or pre-release they often ended up staying in and being unnoticed in stable releases for quite a while um and this is mostly the trickle that we have been seeing over the last you know well we don't really see much anymore but a month ago would have been the trickle that we were seeing you know over a few weeks uh where some super corner case thing just behaved slightly differently and then we get an issue open like hey this this isn't working the way i expected you know is this on purpose or not and oftentimes they weren't and we'd fix them as soon as we heard about them

  5. 44:59

    um there were also ones that we would see from various telemetry systems you know uh monitoring for exceptions crashes uh things like that and obviously those are unintentional when we fix those in the post you you shared this graphic of which models you used