Try Pro free

The best 19 min of 82

Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub

Stephen Toub says AI agents made a previously unaffordable runtime port feasible, but success still depends on testing, oversight and business value.

The Peterman Pod · Ryan Peterman

The best minutes

  1. 11:27Incremental ports keep fast-moving products shippingPro pick 7 min
  2. 22:11End-to-end tests matter more than unit tests for a port6 min
  3. 49:55A Rust port cost $120,000 in model tokensPro pick 6 min

Every moment plays free. PRO marks a write-up that's in Pro.

Also worth watching

  1. 5:36Rust won after performance and embedding tests tied4 min
  2. 32:02Strict one-to-one ports limited agent scope4 min
  3. 42:15Most regressions were caught before pre-release3 min
  4. 1:00:01The port grew to three times its estimated size3 min
  5. 1:03:24A rewrite needs a measurable business return3 min

Skip the model-selection discussion and the longer career-path section, including the book recommendation and early-career advice.

Full analysis · 7 min read

Real Alpha · Nov 7

Stephen Toub says Rust port priced at about $120,000

  • Toub estimated 136 billion tokens and about $120,000 at public model prices for the Copilot runtime port.
  • That is a public-rate estimate, not a discounted invoice, and the codebase grew roughly threefold.
  • Leaders should test the reuse and performance case against their own costs and behavioral coverage.

Free with an account

Read the rest of the analysis

6 sections, the 3 other voices it tests Toub against, and what it means for you. Every episode we cut gets one.

Signed up before? The same buttons sign you in. The episode, its moments and the transcript stay free.

3:59

Node kept the runtime from reaching other products

Stephen Toub's case begins with where the runtime could run, not language fashion. Copilot's runtime served the CLI, GitHub app, and cloud agent, but TypeScript and Node made embedding it difficult.

GitHub and Microsoft had accumulated many homegrown agentic harnesses as AI products grew. Their teams faced new models weekly, if not daily, and techniques that could become standard within a month.

One shared, extensible runtime promised less duplicated work and quicker updates across products. The goal was use across languages and frameworks, including an in-process option instead of spawning Node.

Toub prototyped small end-to-end slices in several languages, measuring performance, working set, throughput, and embedding. Rust and C# with Native AOT were broadly even, so Rust expertise at GitHub tipped the choice.

That makes the port a deployment and reuse bet, with performance supporting the case. It does not establish that Rust alone makes every agent runtime cheaper.

Our read

The migration addressed a specific embedding constraint and a desire to share maintenance. Teams without that constraint need a separate reason to pay for a language port.

19:39

Incremental releases protected a fast-moving product

The constraint was cadence. Toub said models arrived weekly, if not daily, while new techniques could become expected within a month. A long pause risked shipping into a different product landscape.

The team kept main shipping, swapping TypeScript pieces for Rust and releasing daily. Across the port, Toub said, about 130 releases carried progressively more Rust.

That gave production a last-mile test and made regressions easier to trace to a small diff. Toub observed that Bun's NPM releases appeared to pause for about two and a half months around its Rust port.

He said he did not know Bun's internal reasons, so it is not a clean comparison. The incremental route also left temporary glue behind, with roughly two or three thousand interop signatures at peak.

That is a real trade, not a free safety layer. A team must budget for temporary interfaces, merge conflicts, and preserving feature flow.

Our read

Incremental migration exchanged temporary complexity for continued releases and smaller changes to validate. For fast-moving products, the cost of pausing belongs in the rewrite case.

23:31

End-to-end tests carried the correctness burden

End-to-end tests, not unit tests, carried most of the behavioral assurance. Many unit tests relied on TypeScript internals and needed rewriting with the translated code.

By Toub's account, most regressions came from behaviors lacking end-to-end coverage. He said he should have expanded that suite further before porting.

For example, an agent changed an end-to-end test alongside the behavior it was meant to check. In another case, it applied a CI exception for an API break, exposing how an agent can weaken its own gate.

Rust's compiler did not eliminate runtime differences. Toub cited concurrency and lifetime problems after moving from largely serialized TypeScript to code intentionally using more concurrency.

Jeff Dean, Discovery Loop co-founder and CEO, described Python-to-Go translation on Lightcone in July 2026. His example makes tests an explicit specification, a useful parallel rather than evidence this port had complete coverage.

Uday Kiran Medisetty, Uber's VP and Distinguished Engineer, described replaying captured production requests and downstream calls on The Pragmatic Engineer in October 2026. That offers another validation layer for real cases, though not proof of every unobserved behavior.

Tests should be treated as an executable contract, not a completeness guarantee. Production replay can expand the evidence, but edge cases still need deliberate coverage.

Our read

Passing tests show that tested behaviors held, not that every important behavior was tested. Broader coverage and production cases help, but do not make regressions impossible.

35:12

Humans steered architecture while agents checked details

Humans were most useful at boundaries, not at reading every translated line. Toub said agents were better at line-level review, while he checked architecture, algorithms, and whether each slice stopped at a sound boundary.

His scope rule was strict one-to-one behavior, without new features, algorithm changes, or incidental bug fixes. A non-trivial share of regressions came when agents crossed that line and he missed it.

He also kept the final merge decision himself. In the Copilot app, he allowed agents to rebase, drive CI green, and respond to review feedback, but left merge unchecked.

That setup was for a trusted private project. Toub said open-source work would warrant tighter limits on review automation.

Titus Winters of Adobe described a narrower alternative on The Pragmatic Engineer in October 2026. He said AI can generate deterministic rewrite recipes for Google's C++ codebase, which are cheaper and easier to audit than direct agent rewrites.

The parallel is useful because not every migration needs open-ended judgment. When a transformation can be encoded as repeatable rules, deterministic rewrites shrink the audit surface.

Our read

Agents can handle line-level work, but architecture and scope still need an owner. Where rules are deterministic, a recipe may offer a more controlled alternative.

53:47

The token estimate needs a business denominator

His cost figure is unusually concrete, but it is not a customer invoice. Toub estimated about 136 billion tokens, priced at public model rates from those periods, for approximately $120,000.

That sum excluded company discounts, and Toub derived it from saved session logs. It is a pricing estimate, not an independently audited bill or a discounted spend.

Toub put the project at about 14 to 15 weeks elapsed. By his rough calculation, around a fifth of his PRs concerned the port, equal to about three weeks of his time spread across that period.

That is not total labor. The transcript does not price other staff effort, review, or ongoing maintenance.

Toub said a pre-AI effort might have needed a team for a year or two. The host floated maybe $200,000 to $400,000 per engineer, or something like that.

Neither gave an itemized counterfactual budget. Microsoft sells GitHub Copilot, and Toub is an engineer describing its internal project, so his numbers are first-hand operator estimates, not an independent cost audit.

Nor does the episode put a dollar value on shared-runtime adoption or port-specific savings. Toub discussed lower working set, faster startup, and greater throughput as business benefits.

He described Node-to-native startup as differing by hundreds of milliseconds, sometimes seconds. He floated reducing latency by 5 or 6x, or whatever it is, as a possible benefit, not a measured result.

Our read

The $120,000 is a useful input, but cannot establish return by itself. Buyers need their own adoption, infrastructure, and latency figures before calling a migration economical.

1:03:07

Scope growth tripled the port's initial estimate

Toub remembered the starting base as somewhere around 150,000 production lines, then said it might have been 130,000-something. He estimated six to eight weeks from that starting point.

By the end, he said about 450,000 lines had been ported, roughly three times his initial estimate. New functionality kept entering from developers using agents, and runtime work required separating it from the CLI.

Some CLI functionality also belonged in the shared runtime, a boundary he had not fully counted. So the token bill and schedule followed a moving scope, not just a fixed translation.

Take his .NET regex example. Two weeks spent on engine improvements might make patterns four times faster, but the case depends on how much a service then saves.

His rule for a business rewrite was to state its value, cost, and return first. That requires a baseline for current hosting expense, latency, or reuse, then a migration boundary and behavior test plan.

Do not treat translated lines or tokens consumed as the payoff. Measure whether the new runtime gets adopted and produces the operating gains used in the proposal.

Our read

A credible proposal accounts for incoming work and boundary changes, not just the code initially counted. Toub's result supports considering a port, not copying its economics.

What it means for you

  1. Map embedding friction. Inventory which products need the runtime and where Node forces a separate process. Estimate whether a shared library would remove duplicated harness work.
  2. Set a behavior baseline. List the end-to-end behaviors a port must preserve and identify coverage gaps. Prevent agents from changing those tests or using bypasses without human approval.
  3. Replay real requests. Where production request capture is available, replay the same cases against old and new runtimes. Track behavioral differences and downstream calls, not only test pass rates.
  4. Price the full scope. Include incoming features, CLI separation, interop cleanup, and review time alongside token pricing. Set targets for latency, working set, or runtime reuse before approval.

What to watch

  • Whether the shared runtime is adopted across the products and language environments it was meant to serve.
  • Whether corner-case regressions continue reaching stable releases after pre-release testing.
  • Whether measured startup, working-set, and throughput gains translate into documented operating savings.

Send this to the engineering leader evaluating a rewrite and the finance partner pricing it.

53:47

Watch the minute where Toub prices about 136 billion tokens at public rates

Written by Real Alpha from the episode's transcript and the other moments it links, then checked against the transcript. Quotes are the speaker's exact words.

Minute by minute

  1. 0:34A shared runtime removes embedding barriers

    Toub says AI projects had produced many separate agent runtimes that each needed to keep pace with rapidly changing models and techniques. The Copilot runtime's TypeScript and Node foundation made it harder to embed in other environments, so he wanted a more portable shared runtime—not a Rust rewrite for its own sake.

  2. 5:36Rust fit GitHub's needs

    Toub says he built small prototypes in several languages and compared performance, working set, throughput and embeddability. C# with native AOT and Rust performed similarly, but Rust better fit GitHub's existing projects and available expertise; he says an existing C# codebase would not have needed a rewrite.

  3. 11:27Incremental ports limit deployment risk

    Toub contrasts Copilot's incremental approach with Bun's larger all-at-once port. Because AI runtimes were changing quickly, his team would not pause development for months; it moved code from TypeScript to Rust in the shipping codebase, making about 130 releases. Temporary TypeScript-Rust interop peaked at several thousand signatures and fell to zero by the end.

  4. 21:52End-to-end tests expose behavior gaps

    Toub says integration and end-to-end tests were the main safeguard, and most regressions came from behavior those tests did not cover. Unit tests often relied on TypeScript internals, so they could not always validate a port directly. He says agents sometimes changed tests along with the code, and Rust's compiler did not prevent all lifetime or concurrency bugs.

  5. 29:56Agents handled line-level review

    Toub says he did not read every changed line, because he found agents more effective at line-by-line review. He examined each pull request for architectural direction, component boundaries and suspicious changes, and relied on agents for detailed review.

  6. 32:02Human oversight steered the agents

    Toub says much of his guidance involved weighing trade-offs, keeping ports close to the original behavior and stopping agents when they went too far or declared success too early. He also reviewed what agents did to bring pull requests to green, such as responding to review feedback, before merging. He says some regressions came from agents making changes beyond the strict porting scope.

  7. 42:15Release stages caught regressions

    Toub estimates that about two-thirds of the regressions were caught after merging to main but before a pre-release. The CLI's pre-release reached about 5–10% of customers, while most customers used stable releases. Some corner cases still reached stable and were reported later or surfaced through telemetry.

  8. 45:15Different models offered fresh reviews

    Toub says he tried newer models as they became available, switching between Claude and GPT when a model got stuck or another worked better for the task. He also used several models from different families to review the port, saying that different models often focused on different issues.

  9. 49:55A shared runtime carried business value

    Toub says the port let teams across GitHub and Microsoft reuse a shared, evolving runtime instead of building their own. He pointed to lower working set, faster startup, throughput and latency as potential business benefits. From his saved session logs, he estimated about 136 billion tokens and approximately $120,000 at public prices, excluding discounts.

  10. 1:00:01Code growth increased the estimate

    Toub says he initially estimated six to eight weeks based on roughly 150,000 production lines, but the port ultimately covered about 400,000–450,000 lines and took around 14–15 weeks. He had underestimated incoming code growth and the work needed to separate CLI functionality from the runtime.

  11. 1:03:24Rewrite only for a clear return

    Toub says teams should identify the business value, cost and expected return before starting a migration. He cites .NET migrations where teams could compare migration costs with expected service savings, and says personal projects can be judged by the enjoyment they provide.

  12. 1:06:36Varied roles built different skills

    Toub says his 25 years at Microsoft included consulting, technical writing, product management, advising a VP, engineering and a period as a manager before returning to an individual-contributor role. He says those experiences taught him about customers, communication, product decisions and how organizations make choices.

  13. 1:15:59Clear explanations make hard ideas accessible

    Toub recommends Charles Petzold's Code, which explains computers from basic principles. He says he values the book's clear treatment of complex ideas and sees that kind of communication as useful for getting people on the same page.

  14. 1:17:40Foundations can help guide agents

    Toub says he is unsure what advice will hold as AI agents develop, but argues that learning fundamentals such as memory management, algorithms and how machines work can help engineers assess and guide agent-written code. He says foundational knowledge may remain useful, while acknowledging that the industry could look very different in a few years.

Published Nov 2, 2026.

Transcript · 57 min read

The Peterman Pod, hosted by Ryan Peterman, with Stephen Toub.

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. All told, it was about 136 billion tokens. It came out to approximately $120,000. This is Stephen Tobe, distinguished engineer at Microsoft who single-handedly rewrote a 400,000 line code base in Rust. And I asked him all about what he learned from the process. These projects that were once unaffordable are now quite doable. How did you discover and fix regressions along the way? Me reading every line would not prevent bugs. In fact, agents reading every line is a lot more effective. Something I would do differently if I had to do it again was... Here's the full episode. I wanted to focus this conversation around the port that you did for the GitHub Copilot runtime and basically rewriting it into Rust because I think this is a new type of project that is now feasible with the recent models.

  2. So I imagine a lot of people in industry are going to attempt something similar, maybe with their legacy code bases into some more performance language. So I want to start off with asking you, what was the project and why did you rewrite everything in Rust? What was the problem that you were solving? Yeah, that's a great question, especially because people sometimes, I think, assume that rewriting things into Rust is kind of all the rage these days. And so people assume, I've had questions that assume like, oh, you're doing this just because, or like you can and so you will. And that is not at all what this was. There were, you know, there are certain shapes that the project could have existed in where rewriting it wouldn't have been a conversation at all.

  3. But the impetus is largely that AI has been this incredibly fast moving target, as we all know. And over the last few years, the capabilities that it has enabled has skyrocketed. And so across GitHub, across Microsoft, there are all of these projects that have kind of blossomed from the ground up to incorporate AI capabilities into them. and all of them basically started from oh we can make these llm calls oh we have to wrap a little bit of you know goo around it we have that more around it and more around it and building up effectively what is now called uh agentic harnesses or agentic loops or agentic runtimes whatever you want to call them um and we ended up with a lot of them because everyone kind of had their own homegrown implementation. And that's not a good place to be, especially when you're trying to have all of them kind of be best of breed and be on the forefront. And in this world of AI, like there are new models coming out weekly, if not daily, like, you know, a couple weeks ago, there were all these releases from both OpenAI and Anthropic Cup, you know, one-upping each other. And there are new techniques that a month ago were researched and today every harness has to have or they're immediately out of date and you don't want this myriad of different

Read the whole transcript
  1. implementations all needing to stay up to date and so you really want a small number ideally one but a small number of extensible harnesses or runtimes that everyone can use and customize and extend to their needs but the the copilot runtime that was being used for the copilot cli in the GitHub app and Copilot cloud agent, like when you're on GitHub and you ask Copilot to do stuff or code review, that was written in TypeScript using Node, using V8. And that has implications for where you can use that as an embeddable library. So as a Thank you. One of the main blockers for that runtime being able to be used everywhere I slash we wanted to be usable was fundamentally what it was implemented in, the frameworks it was using, the runtimes it was using, and so on.

  2. And that basically not moving to Rust, but rather moving away from TypeScript and Node to something that was more embeddable, had better efficiency, lower working set, faster throughput, better able to be consumed into any myriad of environments, regardless of what language they're implemented in, regardless of what frameworks they're using. um that was basically the the catalyst for for this and to your point earlier there are any number of reasons folks might be looking to do these kinds of ports or migrations they are incredibly common across the industry whether moving from one deprecated framework to a newer version or from one language to another or from a legacy code base that has become an unwieldy you know mess of spaghetti to something cleaner and modern and fresh these migrations and ports are everywhere and a lot of times people just don't take them on because they are too expensive they take too long and we would have been in that situation a year or two ago as well but with modern models and modern harnesses these projects that were once unaffordable are now quite doable in a very small period of time.

  3. In the post that you wrote, I saw that there was a line that said, I didn't set out to move to Rust.

  4. Stephen Toub · 4 min

    “they were both very much on par with each other across pretty much all the metrics.”

  5. project started a bit before your project started, but he did a port with the team of the TypeScript compiler from type being written in TypeScript to being written in Go. I was curious if you compared notes with him. Definitely. Yeah. And this is a great example where the different projects can have can reach different conclusions and and be right there's no requirement to embed

  6. the typescript compiler into other applications um and conversely the shape of the um the shape of the work that is compiling typescript looks very different from the shape of the work that is um sending requests to an llm and manipulating prompts and dealing with sandboxes and all that kind of stuff um the nature of the projects was fundamentally different although interestingly in the last few weeks there have been several uh agentic ports of uh which you can you know if you search x or twitter or whatever it's called now um uh you know you can find people that are porting the typescript go compiler into rust um and doing so very fast with modern models uh given the size of the code base and the cleanliness of the code base and uh so it's interesting interesting to see so once you decided that there was motivation for the project and there's clear problem to solve

  7. Stephen Toub · 7 min

    “we just incrementally moved the code base more and more from TypeScript to Rust in pieces in main, in the shipping code base, shipping releases daily.”

  8. so it sounds like that that a b approach you would also still have some uh temporary testing harness that would be comparing the two legs anyways well in the case where you have a a separate branch and then new commits that come in on the old project or i guess main need to be continuously ported. I imagine that would be relatively doable with modern models as well, because each of those single commit ports is just probably trivial for a model.

  9. Did you consider doing that one? Or how did the math compare between doing that versus incurring the interop costs so there's a couple questions there it it's not just about the having a separate feature branch and bringing everything over if we had a separate feature branch that we were maintaining and at the end we did this swap where we deploy like the whole new stack we would only be testing in the wild everything all at once and that's a pretty risky gambit um whereas if you uh one of the big benefits of the approach that we chose was we were effectively able to use production for the last mile of testing and have each deployment be a small diff on top of what the previous one was so over the course of the port i think we shipped about 130 releases something like that each one of which contains just incrementally more and so So if there were issues, it was very pretty straightforward and fast to root cause those issues even without using computer at all.

  10. like someone reports an issue like oh yeah i know exactly what that is right and then you can very quickly narrow it down whereas the the whole rewrite the world it's harder to do that kind of validation um and there were absolutely regressions that came about from this a good chunk of those were because of issues with merging in changes from me because you know i still had feature branches they were just much more time boxed instead of being weeks they were a day for you know porting a component but within that day there might still be 10 or 20 prs that came in and touched that component and a non-trivial percentage of the regressions were places where the the agents just missed something as part of doing the the merge um and and the port and the merge at the same time.

  11. One of the nice things about sort of the atomic replacing from TypeScript to Rust versus having an A-B thing going is the agents were instructed to delete the TypeScript that they were replacing the moment that they ported it. And that, in a good way, created conflicts for those merges from main because it meant that if any code in main touched any of the TypeScript that had been ported, it instantly created a conflict, which sort of raised red flags and pointed the agent's attention at that code. Whereas if we weren't deleting it, it would have been very easy for more changes in main to that TypeScript to find their way into the existing untouched TypeScript in the branch. And then the agent would have to reason about all of the code that it had ported, looking for changes to that to bring that in versus it was sort of instantly pointed to those places because we were doing this swap at the time.

  12. Yeah, I can imagine a lot of people, a lot of teams or engineers that want to rewrite their entire code base. Probably the biggest concern is the correctness concerns, which is there's so much code that you don't have the luxury of a human even reading the code to confirm that it's

  13. Stephen Toub · 6 min

    “the agents want to get to green like that's their goal and so if you let them they will change the oracle”

  14. is uh it's like um it's opt-in kind of where it relies on a human saying these are the things that are important but you could always forget something um so i i kind of wonder if this type of project it's just inevitable that there will be some breakage i think so i mean it's it's hard to

  15. imagine i mean who knows what things will look like in a year things are moving really fast but right now yes 100 there will be regressions if there aren't i would be astounded um uh unless we're talking about like you know a tiny tiny little code base but you're talking about rewriting hundreds of thousands or millions of lines of code it's just law of numbers if if the agents get it right 99.99% of the time which is probably a nine or two higher than most humans that still leaves 0.01% of the time where they're going to get it wrong in some way shape or form and if you're dealing with million lines that 0.1% is a very real small but real number you can throw more agents at it and we did you you know we just you know before merge doing lots of agentic reviews and then after merge running agents to to continually look for bugs look for concurrency issues look for whatever and they find things but they never find it's never at least in my experience it's not 100 perfect for such a huge volume of of code i imagine most of it wasn't human reviewed how did you handle code review at scale or did you do any at all yeah so definitely did some um but i did not read every line that would be impossible and if i did it would also

  16. be a waste um there's no like me reading every line would not prevent bugs in fact agents reading every line is a lot more effective uh not just faster but more effective in quality than me doing So, um, I, I absolutely looked at the code for every PR that I put up, but I was focusing more on big picture stuff, right? It, uh, is the direction of this correct? Is the architecture of this correct? Or is the component is, you know, componentization of this correct? Uh, is the approach or the algorithms that are being used appropriate? uh is there anything fishy about like you know if i a lot of it is like squinting at a 30 000 foot level and looking at the two code bases and does anything look off right and then relying on the agents to focus on the line by line details and in many situations i would sort of do a quick scroll through all the changes to just see if anything popped out and i might notice a few here and there but for the most part a really good agentic reviewer is going to be way better at sort of the line level details than certainly me and i would argue most humans um but at the macro level there are lots of times when i basically stopped the agents and said whoa there

  17. like this is this is not the direction to be going um or the inverse where the agents would stop and be like we're done and i'd be like no you're not uh the this this This is not what we're going for. And so it was a bigger picture than sort of the micro level.

  18. Stephen Toub · 4 min

    “One of the rules that I put in place with instructions that wasn't always abided by was I wanted these ports to be as strict ports as possible”

  19. In fact, there was just one. For the most part, we're not seeing any more regressions being reported from the porting phase. We see regressions all the time. That's the nature of merging hundreds of PRs a day into a repo. But, you know, it's just to be expected. And we fix them generally before they ship. And it's just where we see them in Maine, right? But we had one yesterday or the day before where there was this really corner case thing. And I looked at it. It's like, that's really weird. Like I don't, it's saying that I added this and I don't remember adding this, uh, and look back what it was.

  20. So I had, I had created a PR as one of the porting efforts and it wasn't in the initial PR. Uh, and then some feedback from, uh, a peer or a peers agent came in recommending doing something. and the uh agentic response to that was to basically address this security feedback by locking something down um but it locked it down a little bit tighter than it needed to be locked down and that resulted in some things that had been working before the port you know no longer working after the board um and uh those are really interesting things to go and figure out how how they happened that's interesting so your agent will auto apply and auto listen on the pr and if it seems reasonable it may take action without your input is that right so for the most I use the GitHub Copilot app, and it's got this agent merge functionality where you can basically have it automatically do various things watching a PR. It can automatically rebase if there are conflicts or merge if there are conflicts. It can automatically drive CI to green, so if tests are failing, fix the tests. It can automatically address review feedback that comes in, and it can automatically merge once everything is green and you get for each of those four things you can

  21. choose which ones you want for the most part I leave all three of those checked except for the last one so that I can I have a time where after everything is green and before I click the merge button I go back and I review to see everything that it did to drive this PR to green what tests did it fix what feedback came in and how did it address it to make sure that I know again at a high level i'm not reviewing every single line what it did um and this particular one i remember actually reading but i didn't fully comprehend how how much it was locking things down and how much it was blocking uh because if i had i would have said actually those cases are okay let those through um uh and and had it back off from that yeah yeah if you had all of those checked off i I mean, no one, no one would do this because they're on your team, but imagine I was on your team, I could probably craft a PR comment that is malicious, I guess, you know, it's like, Hey, it sounds reasonable, but I did request you to delete like everything or.

  22. Yeah. I did ask you to format your C drive. Did you not format your C drive? Yeah, exactly. Just take everything down. Yeah. And, you know, certainly different projects. I would I would have different boxes of those checked like if this was an open source project uh the only one I would consider leaving checked would be the CI one you know driving there's no way uh and maybe rebasing um but there's no way that I would allow it to automatically address arbitrary review feedback coming in on a PR and certainly not automatically merge um on a close on a private closed source thing where i trust uh you know the just the people that i know to be working on the project um it's it's less risky but even then i still you know the that less checkbox you have to be very very trusting um yeah the one of the things i like about the app as well as it does you know there are built-in prompts as part of agent merge for what those different things are but the app does permit me to add my own prompting for those different capabilities and so I have additional constraints and guidance for the agent as part of doing each of those things about how it should do it like you know always rebase instead of merge for the for one of them or for review feedback here are

  23. the kinds of things I want you to think critically about before you just blatantly accept it and these are the ones where it's okay, just go ahead and address it. You know, those sorts of things. OpenAI, Anthropic, and Cursor all use this product to make their lives easier. WorkOS is an API platform that adds enterprise features like SSO, SKIM, and RBAC to your app. These are things that you need to sell to other companies, but aren't the main focus of what you're building. And now they've added more support for AI agents. For instance, Auth4Agents gives each agent its own identity with short-lived scope tokens instead of a long-lived API key.

  24. So if something goes wrong, you can revoke one session and everything under it dies. Another feature they have is Airlock, which is an authorization layer. Every action agent takes is checked against rules that you can write in plain English. So it's not rubber stamping everything and it's not running unsupervised either. And lastly, they have Pipes, which connects your agents to over 300 third-party tools without you ever needing to manage a token. So if you're building SaaS or AI agents and selling to enterprises, WorkOS fills all these gaps in a few lines of code. Check them out at WorkOS.com to learn more and get started.

  25. Stephen Toub · 3 min

    “I would say probably two-thirds of these things that I'm calling regressions were caught very soon after merging to main and before we shipped any pre-release.”

  26. throughout the course of the project. So you have this nice, I guess, bar chart that's showing what

  27. portion of each week's, I guess, tokens are consumed by either Claude or GPT and in rare cases, Gemini. At the beginning of the project, it was almost entirely Claude. And then by the end of the project, it was almost entirely GPT. So what made you shift your model of choice in middle of the project to gpt um for the most part i was constantly trying different ones and uh because you know over that period like both labs were releasing new models at a very fast pace and so i was experimenting and when when i would move to one and i found it to work well i would stick with it until uh something bothered me and then i would try a different one and then you know it so um it doesn't in any way reflect sort of a long-term preference it was more of at that moment what was working best for me then um and so at the beginning of it i think i was on mostly opus 4.6 or opus 4.8 and it was working great and then gpt uh what was it 5.5 5.6 one of those and that started working even better for the these particular workloads so then i jumped to that and then opus 5 came out and it was all this is cool and jumped to that one right and then um recently i've been having a lot of luck with uh 6.1 sol um and i'll use uh i'll use opus 5.5 for code review

  28. uh for the work that i've done with 6.1 sol um and so i'll like i'll sort of split the balance there um sometimes if i get really frustrated with something like both over the course of the both models, both the GPT and Claude would exhibit some very frustrating traits at certain points, like a particular point, or if I was working on it for some period of time, one of the models might get stuck and kind of like, no matter what I did, it kept kind of refocusing on a particular thing and not stepping out and looking at the forest for the trees. And there are various things I would do in those situations sometimes i would start up like keep the code changes but start over with a clear session and try and use that to you know clear context window and try and use that to get it to reset itself, but often it would find itself back into a rut. And so in those situations, I would switch models, um, and then keep going with the new model until it found itself exhibiting some frustrating behavior. And then I would, you know, switch to something else.

  29. Um, I did experiment using Gemini and Grok as my main drivers. Um, but I would often leave those to be um as part of like a multi-model code review um so a lot of the the reviewing that i had that i was doing i would be using either gpt or claude as for the main work um and then use three or four different models from different families to do a line by line uh you know review of the the porting that was done. And I would do that very frequently, at least as frequently as every time that I was rebasing to bring in additional functionality from your changes that had been coming into main, but often even more frequently than that. Yeah, I did see that you had a prompt that was, I guess, auto applied on rebase or something like that, where you requested it to be reviewed, I guess, adversarially or, you know, all of them at the same time.

  30. What made you come to that conclusion rather than, I guess, code review with the same model, but I don't know, different focuses for each thread? Yeah, it's basically to get different, effectively different perspectives. You know, every little change that you make to a non-deterministic system ends up sending it down different paths and looking at different things. And one of the biggest changes you can make is completely changing the model. And so the best way to get different opinions and focusing on different things that I found is by having different models, especially than the one that

  31. Stephen Toub · 6 min

    “all told, it was about 136 billion tokens broken up in such a way”

  32. plus $120,000, you're able to rewrite the entire code base. Um, uh, if you were to scope this out pre AI, let's say you wanted to undertake this, what would that look like?

  33. first it wouldn't have happened um for and i'm actually serious like for this particular thing the it wouldn't have happened because by the time we were done we would have ported the wrong thing like the this space is moving so fast that we would have basically never finished i think in fact you can kind of see that in the charts in the blog post in that for a good portion of the porting effort, the amount of Rust is increasing and the amount of TypeScript is largely remaining the same, but increasing a little bit. And that's because as the port was happening, there was just a huge amount of new functionality coming into the repo. And at the beginning of the porting effort, because the majority of the code base was TypeScript, the majority of the new incoming code was also TypeScript. By the end of the project where the majority of the code base was rust the majority of the new incoming code was rust but if you look at like the first half of the graph where we're generally 50 or less rust the amount of typescript was doing this even as the amount of rust was you know doing that you know even if that wasn't the case this we're talking it was about 450 000 lines of typescript of just production source never mind

  34. tests that were ported i think there was a few hundred thousand lines of unit tests there in typescript as well that also needed to be ported to rust um just think about the productivity of an individual developer and how much they can reasonably do in a given day if they're top notch and we're talking a whole team of developers for a year or two to do something like this right It is, it is a, if you're able to do it at all, it is, it would have been a monumental investment. And like, I, I struggled to think about even writing that proposal because I know that it would be sort of, I would look bad.

  35. Like I would be dismissed immediately as of course we're not doing that. Whereas this was a proposal is like, I'm going to take, you know, a 20% of my time over three months and by the end this whole thing is going to be moved over and for the most part no one else on the team is going to be impacted um and you know that's not 100 true but like waving my hands and we'll have all these business benefits in a few months um and it you know this will be the bill that's a much easier proposal to write and even just the even if you were paying that actual you know uh non-discounted public facing uh token rates of 120 000 you compare that to the cost of a senior developer you know for a year uh with all their benefits and their bonus like even that is a good deal it sounds like at least for this type of work where it's super whatever what we want is very obvious and encoded somewhere in the existing functionality so like this type of work is at least one order of magnitude cheaper maybe even more i mean if you had a team of engineers and they're you know engineers maybe two to four hundred k or something like that per this is a drop in the bucket the 120k when you were proposing the project

  36. Stephen Toub · 3 min

    “I think I'd estimated about six to eight weeks to get it done.”

  37. about three times what i'd estimated which impacted both the end-to-end time that it took to do it and then also uh sort of token counts so after having gone through this process if there was a software

  38. Stephen Toub · 3 min

    “if i invest two weeks in making these changes to the .NET Regex engine, and it makes the patterns that they're using four times faster”

  39. Well, on your LinkedIn, there's no information, but I found an interview you did. Yeah, I'm very, I'm, I basically, I've been at Microsoft for 25 years. I love it. I've jumped around a lot internally, but I've had no interest in going to different companies. And so I haven't really invested in keeping my LinkedIn profile up to date. But so when I, when I did the digging though or like uh listen to an interview that you had done about the different roles your path to distinguished engineer is really interesting because you did a bunch of different roles and it wasn't just sweet on team a and then sweet on team b you held a variety of ones starting as almost a like a forward deployed engineer consultant engineer yeah and then you did technical writing at some point and then you were an advisor to a technical advisor to a VP

  40. oh and you also jumped to a PM at some point from all those different roles that you tried which of them would you say was the most um like you would recommend someone dabble in that direction if they wanted to try something that wasn't the pure SWE role yeah every role that I I did taught music. something different. And so from that perspective, like I have great respect for people that come out of college and want to be developers and become an engineer and that's what they do and they just stay on that path. That's a very valid path. For me, I think I picked up a whole bunch of skills by having more of a circuitous route. So like you mentioned as a forward deployed engineer, it was more at the time was Microsoft's consulting arm. And so I would spend anywhere from three weeks to three months at a particular customer site helping them solve their problems and i would learn a whole lot about their industry i would learn a whole about a bunch a whole lot about their particular breed of problem i'd learn about microsoft's offerings from the outside and places that it was great in places that it wasn't so great um and learn how our customers like their their real struggles and and what they were dealing with and being able

  41. to bring that back to then later building solutions for them was incredibly valuable um when uh i then worked on like msdn magazine and was doing a bunch of technical writing and technical editing and figuring out what i wanted to share with people and how i wanted to share it and some amount of development building things to then talk about like i i gathered a set of skills around how to think through what was really interesting, what was really relevant, how to communicate it in a clear fashion, what didn't need it to be said, what did need to be said, how to tell a story that would really resonate.

  42. From a PM perspective, it became, how do we figure out what actually needs to be built and what are all the pieces that need to come together to really deliver this thing? um obviously from a dev perspective all the coding and the design and the and the architectural and all that kind of stuff the most interesting role for me i think which i am incredibly thankful and glad that i did and will never do again uh was being um sort of as technical advisor for a VP at Microsoft. Um, it was an amazing experience. Uh, it was a couple of years where I was there right hand, sort of like invisible partner, um, shadowing everything they were doing and ghost writing. And after every meeting, having a little, you know, review kind of going through what we heard and what we think we need to do about it and bouncing ideas back and forth um it was incredibly insightful into how things actually work how decisions actually get made um being a part of things that i wouldn't otherwise have been a part of at least not for many many years later um it was awesome it was also incredibly draining incredibly selfless and it took a lot out of me and so I cherish those experiences and what I learned from them

  43. and I would highly encourage anyone who has the opportunity to do so to do that kind of thing for a period of time that role is really interesting because it sounds like you're you're basically a chief of staff to someone super high up but technical so you're not like an executive assistant you're like the you're pulled into all the really I guess privileged conversations and you're providing that that information that they need to make the calls you mentioned that you learned how things are like really done do you have some examples of stuff that someone might learn if they were kind of in that that role or something that was really helpful for you there are lots of times Where if you're sort of rank and file engineer at a company, and you see these decisions that get made, an email gets sent out about some new path we're following, or some partnership we're forming, or some investment that we're making, some acquisition that is being done, winding down a certain product, investing in a new product, spinning up a new team.

  44. like all of those things look like they just sort of come out of thin air and they're you know they're just sort of handed down as this is the finished decision and it's fully formed you don't really get insights into everything that led up to that point and all of the alternatives that were considered and dismissed and why and what the trade-offs were that were factored in and how experienced people in the industry are thinking through those alternatives and those problems and how, which, which things they're choosing to focus on, which things they consider to be irrelevant. Um, all of that is stuff that I didn't have exposure to and being able to be a fly on the wall, uh, just soaking in not to mix metaphors, but, you know, soaking in all of that. And then being able to have conversations about it to, you know, we get out of a meeting and he basically say, so what did you hear? What do you think? Like help, you know, help reason through it. And sometimes I would have valuable insights and sometimes he would say, no, that's, that's not how, how we think about these things. And every one of those was a learning experience for me. Yeah. I've heard engineers say similarly about the value

  45. of trying management even if they ultimately decide they hate it and they don't want to do it again is you yeah you get to see how the sausage is made a little bit more as a manager you see all those decisions uh in managing the org that can be helpful in operating uh just in your your career in general yeah and you know i also i i'm i didn't mention it before but i also did that like i i thought that what I really wanted to be was a manager. And, um, so I basically set things up, whereas a certain point I trade, you know, I jumped around a lot internally at Microsoft and I changed to a role where I had sort of handshake agreement that six months later, I would be able to sort of shift into a lead position and have a team of people. And I arranged all that and it worked out and I became a lead and six months into being a lead, I was like, I'm not sure if this is actually what I thought it was. And six months after that, I was like, Nope, it's definitely not. This is not for me. Um, and then it took me about a year after that further to sort of, safely work my way back to an IC role where I've been since. And again, to your point, super grateful for the experience, incredibly grateful to Microsoft for having an individual

  46. contributor career track that allows one to sort of progress their whole career and continue to be promoted and whatnot, but still not have to have direct reports. For me, what I realized was the thing that i loved about or that i thought the reason i wanted to be a manager was because i thought it was how i could help others and how i could kind of provide uh career guidance or help other people solve problems or whatever it was and that was misguided on my part and it took becoming a manager to realize that i could do all of that as an individual contributor and the things that I loved about being a manager were things that I didn't have to be a manager to do.

  47. Do you have any top book recommendation that maybe impacted you? Yeah, I mean, it's old at this point, but I've read it several times. And I absolutely love it. There's a book by Charles Petzold called Code. It's probably from 2003, maybe I don't know, something in that timeframe. But I love it. And it is one of the things that I really value you is being able to explain complicated concepts in a way that doesn't require the person to know anything about what you're talking about. I think that is a valuable skill, even if you're talking to people that do know what you're talking about, because being able to break down complicated concepts into very clear, concise pieces and communicate that clearly. And then at a 30,000 foot level is a great way of getting everyone onto the same page and sort of arguing your case and pushing things in a direction that you want them to go. And Code is a book where Charles basically does that, where he talks about how computers work from the ground up, teaching people that know nothing about computers, that aren't technically experienced, about binary, about logic gates about circuits about programming about computers about the internet and kind of building things up from basic principles from nothing to a reasonable understanding of how

  48. everything works um and i was blown away by the book and i've since reread it on multiple occasions even though i know everything that's in there i just love the way that it communicates things so if I had one recommendation, that's it. Looking back on all your experience, you've kind of been working in the industry for a while. What professional advice would you give a young engineer just entering the industry today that you believe could be helpful? That's a great question. And it's one I've been struggling with. A couple years ago, I would have had a much more clear answer but with ai i still don't know what the right answer is um a lot of the times that i spend working with ai agents i'm bringing to bear 25 years of experience for better or worse in being able to very quickly sniff out things when they're doing the right thing or they're doing the wrong thing or how I want to guide them. Or an agent says, I'm 100% confident that the right answer is A. And I'll say, yeah, but have you also considered B and C and D because these trade-offs might not be there, right? And then, oh, of course. Like, just those little things, being able to guide them in the right way requires you to have gone through it.

  49. And I'm concerned that because of how AI agents are progressing, that they're taking away the opportunities to have gone through it. And I don't yet know what the answer then is. I'm a very firm believer in really learning kind of the basics and building everything up from core principles. And so like when, you know, learning a language, I still think it's valuable to learn and to start out with a language like C because you're understanding how computers actually work, and you understand what memory management means, and you learn about all the various trade-offs between this algorithm and that algorithm, rather than working at a layer where a lot of that is obscured from you, where the language and environment are providing a ton of services to you, but you don't necessarily understand everything that's happening.

  50. And so from that perspective, I still think that there's value in focusing on those. Core concepts and really internalizing algorithms and how machines actually work and being able to reason and extrapolate from that. And I suspect that that will, in whatever domain you're in, that kind of core primitive knowledge about distributed systems or about databases, that foundational stuff will help to be successful. in a world where the vast, vast, vast majority of the code is actually being written by agents and you're helping to guide them. But I don't know that for sure. And at the rate that these agents are learning and evolving, I don't know what the world's going to look like in two or three or four years. I suspect it'll look vastly different than it does today. I think we're in for a wild ride. Awesome. Well, yeah, thank you so much for your time today, Stephen. I really appreciate it. Pleasure. This has been a fun conversation. Thank you. Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Liskov, Mike Stonebreaker, Mark Brooker, these

  51. were all people that I brought on because someone left a comment. On another note, aside from the podcast i'm working on building the ergonomic keyboard that i wish existed here's a glance at the prototype it's a split keyboard so there's two sides this is in the case but yeah we launched on kickstarter and we hit our goal within eight hours of launching i really appreciate it if you were one of the people who grabbed one of the early units we're now working on the long journey of building the tooling now and so if you still want to pick one up i've left the late pledges open on kickstarter so you can grab one there i'll put a link in the description thank you again for watching the podcast and i'll see you in the next episode

More interviews