# Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub

Markdown copy of https://realalpha.app/v/PLMOVlZC4eU/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

Stephen Toub says AI agents made a previously unaffordable runtime port feasible, but success still depends on testing, oversight and business value.

- Show: The Peterman Pod
- Hosts: [Ryan Peterman](https://realalpha.app/people/ryan-peterman/)
- Guest: [Stephen Toub](https://realalpha.app/people/stephen-toub/) is a Distinguished Engineer at Microsoft.
- Published: Nov 2, 2026
- Length: 1 h 22 m

## The best minutes

- [11:27](https://realalpha.app/v/PLMOVlZC4eU/?t=687) [Incremental ports keep fast-moving products shipping](https://realalpha.app/s/incremental-ports-keep-fast-moving-products-shipping/) (7 min)
- [22:11](https://realalpha.app/v/PLMOVlZC4eU/?t=1331) [End-to-end tests matter more than unit tests for a port](https://realalpha.app/s/end-to-end-tests-matter-more-than-unit-tests-for-a-port/) (6 min)
- [49:55](https://realalpha.app/v/PLMOVlZC4eU/?t=2995) [A Rust port cost $120,000 in model tokens](https://realalpha.app/s/a-rust-port-cost-120-000-in-model-tokens/) (6 min)

## Also worth watching

- [5:36](https://realalpha.app/v/PLMOVlZC4eU/?t=336) [Rust won after performance and embedding tests tied](https://realalpha.app/s/rust-won-after-performance-and-embedding-tests-tied/) (4 min)
- [32:02](https://realalpha.app/v/PLMOVlZC4eU/?t=1922) [Strict one-to-one ports limited agent scope](https://realalpha.app/s/strict-one-to-one-ports-limited-agent-scope/) (4 min)
- [42:15](https://realalpha.app/v/PLMOVlZC4eU/?t=2535) [Most regressions were caught before pre-release](https://realalpha.app/s/most-regressions-were-caught-before-pre-release/) (3 min)
- [1:00:01](https://realalpha.app/v/PLMOVlZC4eU/?t=3601) [The port grew to three times its estimated size](https://realalpha.app/s/the-port-grew-to-three-times-its-estimated-size/) (3 min)
- [1:03:24](https://realalpha.app/v/PLMOVlZC4eU/?t=3804) [A rewrite needs a measurable business return](https://realalpha.app/s/a-rewrite-needs-a-measurable-business-return/) (3 min)

Skip the model-selection discussion and the longer career-path section, including the book recommendation and early-career advice.

## Full analysis: Stephen Toub says Rust port priced at about $120,000

- Toub estimated 136 billion tokens and about $120,000 at public model prices for the Copilot runtime port.
- That is a public-rate estimate, not a discounted invoice, and the codebase grew roughly threefold.
- Leaders should test the reuse and performance case against their own costs and behavioral coverage.

Its 6 sections, what it means for you and what to watch are on [the page](https://realalpha.app/v/PLMOVlZC4eU/#analysis), free with an account:

- [3:59](https://realalpha.app/v/PLMOVlZC4eU/?t=239) Node kept the runtime from reaching other products
- [19:39](https://realalpha.app/v/PLMOVlZC4eU/?t=1179) Incremental releases protected a fast-moving product
- [23:31](https://realalpha.app/v/PLMOVlZC4eU/?t=1411) End-to-end tests carried the correctness burden
- [35:12](https://realalpha.app/v/PLMOVlZC4eU/?t=2112) Humans steered architecture while agents checked details
- [53:47](https://realalpha.app/v/PLMOVlZC4eU/?t=3227) The token estimate needs a business denominator
- [1:03:07](https://realalpha.app/v/PLMOVlZC4eU/?t=3787) Scope growth tripled the port's initial estimate

## Summary

Stephen Toub is a Distinguished Engineer at Microsoft and says he has worked there for 25 years. He led a port of the GitHub Copilot runtime from TypeScript and Node to Rust; the episode description calls it a 400,000-line codebase.

Toub says the Copilot runtime's TypeScript and Node implementation constrained where teams could embed it, and moving to Rust would make a shared runtime easier to use across products. He describes an incremental port shipped in roughly 130 releases, with end-to-end tests, AI code reviews and human architectural oversight; he says regressions were inevitable at that scale. He estimates the work used about 136 billion tokens, cost approximately $120,000 at public model prices and took 14–15 weeks. He also reflects on varied roles at Microsoft and says foundational engineering knowledge can help people guide AI agents, while acknowledging uncertainty about how the work will change.

### Minute by minute

- [0:34](https://realalpha.app/v/PLMOVlZC4eU/?t=34) **A shared runtime removes embedding barriers.** Toub says AI projects had produced many separate agent runtimes that each needed to keep pace with rapidly changing models and techniques. The Copilot runtime's TypeScript and Node foundation made it harder to embed in other environments, so he wanted a more portable shared runtime—not a Rust rewrite for its own sake.
- [5:36](https://realalpha.app/v/PLMOVlZC4eU/?t=336) **Rust fit GitHub's needs.** Toub says he built small prototypes in several languages and compared performance, working set, throughput and embeddability. C# with native AOT and Rust performed similarly, but Rust better fit GitHub's existing projects and available expertise; he says an existing C# codebase would not have needed a rewrite.
- [11:27](https://realalpha.app/v/PLMOVlZC4eU/?t=687) **Incremental ports limit deployment risk.** Toub contrasts Copilot's incremental approach with Bun's larger all-at-once port. Because AI runtimes were changing quickly, his team would not pause development for months; it moved code from TypeScript to Rust in the shipping codebase, making about 130 releases. Temporary TypeScript-Rust interop peaked at several thousand signatures and fell to zero by the end.
- [21:52](https://realalpha.app/v/PLMOVlZC4eU/?t=1312) **End-to-end tests expose behavior gaps.** Toub says integration and end-to-end tests were the main safeguard, and most regressions came from behavior those tests did not cover. Unit tests often relied on TypeScript internals, so they could not always validate a port directly. He says agents sometimes changed tests along with the code, and Rust's compiler did not prevent all lifetime or concurrency bugs.
- [29:56](https://realalpha.app/v/PLMOVlZC4eU/?t=1796) **Agents handled line-level review.** Toub says he did not read every changed line, because he found agents more effective at line-by-line review. He examined each pull request for architectural direction, component boundaries and suspicious changes, and relied on agents for detailed review.
- [32:02](https://realalpha.app/v/PLMOVlZC4eU/?t=1922) **Human oversight steered the agents.** Toub says much of his guidance involved weighing trade-offs, keeping ports close to the original behavior and stopping agents when they went too far or declared success too early. He also reviewed what agents did to bring pull requests to green, such as responding to review feedback, before merging. He says some regressions came from agents making changes beyond the strict porting scope.
- [42:15](https://realalpha.app/v/PLMOVlZC4eU/?t=2535) **Release stages caught regressions.** Toub estimates that about two-thirds of the regressions were caught after merging to main but before a pre-release. The CLI's pre-release reached about 5–10% of customers, while most customers used stable releases. Some corner cases still reached stable and were reported later or surfaced through telemetry.
- [45:15](https://realalpha.app/v/PLMOVlZC4eU/?t=2715) **Different models offered fresh reviews.** Toub says he tried newer models as they became available, switching between Claude and GPT when a model got stuck or another worked better for the task. He also used several models from different families to review the port, saying that different models often focused on different issues.
- [49:55](https://realalpha.app/v/PLMOVlZC4eU/?t=2995) **A shared runtime carried business value.** Toub says the port let teams across GitHub and Microsoft reuse a shared, evolving runtime instead of building their own. He pointed to lower working set, faster startup, throughput and latency as potential business benefits. From his saved session logs, he estimated about 136 billion tokens and approximately $120,000 at public prices, excluding discounts.
- [1:00:01](https://realalpha.app/v/PLMOVlZC4eU/?t=3601) **Code growth increased the estimate.** Toub says he initially estimated six to eight weeks based on roughly 150,000 production lines, but the port ultimately covered about 400,000–450,000 lines and took around 14–15 weeks. He had underestimated incoming code growth and the work needed to separate CLI functionality from the runtime.
- [1:03:24](https://realalpha.app/v/PLMOVlZC4eU/?t=3804) **Rewrite only for a clear return.** Toub says teams should identify the business value, cost and expected return before starting a migration. He cites .NET migrations where teams could compare migration costs with expected service savings, and says personal projects can be judged by the enjoyment they provide.
- [1:06:36](https://realalpha.app/v/PLMOVlZC4eU/?t=3996) **Varied roles built different skills.** Toub says his 25 years at Microsoft included consulting, technical writing, product management, advising a VP, engineering and a period as a manager before returning to an individual-contributor role. He says those experiences taught him about customers, communication, product decisions and how organizations make choices.
- [1:15:59](https://realalpha.app/v/PLMOVlZC4eU/?t=4559) **Clear explanations make hard ideas accessible.** Toub recommends Charles Petzold's Code, which explains computers from basic principles. He says he values the book's clear treatment of complex ideas and sees that kind of communication as useful for getting people on the same page.
- [1:17:40](https://realalpha.app/v/PLMOVlZC4eU/?t=4660) **Foundations can help guide agents.** Toub says he is unsure what advice will hold as AI agents develop, but argues that learning fundamentals such as memory management, algorithms and how machines work can help engineers assess and guide agent-written code. He says foundational knowledge may remain useful, while acknowledging that the industry could look very different in a few years.

## Transcript

The Peterman Pod, hosted by Ryan Peterman, with Stephen Toub.

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[0:00](https://realalpha.app/v/PLMOVlZC4eU/?t=0) All told, it was about 136 billion tokens. It came out to approximately $120,000. This is Stephen Tobe, distinguished engineer at Microsoft who single-handedly rewrote a 400,000 line code base in Rust. And I asked him all about what he learned from the process. These projects that were once unaffordable are now quite doable. How did you discover and fix regressions along the way? Me reading every line would not prevent bugs. In fact, agents reading every line is a lot more effective. Something I would do differently if I had to do it again was... Here's the full episode. I wanted to focus this conversation around the port that you did for the GitHub Copilot runtime and basically rewriting it into Rust because I think this is a new type of project that is now feasible with the recent models.

[0:56](https://realalpha.app/v/PLMOVlZC4eU/?t=56) So I imagine a lot of people in industry are going to attempt something similar, maybe with their legacy code bases into some more performance language. So I want to start off with asking you, what was the project and why did you rewrite everything in Rust? What was the problem that you were solving? Yeah, that's a great question, especially because people sometimes, I think, assume that rewriting things into Rust is kind of all the rage these days. And so people assume, I've had questions that assume like, oh, you're doing this just because, or like you can and so you will. And that is not at all what this was. There were, you know, there are certain shapes that the project could have existed in where rewriting it wouldn't have been a conversation at all.

[1:44](https://realalpha.app/v/PLMOVlZC4eU/?t=104) But the impetus is largely that AI has been this incredibly fast moving target, as we all know. And over the last few years, the capabilities that it has enabled has skyrocketed. And so across GitHub, across Microsoft, there are all of these projects that have kind of blossomed from the ground up to incorporate AI capabilities into them. and all of them basically started from oh we can make these llm calls oh we have to wrap a little bit of you know goo around it we have that more around it and more around it and building up effectively what is now called uh agentic harnesses or agentic loops or agentic runtimes whatever you want to call them um and we ended up with a lot of them because everyone kind of had their own homegrown implementation. And that's not a good place to be, especially when you're trying to have all of them kind of be best of breed and be on the forefront. And in this world of AI, like there are new models coming out weekly, if not daily, like, you know, a couple weeks ago, there were all these releases from both OpenAI and Anthropic Cup, you know, one-upping each other. And there are new techniques that a month ago were researched and today every harness has to have or they're immediately out of date and you don't want this myriad of different

[3:11](https://realalpha.app/v/PLMOVlZC4eU/?t=191) implementations all needing to stay up to date and so you really want a small number ideally one but a small number of extensible harnesses or runtimes that everyone can use and customize and extend to their needs but the the copilot runtime that was being used for the copilot cli in the GitHub app and Copilot cloud agent, like when you're on GitHub and you ask Copilot to do stuff or code review, that was written in TypeScript using Node, using V8. And that has implications for where you can use that as an embeddable library. So as a Thank you. One of the main blockers for that runtime being able to be used everywhere I slash we wanted to be usable was fundamentally what it was implemented in, the frameworks it was using, the runtimes it was using, and so on.

[4:16](https://realalpha.app/v/PLMOVlZC4eU/?t=256) And that basically not moving to Rust, but rather moving away from TypeScript and Node to something that was more embeddable, had better efficiency, lower working set, faster throughput, better able to be consumed into any myriad of environments, regardless of what language they're implemented in, regardless of what frameworks they're using. um that was basically the the catalyst for for this and to your point earlier there are any number of reasons folks might be looking to do these kinds of ports or migrations they are incredibly common across the industry whether moving from one deprecated framework to a newer version or from one language to another or from a legacy code base that has become an unwieldy you know mess of spaghetti to something cleaner and modern and fresh these migrations and ports are everywhere and a lot of times people just don't take them on because they are too expensive they take too long and we would have been in that situation a year or two ago as well but with modern models and modern harnesses these projects that were once unaffordable are now quite doable in a very small period of time.

[5:38](https://realalpha.app/v/PLMOVlZC4eU/?t=338) In the post that you wrote, I saw that there was a line that said, I didn't set out to move to Rust. I set out to move away from Node.js and V8. And that makes me wonder, if you didn't need to go to Rust, did you consider other alternatives? And why did you end up going to Rust specifically? So the very first thing I did was basically create a prototype agentically, because you can now create prototypes super fast in, um, still with TypeScript, but with different runtimes. So, you know, node versus bond versus whatever, uh, as well as I think I did Python, go C sharp, Java, and rust, um, and created a very small end to end sort of steel thread through uh one of these systems kind of backed by a local um chat completion you know dummy server um probably five to ten thousand lines for each of them uh and looked at them in terms of end-to-end performance uh in terms of uh working set throughput uh what kind of scale that we could get if we threw a lot of load at it and then also what kind of whether it could actually meet requirements and what it would take for someone for us with we have currently have six language sdks that are thin bindings on cross on top of the copilot runtime what it would look

[7:08](https://realalpha.app/v/PLMOVlZC4eU/?t=428) like for this to be usable from all of those and in particular right now because we don't want to force or previously because we didn't want to force node uh into the consuming applications process the the sdk basically spawned another process to host the the actual execution of the sessions but we'd really like as an option for it to be in proc as well like pretty much every other library in the world um and so basically evaluating all of these languages stages and runtimes against these performance goals and constraints, and just looking at the data to see what worked and then how well it worked.

[7:51](https://realalpha.app/v/PLMOVlZC4eU/?t=471) And for the most part, TypeScript and Python were very quickly out from, uh, kind of all of the performance perspective and the embeddability goals. Go had pretty good perf and, you know, had a pretty clean code base, but it's not really designed for embeddability. It's not real. Like Interop is not its raison d'etre, right? It's, you can do it, but you're kind of cutting against the grain. And then, and Java is also not, not great for that. Java's perf was okay. The two main contenders ended up being C-sharp, in particular C-sharp with native AOT, so that it could be compiled ahead of time down to assembly and, and Rust.

[8:50](https://realalpha.app/v/PLMOVlZC4eU/?t=530) **Stephen Toub:** And from a perf perspective, they were both very much on par with each other across pretty much all the metrics. In some places, Rust was a little better. In some places, C-sharp was a

[9:03](https://realalpha.app/v/PLMOVlZC4eU/?t=543) little better. And so to your question earlier, if the project was already in C-sharp, we wouldn't be having this conversation. We would have stayed with it. There would have been no reason to rewrite it. But it wasn't. And so that it came down to these criteria. So from a perf perspective, they both look good. From an embeddability perspective, they both look good. C-sharp has very strong interop characteristics as well. And then it came down to primarily softer factors. So there are other projects around GitHub in particular that are built in Rust and very few that are implemented in C Sharp. There are more people around GitHub that know Rust and fewer that know C Sharp. That's not the case within Microsoft, obviously, but within GitHub. And so factoring in those kinds of criteria, we ended up going with Rust. I know Anders recently did, I think the project started a bit before your project started, but he did a port with the team of the TypeScript compiler from type being written in TypeScript to being written in Go. I was curious if you compared notes with him. Definitely. Yeah. And this is a great example where the different projects can have can reach different conclusions and and be right there's no requirement to embed

[10:24](https://realalpha.app/v/PLMOVlZC4eU/?t=624) the typescript compiler into other applications um and conversely the shape of the um the shape of the work that is compiling typescript looks very different from the shape of the work that is um sending requests to an llm and manipulating prompts and dealing with sandboxes and all that kind of stuff um the nature of the projects was fundamentally different although interestingly in the last few weeks there have been several uh agentic ports of uh which you can you know if you search x or twitter or whatever it's called now um uh you know you can find people that are porting the typescript go compiler into rust um and doing so very fast with modern models uh given the size of the code base and the cleanliness of the code base and uh so it's interesting interesting to see so once you decided that there was motivation for the project and there's clear problem to solve i wanted to ask you a little bit about the migration strategy because i've seen different strategies in different projects so for instance you mentioned bun earlier bun it looked like they kind of built up a giant basically a new branch that's just completely entirely in rust and then they shipped all in all at once uh stable rust

[12:00](https://realalpha.app/v/PLMOVlZC4eU/?t=720) release and they did it all in one one go whereas when i was reading what you had done it looked like a lot more incremental and little by little uh how did you think through the the different strategies and uh when might you select one versus the other the interesting thing about bun first i loved uh jared who wrote the the bun post i loved the post that he wrote it was really exciting because that post i wasn't aware that they were doing that port and then that post came out about halfway through our migration and it was really cool to see that this other project of approximately you know order of magnitude same amount of code it was in different language they were in zig rather than typescript but um that they had you know thought that this was a sensible thing to do and had landed it and in the post they talk about it taking i think 11 days for them to get to like 98 of their tests passing or something like that which is really cool and to your point like they They did it sort of in an en masse, like rewrite the world and go.

[13:01](https://realalpha.app/v/PLMOVlZC4eU/?t=781) It is interesting. I am not involved in the project, so I don't know all the details of what's going on behind the scenes. But it is interesting to me that if you look at their NPM page, they were shipping near daily builds. I think a lot of canary builds basically through mid-April, like for a long time, daily builds. And then mid-April, it stops. And then they talk about doing the 11 days of porting, which is an impressive feat to get from nothing to this, you know, a 500,000 line code base ported to Rust in 11 days and have a passing 98 or 99% of the test. But then they didn't have another NPM release of any kind, at least as far as I could see, for another two and a half months.

[13:49](https://realalpha.app/v/PLMOVlZC4eU/?t=829) um so there was this window where yes the the core port took a small period of time but it seems then like there was some aspect of stabilization and things like that happening such that there was this three month gap between releases um which interestingly is also approximately the same amount of time that it took copilot the other 14 and a half weeks that i spent uh porting copilot So, you know, there's lots of reasons why they differ, but fundamentally, it seemed to be about the same amount of time. And we just chose sort of different ways of going about it in the post. Jared makes a point about not wanting to do the incremental approach that co-pilot did because it would incur lots of temporary interrupt junk. And he's right. It did.

[14:41](https://realalpha.app/v/PLMOVlZC4eU/?t=881) um we just chose that approach knowing full well what we were getting ourselves into because of the sort of the constraints that we were under so uh you know for a javascript runtime um obviously fixing bugs and new features like that's all valuable um but you can pause for three months and the world isn't going to fall over right the javascript three months ago isn't all that different from JavaScript today. And you can do sort of a rewrite the world approach, pause everything, and then continue shipping when you're done. Agenda harnesses in the summer of 2026, if you pause for three months, the world you ship into will be very different from the worlds that you started in. And so trying to pause for, for a long period for you know a period of time even if it's a few days was not something that we were willing to entertain um and you know even if we've been able to do it in two weeks that would have been two weeks worth of either blocking the whole team or uh dealing with a constant influx of of changes

[16:00](https://realalpha.app/v/PLMOVlZC4eU/?t=960) **Stephen Toub:** Across a very large code base, and having to rationalize and, you know, all of those changes in and merge all those changes in as ports. Instead, we opted for the approach, which I think made sense for Copilot and made sense for Bon, where we just incrementally moved the code base more and more from TypeScript to Rust in pieces in main, in the shipping code base, shipping releases daily.

[16:30](https://realalpha.app/v/PLMOVlZC4eU/?t=990) And it's just that at the beginning, it was 100% TypeScript and 0% Rust. And at the end, it was 100% Rust and 0% TypeScript. And it was just, you know, more and more over time. And for the most part, it worked out quite well. It did mean that there was, to the point of the post, of the bun post, there was a lot of temporary code that came in and left. I think we were using NAPI RS for interop between TypeScript and Rust. And at the peak point, we had several thousand interop signatures for, I think, two or three thousand for calling between them. And then by the end, it had dropped to zero because there was no more interop required.

[17:14](https://realalpha.app/v/PLMOVlZC4eU/?t=1034) So for this project, it made a lot of sense. There are other approaches. You could do a rewrite the world. You could do, you know, have everyone stop what they're doing and just rewrite in main. You could have a feature branch and allow main to keep going and then constantly be taking ports in. And for the incremental approach, you can do what we did, but rather than do the approach of what I chose, which was sort of atomically swap TypeScript with Rust, you could have sort of an A-B thing going where you have both the TypeScript and the Rust component And then you ship with both and you have a feature flag that allows you to ship between them and do sort of flights and experiments of those components side by side.

[17:57](https://realalpha.app/v/PLMOVlZC4eU/?t=1077) There are various reasons you might choose which. And I think what we chose ended up working out well for us, but all the other options would be valid or probably better for other projects. so it sounds like that that a b approach you would also still have some uh temporary testing harness that would be comparing the two legs anyways well in the case where you have a a separate branch and then new commits that come in on the old project or i guess main need to be continuously ported. I imagine that would be relatively doable with modern models as well, because each of those single commit ports is just probably trivial for a model.

[18:47](https://realalpha.app/v/PLMOVlZC4eU/?t=1127) Did you consider doing that one? Or how did the math compare between doing that versus incurring the interop costs so there's a couple questions there it it's not just about the having a separate feature branch and bringing everything over if we had a separate feature branch that we were maintaining and at the end we did this swap where we deploy like the whole new stack we would only be testing in the wild everything all at once and that's a pretty risky gambit um whereas if you uh one of the big benefits of the approach that we chose was we were effectively able to use production for the last mile of testing and have each deployment be a small diff on top of what the previous one was so over the course of the port i think we shipped about 130 releases something like that each one of which contains just incrementally more and so So if there were issues, it was very pretty straightforward and fast to root cause those issues even without using computer at all.

[20:00](https://realalpha.app/v/PLMOVlZC4eU/?t=1200) like someone reports an issue like oh yeah i know exactly what that is right and then you can very quickly narrow it down whereas the the whole rewrite the world it's harder to do that kind of validation um and there were absolutely regressions that came about from this a good chunk of those were because of issues with merging in changes from me because you know i still had feature branches they were just much more time boxed instead of being weeks they were a day for you know porting a component but within that day there might still be 10 or 20 prs that came in and touched that component and a non-trivial percentage of the regressions were places where the the agents just missed something as part of doing the the merge um and and the port and the merge at the same time.

[20:58](https://realalpha.app/v/PLMOVlZC4eU/?t=1258) One of the nice things about sort of the atomic replacing from TypeScript to Rust versus having an A-B thing going is the agents were instructed to delete the TypeScript that they were replacing the moment that they ported it. And that, in a good way, created conflicts for those merges from main because it meant that if any code in main touched any of the TypeScript that had been ported, it instantly created a conflict, which sort of raised red flags and pointed the agent's attention at that code. Whereas if we weren't deleting it, it would have been very easy for more changes in main to that TypeScript to find their way into the existing untouched TypeScript in the branch. And then the agent would have to reason about all of the code that it had ported, looking for changes to that to bring that in versus it was sort of instantly pointed to those places because we were doing this swap at the time.

[21:58](https://realalpha.app/v/PLMOVlZC4eU/?t=1318) Yeah, I can imagine a lot of people, a lot of teams or engineers that want to rewrite their entire code base. Probably the biggest concern is the correctness concerns, which is there's so much code that you don't have the luxury of a human even reading the code to confirm that it's correct. So, um, how did you discover and fix regressions along the way? So the, the most important thing was our, um, end to end tests, our integration tests or whatever you want to call them. Um, the, we basically had two flavors of tests for copilot. We had unit tests, which were largely sort of you know they're they're testing micro functionality throughout the code base and a lot of them have internals access so they're able to test this little function here that's not exposed or that piece of code there that's not exposed and then there are the end-to-end tests either at the the runtime level the sdk level or the cli is in the same code base and And so its end-to-end tests also effectively exercise the runtime.

[23:12](https://realalpha.app/v/PLMOVlZC4eU/?t=1392) And this isn't 100% true, but the CLI can largely be thought of as sort of just exposing the runtime's capabilities as sort of the lowest layer possible in a human interface. And so we had all of those end-to-end tests to validate behaviors. And by and large, if not 100%, but the vast majority of regressions that we had were because we didn't have end-to-end tests covering that functionality. We generally had unit tests covering it, but the problem with unit tests in this situation is they were also implemented in TypeScript, testing TypeScript with internals access. And so those, in many situations, those unit tests had to be.

[24:00](https://realalpha.app/v/PLMOVlZC4eU/?t=1440) rewritten at the same time that the code was being ported now because of how we did it we got the benefit of um if i replace this little piece down here and then the typescript so this is now rust and the typescript sitting above it was interacting with this through interop this guy had its own unit tests and so indirectly those unit tests are still validating this ported rust component so we got benefits from that but the safeguard was the the integration test for the end-to-end tests across everything. And I had written a document for the team sort of motivating this port and laying out exactly how we would want to go through it and approximate timeframes and approach and strategies and technology we'd use and so on. One of the things that called out was we needed to beef up our end-to-end tests as the very first thing we did. And I did. I think one of my failings and something I would do differently if I had to do it again was I would have done it more.

[24:58](https://realalpha.app/v/PLMOVlZC4eU/?t=1498) because again, the vast majority, it kind of goes without saying, but if you have tests for something and those tests don't regress, then like the behavior is good. As long as you have tests covering all the behaviors you care about. And so the fact that we had any regressions sort of highlights that there were gaps. Now that's not a hundred percent true. We had cases, the funniest or saddest cases were where we um neglected to prevent the agents from rewriting the end-to-end

[25:30](https://realalpha.app/v/PLMOVlZC4eU/?t=1530) **Stephen Toub:** tests at the same time that they were changing the behavior that was being tested you know the agents want to get to green like that's their goal and so if you let them they will change the oracle like they will change the tests at the same time as the code to make it pass or we had

[25:46](https://realalpha.app/v/PLMOVlZC4eU/?t=1546) one case where uh this one didn't slip through i noticed this one but like the the agent was making changes and it ended up changing some public apis and uh ci caught this and said no no no you can't do that but we have an escape hatch in ci where if someone really knows what they're doing they can put a label on the the the pr to basically say no this particular break is okay and the agent found that and applied the label to say like oh this this break is okay even though it you know it wasn't um and then there are other things that are just really hard for you you could have end-to-end tests but they're sort of in the weeds uh things like you know you're you're sending data out on the wire and making sure that every byte that you send in all possible situations is what's expected at the end of the end of the wire that requires an extensive suite and if you have any gaps there's the potential for things to slip through and then there's also you know there's a typescript and rust have very different threading models um and concurrency models and uh as much as people like to say that you know if your rust compiles it's correct and it prevents all these kinds of bugs that's true but it doesn't prevent like uh you know all kinds of bugs and there are

[27:10](https://realalpha.app/v/PLMOVlZC4eU/?t=1630) plenty of concurrency bugs that slip through um and so a bunch of the issues were lifetime issues concurrency issues things that the rust compiler does not does not catch um and when you're coming from a a largely serialized single threaded execution model and going to one where on purpose we wanted much better scalability we wanted things to run concurrently um there are plenty of issues that that creep in there um so anyhow to your question the the number one thing is to make sure that um all of the behaviors that you care about are encoded in tests and then let the agents port the code while maintaining you know keeping everything passing um and making sure that there are guardrails in place to make sure that they don't change the thing that you're using for the validation at the same time that you need that validation to be correct it still feels like a regression is inevitable because the test coverage is uh it's like um it's opt-in kind of where it relies on a human saying these are the things that are important but you could always forget something um so i i kind of wonder if this type of project it's just inevitable that there will be some breakage i think so i mean it's it's hard to

[28:39](https://realalpha.app/v/PLMOVlZC4eU/?t=1719) imagine i mean who knows what things will look like in a year things are moving really fast but right now yes 100 there will be regressions if there aren't i would be astounded um uh unless we're talking about like you know a tiny tiny little code base but you're talking about rewriting hundreds of thousands or millions of lines of code it's just law of numbers if if the agents get it right 99.99% of the time which is probably a nine or two higher than most humans that still leaves 0.01% of the time where they're going to get it wrong in some way shape or form and if you're dealing with million lines that 0.1% is a very real small but real number you can throw more agents at it and we did you you know we just you know before merge doing lots of agentic reviews and then after merge running agents to to continually look for bugs look for concurrency issues look for whatever and they find things but they never find it's never at least in my experience it's not 100 perfect for such a huge volume of of code i imagine most of it wasn't human reviewed how did you handle code review at scale or did you do any at all yeah so definitely did some um but i did not read every line that would be impossible and if i did it would also

[30:16](https://realalpha.app/v/PLMOVlZC4eU/?t=1816) be a waste um there's no like me reading every line would not prevent bugs in fact agents reading every line is a lot more effective uh not just faster but more effective in quality than me doing So, um, I, I absolutely looked at the code for every PR that I put up, but I was focusing more on big picture stuff, right? It, uh, is the direction of this correct? Is the architecture of this correct? Or is the component is, you know, componentization of this correct? Uh, is the approach or the algorithms that are being used appropriate? uh is there anything fishy about like you know if i a lot of it is like squinting at a 30 000 foot level and looking at the two code bases and does anything look off right and then relying on the agents to focus on the line by line details and in many situations i would sort of do a quick scroll through all the changes to just see if anything popped out and i might notice a few here and there but for the most part a really good agentic reviewer is going to be way better at sort of the line level details than certainly me and i would argue most humans um but at the macro level there are lots of times when i basically stopped the agents and said whoa there

[31:49](https://realalpha.app/v/PLMOVlZC4eU/?t=1909) like this is this is not the direction to be going um or the inverse where the agents would stop and be like we're done and i'd be like no you're not uh the this this This is not what we're going for. And so it was a bigger picture than sort of the micro level. I saw in the blog post that you have this cool bar chart where you basically categorized every single message from you to the agents and what your intent was. Looking over that and reflecting on how you guided the agents, where was most of your time spent and what were the most impactful times where you would step in and change something there were two or three even higher level categories of things that i was doing um there was one or two categories where i think this just speaks to the the state of sort of long horizon tasks with models and harnesses today where it was me being frustrated with their progress in one way, shape, or form, and either telling them to keep going in some fashion when they had stopped or telling them to stop when they were going off down bad paths. But the majority were more, were in a category that was more of, there were trade-offs to be made.

[33:18](https://realalpha.app/v/PLMOVlZC4eU/?t=1998) And I wanted to make sure, and reasonable people or reasonable agents could choose multiple paths. and I wanted to make sure that we were going down the right one and a lot of that would be times where I would just have a conversation with the the main agent to basically evaluate where we were the decisions that were needed to be made and look and kind of talking through rubber ducking through sort of the pros and cons of those choices given the current situation we were in And some of that, to your earlier questions about sort of the incremental versus sort of pause the world or Big Bang rewrite approaches, some of those were less about the end-to-end and more about what I wanted to do for this one slice that we were putting up.

[34:16](https://realalpha.app/v/PLMOVlZC4eU/?t=2056) um so you know the the agents would pause at a particular place saying you know i'm done with with this component and i'd look at it and say this is not a good place to leave things in terms of the the the tendrils that this has into other systems you know temporarily or either maybe we need to do just a little bit more to kind of get beyond this architectural boundary and then that I'll clean up a bunch of stuff so that when we merge this, the code base is in a reasonable place. A lot of times it would sort of declare success for the slice. And if you squint, yeah, OK, but if it were do just a little bit more or in some case revert and do a little bit less, it would have left the code base in a better place or situations where it actually tried to do more than I wanted it to do.

[35:12](https://realalpha.app/v/PLMOVlZC4eU/?t=2112) **Stephen Toub:** One of the rules that I put in place with instructions that wasn't always abided by was I wanted these ports to be as strict ports as possible, not removing functionality, not adding functionality, not making any huge changes in algorithmic approaches, not trying to fix arbitrary bugs that it comes across along the way.

[35:35](https://realalpha.app/v/PLMOVlZC4eU/?t=2135) I would have it note those for me so I could revisit those separately, but I wanted these to be very strictly as one-to-one functionality ports as possible. And there were times when I would see it violating those rules, and I would basically have it back off from that and go back to what it previously had, or at least talk through what it did to make sure that I was comfortable with that. particular change being included. A non-trivial percentage of the regressions that we have had were from places where it violated those rules and I didn't notice and it included fixes for things that shouldn't have actually been done.

[36:19](https://realalpha.app/v/PLMOVlZC4eU/?t=2179) In fact, there was just one. For the most part, we're not seeing any more regressions being reported from the porting phase. We see regressions all the time. That's the nature of merging hundreds of PRs a day into a repo. But, you know, it's just to be expected. And we fix them generally before they ship. And it's just where we see them in Maine, right? But we had one yesterday or the day before where there was this really corner case thing. And I looked at it. It's like, that's really weird. Like I don't, it's saying that I added this and I don't remember adding this, uh, and look back what it was.

[37:03](https://realalpha.app/v/PLMOVlZC4eU/?t=2223) So I had, I had created a PR as one of the porting efforts and it wasn't in the initial PR. Uh, and then some feedback from, uh, a peer or a peers agent came in recommending doing something. and the uh agentic response to that was to basically address this security feedback by locking something down um but it locked it down a little bit tighter than it needed to be locked down and that resulted in some things that had been working before the port you know no longer working after the board um and uh those are really interesting things to go and figure out how how they happened that's interesting so your agent will auto apply and auto listen on the pr and if it seems reasonable it may take action without your input is that right so for the most I use the GitHub Copilot app, and it's got this agent merge functionality where you can basically have it automatically do various things watching a PR. It can automatically rebase if there are conflicts or merge if there are conflicts. It can automatically drive CI to green, so if tests are failing, fix the tests. It can automatically address review feedback that comes in, and it can automatically merge once everything is green and you get for each of those four things you can

[38:39](https://realalpha.app/v/PLMOVlZC4eU/?t=2319) choose which ones you want for the most part I leave all three of those checked except for the last one so that I can I have a time where after everything is green and before I click the merge button I go back and I review to see everything that it did to drive this PR to green what tests did it fix what feedback came in and how did it address it to make sure that I know again at a high level i'm not reviewing every single line what it did um and this particular one i remember actually reading but i didn't fully comprehend how how much it was locking things down and how much it was blocking uh because if i had i would have said actually those cases are okay let those through um uh and and had it back off from that yeah yeah if you had all of those checked off i I mean, no one, no one would do this because they're on your team, but imagine I was on your team, I could probably craft a PR comment that is malicious, I guess, you know, it's like, Hey, it sounds reasonable, but I did request you to delete like everything or.

[39:49](https://realalpha.app/v/PLMOVlZC4eU/?t=2389) Yeah. I did ask you to format your C drive. Did you not format your C drive? Yeah, exactly. Just take everything down. Yeah. And, you know, certainly different projects. I would I would have different boxes of those checked like if this was an open source project uh the only one I would consider leaving checked would be the CI one you know driving there's no way uh and maybe rebasing um but there's no way that I would allow it to automatically address arbitrary review feedback coming in on a PR and certainly not automatically merge um on a close on a private closed source thing where i trust uh you know the just the people that i know to be working on the project um it's it's less risky but even then i still you know the that less checkbox you have to be very very trusting um yeah the one of the things i like about the app as well as it does you know there are built-in prompts as part of agent merge for what those different things are but the app does permit me to add my own prompting for those different capabilities and so I have additional constraints and guidance for the agent as part of doing each of those things about how it should do it like you know always rebase instead of merge for the for one of them or for review feedback here are

[41:15](https://realalpha.app/v/PLMOVlZC4eU/?t=2475) the kinds of things I want you to think critically about before you just blatantly accept it and these are the ones where it's okay, just go ahead and address it. You know, those sorts of things. OpenAI, Anthropic, and Cursor all use this product to make their lives easier. WorkOS is an API platform that adds enterprise features like SSO, SKIM, and RBAC to your app. These are things that you need to sell to other companies, but aren't the main focus of what you're building. And now they've added more support for AI agents. For instance, Auth4Agents gives each agent its own identity with short-lived scope tokens instead of a long-lived API key.

[41:48](https://realalpha.app/v/PLMOVlZC4eU/?t=2508) So if something goes wrong, you can revoke one session and everything under it dies. Another feature they have is Airlock, which is an authorization layer. Every action agent takes is checked against rules that you can write in plain English. So it's not rubber stamping everything and it's not running unsupervised either. And lastly, they have Pipes, which connects your agents to over 300 third-party tools without you ever needing to manage a token. So if you're building SaaS or AI agents and selling to enterprises, WorkOS fills all these gaps in a few lines of code. Check them out at WorkOS.com to learn more and get started.

[42:22](https://realalpha.app/v/PLMOVlZC4eU/?t=2542) And I appreciate them for sponsoring my show. We talked a bit about regressions. And you mentioned that a lot of them were caught before they went out to prod, it sounds like. And I think a lot of people might hear regressions and they think, oh, you're taking down the app or, you know, things are exploding all the time or their incidents. Could you explain maybe the further protections after end-to-end testing that made it so these

[42:49](https://realalpha.app/v/PLMOVlZC4eU/?t=2569) **Stephen Toub:** aren't bad regressions? I would say probably two-thirds of these things that I'm calling regressions were caught very soon after merging to main and before we shipped any pre-release.

[43:02](https://realalpha.app/v/PLMOVlZC4eU/?t=2582) Even though we're shipping, at this point, we're shipping, I think, three pre-releases a day. So Then there was, there were some that made it into a pre-release and, um, you know, our form of ring deployments with the CLI is basically pre-release and stable where pre-release on any given day is probably hitting five to 10% of the, um, the audience, um, of the customer base. And then, you know, 90 or 95% is only unstable. And so a, a, a reasonable number of these issues, uh, I don't know exactly what percentage 10 20 percent um found their way into a pre-release and then pretty quickly we'd get an issue opened or someone saying hey like i'm seeing this difference in behavior was this by design or not and so a lot of these were things that were different not necessarily bad but different enough that someone noticed and then asked us like was this on purpose um You And some of those cases were ones where I'd failed to prevent the agent from fixing bugs as part of the port.

[44:08](https://realalpha.app/v/PLMOVlZC4eU/?t=2648) And so some of them are actually desirable changes, but we hadn't intended to, you know, they weren't purposeful in that regard. And then there were a number that made it into stable releases. And the ones that made it into stable releases, because we hadn't noticed them in either main or pre-release they often ended up staying in and being unnoticed in stable releases for quite a while um and this is mostly the trickle that we have been seeing over the last you know well we don't really see much anymore but a month ago would have been the trickle that we were seeing you know over a few weeks uh where some super corner case thing just behaved slightly differently and then we get an issue open like hey this this isn't working the way i expected you know is this on purpose or not and oftentimes they weren't and we'd fix them as soon as we heard about them um there were also ones that we would see from various telemetry systems you know uh monitoring for exceptions crashes uh things like that and obviously those are unintentional when we fix those in the post you you shared this graphic of which models you used throughout the course of the project. So you have this nice, I guess, bar chart that's showing what

[45:27](https://realalpha.app/v/PLMOVlZC4eU/?t=2727) portion of each week's, I guess, tokens are consumed by either Claude or GPT and in rare cases, Gemini. At the beginning of the project, it was almost entirely Claude. And then by the end of the project, it was almost entirely GPT. So what made you shift your model of choice in middle of the project to gpt um for the most part i was constantly trying different ones and uh because you know over that period like both labs were releasing new models at a very fast pace and so i was experimenting and when when i would move to one and i found it to work well i would stick with it until uh something bothered me and then i would try a different one and then you know it so um it doesn't in any way reflect sort of a long-term preference it was more of at that moment what was working best for me then um and so at the beginning of it i think i was on mostly opus 4.6 or opus 4.8 and it was working great and then gpt uh what was it 5.5 5.6 one of those and that started working even better for the these particular workloads so then i jumped to that and then opus 5 came out and it was all this is cool and jumped to that one right and then um recently i've been having a lot of luck with uh 6.1 sol um and i'll use uh i'll use opus 5.5 for code review

[47:10](https://realalpha.app/v/PLMOVlZC4eU/?t=2830) uh for the work that i've done with 6.1 sol um and so i'll like i'll sort of split the balance there um sometimes if i get really frustrated with something like both over the course of the both models, both the GPT and Claude would exhibit some very frustrating traits at certain points, like a particular point, or if I was working on it for some period of time, one of the models might get stuck and kind of like, no matter what I did, it kept kind of refocusing on a particular thing and not stepping out and looking at the forest for the trees. And there are various things I would do in those situations sometimes i would start up like keep the code changes but start over with a clear session and try and use that to you know clear context window and try and use that to get it to reset itself, but often it would find itself back into a rut. And so in those situations, I would switch models, um, and then keep going with the new model until it found itself exhibiting some frustrating behavior. And then I would, you know, switch to something else.

[48:15](https://realalpha.app/v/PLMOVlZC4eU/?t=2895) Um, I did experiment using Gemini and Grok as my main drivers. Um, but I would often leave those to be um as part of like a multi-model code review um so a lot of the the reviewing that i had that i was doing i would be using either gpt or claude as for the main work um and then use three or four different models from different families to do a line by line uh you know review of the the porting that was done. And I would do that very frequently, at least as frequently as every time that I was rebasing to bring in additional functionality from your changes that had been coming into main, but often even more frequently than that. Yeah, I did see that you had a prompt that was, I guess, auto applied on rebase or something like that, where you requested it to be reviewed, I guess, adversarially or, you know, all of them at the same time.

[49:21](https://realalpha.app/v/PLMOVlZC4eU/?t=2961) What made you come to that conclusion rather than, I guess, code review with the same model, but I don't know, different focuses for each thread? Yeah, it's basically to get different, effectively different perspectives. You know, every little change that you make to a non-deterministic system ends up sending it down different paths and looking at different things. And one of the biggest changes you can make is completely changing the model. And so the best way to get different opinions and focusing on different things that I found is by having different models, especially than the one that I used initially. So after you finish the rewrite, what was the concrete impact and the cost?

[50:12](https://realalpha.app/v/PLMOVlZC4eU/?t=3012) So if you look at it from a business perspective, what was the benefit and the cost of this project? So the benefits are STARK. I mean, the benefits are the ability for all these products and services and teams across GitHub and Microsoft to be able to rely on a shared harness that is constantly sort of, you know, best of breed and getting better and better every day. and not have to spend their own resources trying to do the same thing they can focus on their business value and and reuse this component it's the same as with any library that you want to rely on and this library is a particularly large library right you want to be able to focus on your business value and reuse the constantly evolving work for someone that is spending all their time focusing on on that thing and so from a business value perspective it's it's like incalculable across github and microsoft being able to leverage this shared harness whereas before the port a lot of that would have been a harder sell and harder to justify and um part of that is because of very real uh costs uh like performance costs so you know something that was taking 10 times as much working set waving my hands means 10 times less density which means 10

[51:46](https://realalpha.app/v/PLMOVlZC4eU/?t=3106) times as many servers you know those sorts of things um and uh if you can you know make the um if you can scale the number of users If you're able to be on the same machine running concurrently, that has impacts on your bottom line. If you can scale the throughput and reduce the latency by 5 or 6x or whatever it is, that has real benefits. One of the biggest impacts from a perf perspective, a lot of these systems, especially services, a request will come in and the agent loop needs to be started to make one more turn, which could be a very short thing, and then the request is done. And so for these things, the startup cost ends up being one of the dominant factors and the startup of a node application versus the startup of a native Rust or C application is there's a big difference there.

[52:53](https://realalpha.app/v/PLMOVlZC4eU/?t=3173) We're talking hundreds and hundreds of milliseconds difference, or in some cases, seconds. On a per request basis, that latency can be extremely measurable and extremely impactful for bottom line if you're even able to adopt it, because you might not be able to, and then you're back in the previous world that I was talking about. From a business perspective, it's worth a lot to do this port. Um, in terms of what it actually cost us, um, it was about if I, so I, I stored all of my session logs or 99% of my session logs from this effort. And from those session logs, I was then able to sort of deduce the token counts for fresh tokens, cash rights, cash reads, uh, output tokens.

[53:47](https://realalpha.app/v/PLMOVlZC4eU/?t=3227) **Stephen Toub:** And all told, it was about 136 billion tokens broken up in such a way that if you were to use the public pricing of those models at those times, it came out to approximately $120,000.

[54:07](https://realalpha.app/v/PLMOVlZC4eU/?t=3247) This is not factoring any discounts or things that companies might have for what those tokens cost. um and uh this is just sort of what's on the website of you know open ai and anthropic plus my time which you know the it took it was about 14 15 weeks um but i was doing other things at the same time you know a lot of these uh agentic projects if i look at my my co-pilot app that i currently have on my screen I've got about 35 sessions right now for things that I'm working on some of them in the same area some of them are across different projects and so I'm not developers are no longer working on one thing at a time so if you look at the percentage you just as a rough estimate for my time if you look at the percentage of PRs that I was submitting over that same period about a fifth of them give or take was to do with this this project and about 80 was for other things um and so if you figure you know about 20 of about 15 weeks you end up with about three weeks of my time spread out over three months um plus the the token costs um and it's only getting cheaper the the the models are becoming like at cheaper at an astronomical rate i think like when open ai released i'm not gonna remember the exact things but i think like six one

[55:43](https://realalpha.app/v/PLMOVlZC4eU/?t=3343) sol is half the price of six sol and six was half the price of five six salt like the it's just getting cheaper and cheaper and cheaper um and the models are also getting better and better and better, which means that the amount of time that I need to put in to steer the models to do. right thing is also decreasing. Um, and so the costs of these ports, the cost of these migrations is only getting cheaper over time. Wow. So three weeks of, uh, I guess of developers time, plus $120,000, you're able to rewrite the entire code base. Um, uh, if you were to scope this out pre AI, let's say you wanted to undertake this, what would that look like?

[56:28](https://realalpha.app/v/PLMOVlZC4eU/?t=3388) first it wouldn't have happened um for and i'm actually serious like for this particular thing the it wouldn't have happened because by the time we were done we would have ported the wrong thing like the this space is moving so fast that we would have basically never finished i think in fact you can kind of see that in the charts in the blog post in that for a good portion of the porting effort, the amount of Rust is increasing and the amount of TypeScript is largely remaining the same, but increasing a little bit. And that's because as the port was happening, there was just a huge amount of new functionality coming into the repo. And at the beginning of the porting effort, because the majority of the code base was TypeScript, the majority of the new incoming code was also TypeScript. By the end of the project where the majority of the code base was rust the majority of the new incoming code was rust but if you look at like the first half of the graph where we're generally 50 or less rust the amount of typescript was doing this even as the amount of rust was you know doing that you know even if that wasn't the case this we're talking it was about 450 000 lines of typescript of just production source never mind

[57:49](https://realalpha.app/v/PLMOVlZC4eU/?t=3469) tests that were ported i think there was a few hundred thousand lines of unit tests there in typescript as well that also needed to be ported to rust um just think about the productivity of an individual developer and how much they can reasonably do in a given day if they're top notch and we're talking a whole team of developers for a year or two to do something like this right It is, it is a, if you're able to do it at all, it is, it would have been a monumental investment. And like, I, I struggled to think about even writing that proposal because I know that it would be sort of, I would look bad.

[58:33](https://realalpha.app/v/PLMOVlZC4eU/?t=3513) Like I would be dismissed immediately as of course we're not doing that. Whereas this was a proposal is like, I'm going to take, you know, a 20% of my time over three months and by the end this whole thing is going to be moved over and for the most part no one else on the team is going to be impacted um and you know that's not 100 true but like waving my hands and we'll have all these business benefits in a few months um and it you know this will be the bill that's a much easier proposal to write and even just the even if you were paying that actual you know uh non-discounted public facing uh token rates of 120 000 you compare that to the cost of a senior developer you know for a year uh with all their benefits and their bonus like even that is a good deal it sounds like at least for this type of work where it's super whatever what we want is very obvious and encoded somewhere in the existing functionality so like this type of work is at least one order of magnitude cheaper maybe even more i mean if you had a team of engineers and they're you know engineers maybe two to four hundred k or something like that per this is a drop in the bucket the 120k when you were proposing the project

[1:00:07](https://realalpha.app/v/PLMOVlZC4eU/?t=3607) did you have a sense of how much this would cost in tokens before you went and did it yes and the bill ended up being larger than i expected um mostly because the line count ended up being larger than I expected. Um, I hadn't fully factored in the growth rate of how much incoming code there would be. So this was, you know, there were, there were a few things, I think I did a lot of things, right. I did a few things wrong and, you know, 2020 hindsight and all that. One of them I already mentioned, I didn't spend enough time, uh, really fleshing out the end to end tests to make sure that they were as comprehensive as they could have been. I could have done that better. The other thing I could have done better was really factor in the growth rate of code that would need to be ported. So when I looked at the, the code base as part of writing this proposal, it was somewhere on the order of 150,000 lines of production type script, 130, something like that.

[1:01:12](https://realalpha.app/v/PLMOVlZC4eU/?t=3672) I don't remember the exact number. Um, and based on that and some back of the napkin calculations and looking at how long it took to create the prototypes that I'd alluded to earlier for, you know, in the different languages to sort of measure perf and things like that.

[1:01:26](https://realalpha.app/v/PLMOVlZC4eU/?t=3686) **Stephen Toub:** I think I'd estimated about six to eight weeks to get it done.

[1:01:35](https://realalpha.app/v/PLMOVlZC4eU/?t=3695) But when you look at what we actually need to port, I didn't factor in a few things. One, that there were, you know, tens of developers using agents very quickly augmenting the code base with lots of a different additional functionality. And, you know, this code base is one that's it's not public open source, but it's effectively internal open source. So there's a bunch of folks that come in and add the things that they need. And then the core team is helping to review all that. And so there's this very significant influx of functionality that I hadn't accounted for. I also hadn't accounted for the fact that because of how the code base came about, the CLI, the copyright CLI and the runtime were kind of very intertwined, more so than they ideally should have been. And so in order to port just the runtime, and the CLI can be ported separately if we wanted to, but to actually tease them apart to the point where you don't need the CLI, you don't need to deploy the CLI as part of your application that's using the runtime.

[1:02:46](https://realalpha.app/v/PLMOVlZC4eU/?t=3766) There was this effort of teasing them apart, but then there's also an effort of moving functionality that really doesn't need to be in the TUI and that could be shared in the runtime down. And I failed to account for how much of the code that was currently marked as CLI would actually be beneficial to move down. And so over this effort, there was also code that was coming down so by the end um we ported somewhere between four and i think 450 000 lines of typescript so about three times what i'd estimated which impacted both the end-to-end time that it took to do it and then also uh sort of token counts so after having gone through this process if there was a software engineer out there on some team considering doing some sort of rewrite what criteria should they look for to decide if it's worth undertaking it or not it really comes down to what is the business value i mean there's there's the ones you can do for fun where it's your dime and you get to decide you know what fun is worth um just like you know i i really don't like going to casinos because I really don't like what I consider to be losing.

[1:04:00](https://realalpha.app/v/PLMOVlZC4eU/?t=3840) money, gambling, but friends of mine love it. And they consider it paying for a nice worth of entertainment. They're not losing money. They're paying for fun. Right. And so if you're, if you've got your own personal development project and you want to spend tokens on porting that to something else, great. Have fun. Like that sounds awesome. But the benefit, you need to understand what the benefits are. The benefits are your enjoyment. When it comes to a business, it's what is the business value that this is occurring to and then like anything else what is what is it going to cost uh and what is the return on that investment and that needs to be very clear and there are lots of these migration projects that are people all over the world and businesses all over the world are doing even before agents where they have done that cost assessment and they have decided that the return on investment is very much worthwhile. I saw this a lot with .NET, with people moving from .NET framework to .NET core. And we invested a lot in making .NET core incredibly fast. And so there would be very real performance benefits that would come from doing that migration. And they could figure out the cost associated with, well, we're spending X

[1:05:20](https://realalpha.app/v/PLMOVlZC4eU/?t=3920) millions of dollars running our services today and we would save three million dollars a year by moving you know by doing this port and this port is going to take a year and it's going to cost us a developer for that year so they do the math you know this is this is absolutely worthwhile um even at a more micro level not even for ports like what features do you invest in when i was spending a lot of my time working for example on the dotnet um regex engine uh you know i i could look to see how much are we spending we as a company you know or how much is this

[1:06:02](https://realalpha.app/v/PLMOVlZC4eU/?t=3962) **Stephen Toub:** particular service spending on running on regular expressions every year if i invest two weeks in making these changes to the .NET Regex engine, and it makes the patterns that they're using four times faster, what is the return on investment for my two weeks and how much they save as

[1:06:20](https://realalpha.app/v/PLMOVlZC4eU/?t=3980) a service over the course of the year? These kinds of trade-offs are everywhere. And you just need to be very clear about why you're doing it, what it will cost, and what you get for it. And if you can write all that down and you can justify it, then it's a good project to do. If you can't justify it. It's very suspect. When I was doing research for this interview, I looked into your career. Well, on your LinkedIn, there's no information, but I found an interview you did. Yeah, I'm very, I'm, I basically, I've been at Microsoft for 25 years. I love it. I've jumped around a lot internally, but I've had no interest in going to different companies. And so I haven't really invested in keeping my LinkedIn profile up to date. But so when I, when I did the digging though or like uh listen to an interview that you had done about the different roles your path to distinguished engineer is really interesting because you did a bunch of different roles and it wasn't just sweet on team a and then sweet on team b you held a variety of ones starting as almost a like a forward deployed engineer consultant engineer yeah and then you did technical writing at some point and then you were an advisor to a technical advisor to a VP

[1:07:40](https://realalpha.app/v/PLMOVlZC4eU/?t=4060) oh and you also jumped to a PM at some point from all those different roles that you tried which of them would you say was the most um like you would recommend someone dabble in that direction if they wanted to try something that wasn't the pure SWE role yeah every role that I I did taught music. something different. And so from that perspective, like I have great respect for people that come out of college and want to be developers and become an engineer and that's what they do and they just stay on that path. That's a very valid path. For me, I think I picked up a whole bunch of skills by having more of a circuitous route. So like you mentioned as a forward deployed engineer, it was more at the time was Microsoft's consulting arm. And so I would spend anywhere from three weeks to three months at a particular customer site helping them solve their problems and i would learn a whole lot about their industry i would learn a whole about a bunch a whole lot about their particular breed of problem i'd learn about microsoft's offerings from the outside and places that it was great in places that it wasn't so great um and learn how our customers like their their real struggles and and what they were dealing with and being able

[1:08:56](https://realalpha.app/v/PLMOVlZC4eU/?t=4136) to bring that back to then later building solutions for them was incredibly valuable um when uh i then worked on like msdn magazine and was doing a bunch of technical writing and technical editing and figuring out what i wanted to share with people and how i wanted to share it and some amount of development building things to then talk about like i i gathered a set of skills around how to think through what was really interesting, what was really relevant, how to communicate it in a clear fashion, what didn't need it to be said, what did need to be said, how to tell a story that would really resonate.

[1:09:39](https://realalpha.app/v/PLMOVlZC4eU/?t=4179) From a PM perspective, it became, how do we figure out what actually needs to be built and what are all the pieces that need to come together to really deliver this thing? um obviously from a dev perspective all the coding and the design and the and the architectural and all that kind of stuff the most interesting role for me i think which i am incredibly thankful and glad that i did and will never do again uh was being um sort of as technical advisor for a VP at Microsoft. Um, it was an amazing experience. Uh, it was a couple of years where I was there right hand, sort of like invisible partner, um, shadowing everything they were doing and ghost writing. And after every meeting, having a little, you know, review kind of going through what we heard and what we think we need to do about it and bouncing ideas back and forth um it was incredibly insightful into how things actually work how decisions actually get made um being a part of things that i wouldn't otherwise have been a part of at least not for many many years later um it was awesome it was also incredibly draining incredibly selfless and it took a lot out of me and so I cherish those experiences and what I learned from them

[1:11:15](https://realalpha.app/v/PLMOVlZC4eU/?t=4275) and I would highly encourage anyone who has the opportunity to do so to do that kind of thing for a period of time that role is really interesting because it sounds like you're you're basically a chief of staff to someone super high up but technical so you're not like an executive assistant you're like the you're pulled into all the really I guess privileged conversations and you're providing that that information that they need to make the calls you mentioned that you learned how things are like really done do you have some examples of stuff that someone might learn if they were kind of in that that role or something that was really helpful for you there are lots of times Where if you're sort of rank and file engineer at a company, and you see these decisions that get made, an email gets sent out about some new path we're following, or some partnership we're forming, or some investment that we're making, some acquisition that is being done, winding down a certain product, investing in a new product, spinning up a new team.

[1:12:24](https://realalpha.app/v/PLMOVlZC4eU/?t=4344) like all of those things look like they just sort of come out of thin air and they're you know they're just sort of handed down as this is the finished decision and it's fully formed you don't really get insights into everything that led up to that point and all of the alternatives that were considered and dismissed and why and what the trade-offs were that were factored in and how experienced people in the industry are thinking through those alternatives and those problems and how, which, which things they're choosing to focus on, which things they consider to be irrelevant. Um, all of that is stuff that I didn't have exposure to and being able to be a fly on the wall, uh, just soaking in not to mix metaphors, but, you know, soaking in all of that. And then being able to have conversations about it to, you know, we get out of a meeting and he basically say, so what did you hear? What do you think? Like help, you know, help reason through it. And sometimes I would have valuable insights and sometimes he would say, no, that's, that's not how, how we think about these things. And every one of those was a learning experience for me. Yeah. I've heard engineers say similarly about the value

[1:13:47](https://realalpha.app/v/PLMOVlZC4eU/?t=4427) of trying management even if they ultimately decide they hate it and they don't want to do it again is you yeah you get to see how the sausage is made a little bit more as a manager you see all those decisions uh in managing the org that can be helpful in operating uh just in your your career in general yeah and you know i also i i'm i didn't mention it before but i also did that like i i thought that what I really wanted to be was a manager. And, um, so I basically set things up, whereas a certain point I trade, you know, I jumped around a lot internally at Microsoft and I changed to a role where I had sort of handshake agreement that six months later, I would be able to sort of shift into a lead position and have a team of people. And I arranged all that and it worked out and I became a lead and six months into being a lead, I was like, I'm not sure if this is actually what I thought it was. And six months after that, I was like, Nope, it's definitely not. This is not for me. Um, and then it took me about a year after that further to sort of, safely work my way back to an IC role where I've been since. And again, to your point, super grateful for the experience, incredibly grateful to Microsoft for having an individual

[1:15:14](https://realalpha.app/v/PLMOVlZC4eU/?t=4514) contributor career track that allows one to sort of progress their whole career and continue to be promoted and whatnot, but still not have to have direct reports. For me, what I realized was the thing that i loved about or that i thought the reason i wanted to be a manager was because i thought it was how i could help others and how i could kind of provide uh career guidance or help other people solve problems or whatever it was and that was misguided on my part and it took becoming a manager to realize that i could do all of that as an individual contributor and the things that I loved about being a manager were things that I didn't have to be a manager to do.

[1:16:00](https://realalpha.app/v/PLMOVlZC4eU/?t=4560) Do you have any top book recommendation that maybe impacted you? Yeah, I mean, it's old at this point, but I've read it several times. And I absolutely love it. There's a book by Charles Petzold called Code. It's probably from 2003, maybe I don't know, something in that timeframe. But I love it. And it is one of the things that I really value you is being able to explain complicated concepts in a way that doesn't require the person to know anything about what you're talking about. I think that is a valuable skill, even if you're talking to people that do know what you're talking about, because being able to break down complicated concepts into very clear, concise pieces and communicate that clearly. And then at a 30,000 foot level is a great way of getting everyone onto the same page and sort of arguing your case and pushing things in a direction that you want them to go. And Code is a book where Charles basically does that, where he talks about how computers work from the ground up, teaching people that know nothing about computers, that aren't technically experienced, about binary, about logic gates about circuits about programming about computers about the internet and kind of building things up from basic principles from nothing to a reasonable understanding of how

[1:17:35](https://realalpha.app/v/PLMOVlZC4eU/?t=4655) everything works um and i was blown away by the book and i've since reread it on multiple occasions even though i know everything that's in there i just love the way that it communicates things so if I had one recommendation, that's it. Looking back on all your experience, you've kind of been working in the industry for a while. What professional advice would you give a young engineer just entering the industry today that you believe could be helpful? That's a great question. And it's one I've been struggling with. A couple years ago, I would have had a much more clear answer but with ai i still don't know what the right answer is um a lot of the times that i spend working with ai agents i'm bringing to bear 25 years of experience for better or worse in being able to very quickly sniff out things when they're doing the right thing or they're doing the wrong thing or how I want to guide them. Or an agent says, I'm 100% confident that the right answer is A. And I'll say, yeah, but have you also considered B and C and D because these trade-offs might not be there, right? And then, oh, of course. Like, just those little things, being able to guide them in the right way requires you to have gone through it.

[1:18:59](https://realalpha.app/v/PLMOVlZC4eU/?t=4739) And I'm concerned that because of how AI agents are progressing, that they're taking away the opportunities to have gone through it. And I don't yet know what the answer then is. I'm a very firm believer in really learning kind of the basics and building everything up from core principles. And so like when, you know, learning a language, I still think it's valuable to learn and to start out with a language like C because you're understanding how computers actually work, and you understand what memory management means, and you learn about all the various trade-offs between this algorithm and that algorithm, rather than working at a layer where a lot of that is obscured from you, where the language and environment are providing a ton of services to you, but you don't necessarily understand everything that's happening.

[1:19:53](https://realalpha.app/v/PLMOVlZC4eU/?t=4793) And so from that perspective, I still think that there's value in focusing on those. Core concepts and really internalizing algorithms and how machines actually work and being able to reason and extrapolate from that. And I suspect that that will, in whatever domain you're in, that kind of core primitive knowledge about distributed systems or about databases, that foundational stuff will help to be successful. in a world where the vast, vast, vast majority of the code is actually being written by agents and you're helping to guide them. But I don't know that for sure. And at the rate that these agents are learning and evolving, I don't know what the world's going to look like in two or three or four years. I suspect it'll look vastly different than it does today. I think we're in for a wild ride. Awesome. Well, yeah, thank you so much for your time today, Stephen. I really appreciate it. Pleasure. This has been a fun conversation. Thank you. Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Liskov, Mike Stonebreaker, Mark Brooker, these

[1:21:17](https://realalpha.app/v/PLMOVlZC4eU/?t=4877) were all people that I brought on because someone left a comment. On another note, aside from the podcast i'm working on building the ergonomic keyboard that i wish existed here's a glance at the prototype it's a split keyboard so there's two sides this is in the case but yeah we launched on kickstarter and we hit our goal within eight hours of launching i really appreciate it if you were one of the people who grabbed one of the early units we're now working on the long journey of building the tooling now and so if you still want to pick one up i've left the late pledges open on kickstarter so you can grab one there i'll put a link in the description thank you again for watching the podcast and i'll see you in the next episode
