# End-to-end tests matter more than unit tests for a port

Markdown copy of https://realalpha.app/s/end-to-end-tests-matter-more-than-unit-tests-for-a-port/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Stephen Toub](https://realalpha.app/people/stephen-toub/) · Stephen Toub is a Distinguished Engineer at Microsoft.
- Host: [Ryan Peterman](https://realalpha.app/people/ryan-peterman/) · Host, The Peterman Pod · Went from new grad to staff engineer at Instagram in three years.
- Show: The Peterman Pod
- Episode: [Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub](https://realalpha.app/v/PLMOVlZC4eU/), Nov 2, 2026
- Moment: 22:11 to 28:19 of the episode (6 min), playing on https://realalpha.app/s/end-to-end-tests-matter-more-than-unit-tests-for-a-port/; the episode from there: https://realalpha.app/v/PLMOVlZC4eU/?t=1331

## Quote

> “the agents want to get to green like that's their goal and so if you let them they will **change the oracle**”
> — [Stephen Toub](https://realalpha.app/people/stephen-toub/) at [25:28](https://realalpha.app/v/PLMOVlZC4eU/?t=1528) · The Peterman Pod

## Takeaways

1. Most regressions came from behavior without end-to-end test coverage.
2. Agents may change tests to make their code pass.
3. Rust’s compiler doesn’t catch every concurrency or lifetime bug.

## About this segment

Stephen Toub says integration tests caught most migration regressions, while unit tests often had to be rewritten alongside the code. Agents also need guardrails so they can’t change the tests they’re meant to pass.

From “Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub”, Nov 2, 2026. [Watch the full episode](https://realalpha.app/v/PLMOVlZC4eU/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[22:17](https://realalpha.app/v/PLMOVlZC4eU/?t=1337) correct. So, um, how did you discover and fix regressions along the way? So the, the most important thing was our, um, end to end tests, our integration tests or whatever you want to call them. Um, the, we basically had two flavors of tests for copilot. We had unit tests, which were largely sort of you know they're they're testing micro functionality throughout the code base and a lot of them have internals access so they're able to test this little function here that's not exposed or that piece of code there that's not exposed and then there are the end-to-end tests either at the the runtime level the sdk level or the cli is in the same code base and And so its end-to-end tests also effectively exercise the runtime.

[23:12](https://realalpha.app/v/PLMOVlZC4eU/?t=1392) And this isn't 100% true, but the CLI can largely be thought of as sort of just exposing the runtime's capabilities as sort of the lowest layer possible in a human interface. And so we had all of those end-to-end tests to validate behaviors. And by and large, if not 100%, but the vast majority of regressions that we had were because we didn't have end-to-end tests covering that functionality. We generally had unit tests covering it, but the problem with unit tests in this situation is they were also implemented in TypeScript, testing TypeScript with internals access.

[23:56](https://realalpha.app/v/PLMOVlZC4eU/?t=1436) And so those, in many situations, those unit tests had to be. rewritten at the same time that the code was being ported now because of how we did it we got the benefit of um if i replace this little piece down here and then the typescript so this is now rust and the typescript sitting above it was interacting with this through interop this guy had its own unit tests and so indirectly those unit tests are still validating this ported rust component so we got benefits from that but the safeguard was the the integration test for the end-to-end tests across everything. And I had written a document for the team sort of motivating this port and laying out exactly how we would want to go through it and approximate timeframes and approach and strategies and technology we'd use and so on. One of the things that called out was we needed

[24:45](https://realalpha.app/v/PLMOVlZC4eU/?t=1485) to beef up our end-to-end tests as the very first thing we did. And I did. I think one of my failings and something I would do differently if I had to do it again was I would have done it more. because again, the vast majority, it kind of goes without saying, but if you have tests for something and those tests don't regress, then like the behavior is good. As long as you have tests covering all the behaviors you care about. And so the fact that we had any regressions sort of highlights that there were gaps. Now that's not a hundred percent true. We had cases, the funniest or saddest cases were where we um neglected to prevent the agents from rewriting the end-to-end tests at the same time that they were changing the behavior that was being tested you know the agents want to get to green like that's their goal and so if you let them they will change

[25:40](https://realalpha.app/v/PLMOVlZC4eU/?t=1540) the oracle like they will change the tests at the same time as the code to make it pass or we had one case where uh this one didn't slip through i noticed this one but like the the agent was making changes and it ended up changing some public apis and uh ci caught this and said no no no you can't do that but we have an escape hatch in ci where if someone really knows what they're doing they can put a label on the the the pr to basically say no this particular break is okay and the agent found that and applied the label to say like oh this this break is okay even though it you know it wasn't um and then there are other things that are just really hard for you you could have end-to-end tests but they're sort of in the weeds uh things like you know you're you're sending data out on the wire and making sure that every byte that you send in all possible situations is what's expected

[26:37](https://realalpha.app/v/PLMOVlZC4eU/?t=1597) at the end of the end of the wire that requires an extensive suite and if you have any gaps there's the potential for things to slip through and then there's also you know there's a typescript and rust have very different threading models um and concurrency models and uh as much as people like to say that you know if your rust compiles it's correct and it prevents all these kinds of bugs that's true but it doesn't prevent like uh you know all kinds of bugs and there are plenty of concurrency bugs that slip through um and so a bunch of the issues were lifetime issues concurrency issues things that the rust compiler does not does not catch um and when you're coming from a a largely serialized single threaded execution model and going to one where on purpose we wanted much better scalability we wanted things to run concurrently um there are plenty of

[27:37](https://realalpha.app/v/PLMOVlZC4eU/?t=1657) issues that that creep in there um so anyhow to your question the the number one thing is to make sure that um all of the behaviors that you care about are encoded in tests and then let the agents port the code while maintaining you know keeping everything passing um and making sure that there are guardrails in place to make sure that they don't change the thing that you're using for the validation at the same time that you need that validation to be correct it still feels like a regression is inevitable because the test coverage
