# Digg launched a rewrite without load testing or rollback capacity

Markdown copy of https://realalpha.app/s/digg-launched-a-rewrite-without-load-testing-or-rollback/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Will Larson](https://realalpha.app/people/will-larson/) · Will Larson is CTO of Imprint and author on engineering leadership. · Led engineering at Stripe, Uber, Calm and Carta; author of Staff Engineer and The Engineering Executive's Primer.
- Host: [Gergely Orosz](https://realalpha.app/people/gergely-orosz/) · Host, The Pragmatic Engineer · Spent four years at Uber heading mobile, web and backend teams; shipped part of its payments system.
- Show: The Pragmatic Engineer
- Episode: [The revised rules of engineering management with Will Larson](https://realalpha.app/v/Prl858_AovM/), Oct 21, 2026
- Moment: 19:39 to 26:02 of the episode (6 min), playing on https://realalpha.app/s/digg-launched-a-rewrite-without-load-testing-or-rollback/; the episode from there: https://realalpha.app/v/Prl858_AovM/?t=1179

## Quote

> “we made the decision to **launch on Cassandra**. And there's like a joke at the time”
> — [Will Larson](https://realalpha.app/people/will-larson/) at [20:11](https://realalpha.app/v/Prl858_AovM/?t=1211) · The Pragmatic Engineer

## Takeaways

1. The old system could not run alongside the rewrite.
2. The team began load testing just two days before launch.
3. A Python mutable-default bug caused cascading memory failures.

## About this segment

Will Larson recounts Digg’s V4 launch, from Cassandra failing under load to a memory leak that took weeks to find. The episode shows how deadlines, scarce hardware, and technical choices compounded.

From “The revised rules of engineering management with Will Larson”, Oct 21, 2026. [Watch the full episode](https://realalpha.app/v/Prl858_AovM/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[19:45](https://realalpha.app/v/Prl858_AovM/?t=1185) But the launch went pretty poorly. And why did it go poorly? It went poorly for a lot of reasons. So first, the launch technically wasn't really ready. And so we made a lot of really aggressive technology decisions. And again, I came in about three months before this went live. So I don't personally take a ton of responsibility. I also, again, think about where I was in my career. I was basically, it was my second job. I'd been at Yahoo for two years. So I won't own this one. Later in my career, I'll own anything that happened. But this one wasn't really by fault.

[20:16](https://realalpha.app/v/Prl858_AovM/?t=1216) But basically, we made the decision to launch on Cassandra. And there's like a joke at the time, which is that Cassandra was a Trojan horse Facebook had launched to basically ruin competition in the industry. Because anyone who launched on Cassandra was basically screwed afterwards, which I think certainly was true for Dig. And so running Cassandra to replace the sharded MySQL system, the sharded MySQL system was pretty complicated, but it definitely worked.

[20:41](https://realalpha.app/v/Prl858_AovM/?t=1241) And Cassandra, I would argue, basically didn't work. But I remember there was a team response for load testing Cassandra before our launch. And two days before the launch, this other guy, Rich, who was fantastic, he and I were pulled aside and we're like, hey, well, they never did the load testing. you to do the load testing before, you know, in the next two days. And so Rich was just like a software engineer like me. And so we're like, well, I guess we'll install JMeter and try to figure out how JMeter works to do some load testing. And we had this like really interesting conversation with Chris, who's kind of the head of the Cassandra initiative. He's like, hey, like, I can't do the reads on Cassandra. He's like, that's not true. I was like, well, I can't do the writes. He's like, that's also not true. I'm like, okay, Chris, if I'm writing to Cassandra,

[21:25](https://realalpha.app/v/Prl858_AovM/?t=1285) I can't read from Cassandra, but I'm reading from Cassandra. I can't write to Cassandra. He's like, that is true. It's like, well, we're supposed to launch in two days. And then we launched and we didn't fix it. We just launched. And so I think the company thought that we were literally dying. And the company was. The company literally was dying. It was running out of money. But it was like the most interesting day I've ever had professionally.

[21:47](https://realalpha.app/v/Prl858_AovM/?t=1307) Can't remember when I got to the office, probably around 8 a.m. When you say interesting, even stressful. You can judge. Let me like paint the scene a little bit. So it was a launch, but they also decided to have a launch party. So the entire day, there were caterers kind of in tuxedos with like sushi trays, champagne. Oh, wow. And so like the entire office is there to have this like kind of like celebration.

[22:10](https://realalpha.app/v/Prl858_AovM/?t=1330) And then in the center of the room, there was this like table. It might have been like a boardroom table or something that had been pulled out. This giant table, maybe three or four times the size of this where the engineers were sitting. and we didn't have enough hardware to run the new system and the old system concurrently and again this was hardware we owned so they were literally wiping the old servers and installing the new software so we couldn't roll back we didn't didn't have the money to or the and because again we were running out of money but enough budget for the party but like not enough budget for the servers right so we were wiping server by server spinning it up and ultimately like the damn thing didn't turn on it was functionally what what happened right and so we had this um

[22:51](https://realalpha.app/v/Prl858_AovM/?t=1371) you know like down for for maintenance page up for i think roughly 24 hours and then we're able to scrap it while the champagne and the tuxedos are coming around surreal and there were people upstairs getting massages there were you know i remember certain groups who won't be named playing ping pong while we're trying to get this out and it was it was just like it was sort of like a magical realism. I see what you mean by interesting. This is not how I imagined a launch like this. You can never imagine a launch like this. It was just, it was so, it was so surreal. True, like a magical realism moment where there's kind of all this normalcy happening while this like site's literally imploding. And it was just, it was unlike anything I've seen, but it's also like a moment I take a lot of pride in. And so I'd been there

[23:36](https://realalpha.app/v/Prl858_AovM/?t=1416) three months. I think interesting thing is we did get the site kind of working, but there were a lot of issues and so people just gave up and like day by day like people just stopped showing up and like a small group was really the only group left of maybe like six people trying to keep it working and we did and so we in the first two days like the the same guy rich and i wrote like a caching system from scratch on top of redis which helped the site stay on for hours yeah but we still had huge memory leaks and so we're actually like restarting every process every you know like hour or two and the only way to keep it up was rolling restarts of every every process and so it took us it took us like a full month the bug and I have like a blog post I wrote a long time ago about it but basically the core bug was we created an API service that could be used

[24:25](https://realalpha.app/v/Prl858_AovM/?t=1465) for like third parties and it was in Python the people writing it didn't know Python very well and this was Python then calling like another Python gRPC that was hitting Cassandra and And basically, in Python, when you instantiate a default parameter, if it's mutable, I don't know if this is still true, by the way, I hope it's not so true, but at the time it was true, where if you basically put a list as a default parameter, and then you mutated it, like appended IDs into it, it would persist in memory.

[24:53](https://realalpha.app/v/Prl858_AovM/?t=1493) And so each time that function was called, it would have the same diction, or the same list. And so the memory. of the user IDs, basically every time they call this API to look up user IDs to get the user objects back, all the users that were ever passed into it would still be there. And so it just cascaded over and over and over until we restarted. And we installed memcache kind of in between these and the memcache would start falling over and then it would destroy Cassandra. So really finding that bug. It took me, I think, four weeks to actually find it. Once you found it, trivial to fix, but we got it up. And then our new CEO started, I think, that basically the same day that I fixed that. And then basically the entire company got laid off over the next year. So pure victory in the sense that we did solve it. It was working, but I think you can't really have the central

[25:47](https://realalpha.app/v/Prl858_AovM/?t=1547) site be hard to use or down for like a month. But like if you go to the counterfactual, what would the company have done differently? If they had simply kept up the old site, they also would have