Thundering herds have different causes, so the fixes differ

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:16:17

   What is a thundering herd? Genetically, I talk about it in any situation where you've got mismatched capacities. You've got too many requests for too few resources, right? Resource pool's not infinite. Coming back to rule number three. Rule number three. Yeah, so in the chapter, I talk about a whole bunch of different ones from different types of scenarios, right? So you've got everything from the malicious thundering herd, like a denial of service attack, right?

2. 1:16:38

   A digital denial of service attack, malicious attack, often short notice and... there's not that much you can do about them. The answer is use faster or Cloudflare, but I go into details about how that happens. You get things like a kind of internal self-inflicted ones. So a good example would be something like a cache collapse, right? A system relies on the cache, not only for speed of operation, but also for reducing load on other components.

3. 1:17:05

   Yep. Cache gets restarted. You then have a lot of cache misses. You're hitting the origin and the origin goes down. Yeah, originally was not expecting so much load. Yeah, so a common thing there would be, first you'd flush those cache snapshots of disk, reduces the chance of you being out or having high cache misses over a restart. But also a lot of companies I know will just turn their services off until the cache refills.

4. 1:17:26

   And so that's kind of another example of one. RetryStorms, another one, back in 2017, Square had a big outage. They'd restarted Multipass, which was their multifactal authentication system. Multipass in turn pulled its data from Redis that they're using as a database because you can do that, apparently. I don't think it's a database. Well, they're using it as a cache, basically, right? Well, no, I think they were using it as a database.

5. 1:17:49

   There are some people that think Redis is a database. This is a different podcast. I'm thinking of starting a stupid technology podcast where we talk about using technology in a stupid way. So that's a separate conversation. Anyway, so they were putting this data back from Redis and they had a problem that the clients in Multipass were basically, if the Redis calls failed, they'd try again. but without any delay between the retries and they had a hard-coded limit of 500 retries.

6. 1:18:12

   So they just basically smashed Redis and they kept restarting Redis and then they'd restart multipass. And again, this is a long time ago and they fixed it, right? They fixed it by having a more sensible retry policy and externalizing those values. That's not a thundering herd. The last kind of sections or ones I talked about were ones caused by like products being too successful like Twitter and healthcare.gov, right?

7. 1:18:34

   This is where you have an external real legitimate load that takes you out and things you look at all of these as solutions although what's happened is the same the kind of solutions are a bit different so like when i've got a thundering herd that's coming from externally that's legitimate well i've got different things i'll look at like i will have more i'll have a more constrained sign up i'll have fewer invite codes yeah so with healthcare gov one of the issues there was many others but was that they went live at like 36 states at once and they were massively overloaded so in In hindsight, maybe going live in one state would have been a good idea first.

8. 1:19:09

   For example, with Twitter, they were going down all the time. So they could have constrained signups or put tweet limits on you and things like that. Those are things. And they did those things as well, right? And so although they look the same, the mitigations for Eaton was a little bit different. So I kind of quite like talking about that because they were all great stories I found. And of course, I think we're talking about solving it with when you have same resources, but then we get into things like scaling like can you scale horizontally add more resources because there's two sides of this right like whenever you have it and so what point can you do it or are you going to face a bottleneck because you still have a central whatever but like that's a great example of where sometimes mitigations work and sometimes they don't so an example of something like your

9. 1:19:54

   product is successful if you're a startup or whatever and you suddenly got successful that success comes like a wave right the wave of success hits you and then a lot of it recedes you'll get a thousand customers in and maybe you'll keep a hundred maybe right that's just how things work so you need to be up during the peak but then you probably can shrink down again to keep the cost low so if you've built your application to have some dynamic scalability in it and if you're on the public cloud or similar place where you can spin up and spin down that's a viable mechanism right okay we can scale up to handle this and that's fine would i use that as defense against a distributed denial of service attack? No, because then your distributed denial of service attack becomes a distributed denial of cash attack, right? You go from to not only

10. 1:20:43

    you're having no service, but also now no money. Because you hit spinning up, you know, nodes on your Kubernetes cluster and Amazon just keep taking the money. It's great. And you still can't keep up. So that's a perfectly valid mechanism for some things, but you kind of have to know the context in which to say, you can't just scare yourself out of trouble. I think we're going back to a lot of times