Make retries safe with idempotency keys, not just fingerprints

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:08:31

   If I press that down button again, nothing's changed. The lift is still coming. Well, we hope the lift is still coming, right? That's an idempotent operation. Some light switches are actually toggles. I toggle it, light goes. On Toglet, light goes off. That's not an idempotent operation. In computing, we have some operations which are not the idempotent reads. A delete is typically idempotent. But the example I think I use in my introduces concept is the one of payment. I was about to say that I have a lot better idempotent example for you. Yeah, payment is a good one. So I'm going to pay you money.

2. 1:09:05

   So I'm a service that wants to issue a payment, and I send you, please pay Sam, a hundred pounds. this is i'm nlp-ing people back at home right please say pay sam a hundred pounds and i don't get a confirmation back i don't get back a response the problem is from my point of view there are two possibilities the first possibility is that you receive that payment for us and you've done the payment it's been completed and the result has got lost to me for some reason i didn't get the response that's one possibility the other possibility is that you never got that You never carried out the operation.

3. 1:09:40

   Maybe it didn't get to you. Maybe you crashed halfway through. I think like a good example that I always, I've seen it in the past and maybe it's over, but airlines, you're about to buy this airline ticket for $1,000, $2,000. You've checked all of your details and it says like play or reserve. And then often some airlines would take you to an intermediate page which says like, please do not refresh this page or submit it again because you might be double charged.

4. 1:10:05

   And what you really want is you pay, you know like pay and if you you get the spinner and it's not idempotent if you do it again and you get charged twice yeah but it is idempotent if you refresh you you copy the same thing you submit the exact same form although that's interesting so then the question then becomes right so i sent you that payment but my airline ticket whatever it is and i haven't had that response back but i really want to make sure that the airline ticket booking goes through this is you with the browser or the people at home sending Sam 100 pounds you really want to make sure Sam gets his 100 pounds right now when something doesn't work in a distributed system because of rule number two rule number two meaning the thing might not be there rule number one sometimes it can take ages

5. 1:10:48

   yeah and accepting this you just try it again you retry because you really want it to happen and in retrying there's the danger that if the first operation had been processed successfully and you didn't know about it or the first operation have been applied successfully and you do it again you're not getting paid twice and it were two two tickets so that's the problem with an item potency you want to make it safe to retry important operations so to do that we need to make them item potent there's two ways to do it really one that's good and that is one you should do but you kind of have to retro retrofitting is pain and one which is one you just explained which is useful but has problems right the first one is item potency keys yep so you generate you have a unique id you'd normally use a uuid for this yep and so when you

6. 1:11:35

   the unique id which you generate and when you generate two of them they should be different there's like this tiny tiny tiny tiny tiny tiny likelihood of clash which again what happened i think i know but like i think with uuid seven i think it's like a 50 nanosecond chance but it's something you can generate locally without an additional round trips yes your collision chance is extremely low and there's ways to modify that.

7. 1:11:58

   You can work with that for the most part. And this is unless you're Amazon probably. Well this but this is I mean the AWS APIs you see Stripe Adium PayPal they all use the same key scheme. So you send this to your UID so then if you send that to me say pay Sam and here's the idempotency key I don't send you back a response for whatever reason. You retry it you pass the idempotency key if I had carried out the processing I could look at it and say oh have I processed this?

8. 1:12:25

   Yes I have. Okay, great. If I have processed it, even if I get a second or a third or a fourth, have you processed it? It's processed it. I just keep going back to it. Yeah, it's processed. It's processed. It's fine. That's a nice, simple thing to do. The issue with making things with the idempotency key is it requires both the client and server to support it. So it has to be a required fee. That goes through, and it has to be sent to server support. So if you're trying to retrofit this in, it's a pain. But if you're building it, use an IEB on ZQ. Use it, exactly. If anything needs to be idempotent, do that from the beginning. The one you sort of mentioned was, I go back,