What 100 million transactions taught us about idempotency

The first duplicate was funny. A merchant in Amman booked one pickup and got two couriers, ninety seconds apart, both holding a manifest for the same parcel. The second duplicate, three weeks later, was a payout. That one was not funny.

Neither was a bug in the ordinary sense. Both were the system doing exactly what it was told, twice, because a network timeout looks identical to a slow success and our client had been built to try again.

The 3am page

Every integration we run sits between two systems that disagree about time. A courier API accepts a booking in 400 milliseconds on a good day and 40 seconds on a bad one. A bank gateway holds the connection open while a batch job finishes. A government verification service returns 200 with an error inside the body.

When a call takes longer than the client is willing to wait, the client gives up and retries. The server, meanwhile, finished the work. Now there are two shipments, two payment instructions, or two claims against one accident.

The instinct is to make the timeout longer. That trades one failure for a worse one, because a request held open for two minutes is a thread, a connection, and a queue slot that something else needed. The real fix is to make a repeated request harmless.

Three rules we apply

If a retry can hurt you, the retry is not the problem. The endpoint is.

Choosing the key

The tempting key is a hash of the payload. It is also the wrong one, because two legitimate identical bookings do exist: the same merchant, the same address, the same weight, one hour apart. A payload hash silently swallows the second.

We use a caller-generated identifier, scoped per tenant, stored with a short window and the response body:

POST /v1/shipments
Idempotency-Key: 7f3c9a10-4d2e-4b6f-9f21-8c1a5d0e2b44
Tenant: acme-logistics

200 OK        first call   -> work done, result stored
200 OK        replay       -> stored result returned, nothing re-run
409 Conflict  same key,    -> different payload, caller has a bug
              new payload

The window matters. Too short and a retry after a provider outage does real damage. Too long and the store becomes a second database with its own migration problems. Twenty four hours has held up well for shipment and payment work, and seven days for anything a human might resubmit from an email.

What it costs

Guarding writes is not free. Each protected endpoint pays one extra round trip to the key store, and the store itself has to be at least as available as the endpoint it protects. In our case that is a small cost against the alternative:

What we would change

We added the key store late, endpoint by endpoint, under pressure. Doing it again we would put the guard in the framework before the first integration ships, so that a new endpoint is protected by default and opting out is the decision that needs a reason.

We would also log a replay as an ordinary event rather than a warning. For the first month our dashboards looked alarming, because they were counting successful protection as noise.

None of this is novel. It is written down in every payments API guide, and we still learned it the expensive way, twice. If your system talks to anything you do not control, assume every call will arrive more than once, and design the second arrival to be boring.