Duplicate webhooks: four causes, and idempotency fixes one
The same charge lands twice and the reflex is to reach for idempotency. Four different situations look identical from inside your handler, and they are fixed in four different places.
Quentin Mousset4 min read
The same charge is in your database twice. Support noticed before you did. The reflex is to reach for an idempotency key and move on, and it is the right move in exactly one of the four cases below. The other three produce two rows where you expected one and are fixed somewhere else entirely.
An event is not a delivery#
One inbound webhook is one event. What happens next is deliveries, plural.
A single event can produce many of them: one per destination, one per retry,
one per replay you start by hand. Every one of those is a separate POST to a
separate handler, and every one of them can be the second row you are staring
at.
So "I got the event twice" is almost never what happened. Something got delivered twice. Which thing, and to whom, is the entire diagnosis.
Four things that look identical from inside your handler#
1. A retry, because your endpoint failed the first time#
Your handler returned a 500, or timed out, or refused the connection, and the
forwarder tried again. Same event, second attempt.
attempt:[2 TO *]This is not a malfunction, it is the delivery guarantee working. Retries fire
on connection failures, timeouts, 408, 429 and 5xx (what gets retried
and what does not). A 422 is final, because
sending an unchanged payload back into the same rejection is just a slower way
to get the same answer.
The fix is entirely inside your handler. Nothing about the transport should change.
2. Your own fan-out#
Same event, two destinations, two rows. Nothing was duplicated: your bucket has more than one forwarder, and each one delivers independently, with its own queue, clock and breaker.
This one is worth catching early because the "duplicate" is usually a design problem in disguise. If two of your services both write to the same table on the same event, the delivery layer is doing exactly what you configured, and the thing to change is which service owns that write.
3. A replay somebody started#
Including you, last Tuesday, on a filtered set that was wider than you thought.
forward_type:REPLAYEvery delivery is tagged ATTEMPT or REPLAY, and a replayed one also carries
its replay_job_id, so a mystery duplicate resolves to a specific job somebody
ran (replay). This is the case people forget exists,
and it is the easiest of the four to confirm or rule out.
4. Two genuinely different events#
The provider sent two things. They describe one fact in your business, so they look like duplicates to you, but they are unrelated at the transport layer. Payment providers do this routinely, emitting both a charge-level and an intent-level event for one payment.
Nothing in the delivery pipeline caused this and nothing in it can fix it. Your deduplication key is scoped to the wrong thing.
The case nobody can resolve for you, including us#
There is a fifth situation, and it is the one worth being straight about.
When a provider does not see your acknowledgement, it sends the request again. Repost captures inbound requests as they arrive, so those two requests are two events, with two different event ids. From the receiving side they are genuinely two events. Nothing in the HTTP exchange says otherwise.
The only thing that links them is the identifier the provider put inside the payload. Which means the query that answers "did they send this twice" is not about our identifier at all:
request.body_json.id:"evt_1PabcXYZ"Any JSON field in the body is searchable with dot notation, so this works for whatever your provider calls its id (the full field list).
One thing does get smaller here. A queued bucket answers the provider on arrival rather than after your service has finished, so the window where an acknowledgement can go missing no longer contains your application's processing time. Smaller is not zero. A network can still drop the reply.
What you actually change#
| What you found | Where the fix lives |
|---|---|
attempt above 1 | Your handler, which has to tolerate seeing the event twice |
| Two forwarders | Your topology: decide which destination owns the write |
forward_type:REPLAY | Nothing. Narrow the filter next time |
| Two event ids, one payment | Your dedupe key, which is scoped to the wrong object |
| Two event ids, same provider id | Your dedupe key again, this time keyed on the provider's id |
The provider you came here to blame is not in that table. Four of the five rows are your code, which sounds harsh and is actually the good news: they are the ones you can fix today, without waiting on anybody's support queue.
What this does not solve#
Ordering. Two deliveries of one event tell you nothing about the order of two different events, and no provider promises you one. If your state machine depends on sequence, it has to read that sequence out of the payload.
Providers that ship no stable id. Some send you a body and nothing to hold on to. You are then building a key out of business fields, a customer plus an amount plus a rounded timestamp, and that key is a guess with a failure mode: too narrow and you process a real second payment twice, too wide and you drop one your customer actually made. Pick the direction you would rather be wrong in, and write it down next to the code, because the next person will not guess which way you leaned.