Engineering

How webhook delivery behaves when it stops behaving. Retries and their arithmetic, signatures over exact bytes, replay that does not replay the bug, and the failure modes you meet at 2am.

06 articles
Two blocks on black joined by a wide arrow. Left: three stacked migration cards, v1 add order.created, v2 add currency, v3 rename reference, gathered by a brace reading your git history. The arrow is annotated in coral, none of this crosses. Right: a single JSON box holding type order.created, a timestamp and data, with a coral magnifying glass over it hunting for a field called version, next to a coral question mark. Under a dashed line: adding is safe, renaming is silent.EngineeringWebhook versioning: the version your consumers never seeYou version the event. They receive a type, a timestamp and a payload. Which changes break them, which ones are silent, and why the fix is rarely a v2.Quentin Mousset4 min readTwo diagrams on black, split by a vertical rule. Left: a provider cloud feeds a coral box labelled your handler, which splits into three white boxes, API, analytics and billing; a coral outline round all four reads one failure domain. Right: the same cloud feeds an indigo bucket that sends a short arrow marked 200 back to the provider, then splits into three rows, each in its own dashed outline with a queue, a clock and a circuit breaker. Under a dashed line: three destinations, three queues, three clocks.EngineeringFan out to three services, not three failure domainsOne webhook, three consumers, and a handler that quietly became a router. Moving the fan-out upstream of your code gives every destination its own queue, clock and breaker.Quentin Mousset5 min readA schema.repost box lists id String, amount Float and currency String, with currency circled as the field just added. Six dashed arrows fan out to six generated clients: typescript, go, java, kotlin and csharp each carry a check mark, while python carries a coral cross annotated still on the old shape. A brace gathers all six toward repost schema generate --check, where the note reads build fails here, not in production.EngineeringOne webhook schema. Six SDKs that never drift.Hand-maintained clients drift the week you ship. Generate all six from one schema file, and let CI fail the pull request that forgets one.Quentin Mousset2 min readAn evolution chart headed Webhooks Progress Timeline, with five labelled boxes above it: PING 1999, TRACKBACK 2002, WEB HOOKS 2006, WEBSUB 2018, WEBHOOK 2026. The first two figures are a knuckle-walking ape and a stooped hominid, then the line stops evolving: the 2006, 2018 and 2026 figures are the same upright walker, repeated without a single change. An arrow curves back from 2026 to 2006, and the caption reads Same HTTP POST. Different stakes.EngineeringWho invented webhooks? Meet the HTTP POST that never grew upSeveral serious attempts to replace it, and a webhook is still an HTTP POST to a URL you chose. The protocol never moved. What it carries did.Quentin Mousset15 min readA retry timeline for a 30-second base, a ×3 multiplier and a 3-hour cap: attempts land at 0s, 30s, 2m, 6m30, 20m, 1h and 3h02, separated by gaps of 30s, 1m30, 4m30, 13m30, 40m30 and 2h02. Past attempt 7 the cap engages and every further attempt sits three hours after the last, so attempt 14 falls at 24 hours, where the schedule gives up.EngineeringWebhook retry storms and the arithmetic of backoffFixed-interval retries turn one slow consumer into a queue that never drains. What exponential backoff actually costs, and how to size the ceiling.Felipe Schulz10 min readThe same JSON object in two serializations: the producer's 35-byte body and the 31-byte string that comes back from JSON.parse then JSON.stringify. Each one hashes to a different HMAC-SHA256 under the same signing secret, so only the producer's bytes match the webhook-signature header and the re-serialized copy is rejected.EngineeringWebhook signature verification, end to endA webhook signature covers the exact bytes on the wire, not the object you meant to send. Why re-serialization breaks verification, and how to test it.Quentin Mousset11 min read