Fan out to three services, not three failure domains
One webhook, three consumers, and a handler that quietly became a router. Moving the fan-out upstream of your code gives every destination its own queue, clock and breaker.
Quentin Mousset5 min read
Stripe fires one webhook when a charge succeeds, and three things in your system need to know about it. Your API updates the order. Your analytics worker counts it. Your billing service reconciles it. You own one URL, so you receive the event once and hand it to all three.
That handler is now the only thing standing between one provider and three services that have nothing to do with each other. You did not set out to build a router. You built one anyway, and it has a property you did not choose: when any part of it has a bad afternoon, the whole thing has a bad afternoon.
The fan-out you wrote by accident#
It breaks in three ways, and they are not the ones you plan for.
The slowest destination sets your response time. Stripe is holding a connection open while your analytics worker does whatever analytics workers do. Await all three and your p99 is the worst of the three. Fire and forget instead, and you have swapped latency for silence: nothing now knows whether any of them actually ran.
One failure makes replay undecidable. Billing returns a 500. The API and analytics were fine. You have one event, three outcomes, and no record of which was which. Replaying the event sends it to all three again, so the API applies the same order update twice to fix a billing problem.
Your deploy takes all three down together. They share a process, so they
share an outage. That is the definition of a failure domain, and you drew its
boundary with a for loop.
None of this is a coding mistake. It is a placement mistake. The fan-out lives inside the one component that also has to answer the provider on time.
Move the fan-out upstream of your code#
Point the provider at a bucket instead of at your service. A queue bucket answers the provider itself, with a static response you configure, before any destination has been touched. The acknowledgement no longer depends on your slowest service, because it no longer waits for any service at all.
{
"received": true,
"source": "repost"
}Then attach one forwarder per destination. This is the part that does the real work: each forwarder gets its own Pulsar topic, provisioned when you create it. Three destinations means three queues, not one queue with three subscribers. A destination that stops consuming backs up its own topic and nothing else, which is the difference between one service having a problem and you having a problem.
Three destinations, three clocks#
Every delivery control is stored on the forwarder, not on the bucket, so your analytics pipeline and your payment path stop having to agree on anything.
| Setting | Range | What it buys you |
|---|---|---|
| Timeout | 5 to 60 seconds | Ten seconds for the API, sixty for the batch job that earns it |
| Retries | 0 to 3 | Off entirely for a destination where a duplicate costs more than a miss |
| Retry delay | 5 to 60 seconds | The first window, before backoff touches it |
| Backoff multiplier | 0.5x to 3x | How fast that window grows per attempt |
| Max retry delay | up to 30 minutes | The ceiling the window never passes |
| Rate limit | 5 to 500 rps | A cap on what you push at a destination that cannot take it |
Retries wait on floored full jitter: a random delay between zero and the current window, never shorter than one second, with the window growing by the multiplier until it hits the ceiling. If you want the arithmetic behind that choice, we did the sums in a separate post.
One rule worth reading twice, because it is where people expect symmetry and do
not get it: Repost retries connection failures, timeouts, 408, 429 and
5xx. Every other 4xx is final. A 422 is your payload being wrong, and
sending it again unchanged will produce the same 422 three more times.
The breaker, and the two things it refuses to do#
Each external forwarder carries a circuit breaker keyed on that forwarder. When it opens, no HTTP attempt is made at all. The delivery is rescheduled for when the circuit might be half open, so a destination that is down gets quiet instead of getting hammered by the retry schedule of a service that has not noticed yet.
Two of its behaviours are the opposite of what most people assume, and both are deliberate.
A 4xx does not open it. Only 5xx, timeouts and connection failures
count as failures. A 400 means the endpoint is up and your request is wrong.
Taking a destination out of rotation for that would be punishing a healthy
service for your own payload.
It fails open. When the breaker cannot reach the store holding its own state, it lets the request through rather than blocking it. A breaker that denies traffic the moment it loses track of the world is worse than no breaker, because it converts its own outage into yours.
What this does not fix#
Ordering. Three independent queues deliver in three independent orders, and no webhook provider promises you ordering in the first place. If your state machine depends on sequence, it has to derive that sequence from the payload (a version, a timestamp, a sequence number) rather than from arrival. Splitting the fan-out does not make this worse. It makes it visible, which is usually how people find out they had the assumption.
Proxy buckets get exactly one forwarder. If your provider needs a real answer from your application at request time, a challenge response or a status it will act on, then you need a proxy bucket: it holds the provider connection open while a single destination answers, and returns that answer verbatim. One destination, no retries, no rate limiting, no fan-out. Given that bucket mode cannot be changed later, this is the decision to get right before anything else (creating forwarders).
Your handlers still need to be idempotent. Moving the fan-out upstream means each destination now has its own retry schedule, so each one can see the same event twice on its own. That was already true. It is now true three times independently.
The rule#
One destination, one queue, one clock, one breaker. Your handler's job is to handle one event for one service, and every line it spends deciding where else that event should go is a line that will be in the stack trace the first time one of the three has a bad afternoon.
If you want to see it before you commit to it, the delivery history shows each forwarder's attempts separately, which is the same view your own customers can get.