# Fan out to three services, not three failure domains

> One webhook, three consumers, and a handler that quietly became a router. Moving the fan-out upstream of your code gives every destination its own queue, clock and breaker.

- Author: Quentin Mousset
- Published: 2026-08-26
- Canonical: https://repost.sh/blog/fan-out-without-three-failure-domains

![Two diagrams on black, split by a vertical rule. Left: a provider cloud feeds a coral box labelled your handler, which splits into three white boxes, API, analytics and billing; a coral outline round all four reads one failure domain. Right: the same cloud feeds an indigo bucket that sends a short arrow marked 200 back to the provider, then splits into three rows, each in its own dashed outline with a queue, a clock and a circuit breaker. Under a dashed line: three destinations, three queues, three clocks.](/images/blog/figure-fan-out-failure-domains.webp)

Stripe fires one webhook when a charge succeeds, and three things in your system need to know
  about it. Your API updates the order. Your analytics worker counts it. Your billing service
  reconciles it. You own one URL, so you receive the event once and hand it to all three.

That handler is now the only thing standing between one provider and three
services that have nothing to do with each other. You did not set out to build
a router. You built one anyway, and it has a property you did not choose: when
any part of it has a bad afternoon, the whole thing has a bad afternoon.

## The fan-out you wrote by accident

It breaks in three ways, and they are not the ones you plan for.

**The slowest destination sets your response time.** Stripe is holding a
connection open while your analytics worker does whatever analytics workers do.
Await all three and your p99 is the worst of the three. Fire and forget
instead, and you have swapped latency for silence: nothing now knows whether
any of them actually ran.

**One failure makes replay undecidable.** Billing returns a 500. The API and
analytics were fine. You have one event, three outcomes, and no record of which
was which. Replaying the event sends it to all three again, so the API applies
the same order update twice to fix a billing problem.

**Your deploy takes all three down together.** They share a process, so they
share an outage. That is the definition of a failure domain, and you drew its
boundary with a `for` loop.

None of this is a coding mistake. It is a placement mistake. The fan-out lives
inside the one component that also has to answer the provider on time.

## Move the fan-out upstream of your code

Point the provider at a bucket instead of at your service. A queue bucket
answers the provider itself, with a static response you configure, before any
destination has been touched. The acknowledgement no longer depends on your
slowest service, because it no longer waits for any service at all.

```json
{
  "received": true,
  "source": "repost"
}
```

Then attach one forwarder per destination. This is the part that does the real
work: **each forwarder gets its own Pulsar topic, provisioned when you create
it.** Three destinations means three queues, not one queue with three
subscribers. A destination that stops consuming backs up its own topic and
nothing else, which is the difference between one service having a problem and
you having a problem.

> **Bucket mode is a one-way door**: Queue or proxy is decided when the bucket is created and cannot be changed afterwards. Repost rejects the switch, so moving between modes means a new bucket and a new URL at the provider. Choose it before you wire anything up ([response behavior](/docs/buckets/response-behavior)).

## Three destinations, three clocks

Every delivery control is stored on the forwarder, not on the bucket, so your
analytics pipeline and your payment path stop having to agree on anything.

| Setting | Range | What it buys you |
|---|---|---|
| Timeout | 5 to 60 seconds | Ten seconds for the API, sixty for the batch job that earns it |
| Retries | 0 to 3 | Off entirely for a destination where a duplicate costs more than a miss |
| Retry delay | 5 to 60 seconds | The first window, before backoff touches it |
| Backoff multiplier | 0.5x to 3x | How fast that window grows per attempt |
| Max retry delay | up to 30 minutes | The ceiling the window never passes |
| Rate limit | 5 to 500 rps | A cap on what you push at a destination that cannot take it |

Retries wait on floored full jitter: a random delay between zero and the
current window, never shorter than one second, with the window growing by the
multiplier until it hits the ceiling. If you want the arithmetic behind that
choice, [we did the sums in a separate
post](/blog/retry-storms-and-the-arithmetic-of-backoff).

One rule worth reading twice, because it is where people expect symmetry and do
not get it: Repost retries connection failures, timeouts, `408`, `429` and
`5xx`. **Every other `4xx` is final.** A 422 is your payload being wrong, and
sending it again unchanged will produce the same 422 three more times.

## The breaker, and the two things it refuses to do

Each external forwarder carries a circuit breaker keyed on that forwarder.
When it opens, no HTTP attempt is made at all. The delivery is rescheduled for
when the circuit might be half open, so a destination that is down gets quiet
instead of getting hammered by the retry schedule of a service that has not
noticed yet.

Two of its behaviours are the opposite of what most people assume, and both are
deliberate.

**A `4xx` does not open it.** Only `5xx`, timeouts and connection failures
count as failures. A 400 means the endpoint is up and your request is wrong.
Taking a destination out of rotation for that would be punishing a healthy
service for your own payload.

**It fails open.** When the breaker cannot reach the store holding its own
state, it lets the request through rather than blocking it. A breaker that
denies traffic the moment it loses track of the world is worse than no breaker,
because it converts its own outage into yours.

> **What you see when it trips**: Blocked deliveries are rescheduled rather than dropped, so they show up in the delivery history with a status and a next attempt rather than vanishing ([failed deliveries](/docs/history/failed-deliveries)).

## What this does not fix

**Ordering.** Three independent queues deliver in three independent orders, and
no webhook provider promises you ordering in the first place. If your state
machine depends on sequence, it has to derive that sequence from the payload
(a version, a timestamp, a sequence number) rather than from arrival. Splitting
the fan-out does not make this worse. It makes it visible, which is usually how
people find out they had the assumption.

**Proxy buckets get exactly one forwarder.** If your provider needs a real
answer from your application at request time, a challenge response or a status
it will act on, then you need a proxy bucket: it holds the provider connection
open while a single destination answers, and returns that answer verbatim. One
destination, no retries, no rate limiting, no fan-out. Given that bucket mode
cannot be changed later, this is the decision to get right before anything
else ([creating forwarders](/docs/forwarders/create)).

**Your handlers still need to be idempotent.** Moving the fan-out upstream
means each destination now has its own retry schedule, so each one can see the
same event twice on its own. That was already true. It is now true three times
independently.

## The rule

One destination, one queue, one clock, one breaker. Your handler's job is to
handle one event for one service, and every line it spends deciding where else
that event should go is a line that will be in the stack trace the first time
one of the three has a bad afternoon.

If you want to see it before you commit to it, the delivery history shows each
forwarder's attempts separately, which is the same view [your own customers can
get](/blog/dont-build-the-webhook-delivery-table).

**See it on your own events.** Open a live inbox and inspect your first webhook in seconds (50K events/month free).