# Webhook retry storms and the arithmetic of backoff

> Fixed-interval retries turn one slow consumer into a queue that never drains. What exponential backoff actually costs, and how to size the ceiling.

- Author: Felipe Schulz
- Published: 2026-07-27
- Canonical: https://repost.sh/blog/retry-storms-and-the-arithmetic-of-backoff

![A retry timeline for a 30-second base, a ×3 multiplier and a 3-hour cap: attempts land at 0s, 30s, 2m, 6m30, 20m, 1h and 3h02, separated by gaps of 30s, 1m30, 4m30, 13m30, 40m30 and 2h02. Past attempt 7 the cap engages and every further attempt sits three hours after the last, so attempt 14 falls at 24 hours, where the schedule gives up.](/images/blog/figure-backoff-timeline.webp)

Every webhook sender retries. It is the one failure-handling tool the whole ecosystem agrees on: the delivery failed, so try it again. The disagreement is entirely in the timing. So is the difference between a retry policy that absorbs an incident and one that amplifies it. Nobody has ever specified that timing for you: [webhook retries were documented in production in February 2008, and the 2023 specification still only recommends them, in lowercase](/blog/who-invented-webhooks).

This article is the arithmetic of that timing. What happens to a queue when a consumer slows down and the sender retries on a fixed interval. What an exponential schedule actually costs in wall-clock time, and how to size its cap. Why jitter exists, and why a retry budget bounds something a retry count cannot. Every number below is a worked example, small enough to redo on paper.

## Key takeaways

- Fixed-interval retries make load grow with the backlog: at 600 events per minute and a 30-second retry interval, a 10-minute outage ends with the destination receiving 19× its normal traffic.
- Exponential backoff has three knobs (base, multiplier, cap) and one derived number, time-to-give-up. A 30 s base with a ×3 multiplier and a 3-hour cap reaches roughly 24 hours across 14 attempts.
- Backoff spaces one delivery's attempts; jitter de-correlates thousands of them. Without jitter, everything that failed together comes back together, as one spike.
- A retry count bounds one delivery. A storm is an aggregate phenomenon, so the knob that prevents it must bound the aggregate: a retry budget, a per-endpoint rate limit, or both.
- When retries run out, the delivery should become visible evidence you can replay after the fix - never a silent drop.

## Anatomy of a retry storm

A retry storm is what happens when retry pressure grows with the backlog instead of with the traffic. Fixed-interval retries guarantee it: every delivery that has not succeeded yet generates attempts at a constant personal rate, so total load is proportional to how far behind you are. You are furthest behind exactly when the destination is weakest.

Take a system small enough to check by hand:

- Events arrive at a steady **600 per minute**.
- The destination normally answers in 50 ms; the sender times out at 5 s.
- Failed deliveries retry on a **fixed 30-second interval**.
- At minute 0, a dependency of the consumer degrades and every request starts exceeding the timeout. The bug is fixed ten minutes later. Healthy, the consumer absorbs about 30 requests per second.

Every attempt in minute one fails, so 600 deliveries join the backlog. Each backlogged delivery retries twice a minute (once per 30-second interval). Call the backlog entering a minute B: attempts during that minute ≈ 600 new + 2 × B retries.

| Minute | Backlog entering it | Attempts during it |
| --- | --- | --- |
| 1 | 0 | 600 |
| 2 | 600 | 1,800 |
| 5 | 2,400 | 5,400 |
| 10 | 5,400 | 11,400 |
| 11 (fix ships) | 6,000 | 12,600 |

By minute 10 the destination (still broken) is receiving 11,400 attempts a minute, **19× its normal traffic**. The retry policy has converted a slowdown into a load test.

Minute 11 is the part that surprises people. The dependency is fixed, the consumer is healthy, and it does not matter. 12,600 attempts a minute is 210 requests per second against a service that can absorb 30. The queue in front of the consumer grows by roughly 180 requests every second, waiting time blows through the sender's 5-second timeout almost immediately, and the sender marks attempts failed even when the consumer eventually processed them. Duplicate side effects included, for every handler that is not idempotent. Goodput rounds to zero, the backlog keeps growing, and the storm has outlived the outage it was reacting to.

The drain condition falls out of the same arithmetic. A queue empties only while successful throughput exceeds the arrival rate, and fixed-interval retries push offered load up with the backlog, making success less likely the further behind you get. There are two exits: spread the retries until the pressure fits under capacity, or start deleting data.

> **If you operate the receiving end**: Acknowledge, then process. A handler that stores the request and returns 2xx in milliseconds cannot be slow enough to start a storm: the queue moves inside your own system, where you control the drain rate. That decoupling is what [inbound webhook buckets](/docs/buckets/create) are for: Repost stores the complete request before acknowledging it, and you replay from the bucket once your fix ships.

## The arithmetic of exponential backoff

Exponential backoff has three knobs - the base delay, the multiplier, and the cap - plus one derived number, time-to-give-up. Size the give-up time first: it is the only value in the schedule the rest of the business can have an opinion about.

The gap before retry k is `min(cap, base × multiplier^(k-1))`. Here is the full schedule for a 30-second base, a ×3 multiplier, and a 3-hour cap:

| Attempt | Gap before it | Elapsed since first failure |
| --- | --- | --- |
| 1 | none | 0 |
| 2 | 30 s | 30 s |
| 3 | 90 s | 2 min |
| 4 | 4.5 min | 6.5 min |
| 5 | 13.5 min | 20 min |
| 6 | 40.5 min | ~1 h |
| 7 | ~2 h | ~3 h |
| 8 | 3 h (capped) | ~6 h |
| … | 3 h each | … |
| 14 | 3 h | ~24 h |

Read three properties off the table, because they generalize to any parameters you pick:

- **The ramp is front-loaded.** Five attempts land inside the first 20 minutes, where transient failures actually resolve: a rollback, a restarting process, a brief saturation.
- **With any multiplier of 2 or more, each new gap is longer than all previous gaps combined.** The ramp is cheap. The tail is where the wall-clock time lives.
- **Past the cap, time-to-give-up is linear.** Every extra attempt buys exactly one more cap interval. In this schedule, the last five gaps account for 15 of its 24 hours.

The cap is not optional. Uncapped, the same ×3 schedule puts a gap of 18 hours before attempt 9. By attempt 14, the one that closes the capped schedule at 24 hours, the uncapped gap alone is 184 days (`30 s × 3^12`). Exponential growth does not stop at a useful ceiling on its own; you have to put the ceiling there.

Sizing it is a product question disguised as an infrastructure setting: *how stale can this event be and still be worth delivering?* A cache-invalidation ping is worthless within the hour. A billing event is still worth delivering tomorrow. Work backwards from that answer. Give-up time and cap determine the attempt count, while the base and multiplier only shape the first hour. Think of the cap as your steady knock during a long outage: hours-scale caps match how outages actually end, with a human shipping a fix on a human timescale.

## Jitter: spacing is not spreading

Backoff decides when one delivery comes back. Jitter decides whether ten thousand deliveries come back at the same instant. Deliveries that failed at the same moment stay synchronized through every backoff step: the herd returns as one spike, just less often.

Put numbers on it: a destination falls over and 1,000 in-flight deliveries fail within the same second. Pure backoff schedules all 1,000 retries 30 seconds later - a 1,000-request burst inside one second, aimed at a service you already know is struggling. Averaged over the 30-second window those retries represent about 33 requests per second; delivered as a spike, the instantaneous rate is 30× that. Capacity planning is about peaks, and synchronized backoff manufactures peaks.

The fix is to draw each gap at random from the interval backoff computed (commonly called full jitter):

```ts
function retryDelaySeconds(retry: number): number {
  const base = 30;
  const multiplier = 3;
  const cap = 3 * 60 * 60;
  const ideal = Math.min(cap, base * multiplier ** (retry - 1));

  return Math.random() * ideal; // full jitter: anywhere in [0, ideal)
}
```

The same 1,000 retries now spread across the whole window: about 33 requests a second, evenly. Same total work, survivable peak.

One honest footnote: a uniform draw over `[0, ideal]` has an expected value of `ideal / 2`, so a schedule that reads 24 hours on paper runs about 12 in expectation. Either accept that (the give-up time was an order-of-magnitude decision anyway) or draw from the upper half of each window instead.

## Retry budgets, not retry counts

A retry count bounds one delivery. A storm is an aggregate. The knob that prevents storms has to bound the aggregate: a retry budget on the sending side, a rate limit at the destination, or both.

Backoff already collapses aggregate pressure enormously. The fixed-interval storm above had 6,000 backlogged deliveries producing 12,000 retries a minute. The same backlog sitting in the capped tail of the exponential schedule (one attempt per delivery per 3 hours) produces about 33 retries a minute, roughly 360× less. But notice what the number still depends on: the backlog. Grow it to a million backlogged deliveries - a batch import, a long weekend outage - and the capped tail alone offers ~5,600 attempts a minute, while every delivery stays politely inside its per-delivery allowance. The retry count never noticed.

A retry budget bounds the ratio instead: permit retries only up to a fraction of recent first-attempt traffic, say 10%, and defer the rest. Now the destination never sees more than 1.1× its arrival traffic, no matter how deep the backlog is, and the backlog drains through the budget instead of stampeding.

For queue-based webhook delivery there is an equivalent bound at the other end: a per-endpoint rate limit caps the total attempt rate a destination sees, first attempts and retries combined, regardless of backlog depth. And the degenerate case deserves a name: when you already know the endpoint is down, the correct budget is zero. Pausing an endpoint should hold its deliveries, not burn attempts into a wall.

## What a good webhook retry schedule looks like

Front-loaded, capped, jittered, bounded in aggregate, and honest about the end. In practice:

- **One quick first retry**, within ~30 seconds. It catches connection resets and restarts that resolve on their own; more than one quick attempt is an accelerant.
- **An exponential ramp** (a multiplier of 2 to 3) through the first hour, where most transient failures clear.
- **A cap measured in hours**, because destination outages end on human timescales.
- **Jitter on every gap**, so coincident failures do not stay coincident.
- **An aggregate bound**: a retry budget, a per-endpoint rate limit, or both.
- **Respect for `Retry-After`.** When the destination answers 429 or 503 with a [`Retry-After` header](https://datatracker.ietf.org/doc/html/rfc9110#section-10.2.3), that is the consumer telling you its actual capacity. Treat it as a floor, not a suggestion.
- **A visible end.** A deliberately chosen time-to-give-up, and exhaustion as a state you can inspect, with its attempt evidence, rather than a silent drop.

This shape is what [Repost's delivery layer](/docs/send/delivery) implements: transient failures follow a bounded schedule with jitter, `Retry-After` guidance from the destination is honored, and a success stops the chain. Every attempt retains its response status, latency, and error class. Endpoints can be paused so queued deliveries are held instead of failed, and when a schedule does run out, the delivery stays visible and recoverable. Replay starts a new delivery generation without erasing the attempts that came before it. The arithmetic above is why each of those behaviors exists.

## Frequently asked questions

### How many times should I retry a webhook delivery?

Derive the count instead of picking it. Decide how long a delivery is worth attempting (time-to-give-up), pick a cap for the steady interval during long outages, and the attempt count falls out: a 30-second base with a ×3 multiplier and a 3-hour cap reaches about 24 hours in 14 attempts. An event that is stale after an hour deserves a much shorter schedule than one still worth delivering tomorrow.

### Is exponential backoff alone enough to prevent retry storms?

No. Backoff spaces out one delivery's attempts, but deliveries that failed at the same moment remain synchronized and return as simultaneous spikes. Preventing that requires jitter. Aggregate retry pressure also still grows with backlog depth, which is what a retry budget or a per-endpoint rate limit bounds. It takes all three: backoff for the individual schedule, jitter for de-correlation, and an aggregate bound against the storm.

### Should the first webhook retry be immediate?

Near-immediate is useful: a retry within seconds catches connection resets and rolling restarts that resolve on their own. Keep exactly one quick attempt, though. Immediate retries multiply load fastest precisely when the destination is down hard, so everything after the first quick attempt should back off exponentially.

### What should happen when webhook retries are exhausted?

Exhaustion should be a visible state, not a deletion. The delivery needs to remain inspectable with its attempt history (what was sent, when, and what came back) and replayable once the destination is fixed. In Repost, exhausted deliveries stay visible and recoverable, and a replay starts a new delivery generation while preserving the original attempts.

**See the schedule on your own events.** Repost applies bounded, jittered retries to every transient failure and keeps the evidence when a schedule runs out. The Free plan includes the entire product and 50K events a month, no credit card required.