Diagnose webhooks

Find why a webhook failed with bounded searches, stable IDs, event detail, the delivery retry chain, and the DLQ.

Start broad, then narrow by stable IDs. The goal is to gather enough evidence for an audit trail while making as few queries as possible.

flowchart TD
    H["health<br/>healthy enough to continue?"] --> S["events search<br/>find candidates in bounds"]
    S --> G["events get / diff<br/>inspect one event"]
    G --> C["forwards chain<br/>per-attempt timeline"]
    C --> D{Terminal & replayable?}
    D -->|yes| R["replay (remediate)"]
    D -->|validating a fix| W["expect / tail"]
    H -.->|dlq_depth > 0| Q["dlq list<br/>triage dead letters"]
    Q --> D

Start with health

When the question is "is this forwarder healthy enough to continue?", begin with health.

repost health \
  --bucket stripe-prod \
  --forwarder prod-api \
  --window 30m \
  --fail-on 'success_rate < 0.99 || dlq_depth > 0' \
  --json

{
  "schema": "repost.health/v1",
  "data": {
    "bucket": { "id": "bkt_01HV9S", "slug": "stripe-prod", "name": "Stripe production" },
    "window": "30m",
    "since": "2026-06-19T08:00:00Z",
    "until": "2026-06-19T08:30:00Z",
    "forwarders": [
      {
        "forwarder_id": "fwd_01HV9S",
        "forwarder_name": "prod-api",
        "success_rate": 0.9722,
        "failed": 4,
        "pending_retries": 2,
        "dlq_depth": 1,
        "p95_latency_ms": 1840.25,
        "last_delivery_at": "2026-06-19T08:29:51Z"
      }
    ]
  }
}

--fail-on turns a threshold into a process failure (partial_failure, exit 8) so CI and agents do not parse text. It accepts success_rate and dlq_depth with comparisons and ||. Here failed, pending_retries, and dlq_depth are non-zero, so continue into inspection.

Search events within bounds

Search only the bucket and time range the workflow needs.

repost events search 'method:POST AND path:/stripe/*' \
  --bucket stripe-prod \
  --since 30m \
  --limit 20 \
  --json

Page older results with --cursor. It is forward-only and bound to this exact query and window. Don't widen --limit to dodge paging; narrow the query. Full query grammar and cursor rules are in Search & filters.

Inspect an event

Move from search results to a single event by ID.

repost events get evt_01JZ8V1 --json
repost events diff evt_01JZ8V1 evt_01JZ8UZ --json

{
  "schema": "repost.events.get/v1",
  "data": {
    "event": {
      "id": "evt_01JZ8V1",
      "bucket_id": "bkt_01HV9S",
      "method": "POST",
      "path": "/stripe/webhook",
      "received_at": "2026-06-19T08:31:22Z",
      "response_status": 500,
      "request": {
        "headers": {
          "stripe-signature": ["t=1718061000,v1=[redacted:hmac:9f2a1c4d8b07]"],
          "content-type": ["application/json"]
        },
        "signature_header_present": true,
        "body": { "truncated": false, "bytes_total": 1284, "bytes_shown": 1284, "sha256": "6b8d4f0b…", "content": "{\"type\":\"invoice.payment_succeeded\"}" },
        "body_json": { "type": "invoice.payment_succeeded", "data": { "object": { "id": "in_123" } } }
      },
      "response": {
        "status": 500,
        "headers": { "content-type": ["text/plain"] },
        "body": { "truncated": false, "bytes_total": 36, "bytes_shown": 36, "sha256": "91c6f4b2…", "content": "database connection timeout" }
      }
    },
    "redactions": ["request.headers.stripe-signature"]
  }
}

events get truncates the body by default. Always check body.truncated; if it is true, re-fetch with --full for the complete payload. Use events diff to explain why one webhook behaved differently from another.

Credential headers and secret-like fields are redacted; PII is not. See Transcript safety before copying any payload into a transcript.

Inspect delivery attempts

Find failed deliveries with --failed, then pull the retry chain for one event and forwarder.

repost forwards search --bucket stripe-prod --forwarder prod-api --since 30m --failed --json
 
repost forwards chain evt_01JZ8V1 --bucket stripe-prod --forwarder prod-api --json

{
  "schema": "repost.forwards.chain/v1",
  "data": {
    "event_id": "evt_01JZ8V1",
    "forwarder_id": "fwd_01HV9S",
    "attempts": [
      { "forward_id": "fwd_attempt_01JZ8UZ", "attempt": 1, "type": "EXTERNAL", "status": "FAILED", "response_code": 503, "latency_ms": 220, "created_at": "2026-06-19T08:31:25Z", "will_retry": true, "next_retry_at": "2026-06-19T08:32:25Z" },
      { "forward_id": "fwd_attempt_01JZ8V2", "attempt": 3, "type": "EXTERNAL", "status": "FAILED", "response_code": 500, "latency_ms": 1840, "created_at": "2026-06-19T08:33:26Z", "will_retry": false }
    ]
  }
}

forwards search finds candidate attempts fast (use --failed rather than a status: query); forwards chain is the authoritative per-attempt timeline for one event and forwarder: order, response code, latency, retry intent, and terminal status. Search to locate, chain to drill in.

The chain's will_retry / next_retry_at are the forwarder's automatic retries (its retry policy). Deliberately re-sending an event is a replay, a separate, manual action covered in Replay deliveries.

Triage the DLQ

When health reports DLQ depth, inspect the DLQ rows before replaying.

repost dlq list --bucket stripe-prod --forwarder prod-api --limit 20 --json

{
  "schema": "repost.dlq.list/v1",
  "data": {
    "items": [
      { "dlq_id": "01JZ8V6H5Q8YJ5HY8D7M2QW1X2", "event_id": "evt_01JZ8V1", "forward_id": "fwd_attempt_01JZ8V2", "forwarder_id": "fwd_01HV9S", "created_at": "2026-06-19T08:33:40Z" }
    ],
    "has_more": false
  }
}