How to replay failed webhooks without losing events

Most webhook providers retry for a while and then give up. Stripe retries for days, others for a few hours, some not at all. If your endpoint is down longer than the provider’s patience, the event is gone from their side and you have nothing to process. The fix is not a better retry policy on the provider. It is to put something in front of your endpoint that accepts the request every time and keeps the raw payload.

This post walks through the pieces you need to replay failed webhooks without losing or double-processing events, and shows how to do each with Eventeer. The same ideas apply if you build the pieces yourself.

Why webhooks get lost

There are four common ways an event disappears.

1. Your endpoint was down. A deploy, a crash, a database failover. The provider got a 5xx or a timeout and retried until it stopped.
2. Your endpoint answered slowly. Many providers wait only a few seconds. If you do the real work inside the request handler, a slow dependency turns into a failed delivery.
3. Your code accepted the event and then failed. You returned 200, then the job crashed. The provider thinks you handled it.
4. A bug processed it wrongly. The event arrived fine, but your handler had a bug for two days. You need the original payloads again, not new ones.

Cases 1 and 2 are solved by accepting the request somewhere that does not go down. Cases 3 and 4 need the raw event stored, so you can run it again after the fix.

Step 1: accept and store the raw event first

The receiving side should do as little as possible: authenticate, write the body to durable storage, return a 2xx. Parsing, business logic and calls to other systems come later, from the stored copy.

With Eventeer, a webhook listener is that receiving side. You create one and get an unguessable URL:

curl -X POST https://app.eventeer.io/api/v1/listeners \
  -H "Authorization: Bearer $EVENTEER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"stripe-events","type":"webhook","auth_mode":"anonymous"}'

The response includes an ingress_url. Give it to the provider. A request to POST /in/w/<listener_key> accepts JSON, text and form bodies up to 256 KB, stores the event and answers 202 with the stored id:

{"id":"0192f3a4-7b1e-7c55-9a3e-5d2f8b6c1a3f","duplicate":false}

Events are kept for your plan’s retention (7, 30 or 90 days) and are browsable and filterable in the event explorer. Keep the raw body, not a transformed version of it. Whatever you decide to extract later, you can only do it if the original is still there.

If the provider signs its requests, use an hmac listener so unsigned or tampered requests are rejected before they are stored. See how to verify webhook signatures for the details of the scheme.

Step 2: make duplicates harmless

Replay and retry both mean the same event can arrive more than once. Providers say this in their docs: delivery is at-least-once. So every stage needs a way to recognize a repeat.

On the way in, send an Idempotency-Key header when you control the sender. Eventeer dedupes per listener for the retention period:

curl -X POST https://app.eventeer.io/in/w/lst_7Qp2xK9vLmN3rT5wY8zA1b \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: order-1234" \
  -d '{"type":"order.created","amount":42.5}'

The first call returns 202 with "duplicate":false. The same key again returns 200 with "duplicate":true and the original id, and no second event is stored. This is useful when your own services call a listener and may retry after a lost response. For third-party providers you do not set the header; use the event id inside their payload as your own dedupe key downstream.

On the way out, every delivery carries an X-Eventeer-Event-Id header. Your endpoint should record the ids it has processed and skip repeats:

// Express-style handler. `seen` is a table with a unique index on event_id.
app.post("/hooks/eventeer", express.raw({ type: "application/json" }), async (req, res) => {
  const eventId = req.get("X-Eventeer-Event-Id");
  const inserted = await db.insertIfNew("seen_events", { event_id: eventId });
  if (inserted) await queue.add("process-event", { eventId, body: req.body.toString() });
  res.sendStatus(200); // acknowledge repeats too, or the sender keeps retrying
});

Note the order: record the id and enqueue the work in one place, then answer 2xx. Answering 200 for a repeat matters. If you return an error for something you already handled, the sender retries it again.

Step 3: forward with retries and a delivery log

Next, get stored events to your service. In Eventeer that is a destination: an HTTPS URL that receives each matching event as a signed POST.

curl -X POST https://app.eventeer.io/api/v1/destinations \
  -H "Authorization: Bearer $EVENTEER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"billing service","url":"https://billing.example.com/hooks/eventeer",
       "listener_ids":["7c1e4b92-3a5d-4f60-8b2e-9d0a1c3e5f71"]}'

A delivery succeeds on any 2xx within 10 seconds. Otherwise Eventeer retries with backoff at 30 seconds, 2 minutes, 10 minutes, 30 minutes, 1 hour, 3 hours, 6 hours and 12 hours: eight retries over roughly 24 hours. After that the delivery is marked dead. A 410 Gone disables the destination immediately.

So a short outage heals on its own. Your service comes back and the queued deliveries land, each with the same X-Eventeer-Event-Id it would have had the first time. Retries are not billed again.

Step 4: replay after a long outage or a bug

A 24-hour window does not cover a weekend outage, and automatic retries do nothing for case 4, the handler bug. This is where stored events pay off.

First, find out what failed. The delivery log lists deliveries per destination, newest first, and can be filtered by status:

curl "https://app.eventeer.io/api/v1/destinations/dst_8Wm3Kq1Zp6Vt9Nb2Lc5Xr4/deliveries?status=dead&limit=50" \
  -H "Authorization: Bearer $EVENTEER_KEY"

Each entry has attempts, last_status_code, last_error and the event_id. For a handful of failed or dead deliveries, retry them directly:

curl -X POST https://app.eventeer.io/api/v1/deliveries/1c7e9a24-5b3f-4d6e-8a1b-0f2c3d4e5f60/retry \
  -H "Authorization: Bearer $EVENTEER_KEY"

That returns 202 and the delivery goes back to pending. Fix the cause first. Retrying into a still-broken endpoint just burns another attempt.

For a larger window, such as “everything from Friday 18:00 to Monday 09:00”, use the event explorer. Filter by listener and time range, select the events and use the replay-to-destination action. Replayed events go out signed like any other delivery and carry their original event ids, so the dedupe table from Step 2 keeps you safe if some were already processed. Replay is an action in the web app; the API exposes stored events (GET /api/v1/events with listener, from and to filters) and delivery retries, not a bulk replay call.

A checklist you can reuse

• Accept first, process later. Return 2xx as soon as the raw body is stored.
• Keep the raw payload for as long as you might need to replay it. Pick retention to match your longest realistic outage plus time to notice.
• Deduplicate at every stage, keyed on a stable event id.
• Acknowledge duplicates with a 2xx.
• Watch the delivery log, not just your endpoint’s error rate. A rising count of failed deliveries tells you before the 24-hour retry window runs out. You can also alert when a source goes quiet, which catches the opposite problem: events that never arrive.
• Rehearse a replay once. Pick a small time window in staging and confirm your handler is actually idempotent.

What this does not do

Eventeer stores and re-sends events; it does not know whether your handler did the right thing with them. Idempotency in your own database is still your job. And replay is at-least-once, so your consumer must tolerate repeats. If your downstream cannot be made idempotent, put a queue with its own dedupe in front of it.

If you want the receiving side hosted, the webhooks use case page shows the full flow, and pricing lists retention by plan. The endpoint reference is in the API docs.