Alert when a webhook source goes quiet

The worst webhook failures make no noise. A provider disables your endpoint after repeated errors. Someone rotates an API key and the integration stops sending. A deploy changes a URL. Nothing throws, nothing logs, and the dashboards show a flat line that looks like a quiet afternoon. You find out when a customer asks where their order is.

Error alerts do not catch this, because there are no errors. You need an alert on absence: no events from this source for N minutes. This post shows how to set one up with Eventeer, how to add alerts on values inside the payload, and how to put both on a dashboard.

Pick a window that matches the traffic

Before you create anything, decide what “quiet” means for this source. The window should be longer than the longest normal gap and short enough to matter.

• A payment webhook that fires every few seconds in business hours: 5 to 10 minutes is reasonable.
• A device that reports every minute: 5 minutes tolerates a few lost packets.
• A nightly export: the window is more than a day, with a rule that only matters from morning.

Too short and the alert fires during ordinary lulls, and people stop reading it. Rules accept window_min from 1 to 1440 (24 hours), so the longest absence window is one day.

Step 1: receive events on a listener

Alerts watch listeners, so the source needs to send to one:

curl -X POST https://app.eventeer.io/api/v1/listeners \
  -H "Authorization: Bearer $EVENTEER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"payments","type":"webhook","auth_mode":"anonymous"}'

Give the returned ingress_url to the provider. The listener id in the response (a UUID) is what the alert rule refers to.

Step 2: a quiet-for-N-minutes rule

An absence rule fires when a listener, or one device on it, has received no events for window_min minutes:

curl -X POST https://app.eventeer.io/api/v1/alert-rules \
  -H "Authorization: Bearer $EVENTEER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"Payments quiet","kind":"absence",
       "listener_id":"7c1e4b92-3a5d-4f60-8b2e-9d0a1c3e5f71",
       "window_min":10,
       "cooldown_min":60,
       "channels":[{"type":"email","to":["ops@example.com"]}]}'

How it behaves:

• Rules are evaluated every minute.
• When the condition is met, an incident opens, state goes to firing, and the channels are notified.
• While it stays firing, nothing more is sent, unless cooldown_min is set and that many minutes have passed since the last notification. 0, the default, means never re-notify. Here it re-notifies hourly.
• When an event arrives and the listener is no longer quiet, the incident resolves automatically and a resolved notification goes out. No one has to close anything.

To watch one device instead of the whole listener, add "device_id" to the rule. That is the right choice for a fleet where one silent pump matters even though the others are chatty.

The email channel sends to organization members. The webhook channel reuses one of your destinations, which is how you get alerts into a chat tool or an incident system:

"channels": [{"type": "webhook", "destination_id": "a54c0e72-9b3d-4f18-8e6a-3c2d7f1b0e95"}]

The alert request body looks like this, and is signed with the destination’s secret like other deliveries:

{"type":"alert","status":"firing","rule":{"id":"…","name":"Payments quiet"},"value":83.1,"threshold":80,"opened_at":"…"}

(value and threshold are meaningful for threshold rules; an absence rule is about the time since the last event.) Alert notifications to a destination do not go through the delivery log and are not metered. See verifying webhook signatures for how to check them on your side.

Step 3: alert on values, not just silence

Silence is one failure mode. The other is data that arrives and is wrong: payment amounts at zero, temperatures climbing, error codes appearing. For those you define a metric, then a threshold rule over it.

A metric turns events into a time series with one-minute buckets. You choose an aggregation (count, sum, avg, min, max), and for everything except count, a JSON path to a numeric field:

curl -X POST https://app.eventeer.io/api/v1/metrics \
  -H "Authorization: Bearer $EVENTEER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"Order total","aggregation":"sum","value_path":"$.amount","unit":"USD",
       "listener_ids":["7c1e4b92-3a5d-4f60-8b2e-9d0a1c3e5f71"],
       "filter":{"path":"$.type","op":"eq","value":"order.created"}}'

The path syntax supports $, .field and [n], for example $.order.amount. The optional filter narrows which events count. Events where the field is missing or not a number are skipped. Metrics are not backfilled: a new metric starts from the moment you create it, so define them before you need them.

Then a threshold rule on it:

curl -X POST https://app.eventeer.io/api/v1/alert-rules \
  -H "Authorization: Bearer $EVENTEER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"Revenue dropped","kind":"threshold",
       "metric_id":"3a9c5e71-2d4b-4f68-9e0a-7b1c2d3e4f50",
       "operator":"lt","threshold":100,"window_min":15,
       "channels":[{"type":"email","to":["ops@example.com"]}]}'

This fires when the sum over the last 15 minutes is below 100. Operators are gt, gte, lt and lte.

One caution that applies here: a threshold rule with no data in the window is treated as not breached. A sum that drops to zero because events stopped entirely will not trip a lt rule on its own. That is why the absence rule from Step 2 is the companion to this one: use absence for “nothing is arriving” and threshold for “what arrives looks wrong”.

You can read the series yourself too:

curl "https://app.eventeer.io/api/v1/metrics/3a9c5e71-2d4b-4f68-9e0a-7b1c2d3e4f50/series?step=5m" \
  -H "Authorization: Bearer $EVENTEER_KEY"

It returns {"points":[{"t":"…","value":1.2,"count":3}]}, with empty buckets omitted. Steps are 1m, 5m, 1h and 1d. Minute points are kept for 30 days, independent of event retention.

Step 4: put it on a dashboard

Dashboards are built in the app: a 12-column grid with timeseries charts, stat tiles (last, total, average, min or max, with a delta against the previous range) and a live event feed. Time ranges run from 15 minutes to 30 days, and a dashboard can refresh live. Each organization starts with an Overview dashboard showing events per listener, connections and recent events. The dashboards API is read-only for now, so editing happens in the UI.

A useful minimal layout is a stat tile for events in the last hour, a timeseries of the metric you alert on, and an event feed filtered to the listener. When the alert fires, the feed shows what the last event was and when.

Things to get right

• Test the alert. Pause the listener or stop the sender in staging, wait out the window and confirm that the email arrives, and that it resolves when traffic resumes.
• Send alerts somewhere people look. An alert to a mailbox nobody reads is the same as no alert.
• Alert on what you can act on. One rule per source, with a window you have thought about, beats ten noisy ones.
• Know the limit. Alerts watch what reaches Eventeer. If the provider cannot reach the listener at all (DNS, a wrong URL), you see silence, and that is exactly what the absence rule reports.
• Keep the history. Stored events let you answer “what was the last thing we received?” and, after an outage, replay what failed.

The monitoring and alerts use case page shows the pieces together, and pricing lists how many metrics, alert rules and dashboards each plan includes. Every endpoint above is in the API docs.