Webhook Monitoring — A Field Guide

The dangerous webhook failure is not the one that throws an exception; it is the one where an expected event simply never shows up. No error, no alert, just a customer whose subscription quietly never activated. Monitoring webhooks means watching for both the failures you can see and the absence you cannot.

The metrics that matter

Instrument the whole path — delivery in, processing through, results out:

• Delivery success rate — how many inbound events returned 2xx versus errored or timed out.
• Processing success rate — of the events accepted, how many completed versus dead-lettered.
• Processing latency — time from receipt to completion, watched at p95, so backlog shows up as rising latency.
• DLQ depth — the count of failed events waiting; a rising DLQ is your clearest 'something is broken' signal.

Alert on absence, not just errors

Error-based alerting misses the silent failure entirely, because a missing event produces no error. If a source normally sends a steady stream, alert when the rate drops toward zero — the absence of expected traffic is itself the signal. For low-volume but critical events, a heartbeat or a periodic reconciliation against the source of truth catches gaps that a rate alarm would miss.

1. DLQ depth > N                     -> processing is failing
2. delivery success rate < 99%       -> endpoint or signature issue
3. inbound rate drops > 80% vs norm  -> silent delivery failure
4. p95 processing latency rising      -> backlog building

Reconcile as the backstop

Monitoring detects problems; reconciliation proves you did not miss anything. Periodically compare your state against the provider's — poll their API for recent objects and check each has a corresponding processed event. Reconciliation is the only thing that catches an event that was never sent or never received, which no amount of endpoint instrumentation can see.

Webhook monitoring is delivery and processing metrics, a DLQ-depth alarm, an absence alarm on expected traffic, and a reconciliation backstop for the events that vanish entirely. Together they turn silent webhook failures into ones you find out about from a dashboard, not from an angry customer.