Webhooks · Reliability · Event delivery

Webhooks Without Regret: Designing Event Delivery You Can Trust

How to design webhook delivery so consumers can trust it: signatures, idempotency, retries, and a stable event shape.

A webhook is a promise you make to someone else’s server. When something happens in your system, you will tell them about it, and you will do it in a way that survives failure on both sides. It sounds straightforward, and it is — until you have to deliver a message reliably across the internet with nothing but an HTTP call.

The reason webhooks earn a bad reputation is that they concentrate every classic distributed-systems failure into a single, externally visible contract. The consumer sees a POST that sometimes arrives late, sometimes twice, and occasionally not at all, and they have to decide whether to trust the system that sent it. The good news is that the failure modes are well understood, and a handful of practices removes nearly all of the ambiguity.

A webhook is a contract you cannot debug

The defining property of webhook delivery is that you can never see the other side. When your consumer runs their own code, you will not have access to their logs, their database, or their debugging session. If a delivery goes wrong, all you can do is point at what you sent. This changes the design bar: the delivery mechanism itself has to be inspectable.

Build a delivery log from day one. Record the event, the payload, the target URL, the attempt count, the response status, and the timestamp for each attempt. When a consumer reports a problem, a delivery log turns a he-said-she-said into a concrete artifact you can both look at. It is the single most useful thing you can build for debugging webhooks, and it is embarrassingly easy to implement.

Prove the request is yours

Consumers have every reason to be suspicious of an unauthenticated POST claiming to come from your system. The standard remedy is a signature computed over the raw payload and attached as a header, using a shared secret that both sides know and that you never send inside the payload itself. HMAC-SHA256 over the exact request body is the conventional choice.

Three details make or break signature verification. First, the consumer should compare signatures in constant time, so a mismatch does not leak information byte by byte. Second, the payload used for verification must be byte-for-byte identical to what you sent, which is why you should document that the raw body is used, not a re-parsed or reformatted version. Third, include a timestamp in the signed data and reject events older than a small window, which gives you replay protection without asking the consumer to track every event ID.

When you document the scheme, give the consumer the code that verifies it, not just a description. A short reference implementation removes most of the integration friction, because the tricky parts — the exact bytes, the header name, the timing-safe comparison — are then someone else’s solved problem rather than a debugging session.

Idempotency keys: consumers retry, and that is fine

The internet will drop your deliveries, and the correct response is to retry. But the consumer cannot always distinguish “you sent this twice” from “you sent two different events,” and you should not ask them to guess. Every event should carry a stable, unique event ID, and consumers should be able to use that ID to deduplicate.

For events that cause side effects on the consumer side — creating a record, charging a card, sending a message — encourage them to treat the event ID as an idempotency key. You can help by never changing the ID across retries and by documenting that the ID is the deduplication signal. If you support explicit acknowledgment of deliveries, the combination of event ID plus acknowledgment forms a clean retry protocol: you resend until acknowledged, and the consumer can safely ignore anything they have already processed.

Retries with backoff and a dead-letter path

Your delivery mechanism should not give up after one attempt. Exponential backoff with jitter is the industry default for a reason: it spreads load, respects the consumer’s own failure recovery, and eventually succeeds for the majority of transient outages. Pick a modest schedule — seconds, then minutes, then longer — and cap the total window so that retries do not become a background job that runs forever.

Equally important is what happens when the retries run out. Design a dead-letter path in advance: a place where undelivered events go to be examined by a human or by a repair script. The dead-letter path is what keeps a webhook system honest. Without it, events silently disappear into a retry loop, and the consumer discovers the gap weeks later when a record they expected never arrived.

The event shape: stable envelope, evolving payload

Treat the envelope and the payload as two different compatibility surfaces. The envelope — event ID, type, timestamp, and a reference to what the event is about — should be stable from the first version. Consumers build their parsing around it, and you should be able to promise it will not change. The payload can evolve additively: new fields may appear, and consumers who parse defensively will keep working.

Choose event type names that are explicit and versioned in intent, such as order.created or invoice.payment_failed. Broad type names force consumers to inspect the payload to discover what actually happened, which pushes complexity out of your contract and into theirs.

Deliver fast, acknowledge slow

Do not block your own request handling on a slow consumer. Incoming webhook deliveries should be handed to a queue and sent asynchronously, so that a consumer that is down does not back up your own service. The flip side is also true: you should not wait forever for the consumer to acknowledge. Time out deliveries, treat timeouts as failures, and let the retry schedule handle them.

Test webhooks like the rest of your system

Webhook delivery deserves the same test coverage as any other part of the product. Test that a signed payload verifies, that a tampered payload fails, that an old timestamp is rejected, and that retries stop once an acknowledgment arrives. Run these tests in CI so that a change to the signing code cannot silently break every consumer in production.

There is nothing romantic about webhooks. They are plumbing, and the best plumbing is boring. The reward for doing the unglamorous work — signatures, IDs, retries, logs, and a dead-letter path — is that the events you send become something consumers can trust, and a webhook that can be trusted is one nobody regrets building.