A webhook can look perfectly reliable in a demo and still create duplicate leads, repeat invoices or lose an important event in production. The difference is rarely the HTTP request itself. Reliability comes from what the receiving system does when delivery is late, duplicated, out of order or interrupted halfway through processing.
The goal is to make failure boring. A lost event should be detectable, a duplicate should be harmless, and a temporary outage should trigger a controlled retry rather than a cascade. That is the same principle behind dependable workflow automation: design the failure path before the happy path becomes business-critical.
Design as though an event may arrive twice
Many webhook providers retry when they do not receive a successful response quickly. That improves delivery reliability but means your endpoint may see the same logical event more than once. If processing the event twice can create two leads, two invoices or two customer emails, your receiver needs idempotency.
A common pattern is to store a provider event ID or another stable idempotency key before performing the business action. If the same event arrives again, return success without repeating the side effect. This is different from simply checking whether “a similar record” exists, which can collapse legitimate separate events.
Verify authenticity before trusting the payload
Do not assume that a request is genuine because it reached a secret-looking URL. Mature webhook providers usually offer a signature mechanism based on a shared secret or public-key verification. Validate the signature using the raw request body and the provider’s documented method before processing the event.
Store webhook secrets like other credentials, not inside article content, client-side scripts or public repositories. The OWASP Secrets Management Cheat Sheet recommends centralized access control, rotation and auditing for sensitive credentials.
Separate acknowledgement from the expensive work
Webhook endpoints should usually do the minimum synchronous work required to establish that the event is valid and accepted. Heavy CRM updates, AI calls, document generation or multi-system fan-out should happen after the acknowledgement, ideally through a queue or background worker.
| Stage | Purpose | Failure response |
|---|---|---|
| Receive | Read the request and basic headers | Reject malformed input |
| Verify | Authenticate signature and timestamp | Return a clear 4xx |
| Deduplicate | Check event/idempotency key | Return success for a known completed event |
| Persist | Store event before slow processing | Return error if safe storage fails |
| Process | Run business logic asynchronously | Retry with limits and backoff |
Retries need rules, not enthusiasm
Retry transient problems such as a temporary network error or a provider rate limit. Do not retry permanent failures forever. If an email address is structurally invalid or a destination record was intentionally deleted, infinite retries only create noise.
Use bounded retry counts, increasing delays, and a dead-letter or failed-event state after the limit is reached. Our guide to automation error handling explains why failures need an owner and recovery path rather than being buried in logs.
Logs should let you reconstruct the event
A useful webhook log does not need to retain every sensitive field. Record an event identifier, event type, source, received time, verification result, processing status, attempts and the destination record ID where relevant. Redact credentials and minimize personal data.
The log should answer four questions quickly: Did we receive it? Did we trust it? Did we process it? What happened after processing? If the answer requires searching several products manually, incident recovery will be slow.
Design for ordering problems
Events can arrive out of order. An “updated” event may reach you before a delayed “created” event, or two changes may be processed concurrently. Avoid business rules that depend only on arrival time. Where the provider exposes an object version, event timestamp or sequence, use it. Otherwise, fetch the current source-of-truth record before making destructive changes.
Test the failures that demos skip
- Send the same event twice.
- Delay the downstream CRM response beyond your normal timeout.
- Return a 500 and observe the provider retry.
- Send a bad signature.
- Remove a required field.
- Process two updates concurrently.
- Simulate the queue or worker being offline.
If those tests are difficult to run, the automation may also be difficult to recover. A robust design makes failure simulation part of the development workflow, not an emergency exercise.
Keep webhook scope narrow
One endpoint does not need permission to do everything. Give the integration only the APIs and records it needs. This limits damage if a credential leaks and makes debugging easier. For broader automation architecture, see when rules-based automation is better than AI; deterministic event handling should stay deterministic even when AI is used elsewhere in the workflow.
Separate provider retries from your own retries
A webhook provider may retry delivery because your endpoint did not acknowledge the event. Your queue may also retry downstream processing. Treat those as separate layers. Once an event is safely persisted, acknowledge it according to the provider contract; let the worker own CRM or AI retries. Otherwise a slow downstream dependency can cause the provider and your worker to retry simultaneously.
Recovery needs a replay path
For important workflows, authorized operators should be able to replay a failed event after the root cause is fixed. Replay should reuse the original event identity so idempotency protections remain effective. This is safer than copying payloads into a new ad hoc request where provenance is lost.
A useful recovery screen shows the event ID, failure reason, attempts, last attempt and a controlled replay action. That turns incident recovery into an operational process rather than database improvisation.
Design webhooks for failure, not hope
Reliable webhook handling is mostly disciplined accounting: authenticate the sender, record the event, acknowledge quickly, prevent duplicate side effects and retain enough history to replay a failed job. Build those habits before traffic grows; they are much harder to retrofit after a duplicate payment or lost lead becomes the incident that forces the work.