What this project is about
A webhook is an HTTP notification sent when something happens, such as an order being created. The receiving service may be slow, unavailable or briefly disconnected. RelayForge separates accepting that event from delivering it, so the producer can receive an acknowledgement while durable background work continues. The project focuses on making failed delivery visible and recoverable.
The service uses Python, FastAPI, SQLAlchemy and a SQL database, with separate API and worker processes. SQLite supports local evaluation and PostgreSQL is the intended store for multiple workers. Its scope is a compact, inspectable delivery system; neither the architecture notes nor the available tests establish very-high-throughput performance or an operated multi-tenant SaaS service.
How it works
- Accept + persist event
- Lease, sign, deliver or retry
- Inspect dead letters + replay
Store the event before contacting subscribers
The API validates management requests, stores the producer event and creates one delivery record for each matching endpoint. It does not call subscribers during ingestion. Uniqueness constraints prevent duplicate event keys within a source and duplicate event-to-endpoint delivery rows. A matching retry returns the original event; changing the content while retaining its key produces a conflict. Attempt history and audit records make later outcomes inspectable.
Claim work with an expiring lease
A lease records which worker currently owns a delivery and when that ownership expires. Workers claim due rows through conditional database updates, then end the transaction before making network requests. Expired work becomes eligible again after a crash. Before recording a result, the worker checks the current owner and attempt count, preventing a stale worker from overwriting a newer claim's outcome.
Sign requests and explain failures
Outgoing requests carry HMAC-SHA256 signatures over a timestamp and the exact body, enabling the receiver to verify their origin and integrity. Bounded asynchronous concurrency limits simultaneous requests. Transient failures use exponential backoff with randomised delay and bounded Retry-After support. Permanent failures and exhausted retries become dead-lettered deliveries that operators can inspect and explicitly replay without erasing earlier attempts.
The important design choices
Use the database as the durable queue
Keep delivery state, leases and attempts in SQL rather than requiring an additional message broker.
The trade-offThe failure model and local setup stay compact, but polling introduces latency and database load. Queue traffic shares capacity with the API's persistent data.
Acknowledge ambiguous network outcomes
Use at-least-once delivery and stable event and delivery identifiers.
The trade-offA receiver can process a request even when its response is lost. Repeated delivery is therefore possible, and consumers must deduplicate before applying side effects.
A subscriber recovers after temporary failures
An order producer publishes once while a subscriber returns HTTP 503 for its first two delivery attempts.
- The API stores the event and delivery record and acknowledges the producer without waiting for the subscriber.
- A worker records each failed attempt and schedules another attempt according to the retry policy.
- The receiver eventually verifies the signed request, processes its delivery ID once and returns a success response.
What this showsThe delivery reaches a successful state with its failed attempts still available for inspection. This is also the scenario supported by the repository's receiver demonstration.
Illustrative example, not a production incident.
How it was tested—and what it shows
The checks
The repository includes tests for event fan-out, idempotency, signatures, retry timing, lease expiry and stale-worker protection. Worker tests use controlled HTTP responses while preserving signing and persisted state transitions. A PostgreSQL integration test starts competing worker threads and checks that only one claim succeeds. The demo receiver verifies signatures, rejects stale timestamps, deduplicates delivery IDs and deliberately returns temporary failures before succeeding.
What to take from the results
The contribution is a complete delivery lifecycle with evidence for failure and recovery paths, rather than just a successful HTTP call. Important limits remain documented: endpoint secrets are plaintext database fields, DNS validation is separate from connection-time resolution, and the management API uses one shared key. These controls need further work before handling sensitive customer payloads in a production deployment.
What to keep in mind
- At-least-once delivery can produce duplicate messages.
- Secret-management, key rotation and DNS hardening need further work before production use.
Technical notes & source
Built with
Python · FastAPI · SQLAlchemy · PostgreSQL · HMAC
Terms used in this project
- Webhook
- A message one service sends to another when an event happens.
- Dead-letter queue
- A place to inspect messages that could not be delivered after repeated attempts.