October 12, 2026 · 3 min read · Yunus Emre Vurgun
Background Jobs: Idempotency and the Outbox Pattern
A queue can deliver a job more than once. A worker can finish an external action and crash before acknowledging the message. Designing for those events is easier than trying to prove they will never happen.
Separate delivery from the business effect. A message being delivered once, a handler running once, and an order being charged once are different claims. The application must define an operation identity and make the effect safe under the delivery semantics it actually has.
Make one operation identifiable
Assign a stable job or business-operation ID. Persist a deduplication record with the committed effect when the database can make that atomic. If an external service is involved, use its documented idempotency contract or reconcile its result. Recording processed before the effect can lose work; recording it afterward without atomicity can duplicate work.
Test the handler with the same message twice. Then crash it immediately before commit, immediately after commit, and just before acknowledgement. Observe the final domain state rather than merely counting handler invocations.
Publish events through a transactional outbox
BEGIN;
update domain state;
insert outbox event with stable event ID;
COMMIT;
publisher reads unsent outbox events;
publisher sends event;
publisher marks event as sent;The transaction keeps the domain change and the pending publication record together. A separate publisher can retry sending the event. It can still send a duplicate if it crashes after sending but before marking the event, so consumers must handle repeated event IDs appropriately.
The outbox solves the gap between a database write and publication; it does not create universal exactly-once behavior across every external system. Keep the event version and payload contract explicit so replay remains meaningful after code changes.
Bound leases, retries, and concurrency
A visibility timeout or lease lets another worker recover abandoned work. Long tasks may need heartbeats, but a heartbeat must prove ownership before extending the lease. If processing outlasts a lease, two workers can overlap. Test that case directly.
Use a retry budget with backoff and jitter. Move poison jobs aside with enough evidence to diagnose them, and define a replay procedure that preserves operation identity. Bound concurrency by the downstream resource’s capacity. A queue can buffer pressure; it cannot make an overloaded database infinitely fast.
Expose queue age, repeated failures, and job outcomes in observability. A healthy worker process does not establish that useful work is completing.
Reference tables to keep nearby
- Reliable Background Job Patterns — Queue and worker design patterns for duplicate deliveries, failure recovery, concurrency, and bounded work.
Primary sources
These are the specifications and project documentation behind the examples. Check the relevant implementation and version before applying a pattern.