Reliable Background Job Patterns
Queue and worker design patterns for duplicate deliveries, failure recovery, concurrency, and bounded work.
Queue and worker design patterns for duplicate deliveries, failure recovery, concurrency, and bounded work. Each row includes a practical application and a failure to check before using it.
Use it when
- Design a worker that can safely retry after a crash.
- Separate delivery guarantees from business effects.
Watch out for
- Exactly-once delivery claims do not establish exactly-once external effects.
- Patterns must be implemented with the database and queue’s real atomicity guarantees.
The data
10 rows
| Pattern | Mechanism | Use | Failure to test |
|---|---|---|---|
| Idempotent handler | Make repeated application produce the intended single result. | Handle at-least-once message delivery. | Deliver the same job twice. |
| Deduplication key | Persist a stable key with the committed effect. | Prevent repeating one business operation. | Crash between recording the key and committing the effect. |
| Transactional outbox | Commit the domain change and outgoing record together. | Keep a database write aligned with event publication. | Crash after commit but before publication. |
| Visibility timeout | Lease a job for a bounded processing interval. | Recover work from a failed worker. | Run a job longer than the lease. |
| Lease heartbeat | Extend a lease while an owned job is progressing. | Support long-running bounded tasks. | Lose connectivity during an extension. |
| Exponential backoff | Increase delay between repeated failures with jitter. | Reduce pressure on a failing dependency. | Make many workers fail simultaneously. |
| Dead-letter queue | Move repeatedly failing jobs aside with evidence. | Keep a poison job from blocking useful work. | Replay a corrected job without duplicating effects. |
| Concurrency limit | Bound active jobs per resource or tenant. | Protect downstream capacity. | Enqueue a burst above steady-state throughput. |
| Cancellation | Check a cooperative cancellation signal at safe points. | Stop obsolete or user-cancelled work. | Cancel while an external side effect is in flight. |
| Checkpoint | Persist completed progress under a versioned contract. | Resume a long task after a crash. | Resume with a changed code or data version. |
Fields
The shape of this dataset, read from the data itself: every column, its type and a real value. Agents can use this to plan a query without fetching the whole document.
| Section | Field | Type | Example |
|---|---|---|---|
| data | pattern | string | Idempotent handler |
| data | mechanism | string | Make repeated application produce the intended … |
| data | use | string | Handle at-least-once message delivery. |
| data | failure_to_test | string | Deliver the same job twice. |
Fetch the same bytes
The static files contain the same reference rows as the API, with no API rate limit. Live metadata, such as view counts, may differ. Use the API for content negotiation; use the static files for a cacheable snapshot.
curl "https://yjtoon.com/api/dataset/reliable-background-jobs?format=toon"
const res = await fetch( "https://yjtoon.com/static-data/dataset/reliable-background-jobs.toon" ); const toon = await res.text();
Rate limit: 120 requests per minute per IP, no key and no signup. API reference →
Sources and editorial notes
Original editorial reference notes by YJTOON, informed by the linked primary documentation. External specifications retain their own licenses.
Editorial review: . Source documents keep their own licenses.