api: YAML JSON TOON Database
version: 1.0.0
format: json
dataset:
  id: 1016
  slug: reliable-background-jobs
  title: Reliable Background Job Patterns
  description: Queue and worker design patterns for duplicate deliveries, failure recovery, concurrency, and bounded work.
  summary: Queue and worker design patterns for duplicate deliveries, failure recovery, concurrency, and bounded work. Each row includes a practical application and a failure to check before using it.
  use_when:
    - Design a worker that can safely retry after a crash.
    - Separate delivery guarantees from business effects.
  caveats:
    - Exactly-once delivery claims do not establish exactly-once external effects.
    - Patterns must be implemented with the database and queue’s real atomicity guarantees.
  related:
    - observability-signals
    - concurrency-design-patterns
  category: Design Patterns
  category_slug: design-patterns
  tags: design-patterns,reliable background jobs,reference,engineering
  view_count: 0
  created_at: 1791798640
  updated_at: 1791798640
  reviewed_at: "2026-10-12"
  sources:
    - title: AWS SQS delivery semantics
      url: "https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html"
    - title: AWS transactional outbox pattern
      url: "https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html"
  reading:
    - background-jobs-idempotency-and-the-outbox-pattern
  provenance: Original editorial reference notes by YJTOON, informed by the linked primary documentation. External specifications retain their own licenses.
  entry_count: 10
  fields:
    - section: data
      field: pattern
      type: string
      example: Idempotent handler
    - section: data
      field: mechanism
      type: string
      example: Make repeated application produce the intended …
    - section: data
      field: use
      type: string
      example: Handle at-least-once message delivery.
    - section: data
      field: failure_to_test
      type: string
      example: Deliver the same job twice.
data:
  - pattern: Idempotent handler
    mechanism: Make repeated application produce the intended single result.
    use: Handle at-least-once message delivery.
    failure_to_test: Deliver the same job twice.
  - pattern: Deduplication key
    mechanism: Persist a stable key with the committed effect.
    use: Prevent repeating one business operation.
    failure_to_test: Crash between recording the key and committing the effect.
  - pattern: Transactional outbox
    mechanism: Commit the domain change and outgoing record together.
    use: Keep a database write aligned with event publication.
    failure_to_test: Crash after commit but before publication.
  - pattern: Visibility timeout
    mechanism: Lease a job for a bounded processing interval.
    use: Recover work from a failed worker.
    failure_to_test: Run a job longer than the lease.
  - pattern: Lease heartbeat
    mechanism: Extend a lease while an owned job is progressing.
    use: Support long-running bounded tasks.
    failure_to_test: Lose connectivity during an extension.
  - pattern: Exponential backoff
    mechanism: Increase delay between repeated failures with jitter.
    use: Reduce pressure on a failing dependency.
    failure_to_test: Make many workers fail simultaneously.
  - pattern: Dead-letter queue
    mechanism: Move repeatedly failing jobs aside with evidence.
    use: Keep a poison job from blocking useful work.
    failure_to_test: Replay a corrected job without duplicating effects.
  - pattern: Concurrency limit
    mechanism: Bound active jobs per resource or tenant.
    use: Protect downstream capacity.
    failure_to_test: Enqueue a burst above steady-state throughput.
  - pattern: Cancellation
    mechanism: Check a cooperative cancellation signal at safe points.
    use: Stop obsolete or user-cancelled work.
    failure_to_test: Cancel while an external side effect is in flight.
  - pattern: Checkpoint
    mechanism: Persist completed progress under a versioned contract.
    use: Resume a long task after a crash.
    failure_to_test: Resume with a changed code or data version.