background-jobs-designer

Designs background job processing systems with queue integration (BullMQ/Celery), job definitions, retry policies, exponential backoff, idempotent execution, and monitoring hooks. Use when implementing "background jobs", "task queues", "async processing", or "job workers".

⭐2stars🔀0forks📁View on GitHub🕐Updated Dec 31, 2025⬇️2downloads

When & Why to Use This Skill

The Background Jobs Designer is a specialized Claude skill for architecting robust asynchronous processing systems. It provides comprehensive blueprints for integrating task queues like BullMQ and Celery, implementing exponential backoff retry policies, ensuring idempotency, and setting up real-time monitoring to build scalable and fault-tolerant backend infrastructures.

Use Cases

  • E-commerce Platforms: Offloading payment processing and order confirmation emails to background workers to ensure fast response times for customers during checkout.
  • Data Engineering: Managing long-running data synchronization tasks or complex report generation without impacting the performance of the primary web server.
  • API Reliability: Building resilient integrations with external third-party services using automated retry mechanisms and dead-letter queues to handle transient failures.
  • Scalable Infrastructure: Designing distributed worker systems that can be independently scaled to handle high-volume task processing and sudden spikes in traffic.
namebackground-jobs-designer
descriptionDesigns background job processing systems with queue integration (BullMQ/Celery), job definitions, retry policies, exponential backoff, idempotent execution, and monitoring hooks. Use when implementing "background jobs", "task queues", "async processing", or "job workers".

Background Jobs Designer

Design reliable background job processing with retries and monitoring.

Queue Integration

BullMQ (Node.js):

import { Queue, Worker } from "bullmq";

const emailQueue = new Queue("email", {
  connection: { host: "localhost", port: 6379 },
});

// Add job
await emailQueue.add(
  "send-welcome",
  {
    userId: "123",
    email: "user@example.com",
  },
  {
    attempts: 3,
    backoff: { type: "exponential", delay: 2000 },
  }
);

Celery (Python):

from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379')

@app.task(bind=True, max_retries=3)
def send_email(self, user_id, email):
    try:
        # Send email
        pass
    except Exception as exc:
        raise self.retry(exc=exc, countdown=60)

Job Definitions

export interface Job {
  id: string;
  type: string;
  payload: unknown;
  attempts: number;
  maxAttempts: number;
  createdAt: Date;
  processedAt?: Date;
  failedAt?: Date;
  error?: string;
}

export const JOB_TYPES = {
  SEND_EMAIL: "send-email",
  PROCESS_PAYMENT: "process-payment",
  GENERATE_REPORT: "generate-report",
  SYNC_DATA: "sync-data",
} as const;

Retry Strategy

// Exponential backoff
const RETRY_CONFIG = {
  maxAttempts: 5,
  delays: [
    1000, // 1 second
    5000, // 5 seconds
    30000, // 30 seconds
    300000, // 5 minutes
    1800000, // 30 minutes
  ],
};

// Worker with retry
const worker = new Worker("email", async (job) => {
  try {
    await sendEmail(job.data);
  } catch (error) {
    if (job.attemptsMade < RETRY_CONFIG.maxAttempts) {
      throw error; // Will retry
    }
    await handleFailedJob(job, error);
  }
});

Idempotent Jobs

// Track processed jobs
export const processJob = async (job: Job) => {
  // Check if already processed
  const processed = await db.query(
    "SELECT 1 FROM processed_jobs WHERE job_id = $1",
    [job.id]
  );

  if (processed.rows.length > 0) {
    console.log("Job already processed");
    return; // Idempotent
  }

  await db.transaction(async (trx) => {
    // Mark as processed
    await trx("processed_jobs").insert({ job_id: job.id });

    // Do work
    await performWork(job, trx);
  });
};

Monitoring

// Job events
worker.on("completed", (job) => {
  metrics.increment("jobs.completed", { type: job.name });
});

worker.on("failed", (job, err) => {
  metrics.increment("jobs.failed", { type: job.name });
  logger.error("Job failed", { jobId: job.id, error: err });
});

worker.on("stalled", (jobId) => {
  metrics.increment("jobs.stalled");
  logger.warn("Job stalled", { jobId });
});

Best Practices

  • Jobs should be idempotent
  • Use exponential backoff for retries
  • Set reasonable timeouts
  • Monitor queue depth
  • Dead letter queue for failed jobs
  • Log job start/completion
  • Graceful shutdown handling

Output Checklist

  • Queue setup (Redis/RabbitMQ)
  • Job type definitions
  • Retry policy with backoff
  • Idempotency tracking
  • Error handling
  • Monitoring/metrics
  • Dead letter queue
  • Graceful shutdown