queue-observability

majiayu000's avatarfrom majiayu000

Monitor message queues, job processing, and detect backpressure

⭐0stars🔀0forks📁View on GitHub🕐Updated Jan 5, 2026⬇️1downloads
[Monitoring]

When & Why to Use This Skill

This Claude skill provides comprehensive observability for message queues and job processing systems, enabling teams to track system health and performance in real-time. It focuses on detecting backpressure, monitoring consumer lag, and identifying processing bottlenecks across platforms like Kafka and RabbitMQ. By implementing standardized metrics and distributed tracing patterns, it ensures high reliability and visibility into asynchronous decoupled architectures.

Use Cases

  • Detecting System Bottlenecks: Identify when queue depth is increasing because consumers are too slow or processing resources are insufficient.
  • Preventing Data Processing Delays: Monitor Kafka consumer lag to ensure messages are processed within acceptable timeframes and meet service level agreements (SLAs).
  • Troubleshooting High Error Rates: Track processing failures and dead-letter queues to quickly isolate poison messages or downstream service dependencies.
  • Implementing Distributed Tracing: Use producer and consumer patterns to inject and extract trace contexts, maintaining visibility as requests move across queue boundaries.
  • Backpressure Management: Detect and alert on growing queue depth to trigger auto-scaling or rate-limiting before system failure occurs.
namequeue-observability
description"Monitor message queues, job processing, and detect backpressure"
priority2

Queue Observability

Queues decouple systems but hide problems. Track depth, processing time, and lag.

Key Metrics

Metric Type Description
queue.messages.published Counter Messages sent
queue.messages.consumed Counter Messages processed
queue.process.duration Histogram Processing time
queue.process.errors Counter Processing failures
queue.depth Gauge Pending messages
queue.consumer.lag Gauge Behind by N messages

Span Attributes

Attribute Example Required
messaging.system kafka, rabbitmq Yes
messaging.destination orders, notifications Yes
messaging.operation publish, process Yes
job.type send_email Recommended

Producer Pattern

Before: Start span (kind=PRODUCER), inject trace context into headers, start timer
After:  Record duration, increment published, record errors, end span

Consumer Pattern

On receive: Extract trace context, start span (kind=CONSUMER), start timer
After:      Record duration, increment consumed, record errors, end span

Issues to Detect

Issue Detection Cause
Growing depth queue.depth increasing Consumer too slow
Consumer lag lag > 0 and increasing Crashed consumer, slow processing
High error rate errors/consumed > 5% Bad messages, downstream issues

Anti-Patterns

  • No trace context in messages → Traces break at queue boundary
  • No DLQ monitoring → Poison messages go unnoticed
  • Missing depth alerts → Backpressure surprises

References

  • references/platforms/{platform}/worker.md
  • references/patterns/retry-backoff.md