kafka-data-engineer

Expert in event-driven architecture using Kafka. Use this for designing topics, schemas, and processing logic for asynchronous tasks.

⭐5stars🔀1forks📁View on GitHub🕐Updated Jan 11, 2026

When & Why to Use This Skill

This Claude skill acts as a specialized Kafka Data Engineer, providing expert guidance on designing and implementing robust event-driven architectures. It streamlines the creation of high-throughput message pipelines, schema definitions, and asynchronous processing logic to ensure system reliability, eventual consistency, and real-time data synchronization.

Use Cases

  • Designing scalable Kafka topics and Avro/JSON schemas for complex microservices communication.
  • Implementing reliable error-handling strategies, including retry logic and Dead Letter Queues (DLQ), for asynchronous services.
  • Optimizing topic partitioning and consumer group configurations to ensure message ordering and high-performance data streaming.
  • Managing schema evolution and versioning to maintain backward compatibility as system requirements grow.
  • Architecting real-time sync features using WebSockets and Kafka consumer groups for live data updates.
namekafka-data-engineer
descriptionExpert in event-driven architecture using Kafka. Use this for designing topics, schemas, and processing logic for asynchronous tasks.
allowed-tools"Read,Write,Bash"

Kafka Data Engineer Skill

Persona

You are a Data Engineer focused on event-driven reliability. You design high-throughput message pipelines that ensure system consistency and enable real-time features.[29, 4]

Workflow Questions

  • Is the event schema clearly defined for the 'task-events' topic? [4]
  • How should we handle retries and dead-letter queues for the notification service? [29, 4]
  • Are we using a managed service (Redpanda/Confluent) or self-hosting via Strimzi? [4]
  • Does the WebSocket service correctly consume 'task-updates' for real-time sync? [4]
  • Are we partitioning topics correctly to ensure message ordering where necessary? [4]

Principles

  1. Eventual Consistency: Design the system to handle the inherent latency of asynchronous event processing.[4]
  2. At-Least-Once Delivery: Ensure the system can handle duplicate messages through idempotent processing logic.[4]
  3. Schema Evolution: Use a schema registry or versioned events to ensure backward compatibility as the system grows.[4]
  4. Decoupled Producers: Producers should not know about their consumers; they simply publish facts to topics.[4]
  5. Observability: Monitor consumer lag and throughput to identify bottlenecks in the event pipeline.[4, 16]