effect-queues-background

Queue and PubSub patterns, background fibers, and graceful shutdown. Use for decoupling producers/consumers.

⭐5stars🔀1forks📁View on GitHub🕐Updated Jan 11, 2026

When & Why to Use This Skill

This Claude skill facilitates the implementation of robust asynchronous messaging and concurrency patterns using the Effect library. It provides expert guidance on utilizing bounded queues for backpressure management, PubSub for event-driven broadcasting, and background fiber orchestration to ensure system decoupling and graceful shutdowns in high-performance applications.

Use Cases

  • Decoupling high-frequency producers from slow consumers using bounded queues to maintain system stability and prevent memory exhaustion.
  • Implementing scalable event-driven architectures where messages need to be broadcasted to multiple independent subscribers via PubSub patterns.
  • Managing long-running background tasks and concurrent loops with managed lifecycles, ensuring all fibers are safely interrupted during application shutdown.
  • Building resilient task processing pipelines that require explicit resource safety, logging, and error handling in a functional programming environment.
nameeffect-queues-background
descriptionQueue and PubSub patterns, background fibers, and graceful shutdown. Use for decoupling producers/consumers.
allowed-toolsRead, Grep, Glob, Edit, Write

Queues, PubSub & Background

When to use

  • Decoupling producers/consumers with backpressure
  • Broadcasting events to multiple subscribers
  • Running background loops with graceful shutdown

Queue (bounded)

import { Queue } from "effect"
const q = yield* Queue.bounded<string>(32)
yield* Queue.offer(q, "job")
const job = yield* Queue.take(q)

PubSub (broadcast)

import { PubSub } from "effect"
const ps = yield* PubSub.bounded<string>(32)
yield* PubSub.publish(ps, "evt")

Background Fiber

const fiber = yield* Effect.fork(loop)
yield* Fiber.interrupt(fiber)

Guidance

  • Prefer bounded queues to apply natural backpressure
  • Use multiple workers by forking consumers
  • Ensure background fibers are interrupted during shutdown

Pitfalls

  • Unbounded queues lead to memory growth
  • Silent background failures → add logging/metrics
  • Concurrency for pools and interruption
  • Time/Logging for observability of background tasks

Local Source Reference

CRITICAL: Search local Effect source before implementing

The full Effect source code is available at docs/effect-source/. Always search the actual implementation before writing Effect code.

Key Source Files

  • Queue: docs/effect-source/effect/src/Queue.ts
  • PubSub: docs/effect-source/effect/src/PubSub.ts
  • Fiber: docs/effect-source/effect/src/Fiber.ts

Example Searches

# Find Queue patterns
grep -F "bounded" docs/effect-source/effect/src/Queue.ts
grep -F "offer" docs/effect-source/effect/src/Queue.ts
grep -F "take" docs/effect-source/effect/src/Queue.ts

# Study PubSub operations
grep -F "publish" docs/effect-source/effect/src/PubSub.ts
grep -F "subscribe" docs/effect-source/effect/src/PubSub.ts

# Find background fiber patterns
grep -F "fork" docs/effect-source/effect/src/Fiber.ts
grep -F "interrupt" docs/effect-source/effect/src/Fiber.ts

# Look at Queue test examples
grep -F "Queue." docs/effect-source/effect/test/Queue.test.ts

Workflow

  1. Identify the Queue or PubSub API you need
  2. Search docs/effect-source/effect/src/Queue.ts or PubSub.ts for the implementation
  3. Study the types and backpressure patterns
  4. Look at test files for usage examples
  5. Write your code based on real implementations

Real source code > documentation > assumptions

References