understanding-flow

Use when learning walkerOS architecture, understanding data flow, or designing composable event pipelines. Covers Source→Collector→Destination pattern and separation of concerns.

⭐309stars🔀16forks📁View on GitHub🕐Updated Dec 19, 2025

When & Why to Use This Skill

This Claude skill provides a comprehensive guide to the walkerOS architecture, focusing on the Source→Collector→Destination pattern. It enables developers to design and implement modular, composable event pipelines by mastering the separation of concerns, universal push interfaces, and transformer chain configurations for robust data flow management.

Use Cases

  • 1. Designing Composable Event Pipelines: Architecting modular systems where data flows seamlessly from various sources (Browser, Server, Cloud) to multiple destinations like Google Analytics or Data Warehouses.
  • 2. Implementing walkerOS Core Logic: Utilizing the startFlow helper and universal push interfaces to wire up collectors and event capture mechanisms efficiently.
  • 3. Configuring Transformer Chains: Setting up pre-collector and post-collector sequences to handle data validation, enrichment, and redaction at specific points in the pipeline.
  • 4. Decoupling Data Systems: Applying the separation of concerns principle to swap out sources or destinations without impacting the core processing logic, ensuring high maintainability.
nameunderstanding-flow

Understanding walkerOS Flow

Overview

walkerOS follows a Source → Collector → Destination(s) architecture for composable, modular event processing.

Core principle: Separation of concerns. Each component has one job. Components are composable and replaceable.

The Flow Pattern

Sources → [Pre-Transformers] → Collector → [Post-Transformers] → Destinations
(Capture)  (source.next)    (Processing) (dest.before)       (Delivery)

- Browser DOM              - Validation   - Validation        - Google Analytics
- DataLayer                - Enrichment   - Enrichment        - Meta Pixel
- Server HTTP              - Redaction    - Consent check     - Custom API
- Cloud Functions                         - Routing           - Data Warehouse

Key Concepts

Composability

A Flow combines components. You can:

  • Use multiple sources feeding one collector
  • Route events to multiple destinations
  • Swap components without changing others

The Flow Type

See packages/core/src/types/flow.ts for the canonical interface.

// Conceptual structure (see source for full type)
interface Flow {
  sources?: Record<string, Source>;
  collector: Collector;
  destinations?: Record<string, Destination>;
}

Universal Push Interface

All components communicate via push functions:

Component Push Signature Purpose
Source push(input) → events Capture external data
Collector push(event) → void Process and route
Destination push(event, context) → void Transform and deliver

The elb() function is an alias for collector.push - used for component wiring.

startFlow Helper

See packages/collector/src/flow.ts for the startFlow function.

import { startFlow } from '@walkeros/collector';

const { collector, elb } = await startFlow({
  sources: {
    /* ... */
  },
  destinations: {
    /* ... */
  },
});

Separation of Concerns

Concern Handled By NOT Handled By
Event capture Sources Collector, Destinations
Event structure Event model Components
Consent checking Collector Sources, Destinations
Transformation Mapping system Raw push calls
Delivery Destinations Sources, Collector

Transformer Chains

Transformers run at two points in the pipeline, configured via next and before:

Pre-Collector Chain

Runs after source captures event, before collector processing:

sources: {
  browser: {
    code: sourceBrowser,
    next: 'validate'  // First transformer in pre-chain
  }
},
transformers: {
  validate: {
    code: transformerValidator,
    config: { next: 'enrich' }  // Chain continues
  },
  enrich: {
    code: transformerEnrich
    // No next = chain ends, event goes to collector
  }
}

Post-Collector Chain

Runs after collector enrichment, before destination receives event:

destinations: {
  gtag: {
    code: destinationGtag,
    before: 'redact'  // First transformer in post-chain
  }
},
transformers: {
  redact: {
    code: transformerRedact
    // Event then goes to destination
  }
}

Chain Resolution

  • source.next → starts pre-collector chain
  • transformer.config.next → links transformers together
  • destination.before → starts post-collector chain per destination

Skills:

Package READMEs:

Source Files:

Documentation: