understanding-flow
Use when learning walkerOS architecture, understanding data flow, or designing composable event pipelines. Covers Source→Collector→Destination pattern and separation of concerns.
When & Why to Use This Skill
This Claude skill provides a comprehensive guide to the walkerOS architecture, focusing on the Source→Collector→Destination pattern. It enables developers to design and implement modular, composable event pipelines by mastering the separation of concerns, universal push interfaces, and transformer chain configurations for robust data flow management.
Use Cases
- 1. Designing Composable Event Pipelines: Architecting modular systems where data flows seamlessly from various sources (Browser, Server, Cloud) to multiple destinations like Google Analytics or Data Warehouses.
- 2. Implementing walkerOS Core Logic: Utilizing the startFlow helper and universal push interfaces to wire up collectors and event capture mechanisms efficiently.
- 3. Configuring Transformer Chains: Setting up pre-collector and post-collector sequences to handle data validation, enrichment, and redaction at specific points in the pipeline.
- 4. Decoupling Data Systems: Applying the separation of concerns principle to swap out sources or destinations without impacting the core processing logic, ensuring high maintainability.
| name | understanding-flow |
|---|
Understanding walkerOS Flow
Overview
walkerOS follows a Source → Collector → Destination(s) architecture for composable, modular event processing.
Core principle: Separation of concerns. Each component has one job. Components are composable and replaceable.
The Flow Pattern
Sources → [Pre-Transformers] → Collector → [Post-Transformers] → Destinations
(Capture) (source.next) (Processing) (dest.before) (Delivery)
- Browser DOM - Validation - Validation - Google Analytics
- DataLayer - Enrichment - Enrichment - Meta Pixel
- Server HTTP - Redaction - Consent check - Custom API
- Cloud Functions - Routing - Data Warehouse
Key Concepts
Composability
A Flow combines components. You can:
- Use multiple sources feeding one collector
- Route events to multiple destinations
- Swap components without changing others
The Flow Type
See packages/core/src/types/flow.ts for the canonical interface.
// Conceptual structure (see source for full type)
interface Flow {
sources?: Record<string, Source>;
collector: Collector;
destinations?: Record<string, Destination>;
}
Universal Push Interface
All components communicate via push functions:
| Component | Push Signature | Purpose |
|---|---|---|
| Source | push(input) → events |
Capture external data |
| Collector | push(event) → void |
Process and route |
| Destination | push(event, context) → void |
Transform and deliver |
The elb() function is an alias for collector.push - used for component
wiring.
startFlow Helper
See packages/collector/src/flow.ts for
the startFlow function.
import { startFlow } from '@walkeros/collector';
const { collector, elb } = await startFlow({
sources: {
/* ... */
},
destinations: {
/* ... */
},
});
Separation of Concerns
| Concern | Handled By | NOT Handled By |
|---|---|---|
| Event capture | Sources | Collector, Destinations |
| Event structure | Event model | Components |
| Consent checking | Collector | Sources, Destinations |
| Transformation | Mapping system | Raw push calls |
| Delivery | Destinations | Sources, Collector |
Transformer Chains
Transformers run at two points in the pipeline, configured via next and
before:
Pre-Collector Chain
Runs after source captures event, before collector processing:
sources: {
browser: {
code: sourceBrowser,
next: 'validate' // First transformer in pre-chain
}
},
transformers: {
validate: {
code: transformerValidator,
config: { next: 'enrich' } // Chain continues
},
enrich: {
code: transformerEnrich
// No next = chain ends, event goes to collector
}
}
Post-Collector Chain
Runs after collector enrichment, before destination receives event:
destinations: {
gtag: {
code: destinationGtag,
before: 'redact' // First transformer in post-chain
}
},
transformers: {
redact: {
code: transformerRedact
// Event then goes to destination
}
}
Chain Resolution
source.next→ starts pre-collector chaintransformer.config.next→ links transformers togetherdestination.before→ starts post-collector chain per destination
Related
Skills:
- understanding-events skill - Event model
- understanding-transformers skill - Transformer interface and chaining
- understanding-destinations skill - Destination interface
- understanding-sources skill - Source interface
Package READMEs:
- packages/collector/README.md - Collector details
Source Files:
- packages/collector/src/flow.ts - startFlow implementation
Documentation:
- Website: Flow - Flow concept
- Website: Collector - Collector docs