definition-resource-design

colingwuyu's avatarfrom colingwuyu

Guide for designing Definition resources in OptAIC (the "Law"). Use when creating PipelineDef, StoreDef, AccessorDef, OpDef, MLModuleDef, or PortfolioOptimizerDef. Covers interface_spec, input/output schemas, compatibility rules, and embedded guardrail contracts.

⭐0stars🔀0forks📁View on GitHub🕐Updated Jan 3, 2026
[Architecture Design]

When & Why to Use This Skill

This Claude skill serves as a comprehensive architectural framework for designing 'Definition' resources within the OptAIC ecosystem. It provides standardized patterns for creating reusable plugin templates—such as PipelineDef, StoreDef, and MLModuleDef—by establishing the 'Law' or contract that instances must follow. It ensures system integrity through rigorous interface specifications, JSON schema validation, compatibility rules, and embedded guardrail contracts, making it essential for developers building scalable and reliable data or ML pipelines.

Use Cases

  • Designing standardized interface contracts for new plugin types to ensure seamless integration within the OptAIC framework.
  • Defining strict input and output schemas to automate data validation and prevent runtime errors in complex data pipelines.
  • Establishing compatibility rules for resource graphs to manage how upstream and downstream components connect securely.
  • Embedding guardrail contracts to enforce data quality and business logic constraints directly within the resource definition.
  • Creating reusable templates for machine learning modules (MLModuleDef) or optimization algorithms (PortfolioOptimizerDef) to maintain consistency across different implementations.
namedefinition-resource-design
descriptionGuide for designing Definition resources in OptAIC (the "Law"). Use when creating PipelineDef, StoreDef, AccessorDef, OpDef, MLModuleDef, or PortfolioOptimizerDef. Covers interface_spec, input/output schemas, compatibility rules, and embedded guardrail contracts.

Definition Resource Design Patterns

Guide for designing Definition resources that serve as reusable plugin templates. Definitions are the "Law" that Instances must follow.

When to Use

Apply when:

  • Creating new plugin types (PipelineDef, StoreDef, etc.)
  • Designing the interface contract a plugin must implement
  • Specifying input/output schemas for validation
  • Defining compatibility rules for upstream/downstream connections
  • Embedding guardrail contracts that enforce data quality

Core Concept: The Law

Definition resources contain the rules that Instance resources must follow:

Definition = The Law
├── interface_spec        # Abstract interface to implement
├── input_schema          # Expected input format
├── output_schema         # Expected output format
├── compatibility_rules   # What can connect upstream/downstream
└── guardrail_contracts   # Validation rules (enforced by Engine)

Definition Types

Type Purpose Example Implementations
PipelineDef Data ingestion/transform BloombergPipeline, FREDPipeline, ExpressionPipeline
StoreDef Storage backend ParquetStore, SQLiteStore, VirtualStore
AccessorDef Data access pattern SimpleAccessor, PITAccessor, FieldAccessor
OpDef Mathematical operator REF, DELTA, MEAN, CORR, RANK
OpMacroDef Saved expression User-defined formulas
MLModuleDef ML model template XGBoostModule, LSTMModule
PortfolioOptimizerDef Optimization algo MVO, HRP, BlackLitterman, RiskParity

Schema Structure

See references/schema-structure.md for complete JSON schema.

Minimal Definition

definition_metadata = {
    "interface_spec": "optaic.interfaces.BasePipeline",
    "input_schema": {"datasets": {"type": "array"}},
    "output_schema": {"type": "DataFrame", "columns": ["date", "entity", "value"]},
}

Full Definition with Contracts

definition_metadata = {
    "interface_spec": "optaic.interfaces.SignalPipeline",
    "version": "1.0",

    "input_schema": {
        "datasets": {"type": "array", "items": {"$ref": "#/DatasetInstance"}},
        "parameters": {"type": "object", "properties": {...}}
    },

    "output_schema": {
        "type": "DataFrame",
        "columns": ["date", "entity", "value"],
        "value_range": {"min": -1, "max": 1}
    },

    "compatibility_rules": {
        "upstream_types": ["DatasetInstance"],
        "downstream_types": ["SignalInstance", "BacktestInstance"]
    },

    "guardrail_contracts": [
        {"kind": "signal.bounds", "config": {"min": -1, "max": 1, "allow_nan": False}},
        {"kind": "pit.policy", "config": {"knowledge_date_required": True}},
    ]
}

Implementation Checklist

  1. Define interface_spec pointing to abstract base class
  2. Specify input_schema with JSON Schema format
  3. Specify output_schema with expected structure
  4. Define compatibility_rules for resource graph connections
  5. Embed guardrail_contracts for validation
  6. Create extension table in libs/db/models/
  7. Add to ResourceType enum

Reference Files