blockchain-data-collection-validation

terrylica's avatarfrom terrylica

Empirical validation workflow for blockchain data collection pipelines before production implementation. Use when validating data sources, testing DuckDB integration, building POC collectors, or verifying complete fetch-to-storage pipelines for blockchain data.

⭐0stars🔀0forks📁View on GitHub🕐Updated Dec 11, 2025
[Testing and QA]

When & Why to Use This Skill

This Claude skill provides a comprehensive, test-driven framework for the empirical validation of blockchain data collection pipelines. It ensures operational stability and data integrity by systematically testing RPC connectivity, schema accuracy, sustainable rate limits, and DuckDB storage integration before production deployment, helping developers avoid common pitfalls like data loss and provider throttling.

Use Cases

  • RPC Provider Benchmarking: Empirically determine the actual sustainable request-per-second (RPS) limits of blockchain providers (e.g., Alchemy, Infura) to prevent production failures.
  • DuckDB Storage Optimization: Validating batch insertion performance and implementing crash-tested CHECKPOINT patterns to ensure high-speed, reliable data persistence.
  • End-to-End POC Development: Building and verifying Proof-of-Concept collectors to confirm the complete fetch-to-storage workflow for new blockchain networks or data sources.
  • Data Quality & Schema Auditing: Systematically verifying field types and data consistency across large block ranges to ensure the pipeline meets strict architectural constraints.
nameblockchain-data-collection-validation
descriptionEmpirical validation workflow for blockchain data collection pipelines before production implementation. Use when validating data sources, testing DuckDB integration, building POC collectors, or verifying complete fetch-to-storage pipelines for blockchain data.

Blockchain Data Collection Validation

Overview

This skill provides a systematic, test-driven workflow for validating blockchain data collection pipelines before production implementation. Use when building POC collectors, validating new data sources, testing DuckDB integration, or verifying complete fetch-to-storage workflows.

Key principle: Validate every component empirically before implementation—connectivity, schema, rate limits, storage, and complete pipeline.

Validation Workflow

This skill follows a 5-step empirical validation workflow:

Step Purpose Output Success Criteria
1. Connectivity Test basic RPC access Block fetch confirmed Response <500ms, no errors
2. Schema Validate all required fields Field validation report All fields present, types correct
3. Rate Limits Find sustainable RPS Empirical rate (e.g., 5.79 RPS) 100% success over 50+ blocks
4. Pipeline Test fetch→DuckDB flow Complete pipeline working Data persisted, constraints pass
5. Decision Document findings Go/No-Go recommendation All steps passed, timeline calculated

Detailed workflow: See references/validation-workflow.md for complete step-by-step guide with code templates, testing patterns, and success criteria for each step.

Quick start: Create 01_single_block_fetch.py using template in scripts/, then iterate through steps 2-5.

DuckDB Integration Patterns

Critical patterns for data integrity:

  • CHECKPOINT requirement (crash-tested, prevents data loss)
  • Batch INSERT from DataFrame (124K blocks/sec performance)
  • CHECK constraints for schema validation
  • Storage estimates (76-100 bytes/block empirically validated)

Full guide: See references/duckdb-patterns.md for complete DuckDB integration guide with code examples, crash test results, and performance benchmarks.

Common Pitfalls

Critical mistakes to avoid: Skipping empirical rate validation, testing <50 blocks, forgetting DuckDB CHECKPOINT (data loss), ignoring CHECK constraints, and parallel fetching on free tiers.

Real-world examples: LlamaRPC 50 RPS documented → 1.37 RPS sustainable (2.7% of max), parallel fetch worked for 20 blocks → failed at 50.

Full guide: See references/common-pitfalls.md for detailed anti-patterns with problem/reality/solution format and code examples.

Scripts

POC template scripts for empirical validation:

  • poc_single_block.py - Connectivity and schema validation (Steps 1-2)
  • poc_batch_parallel_fetch.py - Parallel fetch testing (Step 3, expect failures)
  • poc_rate_limited_fetch.py - Rate-limited sequential fetch (Step 3, find sustainable rate)
  • poc_complete_pipeline.py - Complete fetch→DuckDB pipeline (Step 4)

Templates and usage: See scripts/README.md for complete code templates, usage examples, and testing progression guide.

References

Workflow Documentation

  • references/validation-workflow.md - Complete 5-step workflow with detailed guidance, code examples, and success criteria
  • references/common-pitfalls.md - Anti-patterns to avoid with problem/reality/solution format
  • references/example-workflow.md - Complete case study: Validating Alchemy for Ethereum collection

Technical Patterns

  • references/duckdb-patterns.md - DuckDB integration patterns (CHECKPOINT, batch INSERT, constraints, performance)
  • references/ethereum-collector-poc-findings.md - Ethereum collector POC case study with rate limit discovery

Scripts

  • scripts/README.md - Complete script templates and testing progression guide
  • scripts/poc_single_block.py - Connectivity and schema validation template
  • scripts/poc_batch_parallel_fetch.py - Parallel fetch testing template
  • scripts/poc_rate_limited_fetch.py - Rate-limited fetch template
  • scripts/poc_complete_pipeline.py - Complete pipeline template

Example Workflow

Case study: Validating Alchemy for Ethereum collection → ✅ GO at 5.79 RPS sustained (26 days for 13M blocks, HIGH confidence).

Full walkthrough: See references/example-workflow.md for complete step-by-step case study showing all 5 validation steps with actual test results and final decision.

When to Use This Skill

Invoke this skill when:

  • Validating a new blockchain RPC provider before implementation
  • Testing DuckDB integration for blockchain data
  • Building POC collector for new blockchain
  • Verifying complete fetch-to-storage pipeline
  • Investigating data quality issues
  • Planning production collector implementation
  • Need empirical validation before committing to architecture

This skill pairs well with:

  • blockchain-rpc-provider-research - For comparing multiple providers before validation
  • Project scratch investigations in scratch/ethereum-collector-poc/ and scratch/duckdb-batch-validation/