Use when designing data platforms, choosing between data lakes/lakehouses/warehouses, or implementing data mesh patterns. Covers modern data architecture approaches.
This Claude skill provides expert guidance on designing modern data platforms, enabling users to navigate complex decisions between data lakes, warehouses, and lakehouses. It covers essential patterns like Data Mesh, Medallion architecture, and advanced data modeling to ensure scalable, high-quality data infrastructure and optimized storage strategies.
Use Cases
Strategic Decision Making: Compare and select the optimal storage strategy (Lake vs. Warehouse vs. Lakehouse) based on specific business needs, query performance requirements, and cost constraints.
Data Mesh Implementation: Design a decentralized data architecture that promotes domain ownership and treats data as a product to eliminate organizational bottlenecks and improve scalability.
Infrastructure Optimization: Apply Medallion architecture principles (Bronze, Silver, Gold layers) to structure data pipelines for maximum reliability, data quality, and governance.
Technology Selection: Evaluate and choose between modern table formats like Delta Lake, Apache Iceberg, and Apache Hudi for ACID compliance, schema evolution, and time-travel capabilities.
Data Modeling: Transition from traditional star schemas to flexible, auditable patterns like Data Vault for complex, enterprise-grade data environments.
name
data-architecture
description
Use when designing data platforms, choosing between data lakes/lakehouses/warehouses, or implementing data mesh patterns. Covers modern data architecture approaches.
allowed-tools
Read, Glob, Grep
Data Architecture
Modern data architecture patterns including data lakes, lakehouses, data mesh, and data platform design.
When to Use This Skill
Choosing between data lake, warehouse, and lakehouse
Designing a modern data platform
Implementing data mesh principles
Planning data storage strategy
Understanding data architecture trade-offs
Data Architecture Evolution
Generation 1: Data Warehouse (1990s-2000s)
- Structured data only
- ETL into warehouse
- Star/snowflake schemas
- SQL-based analytics
Generation 2: Data Lake (2010s)
- All data types (structured, semi, unstructured)
- Schema-on-read
- Hadoop/HDFS based
- Cheap storage, complex processing
Generation 3: Lakehouse (2020s)
- Best of both: lake flexibility + warehouse features
- ACID transactions on lake
- Schema enforcement optional
- Unified analytics and ML
Architecture Comparison
Data Warehouse
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Sources │ ──► │ ETL │ ──► │ Warehouse │
│ (Structured)│ │ (Transform) │ │ (Star/Snow) │
└─────────────┘ └─────────────┘ └─────────────┘
│
▼
┌─────────────┐
│ BI │
│ Analytics │
└─────────────┘
Characteristics:
- Schema-on-write
- Optimized for SQL queries
- Structured data only
- High data quality
- Expensive storage
Best for:
- Business intelligence
- Financial reporting
- Structured analytics
Data Lake
┌─────────────┐ ┌─────────────┐
│ Sources │ ──► │ Data Lake │
│ (All) │ │ (Raw) │
└─────────────┘ └─────────────┘
│
┌────────────────┼────────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ ML │ │ ETL │ │ Spark │
│ Training│ │ to DW │ │ Analysis│
└─────────┘ └─────────┘ └─────────┘
Characteristics:
- Schema-on-read
- All data types
- Cheap storage
- Flexible processing
- Risk of "data swamp"
Best for:
- Data science/ML
- Unstructured data
- Experimental analysis
Data Lakehouse
┌─────────────┐ ┌─────────────────────────────────┐
│ Sources │ ──► │ Data Lakehouse │
│ (All) │ │ ┌──────────────────────────┐ │
└─────────────┘ │ │ Metadata Layer │ │
│ │ (Delta/Iceberg/Hudi) │ │
│ └──────────────────────────┘ │
│ ┌──────────────────────────┐ │
│ │ Storage Layer │ │
│ │ (Object Storage) │ │
│ └──────────────────────────┘ │
└─────────────────────────────────┘
│
┌────────────────────┼────────────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ SQL │ │ ML │ │ Stream │
│ BI │ │ Workload│ │ Process │
└─────────┘ └─────────┘ └─────────┘
Characteristics:
- ACID transactions
- Schema evolution
- Time travel
- Unified batch/streaming
- Open formats
Best for:
- Unified analytics
- Both BI and ML
- Modern data platforms
Architecture Selection Guide
Factor
Warehouse
Lake
Lakehouse
Data types
Structured
All
All
Query performance
Excellent
Poor-Medium
Good
Data quality
High
Variable
Configurable
Cost
High
Low
Medium
ML workloads
Limited
Excellent
Excellent
Real-time
Limited
Good
Good
Governance
Strong
Weak
Strong
Complexity
Low
High
Medium
Decision Tree:
Is data mostly structured with BI focus?
├── Yes → Data Warehouse
└── No
└── Need ML + BI on same data?
├── Yes → Lakehouse
└── No
└── Primarily ML/unstructured?
├── Yes → Data Lake
└── No → Lakehouse
Data Mesh = Decentralized data architecture
Four Principles:
1. Domain Ownership
- Data owned by domain teams
- Not centralized data team
2. Data as a Product
- Treat data like a product
- Quality, discoverability, usability
3. Self-Serve Platform
- Platform enables domain teams
- Reduces friction
4. Federated Governance
- Global standards
- Local implementation
Landing Zone: Raw files from sources
Raw Zone: Structured raw data
Curated Zone: Transformed, quality-checked
Consumption Zone: Ready for analytics
Sandbox Zone: Exploration and experimentation