etl-designer
Design ETL/ELT pipelines with proper orchestration, error handling, and monitoring. Use when building data pipelines, designing data workflows, or implementing data transformations.
When & Why to Use This Skill
The ETL Designer Claude skill is a specialized assistant for architecting robust ETL/ELT pipelines and data workflows. It provides expert guidance on implementing scalable data transformations, workflow orchestration with Apache Airflow, and ensuring system reliability through idempotent operations and comprehensive error handling.
Use Cases
- Automating Data Integration: Designing end-to-end pipelines to extract data from various sources and load it into centralized data warehouses like Snowflake or BigQuery.
- Optimizing Pipeline Performance: Implementing incremental loading strategies and idempotent logic to handle large-scale datasets efficiently while minimizing resource consumption.
- Building Resilient Workflows: Developing sophisticated error-handling, retry mechanisms, and monitoring alerts within Airflow DAGs to ensure high data availability and pipeline health.
| name | etl-designer |
|---|---|
| description | Design ETL/ELT pipelines with proper orchestration, error handling, and monitoring. Use when building data pipelines, designing data workflows, or implementing data transformations. |
ETL Designer
Design robust ETL/ELT pipelines for data processing.
Quick Start
Use Airflow for orchestration, implement idempotent operations, add error handling, monitor pipeline health.
Instructions
Airflow DAG Structure
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'data-team',
'retries': 3,
'retry_delay': timedelta(minutes=5),
'email_on_failure': True,
'email': ['alerts@company.com']
}
with DAG(
'etl_pipeline',
default_args=default_args,
schedule_interval='0 2 * * *', # Daily at 2 AM
start_date=datetime(2024, 1, 1),
catchup=False
) as dag:
extract = PythonOperator(
task_id='extract_data',
python_callable=extract_from_source
)
transform = PythonOperator(
task_id='transform_data',
python_callable=transform_data
)
load = PythonOperator(
task_id='load_to_warehouse',
python_callable=load_to_warehouse
)
extract >> transform >> load
Incremental Processing
def extract_incremental(last_run_date):
query = f"""
SELECT * FROM source_table
WHERE updated_at > '{last_run_date}'
"""
return pd.read_sql(query, conn)
Error Handling
def safe_transform(data):
try:
transformed = transform_data(data)
return transformed
except Exception as e:
logger.error(f"Transform failed: {e}")
send_alert(f"Pipeline failed: {e}")
raise
Best Practices
- Make operations idempotent
- Use incremental processing
- Implement proper error handling
- Add monitoring and alerts
- Use data quality checks
- Document pipeline logic