spark-sql-optimizer
Spark Sql Optimizer - Auto-activating skill for Data Pipelines.Triggers on: spark sql optimizer, spark sql optimizerPart of the Data Pipelines skill category.
When & Why to Use This Skill
The Spark SQL Optimizer skill is a specialized AI assistant designed to enhance data engineering efficiency by providing automated guidance for query optimization and performance tuning. It streamlines the development of high-performance data pipelines by generating production-ready Spark SQL code, implementing industry best practices for ETL processes, and validating configurations against common data engineering standards.
Use Cases
- Query Performance Tuning: Optimizing slow-running Spark SQL queries to reduce execution time and resource consumption by addressing bottlenecks like data skew and inefficient joins.
- ETL Pipeline Development: Generating optimized Spark code and configurations for complex data transformation tasks and workflow orchestration within data engineering environments.
- Best Practice Implementation: Providing step-by-step guidance on applying advanced Spark features such as broadcast joins, partitioning strategies, and caching to ensure scalable data processing.
- Code Validation and Review: Validating Spark SQL implementations against industry standards to ensure reliability and maintainability in production-grade data pipelines.
| name | spark-sql-optimizer |
|---|---|
| description | | |
| Triggers on | spark sql optimizer, spark sql optimizer |
| allowed-tools | Read, Write, Edit, Bash, Grep |
| version | 1.0.0 |
| license | MIT |
| author | Jeremy Longshore <jeremy@intentsolutions.io> |
Spark Sql Optimizer
Purpose
This skill provides automated assistance for spark sql optimizer tasks within the Data Pipelines domain.
When to Use
This skill activates automatically when you:
- Mention "spark sql optimizer" in your request
- Ask about spark sql optimizer patterns or best practices
- Need help with data pipeline skills covering etl, data transformation, workflow orchestration, and streaming data processing.
Capabilities
- Provides step-by-step guidance for spark sql optimizer
- Follows industry best practices and patterns
- Generates production-ready code and configurations
- Validates outputs against common standards
Example Triggers
- "Help me with spark sql optimizer"
- "Set up spark sql optimizer"
- "How do I implement spark sql optimizer?"
Related Skills
Part of the Data Pipelines skill category. Tags: etl, airflow, spark, streaming, data-engineering