spark-docker
Manage Docker Spark infrastructure for local development or production (S3). Use this skill to start, stop, or check status of Spark clusters. Invoke with /spark-docker.
When & Why to Use This Skill
The spark-docker skill provides a robust framework for managing Docker-based Apache Spark infrastructure, optimized for both local development and production-grade S3 data warehousing. It streamlines the lifecycle of Spark clusters—including starting, stopping, and status monitoring—while offering pre-configured support for Spark 3.5.5, RDS MySQL metastores, and S3 integration to ensure high-performance data processing.
Use Cases
- Development Environment Setup: Rapidly deploy a local Spark 3.5.5 cluster using Docker Compose to test data engineering pipelines before production.
- Cloud Data Warehousing: Manage Spark clusters that utilize AWS S3 as a primary data store and RDS for persistent Hive metastore management.
- Infrastructure Monitoring: Use simple commands to check cluster health, inspect container logs, and access the Spark-SQL shell for ad-hoc data querying.
- Performance Optimization: Apply expert troubleshooting configurations to resolve common Spark issues like Out-of-Memory (OOM) errors and database locking conflicts.
| name | spark-docker |
|---|---|
| description | Manage Docker Spark infrastructure for local development or production (S3). Use this skill to start, stop, or check status of Spark clusters. Invoke with /spark-docker. |
Spark Docker Management
This skill manages the Docker-based Spark infrastructure for the wc_simd project.
Use
spark_docker_s3/for all Spark work. The HDFS-basedspark_docker/is deprecated.
Primary Stack: spark_docker_s3/
- Spark 3.5.5 + S3 warehouse + RDS MySQL metastore
- Uses EC2 instance profile for S3 access
- Configure
.envfrom.env.examplewith S3 bucket and RDS credentials
Commands
Start Spark
cd spark_docker_s3 && docker compose up -d --build
First time setup requires INIT_HIVE_SCHEMA=true in .env to create metastore tables.
Stop Spark
cd spark_docker_s3 && docker compose down
Check Status
cd spark_docker_s3 && docker compose ps
View Logs
cd spark_docker_s3 && docker compose logs -f spark
Access Spark Shell
docker exec -it spark /opt/spark/bin/spark-sql
Troubleshooting
OOM Errors
Add to Spark config:
spark.sql.orc.enableVectorizedReader=falsespark.sql.parquet.columnarReaderBatchSize=256
Derby Lock Issues
Use MySQL-backed metastore (already configured in Docker stacks).
Host Service Access from Spark
Use Docker gateway IP 172.19.0.1 instead of localhost.
Deprecated: spark_docker/ (HDFS-based)
DEPRECATED: This stack is no longer maintained. Use
spark_docker_s3/instead.
The old HDFS-based local stack in spark_docker/ required:
127.0.0.1 hadoop-namenodein/etc/hosts- Local HDFS storage (not portable across machines)