spark-docker
Manage Docker Spark infrastructure for local development or production (S3). Use this skill to start, stop, or check status of Spark clusters. Invoke with /spark-docker.
📖 中文导读:Manage Docker Spark infrastructure for local development or production (S3). Use this skill to start, stop, or check status of Spark clusters. Invoke with /spark-docker. 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。
name: spark-docker description: Manage Docker Spark infrastructure for local development or production (S3). Use this skill to start, stop, or check status of Spark clusters. Invoke with /spark-docker.
Spark Docker Management
This skill manages the Docker-based Spark infrastructure for the wc_simd project.
Use
spark_docker_s3/for all Spark work. The HDFS-basedspark_docker/is deprecated.
Primary Stack: spark_docker_s3/
- Spark 3.5.5 + S3 warehouse + RDS MySQL metastore
- Uses EC2 instance profile for S3 access
- Configure
.envfrom.env.examplewith S3 bucket and RDS credentials
Commands
Start Spark
cd spark_docker_s3 && docker compose up -d --build
First time setup requires INIT_HIVE_SCHEMA=true in .env to create metastore tables.
Stop Spark
cd spark_docker_s3 && docker compose down
Check Status
cd spark_docker_s3 && docker compose ps
View Logs
cd spark_docker_s3 && docker compose logs -f spark
Access Spark Shell
docker exec -it spark /opt/spark/bin/spark-sql
Troubleshooting
OOM Errors
Add to Spark config:
spark.sql.orc.enableVectorizedReader=falsespark.sql.parquet.columnarReaderBatchSize=256
Derby Lock Issues
Use MySQL-backed metastore (already configured in Docker stacks).
Host Service Access from Spark
Use Docker gateway IP 172.19.0.1 instead of localhost.
Deprecated: spark_docker/ (HDFS-based)
DEPRECATED: This stack is no longer maintained. Use
spark_docker_s3/instead.
The old HDFS-based local stack in spark_docker/ required:
127.0.0.1 hadoop-namenodein/etc/hosts- Local HDFS storage (not portable across machines)