sbt-benchmark
Use PROACTIVELY for JMH benchmarks in this Spark protobuf project. Handles sbt commands, always cleans before benchmarking, saves output to /tmp logs, and uses log-reader agent to parse results.
When & Why to Use This Skill
This Claude skill automates JMH (Java Microbenchmark Harness) performance testing within Spark protobuf projects. It optimizes the benchmarking lifecycle by managing complex sbt commands, enforcing clean builds to eliminate incremental compilation errors, and integrating log-parsing agents for precise performance data extraction and analysis.
Use Cases
- Parser Performance Comparison: Execute side-by-side JMH benchmarks to compare the efficiency of different Scala parser implementations, such as inline vs. generated wire format parsers.
- Automated Regression Testing: Proactively run specific performance benchmarks after code modifications to ensure no latency regressions are introduced in the Spark protobuf backport.
- Background Benchmark Monitoring: Launch long-running performance tests in the background using automated monitoring scripts that provide real-time progress updates and parse results upon completion.
- Build Environment Optimization: Ensure reliable benchmarking results by automatically executing 'sbt clean' and managing shaded JAR builds via 'sbt assembly' before test execution.
| name | sbt-benchmark |
|---|---|
| description | Use PROACTIVELY for JMH benchmarks in this Spark protobuf project. Handles sbt commands, always cleans before benchmarking, saves output to /tmp logs, and uses log-reader agent to parse results. |
| allowed-tools | Bash, Glob, Grep, Read, Edit, AskUserQuestion, Task, BashOutput, KillShell |
Expert Scala/sbt build engineer specializing in JMH benchmarking for the Spark protobuf backport project.
Workflow
1. Clarify Scope
If ambiguous, examine bench/ and ask which benchmarks to run. Never run all benchmarks without asking.
2. Execute
CRITICAL: Always sbt clean before benchmarks to avoid incremental compilation errors.
# Specific benchmark
sbt clean bench/Test/compile 'bench/Jmh/run .*<BenchmarkName>.*' | tee /tmp/jmh_<name>.log
# Parser comparison
sbt clean bench/Test/compile 'bench/Jmh/run .*Scalar.*(anInlineParser|generatedWireFormatParser).*' | tee /tmp/jmh_scalar_comparison.log
# All benchmarks (ask first)
sbt clean jmh | tee /tmp/jmh_all.log
Run in background (run_in_background: true) and monitor with .claude/skills/sbt-benchmark/scripts/monitor-jmh.sh -t <timeout> /tmp/jmh_<name>.log (prefer using 30s for first monitor, then ETA). Do NOT use BashOutput.
3. Parse and Present
Use log-reader agent to parse /tmp/jmh_<name>.log and extract result table. Present raw JMH results without interpretation unless anomalies detected (noise, errors, warnings).
sbt Commands
Key commands (see build.sbt for complete list):
sbt clean- Required before JMHsbt jmh/sbt jmhQuick- Full/quick benchmarkssbt unitTests/propertyTests/integrationTests/allTestTiers- Test tierssbt assembly- Build shaded JAR
Benchmark Parameters
To discover parameters: Read bench/ source files, look for @Param annotations, @Benchmark methods, and @State classes. Use -p paramName=value for custom values.
Performance Analysis
Only when user requests. Use unambiguous language: "1.5x speedup" (old/new), "33% reduction", "2000ns → 1500ns". Not: "1.5x faster" (ambiguous).
Monitoring
- Live:
.claude/skills/sbt-benchmark/scripts/monitor-jmh.sh -t <timeout> /tmp/jmh.log- Real-time progress, auto-exits on completion/timeout (e.g.,-t 30for 30s) - Status:
.claude/skills/sbt-benchmark/scripts/jmh-status.sh /tmp/jmh.log- One-shot status check