sbt-benchmark

Use PROACTIVELY for JMH benchmarks in this Spark protobuf project. Handles sbt commands, always cleans before benchmarking, saves output to /tmp logs, and uses log-reader agent to parse results.

⭐5stars🔀1forks📁View on GitHub🕐Updated Jan 11, 2026

When & Why to Use This Skill

This Claude skill automates JMH (Java Microbenchmark Harness) performance testing within Spark protobuf projects. It optimizes the benchmarking lifecycle by managing complex sbt commands, enforcing clean builds to eliminate incremental compilation errors, and integrating log-parsing agents for precise performance data extraction and analysis.

Use Cases

  • Parser Performance Comparison: Execute side-by-side JMH benchmarks to compare the efficiency of different Scala parser implementations, such as inline vs. generated wire format parsers.
  • Automated Regression Testing: Proactively run specific performance benchmarks after code modifications to ensure no latency regressions are introduced in the Spark protobuf backport.
  • Background Benchmark Monitoring: Launch long-running performance tests in the background using automated monitoring scripts that provide real-time progress updates and parse results upon completion.
  • Build Environment Optimization: Ensure reliable benchmarking results by automatically executing 'sbt clean' and managing shaded JAR builds via 'sbt assembly' before test execution.
namesbt-benchmark
descriptionUse PROACTIVELY for JMH benchmarks in this Spark protobuf project. Handles sbt commands, always cleans before benchmarking, saves output to /tmp logs, and uses log-reader agent to parse results.
allowed-toolsBash, Glob, Grep, Read, Edit, AskUserQuestion, Task, BashOutput, KillShell

Expert Scala/sbt build engineer specializing in JMH benchmarking for the Spark protobuf backport project.

Workflow

1. Clarify Scope

If ambiguous, examine bench/ and ask which benchmarks to run. Never run all benchmarks without asking.

2. Execute

CRITICAL: Always sbt clean before benchmarks to avoid incremental compilation errors.

# Specific benchmark
sbt clean bench/Test/compile 'bench/Jmh/run .*<BenchmarkName>.*' | tee /tmp/jmh_<name>.log

# Parser comparison
sbt clean bench/Test/compile 'bench/Jmh/run .*Scalar.*(anInlineParser|generatedWireFormatParser).*' | tee /tmp/jmh_scalar_comparison.log

# All benchmarks (ask first)
sbt clean jmh | tee /tmp/jmh_all.log

Run in background (run_in_background: true) and monitor with .claude/skills/sbt-benchmark/scripts/monitor-jmh.sh -t <timeout> /tmp/jmh_<name>.log (prefer using 30s for first monitor, then ETA). Do NOT use BashOutput.

3. Parse and Present

Use log-reader agent to parse /tmp/jmh_<name>.log and extract result table. Present raw JMH results without interpretation unless anomalies detected (noise, errors, warnings).

sbt Commands

Key commands (see build.sbt for complete list):

  • sbt clean - Required before JMH
  • sbt jmh / sbt jmhQuick - Full/quick benchmarks
  • sbt unitTests / propertyTests / integrationTests / allTestTiers - Test tiers
  • sbt assembly - Build shaded JAR

Benchmark Parameters

To discover parameters: Read bench/ source files, look for @Param annotations, @Benchmark methods, and @State classes. Use -p paramName=value for custom values.

Performance Analysis

Only when user requests. Use unambiguous language: "1.5x speedup" (old/new), "33% reduction", "2000ns → 1500ns". Not: "1.5x faster" (ambiguous).

Monitoring

  • Live: .claude/skills/sbt-benchmark/scripts/monitor-jmh.sh -t <timeout> /tmp/jmh.log - Real-time progress, auto-exits on completion/timeout (e.g., -t 30 for 30s)
  • Status: .claude/skills/sbt-benchmark/scripts/jmh-status.sh /tmp/jmh.log - One-shot status check