bench

Run benchmark and analyze PDF parsing performance

⭐820stars🔀43forks📁View on GitHub🕐Updated Jan 9, 2026⬇️1downloads

When & Why to Use This Skill

This Claude skill automates the benchmarking and performance analysis of PDF parsing engines. It streamlines the evaluation of document extraction quality by measuring critical metrics such as reading order (NID), table structure (TEDS), and heading accuracy (MHS). By comparing real-time results against established thresholds, it provides automated regression detection to ensure the reliability of document processing pipelines.

Use Cases

  • Regression Testing: Automatically verify if recent code changes have negatively impacted the accuracy of PDF data extraction or table recognition.
  • Performance Optimization: Measure and analyze processing speed (seconds per document) to identify bottlenecks in the document parsing workflow.
  • Quality Assurance: Validate PDF parsing models against specific document IDs to ensure high-precision extraction for complex layouts and heading structures.
  • Threshold Compliance: Compare current extraction metrics against predefined performance benchmarks to determine if a build is ready for production deployment.
namebench
descriptionRun benchmark and analyze PDF parsing performance

/bench

Builds Java and runs the full benchmark suite, then analyzes results.

Execution Steps

  1. Run ./scripts/bench.sh
  2. Extract metrics from tests/benchmark/prediction/opendataloader/evaluation.json using:
    jq '{summary, metrics, table_detection, speed}' tests/benchmark/prediction/opendataloader/evaluation.json
    
  3. Output summary:
    • NID (reading order)
    • TEDS (table structure)
    • MHS (heading structure)
    • Table Detection F1/Precision/Recall
    • Speed (s/doc, total time)
  4. Compare with thresholds in tests/benchmark/thresholds.json
  5. Warn if regression detected

Options

  • /bench --doc-id 01030000000189 - Run for a specific document only

Notes

  • Benchmark won't run if Java build fails
  • First run installs Python dependencies via uv sync