bench
Run benchmark and analyze PDF parsing performance
When & Why to Use This Skill
This Claude skill automates the benchmarking and performance analysis of PDF parsing engines. It streamlines the evaluation of document extraction quality by measuring critical metrics such as reading order (NID), table structure (TEDS), and heading accuracy (MHS). By comparing real-time results against established thresholds, it provides automated regression detection to ensure the reliability of document processing pipelines.
Use Cases
- Regression Testing: Automatically verify if recent code changes have negatively impacted the accuracy of PDF data extraction or table recognition.
- Performance Optimization: Measure and analyze processing speed (seconds per document) to identify bottlenecks in the document parsing workflow.
- Quality Assurance: Validate PDF parsing models against specific document IDs to ensure high-precision extraction for complex layouts and heading structures.
- Threshold Compliance: Compare current extraction metrics against predefined performance benchmarks to determine if a build is ready for production deployment.
| name | bench |
|---|---|
| description | Run benchmark and analyze PDF parsing performance |
/bench
Builds Java and runs the full benchmark suite, then analyzes results.
Execution Steps
- Run
./scripts/bench.sh - Extract metrics from
tests/benchmark/prediction/opendataloader/evaluation.jsonusing:jq '{summary, metrics, table_detection, speed}' tests/benchmark/prediction/opendataloader/evaluation.json - Output summary:
- NID (reading order)
- TEDS (table structure)
- MHS (heading structure)
- Table Detection F1/Precision/Recall
- Speed (s/doc, total time)
- Compare with thresholds in
tests/benchmark/thresholds.json - Warn if regression detected
Options
/bench --doc-id 01030000000189- Run for a specific document only
Notes
- Benchmark won't run if Java build fails
- First run installs Python dependencies via uv sync