bench-debug

Debug specific document parsing failures

⭐820stars🔀43forks📁View on GitHub🕐Updated Jan 9, 2026⬇️1downloads

When & Why to Use This Skill

The bench-debug skill is a specialized developer tool designed to diagnose and resolve complex document parsing failures. By automating the comparison between predicted Markdown outputs and ground-truth references, it provides a deep-dive analysis of structural inaccuracies in PDF extraction. It specifically targets metrics like TEDS for tables, MHS for headings, and NID for reading order, helping engineers pinpoint root causes within Java core components and suggest targeted code improvements.

Use Cases

  • Table Extraction Troubleshooting: Analyze specific documents where table boundaries or structures failed to detect, identifying necessary threshold adjustments in TableDetector.java.
  • Reading Order Optimization: Diagnose NID (Negative Information Density) score drops in multi-column layouts to improve the logical flow of extracted text.
  • Heading Hierarchy Debugging: Resolve MHS (Markdown Hierarchy Score) mismatches by comparing heading levels in the prediction against the original PDF structure.
  • Root Cause Mapping: Automatically link parsing discrepancies to specific Java classes and methods, providing a clear path for code-level fixes and impact assessment.
  • Regression Analysis: Run targeted benchmarks on problematic document IDs to verify if recent code changes have successfully resolved known parsing issues.
namebench-debug
descriptionDebug specific document parsing failures

/bench-debug

Compares parsing output with ground-truth for a specific document and analyzes failure causes.

Usage

/bench-debug 01030000000189

Execution Steps

  1. Run benchmark for the specific document

    ./scripts/bench.sh --doc-id <doc_id>
    
  2. Compare files

    • Ground-truth: tests/benchmark/ground-truth/markdown/<doc_id>.md
    • Prediction: tests/benchmark/prediction/opendataloader/markdown/<doc_id>.md
    • Original PDF: tests/benchmark/pdfs/<doc_id>.pdf
  3. Analyze differences

    • Missing/extra text locations
    • Table structure differences (TEDS score causes)
    • Heading level mismatches (MHS score causes)
    • Reading order errors (NID score causes)
  4. Identify root causes

    • Which PDF elements caused the issue
    • Which Java core components are involved
  5. Suggest improvements

    • Java classes/methods that need modification
    • Expected impact scope

Reference Files

  • ground-truth/reference.json: Per-document element info (categories, coordinates, etc.)
  • java/opendataloader-pdf-core/: Core parsing logic

Example Output

Document 01030000000189 Analysis:

Overall: 0.2763 (one of the worst performing documents)

Issues:
1. 2 of 3 tables not detected (TEDS: 0.15)
   - Table boundary detection failed
   - Related code: TableDetector.java

2. Reading order errors (NID: 0.45)
   - Multi-column layout handling failed
   - Related code: ColumnDetector.java

Recommended Actions:
- Adjust clustering threshold in TableDetector
- Improve multi-column detection logic