golden-dataset-validation
Validation rules, schema checks, duplicate detection, and coverage analysis for golden dataset integrity
When & Why to Use This Skill
The Golden Dataset Validation skill is designed to automate the quality assurance process for AI evaluation datasets. It ensures data integrity by performing rigorous schema checks, detecting semantic duplicates, and analyzing dataset coverage, providing a robust framework for maintaining high-quality benchmarks for AI agents and models.
Use Cases
- Automating schema and format validation for new data entries to maintain a consistent and error-free golden dataset.
- Detecting and blocking semantically similar or duplicate content to prevent data redundancy and ensure evaluation variety.
- Analyzing dataset distribution across different domains and difficulty levels to identify and fill coverage gaps.
- Verifying referential integrity between evaluation queries and source document sections to ensure benchmark reliability.
- Integrating as a pre-commit hook or CI/CD step to audit dataset quality before deploying AI evaluation pipelines.
| name | golden-dataset-validation |
|---|---|
| description | Use when validating golden dataset quality. Runs schema checks, duplicate detection, and coverage analysis to ensure dataset integrity for AI evaluation. |
| context | fork |
| agent | data-pipeline-engineer |
| version | 1.0.0 |
| author | SkillForge AI Agent Hub |
| tags | [golden-dataset, validation, integrity, schema, duplicate-detection, 2025] |
| user-invocable | false |
Golden Dataset Validation
Ensure data integrity, prevent duplicates, and maintain quality standards
Overview
This skill provides comprehensive validation patterns for the golden dataset, ensuring every entry meets quality standards before inclusion.
When to use this skill:
- Validating new documents before adding
- Running integrity checks on existing dataset
- Detecting duplicate or similar content
- Analyzing coverage gaps
- Pre-commit validation hooks
Schema Validation
Document Schema (v2.0)
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["id", "title", "source_url", "content_type", "sections"],
"properties": {
"id": {
"type": "string",
"pattern": "^[a-z0-9-]+$",
"description": "Unique kebab-case identifier"
},
"title": {
"type": "string",
"minLength": 10,
"maxLength": 200
},
"source_url": {
"type": "string",
"format": "uri",
"description": "Canonical source URL (NOT placeholder)"
},
"content_type": {
"type": "string",
"enum": ["article", "tutorial", "research_paper", "documentation", "video_transcript", "code_repository"]
},
"bucket": {
"type": "string",
"enum": ["short", "long"]
},
"tags": {
"type": "array",
"items": {"type": "string"},
"minItems": 2,
"maxItems": 10
},
"sections": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"required": ["id", "title", "content"],
"properties": {
"id": {"type": "string", "pattern": "^[a-z0-9-/]+$"},
"title": {"type": "string"},
"content": {"type": "string", "minLength": 50},
"granularity": {"enum": ["coarse", "fine", "summary"]}
}
}
}
}
}
Query Schema
{
"type": "object",
"required": ["id", "query", "difficulty", "expected_chunks", "min_score"],
"properties": {
"id": {"type": "string", "pattern": "^q-[a-z0-9-]+$"},
"query": {"type": "string", "minLength": 5, "maxLength": 500},
"modes": {"type": "array", "items": {"enum": ["semantic", "keyword", "hybrid"]}},
"category": {"enum": ["specific", "broad", "negative", "edge", "coarse-to-fine"]},
"difficulty": {"enum": ["trivial", "easy", "medium", "hard", "adversarial"]},
"expected_chunks": {"type": "array", "items": {"type": "string"}, "minItems": 1},
"min_score": {"type": "number", "minimum": 0, "maximum": 1}
}
}
Validation Rules Summary
| Rule | Purpose | Severity |
|---|---|---|
| No Placeholder URLs | Ensure real canonical URLs | Error |
| Unique Identifiers | No duplicate doc/query/section IDs | Error |
| Referential Integrity | Query chunks reference valid sections | Error |
| Content Quality | Title/content length, tag count | Warning |
| Difficulty Distribution | Balanced query difficulty levels | Warning |
Quick Reference
Duplicate Detection Thresholds
| Similarity | Action |
|---|---|
| >= 0.90 | Block - Content too similar |
| >= 0.85 | Warn - High similarity detected |
| >= 0.80 | Note - Similar content exists |
| < 0.80 | Allow - Sufficiently unique |
Coverage Requirements
| Metric | Minimum |
|---|---|
| Tutorials | >= 15% of documents |
| Research papers | >= 5% of documents |
| Domain coverage | >= 5 docs per expected domain |
| Hard queries | >= 10% of queries |
| Adversarial queries | >= 5% of queries |
Difficulty Distribution Requirements
| Level | Minimum Count |
|---|---|
| trivial | 3 |
| easy | 3 |
| medium | 5 |
| hard | 3 |
References
For detailed implementation patterns, see:
references/validation-rules.md- URL validation, ID uniqueness, referential integrity, content quality, and duplicate detection codereferences/quality-metrics.md- Coverage analysis, pre-addition validation workflow, full dataset validation, and CLI/hook integration
Related Skills
golden-dataset-curation- Quality criteria and workflowsgolden-dataset-management- Backup/restore operationspgvector-search- Embedding-based duplicate detection
Version: 1.0.0 (December 2025) Issue: #599
Capability Details
schema-validation
Keywords: schema, validation, schema check, format validation Solves:
- Validate entries against document schema
- Check required fields are present
- Verify data types and constraints
duplicate-detection
Keywords: duplicate, detection, deduplication, similarity check Solves:
- Detect duplicate or near-duplicate entries
- Use semantic similarity for fuzzy matching
- Prevent redundant entries in dataset
referential-integrity
Keywords: referential, integrity, foreign key, relationship Solves:
- Verify relationships between documents and queries
- Check source URL mappings are valid
- Ensure cross-references are consistent
coverage-analysis
Keywords: coverage, analysis, distribution, completeness Solves:
- Analyze dataset coverage across domains
- Identify gaps in difficulty distribution
- Report coverage metrics and recommendations