grading
Evaluates document relevance to queries using binary scoring (yes/no) with reasoning. Use when filtering retrieved documents, determining relevance for answer generation, or when the user mentions grading, filtering, or document quality.
When & Why to Use This Skill
The Document Grading skill provides an automated mechanism to evaluate the relevance of retrieved documents against specific queries. By utilizing binary scoring and structured reasoning, it ensures that only high-quality, pertinent information is used for answer generation, effectively reducing hallucinations and enabling adaptive retrieval workflows in RAG (Retrieval-Augmented Generation) systems.
Use Cases
- RAG Pipeline Optimization: Automatically filter out noise and irrelevant documents retrieved from vector databases like ChromaDB to improve the accuracy of AI-generated answers.
- Adaptive Retrieval Logic: Trigger query rewriting and retry mechanisms when initial retrieval fails to produce relevant documents, ensuring robust information discovery for vague queries.
- Quality Assurance for Knowledge Bases: Audit and grade the relevance of internal documentation against common user queries to identify gaps or low-quality content in the knowledge base.
- Transparent Decision Making: Provide clear, AI-generated reasoning for document inclusion or exclusion, enhancing the interpretability and debuggability of agentic workflows.
| name | grading |
|---|---|
| description | Evaluates document relevance to queries using binary scoring (yes/no) with reasoning. Use when filtering retrieved documents, determining relevance for answer generation, or when the user mentions grading, filtering, or document quality. |
Document Grading
Instructions
Grade documents for relevance using the functions in components/grader.py. The grading uses Claude API with structured outputs for consistent binary decisions.
Default workflow:
- Retrieve documents from ChromaDB or web search
- Call
grade_documents(query, documents)to grade all documents - Call
filter_relevant_documents(graded_docs)to get only relevant ones - Check if
len(relevant_docs) == 0to trigger retry logic
Key functions:
# Grade multiple documents (preferred)
graded_docs = grade_documents(query, documents)
# Filter to relevant only
relevant_docs = filter_relevant_documents(graded_docs)
# Grade single document (if needed)
result = grade_document(query, document)
Grading returns:
relevant: Boolean (True/False)reasoning: Explanation of decision
Critical for adaptive retry: When ALL documents are graded as not relevant (len(relevant_docs) == 0), the pipeline triggers query rewriting and retry.
Implementation: components/grader.py, uses GRADING_MODEL from config.py (default: Haiku 4.5), temperature 0.
Examples
Example 1: Standard grading workflow
from components.grader import grade_documents, filter_relevant_documents
from tools.chromadb_tool import query_chromadb
# Input
docs = query_chromadb("gaming laptops", top_k=5)
# Retrieved: 5 documents from catalog
# Grade
graded = grade_documents("gaming laptops", docs)
relevant = filter_relevant_documents(graded)
# Output
# Found 3 relevant documents out of 5 total
# Each has grading_result: {relevant: True, reasoning: "..."}
Example 2: Triggering retry
# Input: Vague query
docs = query_chromadb("fast computer", top_k=5)
graded = grade_documents("fast computer", docs)
relevant = filter_relevant_documents(graded)
# Output
# len(relevant) == 0
# → Triggers query rewrite and retry
Example 3: In adaptive RAG pipeline
# After retrieval
graded_documents = grade_documents(current_query, retrieved_documents)
relevant_documents = filter_relevant_documents(graded_documents)
if len(relevant_documents) == 0:
# No relevant docs - rewrite query and retry
if num_retries < RETRY_LIMIT:
rewrite_result = rewrite_query(current_query, previous_context)
current_query = rewrite_result['rewritten_query']
num_retries += 1
# Loop back to retrieval
else:
# Found relevant docs - proceed to generation
generate_answer(original_query, relevant_documents)