chunking-strategies

jpoutrin 的头像作者 jpoutrin

RAG系统的文档分块策略。在实现文档处理管道以根据文档类型和检索要求确定最佳分块方法时使用。

📖 中文导读:RAG系统的文档分块策略。在实现文档处理管道以根据文档类型和检索要求确定最佳分块方法时使用。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: chunking-strategies description: Document chunking strategies for RAG systems. Use when implementing document processing pipelines to determine optimal chunking approaches based on document type and retrieval requirements.

Chunking Strategies Skill

This skill provides chunking strategies for RAG document processing.

Chunking Methods

1. Fixed-Size Chunking

def fixed_size_chunk(text: str, chunk_size: int = 500, overlap: int = 50):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

2. Semantic Chunking

Split on natural boundaries (sentences, paragraphs).

def semantic_chunk(text: str, max_tokens: int = 500):
    paragraphs = text.split("\n\n")
    chunks = []
    current_chunk = []
    current_tokens = 0

    for para in paragraphs:
        para_tokens = count_tokens(para)
        if current_tokens + para_tokens > max_tokens:
            chunks.append("\n\n".join(current_chunk))
            current_chunk = [para]
            current_tokens = para_tokens
        else:
            current_chunk.append(para)
            current_tokens += para_tokens

    if current_chunk:
        chunks.append("\n\n".join(current_chunk))
    return chunks

3. Recursive Chunking

Hierarchical splitting on multiple separators.

SEPARATORS = ["\n\n", "\n", ". ", " "]

def recursive_chunk(text: str, max_size: int, separators: list[str]):
    if len(text) <= max_size:
        return [text]

    sep = separators[0] if separators else ""
    chunks = []
    parts = text.split(sep)

    for part in parts:
        if len(part) <= max_size:
            chunks.append(part)
        elif len(separators) > 1:
            chunks.extend(recursive_chunk(part, max_size, separators[1:]))
        else:
            chunks.append(part[:max_size])

    return chunks

Chunking by Document Type

Document Type Recommended Strategy Chunk Size
Technical docs Semantic (headers) 500-1000 tokens
Legal documents Semantic (sections) 1000-2000 tokens
Code Function/class based 200-500 tokens
Conversations Message boundaries 100-300 tokens
General text Recursive 300-500 tokens

Chunk Enrichment

@dataclass
class EnrichedChunk:
    content: str
    metadata: dict
    summary: str  # LLM-generated
    keywords: list[str]
    parent_id: str  # For hierarchical retrieval

Best Practices

  • Add overlap between chunks (10-20%)
  • Preserve semantic boundaries
  • Include metadata (source, position)
  • Consider hierarchical chunking for long docs
  • Test retrieval quality with different sizes