intelligent-text-chunking
将大文本分割成有意义且针对 AI 优化的块,同时保持语义连贯性和文档结构。在为 AI 训练或 RAG 系统处理大型文档时使用,或者在需要分解内容同时保持上下文和关系时使用。
📖 中文导读:将大文本分割成有意义且针对 AI 优化的块,同时保持语义连贯性和文档结构。在为 AI 训练或 RAG 系统处理大型文档时使用,或者在需要分解内容同时保持上下文和关系时使用。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。
name: intelligent-text-chunking description: Split large texts into meaningful, AI-optimized chunks while preserving semantic coherence and document structure. Use when processing large documents for AI training, RAG systems, or when you need to break down content while maintaining context and relationships.
Intelligent Text Chunking
Overview
A sophisticated text segmentation skill that splits large texts into meaningful, AI-optimized chunks while preserving semantic coherence, document structure, and contextual relationships. Essential for processing large documents for AI training, RAG systems, and memory-constrained applications.
Capabilities
- Semantic Awareness: Respects topic boundaries and meaning transitions
- Structure Preservation: Maintains document hierarchy and formatting context
- Multi-language Support: Handles accented characters and diverse writing systems
- Configurable Strategies: Multiple chunking approaches for different use cases
- Context Preservation: Intelligent overlap management for context continuity
- Quality Optimization: Balances chunk size with content coherence
Chunking Strategies
1. Semantic Chunking
Split text based on meaning and topic boundaries rather than arbitrary size limits.
2. Structural Chunking
Follow document organization (headings, sections, lists) for natural divisions.
3. Fixed-Size Chunking
Create consistent-sized chunks with intelligent boundary selection.
4. Sliding Window Chunking
Create overlapping chunks for enhanced context preservation.
Implementation Examples
Language-Aware Processing
# Multi-language sentence detection
sentence_patterns = {
'english': ['.', '!', '?', ';'],
'chuukese': ['.', '!', '?'],
'general': ['.', '!', '?', ';', '。', '!', '?']
}
def detect_language_patterns(text):
has_accents = bool(re.search(r'[áéíóúàèìòùāēīōūâêîôû]', text))
return 'chuukese' if has_accents else 'english'
Basic Usage
from .intelligent_chunker import IntelligentTextChunker, ChunkType
chunker = IntelligentTextChunker(
max_chunk_size=1024,
overlap_ratio=0.15,
preserve_sentences=True
)
chunks = chunker.chunk_document(text, ChunkType.SEMANTIC)
Best Practices
- Size Balancing: Balance chunk size with content coherence
- Context Preservation: Use appropriate overlap for your use case
- Language Awareness: Configure for specific languages when known
- Quality Validation: Check chunk quality with sample reviews
- Use Case Optimization: Choose strategy based on downstream use
Dependencies
re: Regular expression pattern matchingspacy: Advanced sentence segmentationnltk: Natural language processing utilitieslangdetect: Language identification support