rag-research

Use this skill when users ask about RAG (Retrieval-Augmented Generation), semantic search, document indexing, embeddings, vector databases, or chunking strategies. Trigger phrases include "how does rag-research work", "improve search results", "RAG optimization", "embedding model", "chunking strategy", "vector database".

⭐2stars🔀0forks📁View on GitHub🕐Updated Dec 31, 2025⬇️2downloads

When & Why to Use This Skill

This Claude skill provides a comprehensive framework for building and optimizing Retrieval-Augmented Generation (RAG) systems. It offers expert guidance on document indexing pipelines, embedding model selection (such as the BAAI/bge series), and advanced chunking strategies to significantly improve semantic search accuracy and information retrieval quality for AI-driven knowledge management.

Use Cases

  • Optimizing Search Relevance: Fine-tuning chunk sizes and overlap parameters to ensure high-quality context retrieval for complex technical manuals and documentation.
  • Embedding Model Selection: Evaluating and implementing different embedding models based on the trade-off between inference speed and retrieval accuracy for specific use cases.
  • Document Pipeline Troubleshooting: Diagnosing and resolving issues related to poor search results, missing information, or PDF extraction failures using Mistral OCR.
  • Vector Database Management: Implementing best practices for storing, filtering, and querying high-dimensional vectors in systems like Qdrant to maintain a scalable knowledge base.
namerag-research
descriptionUse this skill when users ask about RAG (Retrieval-Augmented Generation), semantic search, document indexing, embeddings, vector databases, or chunking strategies. Trigger phrases include "how does rag-research work", "improve search results", "RAG optimization", "embedding model", "chunking strategy", "vector database".

RAG Research Skill

Use this skill when users ask about RAG (Retrieval-Augmented Generation), semantic search, document indexing, embeddings, vector databases, or chunking strategies. This skill provides best practices for working with the rag-research plugin and optimizing document retrieval.

When to Use

Trigger this skill when users:

  • Ask about RAG, embeddings, or semantic search concepts
  • Want to optimize their document indexing strategy
  • Need help with chunking or retrieval quality
  • Ask "how does rag-research work?" or "how to improve search results?"
  • Troubleshoot poor search results or missing information

Core Concepts

Document Indexing Pipeline

  1. Load Document: Extract text from PDF, Markdown, or Text files
  2. Chunk Text: Split into overlapping segments (default: 512 chars, 50 overlap)
  3. Generate Embeddings: Convert chunks to vectors using FastEmbed (BAAI/bge-small-en-v1.5)
  4. Store in Qdrant: Persist vectors with metadata for retrieval

Embedding Models

The plugin uses FastEmbed with ONNX Runtime for efficient CPU inference:

Model Dimensions Speed Quality Use Case
BAAI/bge-small-en-v1.5 384 Fast Good Default, general use
BAAI/bge-base-en-v1.5 768 Medium Better Higher accuracy needs
BAAI/bge-large-en-v1.5 1024 Slow Best Maximum quality

Chunking Strategies

Chunk size affects retrieval quality:

  • Smaller chunks (256-512): More precise, may lose context
  • Larger chunks (1024+): More context, may dilute relevance
  • Overlap (10-20%): Prevents information loss at boundaries

Recommend: Start with defaults (512/50), adjust based on results.

Search Quality Tips

  1. Use specific queries: "Mistral OCR API configuration" > "OCR"
  2. Check coverage: Run /rag-research:list to verify documents are indexed
  3. Increase limit: Use --limit 20 for comprehensive research
  4. Review scores: Scores > 0.7 are highly relevant, < 0.5 may be tangential

Troubleshooting

Poor Search Results

  1. Check if document is indexed: /rag-research:list --filter "keyword"
  2. Re-index with different chunking: Adjust CHUNK_SIZE in .env
  3. Use more specific queries: Add domain-specific terms
  4. Verify embeddings: Check model compatibility

PDF Extraction Issues

  1. Enable Mistral OCR: Set MISTRAL_API_KEY in .env for scanned PDFs
  2. Fallback to pypdf: Use --no-ocr flag for text-based PDFs
  3. Check file permissions: Ensure PDF is readable

Database Issues

  1. Reset database: rm -rf ~/.rag-research and re-index
  2. Check disk space: Qdrant needs space for vectors
  3. Verify installation: uv run rag-research stats

Configuration Reference

See references/configuration.md for detailed settings documentation.

Examples

See references/examples.md for common usage patterns.