rag-pipeline

Use when building Retrieval-Augmented Generation systems. Covers document chunking, embedding generation, vector indexing, semantic search, context building, and integration of retrieval with LLM completion for accurate Q&A on Physical AI textbook content.

⭐5stars🔀1forks📁View on GitHub🕐Updated Jan 11, 2026⬇️2downloads

When & Why to Use This Skill

This Claude skill provides a comprehensive framework for building and maintaining Retrieval-Augmented Generation (RAG) systems. It streamlines the entire pipeline—from document ingestion and semantic chunking to vector indexing with Qdrant and context-aware LLM completion. By utilizing OpenAI embeddings and GPT-4, it ensures highly accurate, source-cited Q&A capabilities for complex technical content, significantly reducing hallucinations and improving information retrieval efficiency.

Use Cases

  • Technical Documentation Assistant: Build an AI-powered Q&A system for complex textbooks or manuals that provides precise answers with direct source citations.
  • Semantic Search Implementation: Develop a high-performance search engine that retrieves relevant document sections based on conceptual meaning rather than simple keyword matching.
  • Enterprise Knowledge Management: Create a centralized RAG pipeline to ingest internal Docusaurus or Markdown files, making corporate knowledge instantly accessible via natural language.
  • Context-Aware Research Tools: Enhance LLM performance by automatically retrieving and injecting the most relevant 3000-token context windows into prompts for specialized domain analysis.
namerag-pipeline
descriptionUse when building Retrieval-Augmented Generation systems. Covers document chunking, embedding generation, vector indexing, semantic search, context building, and integration of retrieval with LLM completion for accurate Q&A on Physical AI textbook content.

RAG Pipeline Skill

Quick Start Workflow

When building or maintaining the RAG pipeline:

  1. Content Ingestion (One-time setup)

    • Read all Docusaurus markdown files from /docs
    • Chunk text (800 chars, 200 overlap)
    • Generate embeddings with OpenAI ada-002
    • Upsert to Qdrant with metadata
  2. Query Flow (Runtime)

    • Receive user question
    • Generate query embedding
    • Search Qdrant (top 5 results, score >= 0.7)
    • Build context from relevant chunks
    • Pass to OpenAI GPT-4 with context
    • Return answer + sources
  3. Continuous Improvement

    • Monitor search quality (are results relevant?)
    • Adjust chunk size if needed
    • Update score thresholds
    • Add filters for specific chapters

Standard Architecture

User Question
    ↓
[Generate Embedding]
    ↓ 
[Search Qdrant]
    ↓
[Extract Top 5 Chunks]
    ↓
[Build Context String]
    ↓
[GPT-4 with Context]
    ↓
AI Answer + Sources

Key Parameters

  • Chunk size: 800 characters
  • Overlap: 200 characters
  • Embedding model: text-embedding-ada-002
  • LLM model: gpt-4 or gpt-3.5-turbo
  • Search limit: 5 chunks
  • Score threshold: 0.7
  • Context window: ~3000 tokens max

Best Practices

For Physical AI textbook RAG:

  • Preserve code blocks when chunking
  • Include chapter/section in metadata
  • Cite sources in responses
  • Cache embeddings for popular queries
  • Log all queries for analytics
  • Handle "no results" gracefully

Knowledge Base

Detailed guides available:

  • Chunking Strategies → references/chunking.md
  • Ingestion Script → references/ingestion-script.md
  • Query Pipeline → references/query-pipeline.md
  • Context Building → references/context-building.md
  • Error Handling → references/error-handling.md