rag-pipeline

Details on the Retrieval Augmented Generation pipeline, Ingestion, and Vector Search.

⭐1stars🔀0forks📁View on GitHub🕐Updated Dec 13, 2025⬇️2downloads

When & Why to Use This Skill

This Claude skill provides a comprehensive Retrieval Augmented Generation (RAG) pipeline, enabling agents to handle complex data ingestion, vector-based search, and sophisticated context management. By integrating Qdrant for vector storage and utilizing advanced embedding models, it allows for the creation of highly context-aware agents that can retrieve and process information from local documentation with high precision.

Use Cases

  • Technical Documentation Assistant: Automatically ingest and index MDX-based documentation to provide precise, context-rich answers to complex technical queries.
  • Personalized AI Interaction: Leverage user-specific software and hardware backgrounds to tailor the RAG prompt, ensuring the agent's responses are optimized for the user's specific environment.
  • Automated Knowledge Base Management: Streamline the workflow of scanning, chunking, and embedding new project files into a vector database for real-time information retrieval.
  • Agentic Workflow Integration: Use the custom Agent class to wrap LLM calls with retrieved context, facilitating the development of multi-agent systems that share a unified knowledge source.
nameRAG Pipeline
descriptionDetails on the Retrieval Augmented Generation pipeline, Ingestion, and Vector Search.

RAG Pipeline Logic

Ingestion

  • Script: backend/ingest.py
  • Process:
    1. Scans docs/.
    2. Cleans MDX (removes frontmatter/imports).
    3. Chunks text (1000 chars, 100 overlap).
    4. Embeds using models/text-embedding-004.
    5. Upserts to Qdrant collection physical_ai_book.
  • Run: python backend/ingest.py

Vector Search (Qdrant)

  • Client: qdrant-client
  • Collection: physical_ai_book
  • Vector Size: 768 (Gecko-004)
  • Similarity: Cosine

Prompt Engineering

  • File: backend/utils/helpers.py.
  • RAG Prompt: Constructs a prompt containing retrieved context chunks.
  • Personalization: backend/personalization.py creates system instructions based on software_background and hardware_background of the user.

Agentic Flow

We use a custom Agent class (backend/agents.py) that wraps the LLM calls, allowing for future expansion into multi-agent workflows.