Qdrant Manager

Agent: AIEngineer

Manage Qdrant Cloud collections for the Physical AI textbook RAG system. This skill handles vectorization of textbook chapters, metadata tagging, and collection management for efficient retrieval-augmented generation.

Quick Setup

# Set environment variables
export QDRANT_URL="https://xxx.qdrant.io"
export QDRANT_API_KEY="your-api-key"
export OPENAI_API_KEY="sk-..."

# Initialize collection
.claude/skills/qdrant-manager/scripts/setup.sh --init

# Vectorize all chapters
.claude/skills/qdrant-manager/scripts/setup.sh --vectorize docs/

# Query the collection
.claude/skills/qdrant-manager/scripts/setup.sh --query "What is ROS 2?"

Command Options

Option	Description	Default
--init	Initialize/create the collection	-
--vectorize PATH	Vectorize markdown files in PATH	-
--query TEXT	Query the collection	-
--delete	Delete the collection	-
--info	Show collection info	-
--list	List all collections	-
--collection NAME	Collection name	textbook_chapters
--model MODEL	Embedding model	text-embedding-3-small
--batch-size N	Batch size for vectorization	10
--top-k N	Number of results for query	5
-h, --help	Show help message	-

What It Does

1. Collection Management

Creates and manages Qdrant collections with optimized settings for textbook content:

# Collection schema
{
    "vectors": {
        "size": 1536,  # text-embedding-3-small
        "distance": "Cosine"
    },
    "on_disk_payload": True  # Optimize for large documents
}

2. Chapter Vectorization

Processes markdown files with intelligent chunking:

• Chunk Strategy: Split by headers (##, ###) for semantic coherence
• Overlap: 100 tokens overlap between chunks
• Max Chunk Size: 500 tokens per chunk
• Metadata Extraction: Title, chapter, section, language, keywords

3. Metadata Schema

Each vector point includes rich metadata for filtered retrieval:

{
    "id": "uuid-v4",
    "vector": [0.1, 0.2, ...],
    "payload": {
        "content": "The actual text content...",
        "source": "docs/chapter-1/index.md",
        "chapter": 1,
        "section": "Introduction",
        "title": "Introduction to Physical AI & ROS 2",
        "language": "en",
        "locale": "en-US",
        "keywords": ["ros2", "physical-ai", "robotics"],
        "word_count": 245,
        "created_at": "2026-01-02T00:00:00Z",
        "chunk_index": 0,
        "total_chunks": 5
    }
}

4. RAG Query Support

Semantic search with metadata filtering:

# Example: Query with language filter
results = client.search(
    collection_name="textbook_chapters",
    query_vector=embed("What is ROS 2?"),
    query_filter={
        "must": [
            {"key": "language", "match": {"value": "en"}}
        ]
    },
    limit=5
)

Bundled Resources

1. Python Dependencies

File: requirements.txt

qdrant-client>=1.7.0
openai>=1.0.0
tiktoken>=0.5.0
python-dotenv>=1.0.0
rich>=13.0.0

2. Collection Configuration

File: assets/collection_config.json

{
    "vectors": {
        "size": 1536,
        "distance": "Cosine"
    },
    "optimizers_config": {
        "indexing_threshold": 20000
    },
    "on_disk_payload": true,
    "replication_factor": 1
}

3. Chunking Configuration

File: assets/chunking_config.json

{
    "chunk_size": 500,
    "chunk_overlap": 100,
    "separators": ["## ", "### ", "\n\n", "\n"],
    "length_function": "tiktoken",
    "model": "text-embedding-3-small"
}

4. Vectorization Script

File: scripts/vectorize.py

from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance
from openai import OpenAI
import tiktoken
import uuid
import json
from pathlib import Path

class TextbookVectorizer:
    def __init__(self, qdrant_url, qdrant_api_key, openai_api_key):
        self.qdrant = QdrantClient(url=qdrant_url, api_key=qdrant_api_key)
        self.openai = OpenAI(api_key=openai_api_key)
        self.encoder = tiktoken.encoding_for_model("text-embedding-3-small")

    def create_collection(self, name: str, vector_size: int = 1536):
        self.qdrant.create_collection(
            collection_name=name,
            vectors_config=VectorParams(size=vector_size, distance=Distance.COSINE)
        )

    def chunk_markdown(self, content: str, source: str) -> list:
        # Split by headers for semantic coherence
        chunks = []
        current_chunk = ""
        current_section = "Introduction"

        for line in content.split("\n"):
            if line.startswith("## "):
                if current_chunk:
                    chunks.append({"content": current_chunk, "section": current_section})
                current_section = line[3:].strip()
                current_chunk = line + "\n"
            else:
                current_chunk += line + "\n"
                if len(self.encoder.encode(current_chunk)) > 500:
                    chunks.append({"content": current_chunk, "section": current_section})
                    current_chunk = ""

        if current_chunk:
            chunks.append({"content": current_chunk, "section": current_section})

        return chunks

    def embed(self, text: str) -> list:
        response = self.openai.embeddings.create(
            input=text,
            model="text-embedding-3-small"
        )
        return response.data[0].embedding

    def vectorize_file(self, filepath: Path, collection: str):
        content = filepath.read_text()
        chunks = self.chunk_markdown(content, str(filepath))

        points = []
        for i, chunk in enumerate(chunks):
            vector = self.embed(chunk["content"])
            points.append(PointStruct(
                id=str(uuid.uuid4()),
                vector=vector,
                payload={
                    "content": chunk["content"],
                    "source": str(filepath),
                    "section": chunk["section"],
                    "chunk_index": i,
                    "total_chunks": len(chunks)
                }
            ))

        self.qdrant.upsert(collection_name=collection, points=points)
        return len(points)

5. Input Variables

Variable	Required	Default	Description
qdrant_url	Yes	-	Qdrant Cloud cluster URL
qdrant_api_key	Yes	-	Qdrant API key
collection_name	No	textbook_chapters	Collection name
embedding_model	No	text-embedding-3-small	OpenAI embedding model
vector_size	No	1536	Vector dimension

Usage Instructions

Step 1: Set Environment Variables

# Add to .env file
QDRANT_URL=https://your-cluster.qdrant.io
QDRANT_API_KEY=your-api-key
OPENAI_API_KEY=sk-...

Step 2: Initialize Collection

.claude/skills/qdrant-manager/scripts/setup.sh --init --collection textbook_chapters

Step 3: Vectorize Chapters

# Vectorize English chapters
.claude/skills/qdrant-manager/scripts/setup.sh --vectorize docs/

# Vectorize Urdu translations
.claude/skills/qdrant-manager/scripts/setup.sh --vectorize i18n/ur/docusaurus-plugin-content-docs/current/

Step 4: Query for RAG

# Simple query
.claude/skills/qdrant-manager/scripts/setup.sh --query "How do I install ROS 2?"

# Query with top-k
.claude/skills/qdrant-manager/scripts/setup.sh --query "Gazebo simulation" --top-k 10

Step 5: Verify Collection

# Show collection info
.claude/skills/qdrant-manager/scripts/setup.sh --info

# List all collections
.claude/skills/qdrant-manager/scripts/setup.sh --list

Verification Checklist

• QDRANT_URL environment variable set
• QDRANT_API_KEY environment variable set
• OPENAI_API_KEY environment variable set
• Collection created successfully
• Chapters vectorized without errors
• Query returns relevant results
• Metadata filtering works correctly

Integration with RAG Chatbot

This skill provides the vector storage for the RAG chatbot (Feature 002). The chatbot queries this collection to retrieve relevant context for answering user questions.

# In RAG chatbot
from qdrant_client import QdrantClient

client = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY)

def get_context(query: str, language: str = "en") -> list:
    query_vector = embed(query)
    results = client.search(
        collection_name="textbook_chapters",
        query_vector=query_vector,
        query_filter={"must": [{"key": "language", "match": {"value": language}}]},
        limit=5
    )
    return [hit.payload["content"] for hit in results]

Troubleshooting

Issue	Solution
Connection refused	Check QDRANT_URL is correct and cluster is running
Authentication failed	Verify QDRANT_API_KEY is valid
Embedding rate limit	Reduce --batch-size or add delays
Collection not found	Run --init first to create collection
Out of memory	Use smaller --batch-size
Slow vectorization	Use --batch-size 20 for faster processing

Requirements

• Python 3.9+
• Qdrant Cloud account (or self-hosted Qdrant)
• OpenAI API key (for embeddings)
• pip packages: qdrant-client, openai, tiktoken

Cost Considerations

Resource	Cost
Qdrant Cloud Free Tier	1GB storage, 1M vectors
OpenAI text-embedding-3-small	$0.02 / 1M tokens
Estimated for 3 chapters	~$0.01 (one-time)

Related

• Qdrant Documentation
• OpenAI Embeddings
• Skill: vercel-fastapi-link - Backend API with RAG endpoints
• Feature: 002-rag-chatbot - RAG chatbot specification

Changelog

v1.0.0 (2026-01-02)

Initial Release

• Collection creation and management
• Markdown chunking with header-based splitting
• OpenAI embedding integration (text-embedding-3-small)
• Metadata extraction (chapter, section, language, keywords)
• Query support with filtering
• Batch vectorization with progress tracking