This Claude skill provides comprehensive technical guidance on vector database architecture, embedding storage, and similarity search optimization. It focuses on helping developers and architects navigate the complexities of Approximate Nearest Neighbor (ANN) algorithms like HNSW and IVF, select the right vector store based on scale, and design high-performance semantic search systems for RAG and AI applications.
Use Cases
Case 1: Selecting the optimal vector database (e.g., Pinecone, Milvus, Weaviate, or pgvector) by evaluating trade-offs between managed services and self-hosted solutions.
Case 2: Optimizing search latency and recall by tuning ANN algorithm parameters such as HNSW's M, ef_construction, and IVF's nprobe.
Case 3: Designing hybrid search architectures that combine dense vector embeddings with metadata filtering to improve retrieval precision.
Case 4: Scaling vector infrastructure for massive datasets through sharding strategies, memory optimization (PQ quantization), and cost estimation.
name
vector-databases
description
Vector database selection, embedding storage, approximate nearest neighbor (ANN) algorithms, and vector search optimization. Use when choosing vector stores, designing semantic search, or optimizing similarity search performance.
Need managed, don't want operations?
├── Yes → Pinecone (simplest) or Weaviate Cloud
└── No (self-hosted)
└── Already using PostgreSQL?
├── Yes, <1M vectors → pgvector
└── No
└── Need maximum performance at scale?
├── Yes → Milvus or Qdrant
└── No
└── Prototyping/development?
├── Yes → Chroma
└── No → Qdrant (balanced choice)
ANN Algorithms
Algorithm Overview
Exact KNN:
• Search ALL vectors
• O(n) time complexity
• Perfect accuracy
• Impractical at scale
Approximate NN (ANN):
• Search SUBSET of vectors
• O(log n) to O(1) complexity
• Near-perfect accuracy
• Practical at any scale
HNSW (Hierarchical Navigable Small World)
Layer 3: ○───────────────────────○ (sparse, long connections)
│ │
Layer 2: ○───○───────○───────○───○ (medium density)
│ │ │ │ │
Layer 1: ○─○─○─○─○─○─○─○─○─○─○─○─○ (denser)
│││││││││││││││││││││││
Layer 0: ○○○○○○○○○○○○○○○○○○○○○○○○○ (all vectors)
Search: Start at top layer, greedily descend
• Fast: O(log n) search time
• High recall: >95% typically
• Memory: Extra graph storage
HNSW Parameters:
Parameter
Description
Trade-off
M
Connections per node
Memory vs. recall
ef_construction
Build-time search width
Build time vs. recall
ef_search
Query-time search width
Latency vs. recall
IVF (Inverted File Index)
Clustering Phase:
┌─────────────────────────────────────────┐
│ Cluster vectors into K centroids │
│ │
│ ● ● ● ● │
│ /│\ /│\ /│\ /│\ │
│ ○○○○○ ○○○○○ ○○○○○ ○○○○○ │
│ Cluster 1 Cluster 2 Cluster 3 Cluster 4│
└─────────────────────────────────────────┘
Search Phase:
1. Find nprobe nearest centroids
2. Search only those clusters
3. Much faster than exhaustive