vector-search
Semantic search using vector embeddings and similarity metrics. Apply when implementing search with embeddings (OpenAI, Sentence Transformers) or distance-based ranking. Covers cosine similarity, Euclidean distance, dot product, and threshold selection.
When & Why to Use This Skill
This Claude skill provides a comprehensive framework for implementing semantic search and similarity-based ranking using vector embeddings. It covers essential similarity metrics—including Cosine Similarity, Euclidean Distance, and Dot Product—while providing technical guidance on threshold optimization and database integration (e.g., pgvector). It is designed to help developers build robust retrieval systems, enhance RAG (Retrieval-Augmented Generation) workflows, and avoid common pitfalls in vector-based data processing.
Use Cases
- Semantic Search Implementation: Developing search engines for text, images, or audio that retrieve results based on conceptual meaning rather than simple keyword matching.
- Recommendation Systems: Building 'find similar' features by calculating distance metrics between item embeddings to rank the most relevant content for users.
- AI Agent Memory & RAG: Optimizing Retrieval-Augmented Generation by accurately fetching context from vector databases to improve the accuracy and relevance of LLM responses.
- Data Clustering and Classification: Grouping high-dimensional data points or classifying new inputs based on their proximity to known clusters in an embedding space.
- Database Query Optimization: Writing efficient SQL queries for vector extensions like pgvector to handle high-performance similarity searches at scale.
| name | vector-search |
|---|---|
| description | Semantic search using vector embeddings and similarity metrics. Apply when implementing search with embeddings (OpenAI, Sentence Transformers) or distance-based ranking. Covers cosine similarity, Euclidean distance, dot product, and threshold selection. |
| type | skill |
| version | 1.0.0 |
Vector Search Skill
Implement semantic search and similarity-based ranking using vector embeddings.
What This Skill Provides
- Similarity Metrics: Cosine similarity, Euclidean distance, dot product
- Range Understanding: Mathematical domains for each metric
- Threshold Selection: Setting appropriate minScore/maxDistance values
- Implementation Patterns: Database queries, filtering, ranking
When to Use
- Implementing semantic search (text, images, audio)
- Ranking results by similarity to query
- Recommendation systems (find similar items)
- Clustering or classification based on embeddings
- When user mentions: "vector search", "embeddings", "similarity", "semantic search"
Primitives Included
- Instructions:
similarity-metrics.instructions.md- Metric selection and threshold guidance - Instructions:
embedding-integration.instructions.md- OpenAI, Sentence Transformers patterns
Key Concepts
Similarity Metrics Cheat Sheet
| Metric | Range | When to Use |
|---|---|---|
| Cosine Similarity | [-1, 1] | Text embeddings, direction matters |
| Euclidean Distance | [0, ∞) | Spatial data, magnitude matters |
| Dot Product | (-∞, ∞) | Raw similarity, not normalized |
| Jaccard Similarity | [0, 1] | Set overlap, binary features |
Critical: Understand the Range
Most common bug: Setting minScore=0 for cosine similarity and filtering out valid negative scores.
Example:
// ❌ Wrong - filters out negative scores
function search(embedding, minScore = 0) {
return results.filter(r => r.score >= minScore);
// Cosine can be negative!
}
// ✅ Correct - includes full range
function search(embedding, minScore = -1) {
return results.filter(r => r.score >= minScore);
// -1 to 1 for cosine similarity
}
Example: Semantic Bird Search
// Generate embedding for query
const queryEmbedding = await openai.embeddings.create({
model: 'text-embedding-3-small',
input: 'red bird with black wings'
});
// Search with cosine similarity
const results = await db.query(`
SELECT
id,
name,
1 - (embedding <=> $1) as similarity
FROM birds
WHERE 1 - (embedding <=> $1) >= $2
ORDER BY similarity DESC
LIMIT 10
`, [queryEmbedding.data[0].embedding, 0.3]);
Note: PostgreSQL pgvector uses <=> for cosine distance (0 to 2), so 1 - distance gives similarity (-1 to 1).
Dependencies
- Database: SQLite with BLOB, PostgreSQL with pgvector, or specialized vector DB
- Embedding Model: OpenAI API, Sentence Transformers, or custom model
- Math Library: For similarity calculations if not database-native
Related Skills: fullstack-expertise, claude-framework (E-1 validation)