koan-vector-migration
Vector export/import, embedding caching, provider migration
When & Why to Use This Skill
This Claude skill facilitates seamless vector database migration and embedding management by enabling the export and import of vectors without regenerating them via AI APIs. It focuses on cost optimization and infrastructure flexibility, allowing developers to cache embeddings and move data between providers like ElasticSearch and Weaviate with zero additional AI processing costs.
Use Cases
- Migrating vector datasets from Weaviate to ElasticSearch (or vice versa) without paying for redundant AI embedding generation.
- Implementing an embedding cache layer to reduce latency and API expenses during large-scale data indexing operations.
- Switching AI embedding providers while maintaining existing vector data to ensure consistency across different infrastructure setups.
- Creating portable backups of vector embeddings for disaster recovery, archival, or offline development environments.
| name | koan-vector-migration |
|---|---|
| description | Vector export/import, embedding caching, provider migration |
Koan Vector Migration
Core Principle
Export vectors without regenerating via AI. Cache embeddings to enable zero-cost vector database migration.
Vector Export (DATA-0078)
Export Vectors
var vectorRepo = serviceProvider.GetRequiredService<IVectorSearchRepository<Media, string>>();
await foreach (var batch in vectorRepo.ExportAllAsync(batchSize: 100, ct))
{
// batch.Id: Entity identifier
// batch.Embedding: float[] vector
// batch.Metadata: Optional metadata
// Cache the embedding
var contentHash = EmbeddingCache.ComputeContentHash(embeddingText);
await cache.SetAsync(contentHash, modelId, batch.Embedding, ct);
}
Provider Support
- ✅ ElasticSearch: Scroll API (default batch: 1000)
- ✅ Weaviate: GraphQL pagination (default batch: 100)
- ⏳ Qdrant: Planned
- ⏳ Milvus: Planned
- ❌ Pinecone: Not supported (throws NotSupportedException)
Migration Pattern
1. Export vectors from Provider A → Cache
2. Switch configuration to Provider B
3. Import vectors from Cache → Provider B
Result: Zero AI API calls for migration
Example: Weaviate → ElasticSearch
// Step 1: Export from Weaviate
using (EntityContext.Adapter("weaviate"))
{
var vectorRepo = sp.GetRequiredService<IVectorSearchRepository<Media, string>>();
await foreach (var batch in vectorRepo.ExportAllAsync(batchSize: 100, ct))
{
await cache.SetAsync(batch.Id, "ada-002", batch.Embedding, ct);
}
}
// Step 2: Switch to ElasticSearch in appsettings.json
{
"Koan": {
"Data": {
"Sources": {
"Vectors": {
"Adapter": "elasticsearch",
"ConnectionString": "http://localhost:9200"
}
}
}
}
}
// Step 3: Import to ElasticSearch
foreach (var mediaId in allMediaIds)
{
var embedding = await cache.GetAsync(mediaId, "ada-002", ct);
if (embedding != null)
{
var media = new Media { Id = mediaId, Embedding = embedding };
await media.Save();
}
}
When This Skill Applies
- ✅ Migrating vector databases
- ✅ Caching embeddings
- ✅ AI provider switches
- ✅ Cost optimization
- ✅ Vector backup/restore
Reference Documentation
- CLAUDE.md: Lines 96-123 (Vector Export for Migration)
- ADR: DATA-0078 (Vector export specification)
- Guide:
docs/guides/ai-vector-howto.md