@kb-labs/mind-engine

Provides a pluggable adapter pattern for integrating multiple embedding model providers (OpenAI, Anthropic, local models, etc.) through a unified interface. The engine abstracts provider-specific API signatures, authentication, and response formats into standardized adapter implementations, allowing runtime switching between embedding backends without application code changes.

Abstracts vector database operations (insert, search, delete, update) across heterogeneous backends (Pinecone, Weaviate, Milvus, in-memory stores) through a unified CRUD interface. Handles vector normalization, metadata filtering, similarity search configuration, and result ranking without exposing backend-specific query syntax or connection management.

Automatically expands user queries through synonym generation, paraphrasing, or semantic decomposition to improve retrieval coverage. Generates multiple query variants and executes parallel searches, then deduplicates and merges results to find documents that might be missed by literal query matching. Supports custom expansion strategies and LLM-based reformulation.

Reranks vector search results using secondary relevance signals (cross-encoder models, BM25 scores, domain-specific heuristics) to improve ranking quality beyond initial similarity scores. Combines multiple ranking signals through learned or rule-based fusion, enabling fine-grained relevance tuning without re-embedding documents.

Coordinates the end-to-end retrieval-augmented generation workflow: document ingestion → chunking → embedding → vector storage → query retrieval → context assembly. Manages data flow between components, handles batch processing, and provides hooks for custom preprocessing or postprocessing steps at each stage without requiring manual pipeline wiring.

Executes vector similarity search combined with structured metadata filtering, enabling hybrid queries that find semantically similar documents while respecting categorical, temporal, or permission-based constraints. Translates filter expressions into backend-specific query syntax and ranks results by relevance score with optional reranking strategies.

Automatically segments documents into semantically coherent chunks using configurable strategies (fixed-size, semantic boundaries, recursive splitting) while preserving metadata and context. Handles multiple input formats (text, markdown, structured data) and applies preprocessing transformations (normalization, deduplication, encoding) before embedding to optimize retrieval quality.

Processes large document collections through embedding providers in batches, aggregating requests to minimize API calls and costs. Implements request deduplication, caching of previously computed embeddings, and intelligent batching strategies that respect provider rate limits and token budgets while tracking embedding costs per document.

Retrieves relevant document chunks from the vector store and assembles them into a coherent context block formatted for LLM consumption. Handles context window constraints, result ranking, deduplication of overlapping chunks, and optional reranking to maximize relevance while staying within token budgets. Produces formatted prompts ready for LLM inference.

Tracks versions of embedded documents and vector store snapshots, enabling rollback to previous knowledge base states. Maintains version metadata (timestamp, document count, embedding model used) and supports selective rollback of specific documents or entire snapshots without rebuilding embeddings from scratch.

Handles embedding and retrieval across documents in multiple languages using language-aware embedding models and optional translation strategies. Automatically detects document language, selects appropriate embedding models, and enables cross-language semantic search through multilingual embedding spaces or translation-based approaches.

Provides tools to evaluate embedding model quality on custom datasets through metrics like retrieval precision, recall, NDCG, and MRR. Supports A/B testing different embedding models against the same query set and benchmarks latency, cost, and quality tradeoffs to guide model selection decisions.