e5-base-v2

Generates dense vector embeddings (768-dimensional) for sentences and documents using a BERT-based architecture trained with contrastive learning on 1B+ sentence pairs. The model uses a masked language modeling objective combined with in-batch negatives and hard negative mining to learn representations where semantically similar sentences cluster together in embedding space. Supports 100+ languages through multilingual BERT pretraining, enabling cross-lingual semantic search without language-specific fine-tuning.

Computes cosine similarity between embeddings of sentences in different languages by leveraging multilingual BERT's shared embedding space, enabling cross-lingual retrieval without language-specific alignment or translation. The model transfers semantic understanding across languages through shared subword tokenization and joint pretraining, allowing queries in one language to retrieve relevant documents in another language with minimal performance degradation.

Provides embeddings optimized for retrieval-augmented generation pipelines, where embeddings are used to retrieve relevant documents from a knowledge base to augment LLM prompts. The model's embeddings are designed for high recall on semantic search (retrieving all relevant documents) while maintaining precision for ranking. Integration with vector databases enables efficient retrieval at scale, and the embeddings are compatible with popular RAG frameworks (LangChain, LlamaIndex, Haystack).

Processes multiple sentences or documents in parallel through the model, automatically batching inputs to maximize GPU/CPU utilization and converting outputs to multiple formats (PyTorch tensors, NumPy arrays, ONNX, OpenVINO). The implementation handles variable-length sequences through dynamic padding, manages memory efficiently for large batches, and supports multiple serialization formats for downstream integration with vector databases or ML pipelines.

Ranks documents or sentences by semantic similarity to a query using multiple distance metrics (cosine, euclidean, dot product) computed directly on embedding vectors. The implementation supports both dense-only ranking and hybrid ranking (combining semantic similarity with BM25 keyword scores), enabling flexible relevance tuning for different use cases through metric selection and score normalization.

Exports embeddings in formats compatible with major vector databases (Pinecone, Weaviate, Milvus, Qdrant, Chroma) through standardized serialization and metadata handling. The model outputs embeddings with optional metadata (document IDs, text, timestamps) that can be directly ingested into vector stores, supporting both batch indexing and streaming updates with automatic schema mapping.

Enables domain-specific adaptation by fine-tuning the base model on custom sentence pairs using contrastive learning (triplet loss, in-batch negatives). The fine-tuning process preserves the pretrained multilingual knowledge while optimizing embeddings for domain-specific similarity patterns, supporting both supervised pairs (positive/negative examples) and weak supervision from domain data. Training uses the sentence-transformers library's built-in loss functions and data loaders, enabling efficient adaptation with minimal code.

Exports the model to ONNX (Open Neural Network Exchange) and OpenVINO intermediate representation formats, enabling deployment on edge devices, mobile platforms, and on-premise servers without PyTorch dependencies. The export process converts the model graph and weights to standardized formats, supporting quantization (int8, fp16) for reduced model size and inference latency. Exported models run on CPUs, GPUs, and specialized accelerators (Intel VPU, ARM processors) with minimal performance degradation.

Provides standardized evaluation against the Massive Text Embedding Benchmark (MTEB) covering 56+ tasks across 8 categories (clustering, reranking, retrieval, semantic similarity, STS, summarization, classification, paraphrase detection). The model's performance is pre-computed and published on the MTEB leaderboard, enabling comparison against 100+ other embedding models. Users can run local MTEB evaluation to measure performance on custom datasets using the same standardized metrics (NDCG@10 for retrieval, Spearman correlation for STS, etc.).

Handles text preprocessing for 100+ languages through multilingual BERT's tokenizer, automatically detecting language and applying appropriate tokenization, lowercasing, and special token handling. The preprocessing pipeline normalizes text (whitespace, punctuation), handles out-of-vocabulary words through subword tokenization, and manages sequence length constraints (512 tokens max) through truncation or chunking. Language detection is implicit through the tokenizer's multilingual vocabulary, requiring no explicit language specification.

Groups similar documents or sentences into clusters using embeddings and clustering algorithms (K-means, hierarchical clustering, DBSCAN) applied to the 768-dimensional embedding space. The clustering leverages the semantic structure learned by the model, where similar texts naturally cluster together. Users can specify the number of clusters or use automatic cluster detection, and retrieve cluster assignments and centroids for downstream analysis or organization.