nomic-embed-text-v2-moe

Generates dense vector embeddings (768-dimensional) for sentences and documents across 19 languages using a Mixture-of-Experts (MoE) architecture that routes inputs to specialized expert transformers based on language and semantic content. The model uses nomic_bert as its backbone with learned gating mechanisms to dynamically select which expert sub-networks process each token, enabling efficient cross-lingual semantic understanding without language-specific fine-tuning.

Computes semantic similarity between sentence pairs by encoding both inputs through the MoE embedding pipeline and applying learned pooling mechanisms (mean pooling with attention weighting) to aggregate token-level representations into sentence-level vectors, then computing cosine similarity. The model is trained on contrastive objectives (InfoNCE-style losses) to maximize similarity for semantically related pairs and minimize it for negatives, enabling direct similarity prediction without additional classification layers.

Processes multiple sentences or documents in parallel through the MoE architecture, with the gating network dynamically routing each input sequence to different expert combinations based on learned routing weights. Batch processing leverages GPU/TPU parallelism while the sparse expert routing reduces per-sample compute by activating only top-k experts (typically 2-4 out of 8-16 total experts) per token, enabling efficient large-scale embedding generation without proportional memory growth.

Provides frozen sentence embeddings that serve as input features for downstream supervised tasks (classification, clustering, regression) without requiring fine-tuning of the embedding model itself. The 768-dimensional embeddings are designed to be task-agnostic and semantically rich, allowing practitioners to train lightweight task-specific heads (linear classifiers, clustering algorithms) on top of the embeddings while keeping the base model frozen, reducing training data requirements and computational cost.

Encodes text from 19 languages (English, Spanish, French, German, Italian, Portuguese, Polish, Dutch, Turkish, Japanese, Vietnamese, Russian, Indonesian, Arabic, and others) into a shared semantic space where cross-lingual synonyms and translations have similar embeddings. The MoE architecture includes language-aware expert routing that specializes different experts for different language families (e.g., Romance languages, East Asian languages, Semitic languages), while the shared embedding space enables zero-shot cross-lingual retrieval and similarity matching without language-specific alignment.

Model weights are distributed in safetensors format (a safer, faster alternative to pickle-based PyTorch checkpoints) enabling secure model loading without arbitrary code execution risks. The architecture is compatible with quantization frameworks (GPTQ, AWQ, bitsandbytes) allowing practitioners to reduce model size and inference latency through post-training quantization without retraining, supporting int8 and int4 quantization for deployment on resource-constrained devices while maintaining embedding quality.