FastEmbed

Generates fixed-size dense vector representations for text using ONNX-compiled transformer models (default: BAAI/bge-small-en-v1.5). Implements automatic model downloading, caching, and batch processing with configurable pooling strategies (mean, cls, last-token). ONNX Runtime provides CPU-optimized inference without PyTorch dependencies, enabling 5-10x faster embedding generation than traditional Sentence Transformers on CPU-only environments.

Generates sparse token-weighted embeddings using SPLADE, BM25, or BM42 models that produce high-dimensional vectors with mostly zero values. Each non-zero dimension corresponds to a vocabulary token with a learned importance weight. Sparse embeddings enable hybrid search by combining dense semantic matching with traditional lexical matching, supporting both keyword recall and semantic relevance in a single query.

Provides optional GPU acceleration for embedding inference through separate fastembed-gpu package that replaces CPU ONNX Runtime with CUDA-accelerated ONNX Runtime. Maintains identical API and model compatibility, enabling seamless CPU-to-GPU migration without code changes. GPU acceleration provides 10-50x speedup for batch processing depending on batch size and GPU model, with automatic device selection (CUDA, ROCm, or fallback to CPU).

Provides direct integration with Qdrant vector database's native late interaction search API, enabling token-level matching without custom scoring logic. Automatically formats late interaction embeddings (token-level vectors) into Qdrant's expected format and supports Qdrant's built-in late interaction scoring algorithm. Enables end-to-end pipelines where FastEmbed generates embeddings and Qdrant handles efficient retrieval with token-level matching.

Generates token-level embeddings where each token in the input text receives its own embedding vector, enabling fine-grained matching at the token level rather than document level. Implements ColBERT architecture via ONNX Runtime, producing a matrix of embeddings (one per token) that supports late interaction scoring where query tokens are matched against document tokens individually. This enables more precise relevance scoring than dense embeddings alone.

Generates fixed-size dense vector representations for images using CLIP and similar vision-language models compiled to ONNX format. Handles image preprocessing (resizing, normalization) automatically and produces embeddings in the same vector space as text embeddings from the same model, enabling cross-modal search where images and text can be compared directly. Supports batch processing of images with configurable batch sizes for memory management.

Generates token-level embeddings for document images (PDFs, scanned documents) using ColPali architecture, producing per-token embeddings that capture both visual and textual information from document images. Enables fine-grained matching where query tokens are matched against document image tokens, supporting precise document retrieval without OCR. Implements visual token extraction via ONNX Runtime with late interaction scoring for document-level relevance.

Scores relevance of text pairs (query-document, sentence-pair) using cross-encoder models compiled to ONNX format. Takes paired text inputs and produces scalar relevance scores (typically 0-1) indicating semantic similarity or relevance. Implements efficient batch processing of multiple pairs and supports various cross-encoder architectures (MS MARCO, NLI-based). Used as a reranking layer after initial retrieval to refine results with higher precision.

Manages lifecycle of embedding models including automatic download from Hugging Face Hub, local caching with version tracking, and cache invalidation. Implements smart caching that stores models in a configurable directory (~/.cache/fastembed by default) and reuses cached models across sessions. Supports model versioning to enable reproducible embeddings and handles concurrent access to cached models safely.

Processes multiple documents/images in batches with automatic CPU parallelization using ONNX Runtime's built-in threading. Implements configurable batch sizes to balance memory usage and throughput, with intelligent batching that groups inputs for efficient tensor operations. Automatically distributes batch computation across available CPU cores without explicit threading code, achieving near-linear speedup with core count for large batches.

Leverages ONNX Runtime's graph compilation and operator fusion to optimize embedding model inference. Automatically applies graph transformations including operator fusion (combining multiple ops into single fused kernel), constant folding, and memory layout optimization. Eliminates PyTorch overhead entirely by running compiled ONNX graphs directly, achieving 5-10x CPU speedup vs PyTorch-based alternatives while maintaining identical numerical outputs.

Provides unified Python API for switching between different embedding strategies (dense, sparse, late interaction, image, multimodal) without changing application code. Implements factory pattern where model selection is decoupled from inference logic, enabling A/B testing of different embedding models and strategies. Supports mixing multiple embedding types in single pipeline (e.g., dense + sparse for hybrid search) with automatic output format handling.