infinity-emb

Accumulates incoming embedding requests into optimally-sized batches using a BatchHandler that balances latency and throughput, then executes batches on GPU/accelerator hardware via backend-specific inference pipelines (PyTorch, ONNX/TensorRT, CTranslate2, AWS Neuron). The system uses multi-threaded tokenization to parallelize text preprocessing while batches are formed, reducing end-to-end latency by overlapping I/O and compute.

Manages multiple embedding/reranking models simultaneously within a single server process using AsyncEngineArray, which routes incoming requests to the appropriate AsyncEmbeddingEngine instance based on model ID. Each model maintains its own inference pipeline, GPU memory allocation, and batch queue, enabling efficient resource sharing and model hot-swapping without server restart.

Provides a Python SDK (AsyncEmbeddingEngine, AsyncEngineArray) for programmatic embedding generation without HTTP overhead, enabling direct in-process inference for Python applications. The SDK supports async/await patterns for non-blocking inference and batch operations, with automatic model loading and GPU memory management.

Implements a FastAPI-based REST server that exposes embedding, reranking, and classification models via HTTP endpoints. The server handles request routing, response formatting, error handling, and OpenAPI documentation generation, with support for both OpenAI and Cohere API formats.

Provides a command-line interface (infinity_emb command) for starting the embedding server with configuration via CLI arguments or environment variables. The CLI handles model loading, server startup, and configuration management, enabling one-command deployment without writing Python code.

Provides Docker images and docker-compose configuration for containerized deployment of Infinity, with pre-built images for different hardware backends (CUDA, ROCM, CPU). The Dockerfile handles dependency installation, model caching, and server startup, enabling reproducible deployments across environments.

Implements a caching layer that deduplicates identical embedding requests and returns cached results, reducing redundant inference. The cache stores embeddings by input text hash and returns cached results for repeated queries, with configurable cache size and TTL.

Supports pre-loading models into GPU memory on server startup, eliminating cold-start latency for the first request. The system can warm up multiple models simultaneously and verify they load correctly before accepting requests.

Exposes a REST API endpoint that mirrors OpenAI's embeddings API specification, accepting requests with text input and returning embedding vectors in OpenAI format (with usage statistics). This compatibility layer enables drop-in replacement of OpenAI API calls with local Infinity instances by simply changing the base URL, without modifying client code.

Provides a REST API endpoint that implements Cohere's reranking API specification, accepting a query and list of documents, then returning relevance scores for each document. This enables using open-source reranking models (e.g., mxbai-rerank-xlarge) as a drop-in replacement for Cohere's reranking service without changing client code.

Generates embeddings for both text and images using CLIP-based models (e.g., openai/clip-vit-base-patch32), producing aligned vector representations in a shared embedding space. The system handles image preprocessing (resizing, normalization), tokenization, and dual-stream inference through a unified embedding pipeline that supports batch processing of mixed text and image inputs.

Generates embeddings for audio files using CLAP (Contrastive Language-Audio Pre-training) models, producing aligned embeddings in a shared space with text. The system handles audio preprocessing (resampling, normalization), spectrogram generation, and inference through the embedding pipeline, enabling audio-text cross-modal retrieval.

Executes text classification models (e.g., sentiment analysis, topic classification) that produce logits or probabilities for predefined classes. The system batches classification requests and returns class predictions with confidence scores, supporting both multi-class and multi-label classification through the unified inference pipeline.

Compiles and executes models using ONNX Runtime with TensorRT optimization, converting PyTorch/HuggingFace models to ONNX format and applying GPU-specific optimizations (quantization, kernel fusion, memory optimization). This backend provides 2-10x speedup over PyTorch inference for compatible models while reducing memory footprint.

Executes models using CTranslate2, a C++ inference engine optimized for CPU and GPU inference with support for model quantization and efficient memory management. This backend enables fast inference on CPU-only hardware and provides 5-20x speedup over PyTorch on CPU by using optimized kernels and reduced precision arithmetic.

Executes models on AWS Inferentia and Trainium accelerators using AWS Neuron SDK, providing optimized inference on AWS-specific hardware. This backend compiles models to Neuron format and executes them on Inferentia chips, offering cost-effective inference at scale with lower power consumption than GPUs.