CTranslate2

Executes pre-trained encoder-decoder transformer models (Transformer base/big, NLLB, BART, mBART, Pegasus, T5, Whisper) through a custom C++ runtime that applies layer fusion, padding removal, and in-place operations to accelerate inference. The Translator component manages the encoder-decoder pipeline, handling variable-length input sequences and generating target sequences with configurable decoding strategies. Supports batch processing with automatic reordering to maximize throughput while maintaining low latency.

Implements the Generator component for decoder-only transformer models (Llama, Mistral, Falcon, MPT, GPT-2, OPT, BLOOM, Qwen2, Gemma, CodeGen) using a custom C++ runtime with KV-cache management, dynamic batching, and advanced decoding strategies (beam search, sampling, nucleus sampling, top-k). The Generator manages autoregressive token generation with support for interactive generation, prefix constraints, and early stopping. Tensor parallelism distributes inference across multiple GPUs for models exceeding single-GPU memory.

Provides multiple decoding strategies for text generation including greedy decoding, beam search with configurable beam width, temperature-based sampling, nucleus (top-p) sampling, and top-k sampling. Supports advanced features like length penalties, coverage penalties, and vocabulary constraints to guide generation toward desired outputs. Decoding strategies are compiled into the inference graph at model conversion time and cannot be changed at runtime. Supports early stopping based on EOS token or maximum length.

Allows definition of custom transformer architectures through ModelSpec configuration files that specify layer types, attention patterns, activation functions, and other architectural details. The ModelSpec abstraction decouples model architecture from the inference engine, enabling support for novel transformer variants without modifying core CTranslate2 code. Supports encoder-decoder, decoder-only, and encoder-only architectures with flexible layer composition. Custom architectures must be defined before model conversion; runtime architecture changes are not supported.

Automatically fuses multiple transformer layers (e.g., linear projection + activation + normalization) into single optimized kernels during model conversion, reducing memory bandwidth and kernel launch overhead. Padding removal eliminates unnecessary computation on padding tokens by tracking sequence lengths and skipping padded positions in attention and feed-forward layers. These optimizations are applied at the C++ level and are transparent to users. Combined effect is 2-5x latency reduction compared to unfused implementations.

Detects CPU capabilities at runtime and automatically selects optimized backend implementations (AVX, AVX2, AVX-512, NEON for ARM64) without requiring manual configuration. The CPU dispatch layer in CTranslate2 profiles the host CPU's instruction set support and routes tensor operations to the fastest available implementation. Supports x86-64 and AArch64/ARM64 processors with architecture-specific GEMM kernels and SIMD operations. No performance penalty for unsupported instruction sets; gracefully falls back to portable implementations.

Converts model weights and activations to reduced-precision formats (INT8, INT16, FP16, BF16, INT4) during model conversion, reducing memory footprint and accelerating inference without retraining. The quantization pipeline applies per-layer or per-channel quantization with learned scale factors and zero points. Supports mixed-precision inference where different layers use different precisions based on sensitivity analysis. Automatic precision selection recommends optimal quantization levels per layer to maximize accuracy-speed tradeoff.

Converts pre-trained transformer models from multiple training frameworks (Hugging Face Transformers, OpenNMT-py, OpenNMT-tf, Fairseq, Marian, OPUS-MT) into CTranslate2's optimized binary format. The conversion pipeline extracts weights, applies layer fusion, computes quantization scale factors, and generates architecture-specific execution graphs. Conversion is a one-time offline process that produces a portable model file compatible with any CTranslate2 runtime. Supports custom model architectures via ModelSpec configuration.

Manages multiple inference requests in parallel using dynamic batch reordering to maximize GPU/CPU utilization while maintaining per-request latency SLAs. The batch processing layer automatically reorders requests based on sequence length and model architecture to minimize padding overhead. Asynchronous execution allows clients to submit requests without blocking, with results available via callback or polling. Supports variable batch sizes and dynamic batching where requests are grouped at runtime rather than pre-allocated.

Distributes inference across multiple GPUs using tensor parallelism, where each GPU processes a different part of the model's tensors. The ModelReplica abstraction manages GPU allocation and communication, transparently splitting large models (70B+ parameters) across multiple GPUs. Supports both intra-layer parallelism (splitting weight matrices) and inter-layer parallelism (assigning different layers to different GPUs). Communication overhead is minimized through optimized all-reduce operations and overlapping computation with communication.

Implements the Whisper component for efficient speech-to-text inference on pre-trained Whisper models (tiny, base, small, medium, large). Handles audio preprocessing (resampling to 16kHz, mel-spectrogram computation, padding) and runs the encoder-decoder transformer pipeline optimized for audio input. Supports variable-length audio with automatic padding removal. Decoding strategies include greedy decoding, beam search, and language-aware decoding with vocabulary constraints.

Implements the Encoder component for encoder-only transformer models (BERT, DistilBERT, XLM-RoBERTa) optimized for text classification, semantic similarity, and embedding generation. The encoder processes input sequences through the transformer stack and outputs contextualized token embeddings or pooled sentence embeddings. Supports batch processing with dynamic padding removal and layer fusion optimizations. No decoding stage; output is raw embeddings or classification logits.

Leverages NVIDIA CUDA for GPU acceleration of tensor operations, with automatic GPU memory management and optimization. The GPU backend implements fused kernels for common operations (attention, layer normalization, GEMM) and manages GPU memory allocation to minimize fragmentation. Supports multiple GPUs with automatic device selection and load balancing. Memory optimization techniques include in-place operations, activation checkpointing, and dynamic memory allocation based on batch size.