CTranslate2

Executes encoder-decoder transformer models (Transformer base/big, M2M-100, NLLB, BART, mBART, Pegasus, T5, Whisper) through a specialized ctranslate2.Translator class that manages bidirectional attention computation, cross-attention between encoder and decoder stacks, and autoregressive decoding with configurable beam search or greedy strategies. The runtime applies layer fusion, padding removal, and in-place operations to accelerate the encoder-decoder forward pass while maintaining numerical stability across FP32, FP16, BF16, INT16, and INT8 precision modes.

Implements ctranslate2.Generator for autoregressive text generation from decoder-only models (GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, MPT, Llama, Mistral, Gemma, Falcon, Qwen2) using a custom decoding loop that supports beam search, sampling, nucleus sampling, and repetition penalties. The generator manages KV-cache reuse across generation steps, applies vocabulary filtering at each step, and supports early stopping via length penalties or custom stopping criteria, all while maintaining sub-linear memory growth during long-sequence generation.

Implements vocabulary mapping that restricts the decoder's output vocabulary to a subset of tokens, and token filtering that applies constraints during generation (e.g., disallow certain tokens, enforce token sequences). The mapping is applied at inference time without retraining, enabling use cases like domain-specific vocabulary restriction, preventing toxic outputs, or enforcing structured output formats. Token filtering supports regex patterns, token ID lists, and custom filtering functions.

Implements multiple decoding strategies for autoregressive generation: beam search (with configurable beam width and length penalty), greedy decoding, sampling (with temperature and top-k/top-p filtering), and repetition penalties that discourage repeated tokens. Each strategy is configurable at inference time without retraining, enabling users to trade off between output quality (beam search) and latency (greedy/sampling).

Implements multiple decoding strategies (greedy, beam search, sampling with top-k/top-p, temperature scaling, repetition penalty) that can be configured at inference time without reloading the model. The implementation is integrated into the Generator component and supports both encoder-decoder and decoder-only models, enabling diverse output generation from a single model.

Provides a quantization pipeline supporting FP32, FP16, BF16, INT16, INT8, and INT4 precision modes, with automatic ISA-aware backend selection that chooses optimal compute kernels for the target CPU (x86-64 with AVX2/AVX-512, ARM64 with NEON/SVE) or GPU (CUDA, Metal). The quantization is applied at model conversion time via ct2-transformers-converter, which uses per-channel weight quantization for linear layers and per-tensor quantization for activations, enabling 4-8x memory reduction with <2% accuracy loss on standard benchmarks.

Implements a batch processing pipeline that accepts multiple inference requests, dynamically reorders them by sequence length to minimize padding waste, and executes them in parallel across multiple GPUs or CPU cores using a thread pool. The reordering strategy groups similar-length sequences together, reducing the effective batch size for padding computation while maintaining throughput. Asynchronous execution via futures allows non-blocking submission of requests, enabling pipelined inference where new requests are queued while previous batches are still computing.

Provides ct2-transformers-converter CLI tool that automatically detects model architecture (encoder-decoder, decoder-only, encoder-only), extracts weights and configuration from Hugging Face model hub, applies CTranslate2 optimizations (layer fusion, operator specialization), and exports to a binary format with metadata. The converter handles vocabulary mapping, special token preservation, and quantization configuration, supporting 100+ model architectures without manual layer mapping.

Implements ctranslate2.models.Whisper for running OpenAI Whisper models with optimized encoder-decoder inference, automatic audio resampling to 16kHz, mel-spectrogram computation, and token-level timestamp prediction. The Whisper component handles multi-language detection, task specification (transcription vs translation), and confidence scoring per token, all within the CTranslate2 optimized runtime without requiring librosa or torchaudio dependencies.

Implements ctranslate2.Encoder for running encoder-only models (BERT, DistilBERT, XLM-RoBERTa) that produce contextual embeddings or classification logits without autoregressive decoding. The encoder applies the same optimizations as other model types (layer fusion, padding removal, quantization) and supports pooling strategies (mean, max, CLS token) for generating fixed-size embeddings from variable-length sequences.

Implements distributed inference by partitioning model weights across multiple GPUs, with each GPU computing a portion of the matrix multiplications and communicating intermediate results via all-reduce operations. The tensor parallelism strategy is transparent to the user; model loading automatically detects available GPUs and distributes weights, with synchronization handled internally via NCCL (NVIDIA Collective Communications Library) or custom communication backends.

Implements automatic CPU backend selection at runtime that detects available instruction sets (AVX2, AVX-512, VNNI for x86-64; NEON, SVE for ARM64) and selects optimized compute kernels for matrix multiplication, quantization, and activation functions. The dispatch mechanism is transparent; a single compiled binary runs on heterogeneous CPUs and automatically uses the fastest available kernels without recompilation or manual configuration.

Provides specialized converters (ct2-opennmt-py-converter, ct2-opennmt-tf-converter, ct2-fairseq-converter, ct2-marian-converter) that extract weights and configuration from non-Hugging Face frameworks, apply CTranslate2 optimizations, and export to binary format. Each converter handles framework-specific model definitions, checkpoint formats, and vocabulary structures, enabling users to leverage existing models trained in these frameworks without retraining.