ExLlamaV2

Executes inference on EXL2-quantized models using dynamic per-token bit allocation, where different weight matrices are quantized to different bit depths (2-8 bits) based on sensitivity analysis. The framework loads quantized weights directly into VRAM and performs mixed-precision matrix multiplications, automatically selecting optimal bit widths per layer to balance quality and memory footprint without requiring full dequantization.

Loads and executes inference on GPTQ-quantized models using group-wise quantization, where weight matrices are divided into groups and each group is quantized independently with a shared scale factor. The framework performs fused dequantization-and-multiplication operations in GPU kernels to avoid materializing full-precision weights in VRAM, enabling inference on models that would otherwise exceed GPU memory.

Processes multiple sequences of different lengths in a single batch by padding shorter sequences to the longest sequence length and applying attention masks to ignore padding tokens. The framework automatically handles padding, mask generation, and unpadding of outputs, allowing efficient batched inference without manual sequence length management.

Quantizes full-precision models to EXL2 or GPTQ formats by analyzing layer sensitivity to quantization and selecting appropriate bit widths. For EXL2, the framework performs a sensitivity analysis pass to identify which layers tolerate lower bit depths, then quantizes each layer independently. For GPTQ, it uses group-wise quantization with configurable group size and bit width.

Provides an HTTP API compatible with OpenAI's chat completion and text completion endpoints, allowing drop-in replacement of OpenAI with local ExLlamaV2 inference. The API handles request parsing, model loading, inference execution, and response formatting, supporting streaming responses and standard sampling parameters.

Extends the context window of models beyond their training length using position interpolation (PI) or Rotary Position Embedding (RoPE) scaling. These techniques adjust positional encodings to accommodate longer sequences without retraining, allowing inference on sequences longer than the model's original training context.

Integrates Flash Attention 2 kernels to compute self-attention in O(N) memory and reduced FLOPs by fusing the attention computation (QK^T, softmax, attention dropout, value multiplication) into a single GPU kernel that operates on blocks of the query/key/value matrices. This avoids materializing the full NxN attention matrix in memory, enabling longer context windows and faster inference on the same hardware.

Implements a request queue and scheduler that batches multiple inference requests of varying lengths into a single GPU batch, automatically padding shorter sequences and scheduling requests to maximize GPU utilization. The scheduler uses a token-budget approach where it accumulates requests until adding another would exceed a configurable token limit, then executes the batch and immediately begins accumulating the next batch.

Implements speculative decoding where a smaller, faster draft model generates candidate tokens, and the main model validates them in parallel. If the draft model's predictions match the main model's top-1 choice, multiple tokens are accepted in a single forward pass; otherwise, the main model's prediction is used. This reduces the number of main model forward passes required to generate a sequence, achieving 1.5-2x speedup with minimal quality loss.

Loads Low-Rank Adaptation (LoRA) adapter weights and applies them to the base model during inference by computing the low-rank update (LoRA_A @ LoRA_B) and adding it to the original weight matrices. Supports multiple LoRA adapters with weighted combination, allowing fine-tuned behavior without modifying the base model weights or requiring full model retraining.

Generates tokens one at a time (or in small groups with speculative decoding) and streams them to the caller, supporting multiple sampling strategies including temperature scaling, top-k filtering, top-p (nucleus) sampling, and repetition penalty. The framework maintains generation state (KV cache, sequence length) across token steps, allowing the caller to interrupt or modify sampling parameters mid-generation.

Manages the Key-Value (KV) cache that stores intermediate attention computations across token generation steps. The framework automatically allocates cache space, reuses cache entries for identical prefixes (e.g., in batch processing), and evicts old cache entries when VRAM is exhausted. This reduces memory overhead and enables longer sequences without running out of VRAM.

Distributes model weights and computation across multiple GPUs using tensor parallelism, where each GPU holds a partition of the weight matrices and performs partial matrix multiplications. The framework automatically splits tensors along the appropriate dimensions, synchronizes partial results via all-reduce operations, and overlaps communication with computation to minimize latency.

Supports fine-tuning of quantized models by computing gradients through quantized weight matrices using straight-through estimators (STE) or other gradient approximations. The framework keeps weights quantized during forward and backward passes, avoiding full-precision weight materialization and enabling efficient fine-tuning on consumer GPUs.