ExLlamaV2

Executes inference on EXL2-format quantized models using a dynamic token allocation system that adjusts per-layer quantization precision based on available VRAM and batch size. The framework implements row-wise quantization with per-token scaling factors, enabling sub-4-bit effective precision while maintaining quality. This approach allows models to fit on consumer GPUs (8-24GB) that would normally require 40GB+ for full precision.

Loads and executes inference on GPTQ-quantized models using group-wise quantization with learned scaling factors per group. ExLlamaV2 implements optimized CUDA kernels for GPTQ dequantization that fuse multiple operations (scaling, addition, activation) into single kernel calls, reducing memory bandwidth overhead. Supports variable group sizes (32-128) and mixed-precision configurations where different layers use different bit-widths.

Caches key-value (KV) pairs from previous tokens to avoid recomputing attention for the entire conversation history on each new token. Implements a sliding-window KV cache that stores only the most recent N tokens' KV pairs, reducing memory overhead while maintaining context awareness. Supports cache invalidation and reuse across multiple conversation turns, with automatic cache size management based on available VRAM.

Caches computed activations for common prompt prefixes (e.g., system prompts, few-shot examples) and reuses them across multiple inference requests with different suffixes. Uses prefix matching to identify when a new prompt shares a prefix with a cached prompt, then skips recomputation for the shared portion. Supports hierarchical caching where different prefix lengths are cached separately, enabling fine-grained reuse.

Provides tools to evaluate quantized models and measure quality degradation compared to full-precision baselines. Implements multiple evaluation metrics: perplexity on standard benchmarks (WikiText, C4), task-specific metrics (BLEU for translation, F1 for QA), and custom metrics. Supports side-by-side comparison of multiple quantized variants to identify optimal quantization parameters for specific quality targets.

Converts between quantization formats (e.g., GPTQ to EXL2) and optimizes quantized models for specific hardware. The framework analyzes model architecture and hardware capabilities to recommend optimal quantization parameters (bit-width, group size) and performs format conversion with minimal quality loss. Supports batch conversion of multiple models and provides quality metrics (perplexity, task-specific benchmarks) to validate conversions.

Integrates Flash Attention 2 algorithm to compute attention with O(N) memory complexity instead of O(N²), using tiling and recomputation to avoid materializing the full attention matrix. ExLlamaV2 wraps Flash Attention 2 with custom CUDA kernels that optimize for quantized weight access patterns and support variable sequence lengths without padding overhead. Automatically falls back to standard attention for unsupported configurations (e.g., custom attention masks).

Implements dynamic batching that groups multiple inference requests into a single forward pass, with adaptive batch size scheduling that adjusts batch size based on available VRAM and latency targets. The scheduler uses a token-budget approach: it accumulates requests until the total token count would exceed the budget, then executes the batch. Supports variable-length sequences within a batch without padding waste through ragged tensor operations.

Implements speculative decoding by running a smaller draft model (e.g., 7B) to generate candidate tokens, then verifying them with the main model (e.g., 70B) in parallel. Uses a rejection sampling approach: if the draft model's probability for a token exceeds the main model's probability, the token is accepted; otherwise, it's rejected and the main model generates a replacement. This reduces the number of main model forward passes by 2-4x while maintaining identical output distribution.

Loads Low-Rank Adaptation (LoRA) adapters and applies them during inference by computing the low-rank update matrices (A and B) and adding them to the base model weights. ExLlamaV2 implements two strategies: (1) weight merging, which fuses LoRA weights into the base model before inference (faster but requires model reloading for adapter switching), and (2) on-the-fly application, which computes LoRA updates during forward passes (slower but supports dynamic adapter switching). Supports multiple concurrent LoRA adapters with weighted combination.

Distributes model weights across multiple GPUs using tensor parallelism, where each GPU holds a partition of the weight matrices and computes a portion of the matrix multiplication. ExLlamaV2 implements column-wise and row-wise partitioning strategies with all-reduce communication to synchronize partial results across GPUs. Supports both intra-node (NVLink) and inter-node (PCIe/Ethernet) communication with automatic topology detection and optimization.

Generates tokens one at a time with callback functions invoked for each generated token, enabling real-time streaming output to clients without buffering the entire response. Implements a generator pattern where the inference loop yields control after each token, allowing the application to process or transmit the token before requesting the next one. Supports early stopping based on callback return values (e.g., stop if user disconnects) and token filtering/transformation before output.

Provides configurable sampling strategies that control token generation randomness and diversity. Implements temperature scaling (adjusts logit distribution), top-k filtering (keeps only k highest-probability tokens), top-p (nucleus sampling, keeps tokens until cumulative probability reaches p), and repetition penalty (reduces probability of recently-generated tokens). Supports combining multiple strategies simultaneously and per-token customization of parameters.

Provides tools to convert full-precision models to GPTQ or EXL2 quantized formats, with options for calibration data selection, quantization parameters (bit-width, group size), and post-quantization optimization. The conversion process uses a layer-by-layer approach: for each layer, it computes optimal quantization parameters by minimizing reconstruction error on calibration data, then applies quantization and stores the result. Supports mixed-precision quantization where different layers use different bit-widths based on sensitivity analysis.