llmcompressor

Applies quantization algorithms (GPTQ, AWQ, AutoRound) to pre-trained models in a single forward pass without requiring fine-tuning, using a modifier-based architecture that injects quantization observers into the model graph during a calibration phase. The system traces model execution on representative data, collects activation statistics via the observer system, and applies learned quantization parameters without gradient updates, enabling sub-hour compression of 70B+ parameter models on consumer hardware.

Implements a composable modifier system where each compression technique (quantization, pruning, distillation) is a discrete Modifier object that hooks into model layers via PyTorch's forward/backward passes. The CompressionSession manages modifier lifecycle, state persistence, and execution order, allowing multi-stage compression recipes where modifiers can be applied sequentially or in parallel with dependency tracking. State is serialized to disk between stages, enabling resumable compression workflows.

Extends compression techniques to multimodal models (vision-language models like LLaVA, CLIP) by handling both vision and language components with architecture-aware compression. Applies quantization/pruning to vision encoders and language models separately, with special handling for cross-modal alignment layers. Supports calibration on image-text pairs and validates compression on multimodal tasks (visual QA, image captioning).

Serializes compressed models to the compressed-tensors format, which combines safetensors (weight storage) with JSON metadata (quantization scales, zero-points, sparsity masks, pruning info). This format is natively supported by vLLM's inference engine, enabling zero-copy loading of quantized weights and automatic kernel selection based on quantization scheme. Metadata includes algorithm version, calibration info, and hardware targets for reproducibility.

Enables quantization-aware training (QAT) and pruning-during-training by injecting quantization observers and pruning masks into the model during fine-tuning. Modifiers hook into the backward pass to simulate quantization error and update pruning masks based on gradients. Supports both full fine-tuning and parameter-efficient methods (LoRA, QLoRA) with compression, enabling task-specific optimization of quantization/pruning parameters.

Provides a declarative YAML-based recipe system for defining compression pipelines without writing Python code. Recipes specify modifier sequences, algorithm parameters, calibration data, and evaluation metrics in structured YAML, which the framework parses and executes via the CompressionSession. Supports recipe composition (include other recipes), conditional execution (apply modifier if condition met), and parameter sweeps for hyperparameter tuning.

Provides built-in evaluation utilities for measuring compression impact on model accuracy across multiple metrics: perplexity on language modeling, accuracy on classification tasks, BLEU on translation, and custom task-specific metrics. Supports both calibration-set evaluation (fast) and held-out test-set evaluation (accurate), with automatic metric computation and logging. Integrates with HuggingFace Evaluate library for standard benchmark support.

Provides comprehensive logging and monitoring of compression process, including per-layer quantization statistics (scales, zero-points, clipping rates), pruning masks, modifier execution timing, and memory usage. Logs are structured (JSON) and can be exported to monitoring systems (Weights & Biases, TensorBoard). Includes real-time progress tracking and compression statistics visualization.

Instruments model layers with Observer objects that capture activation statistics (min, max, mean, percentiles) during forward passes on calibration data. These statistics inform quantization scale/zero-point computation for activation quantization (W8A8, W4A8) via algorithms like SmoothQuant that redistribute quantization burden from activations to weights. The observer system supports pluggable collection strategies (per-channel, per-token, per-group) and can be toggled on/off without recompiling the model.

Implements GPTQ (Generative Pre-trained Transformer Quantization) which computes the Hessian matrix of the model's loss with respect to weights, then greedily quantizes weights in order of least importance (lowest Hessian diagonal values). This approach minimizes quantization error by prioritizing precision for weights that most affect model output. The implementation uses layer-wise Hessian computation to fit in GPU memory and supports both per-channel and per-group quantization granularity.

Implements AWQ (Activation-Weighted Quantization) which scores weight importance by the magnitude of activations flowing through each weight during calibration. Weights connected to high-activation channels are preserved at higher precision, while low-activation weights are quantized more aggressively. This approach is faster than GPTQ (no Hessian computation) and often achieves comparable accuracy by leveraging the observation that activation magnitude correlates with weight importance in transformers.

Implements AutoRound, a quantization method that learns optimal rounding for quantized weights by minimizing task loss (e.g., language modeling loss) rather than reconstruction error. Uses a differentiable rounding function and gradient-based optimization to adjust rounding decisions per weight, enabling fine-grained control over quantization-accuracy tradeoff. Supports both weight-only and activation quantization with layer-wise or global optimization strategies.

Applies pruning techniques that remove weights or neurons based on magnitude, gradient, or learned importance scores. Supports both structured pruning (remove entire channels/heads) and unstructured pruning (remove individual weights), with layer-wise mask generation and application. Pruning can be applied one-shot (magnitude-based) or during fine-tuning (gradient-based), with automatic sparsity target enforcement via iterative pruning or lottery ticket discovery.

Enables compression of models larger than GPU memory by streaming layers from disk to GPU sequentially, processing each layer's compression independently, then offloading to disk. Uses a layer-by-layer execution strategy that traces the model graph to identify layer boundaries, loads only necessary layers into GPU memory, and maintains intermediate activations on disk. This approach reduces peak memory usage from 3-4x model size to ~1.5x, enabling 70B+ model compression on 24GB GPUs.

Distributes compression workload across multiple GPUs using PyTorch's distributed training infrastructure (FSDP, DDP). Splits model layers across GPUs, synchronizes calibration data and quantization parameters, and coordinates modifier execution across devices. Supports both data-parallel (replicate model on each GPU) and model-parallel (shard model across GPUs) strategies, with automatic gradient synchronization for fine-tuning-based compression.

Applies compression techniques to MoE models (e.g., Mixtral, Phi-MoE) with expert-level granularity, enabling selective compression of high-utilization experts while preserving precision for rarely-used experts. Tracks expert routing statistics during calibration, applies quantization/pruning per-expert, and maintains router weights at full precision. Supports both dense and sparse MoE architectures with automatic expert grouping for efficient computation.