AutoGPTQ

Implements the GPTQ quantization algorithm to compress model weights to 2/3/4/8-bit precision while maintaining activation precision, using a layer-wise quantization process that calibrates quantization parameters against representative data samples. The framework supports configurable group sizes (typically 128) and activation description (desc_act) flags to balance compression ratio against accuracy preservation, enabling up to 4x memory reduction compared to FP16 models.

Provides pluggable backend implementations (CUDA, Exllama/ExllamaV2, Marlin, Triton, ROCm, HPU) that execute quantized matrix multiplications using specialized low-level kernels optimized for each hardware target. The framework abstracts backend selection through a factory pattern (AutoGPTQForCausalLM), automatically selecting the fastest available kernel based on GPU architecture and quantization configuration, with fallback chains for unsupported configurations.

Provides utilities for batching quantization and inference operations across multiple models or datasets, with automatic batching, scheduling, and result aggregation. The pipeline supports mixed quantization configs (different bit-widths, group sizes) in single batch, with automatic GPU memory management and fallback to CPU if GPU memory exhausted. Batch processing enables efficient resource utilization when quantizing or inferencing multiple models.

Provides validation utilities to check quantization config compatibility with target model architecture and hardware, detecting invalid configurations before quantization begins. The validator checks bit-width support, group size constraints, backend availability, and GPU architecture compatibility, providing detailed error messages and suggestions for valid configurations. Validation prevents wasted compute on incompatible configs and ensures reproducibility across environments.

Provides a plugin architecture for adding support to new model architectures through subclassing BaseGPTQForCausalLM and implementing architecture-specific quantization logic (layer mapping, fused operations, attention patterns). The framework includes pre-built implementations for 30+ architectures (Llama, Mistral, Falcon, Qwen, Yi, etc.) with automatic model detection via HuggingFace config, enabling quantization of custom or emerging models by implementing a minimal set of required methods.

Implements a calibration pipeline that processes representative data samples through the model to compute per-group quantization scales and zero-points that minimize reconstruction error. The process uses Hessian-based optimization (second-order information) to determine optimal quantization parameters, with support for both symmetric and asymmetric quantization schemes, enabling data-aware compression that preserves model accuracy better than blind quantization.

Integrates with PEFT (Parameter-Efficient Fine-Tuning) library to enable LoRA and other adapter-based fine-tuning on frozen quantized weights, allowing model adaptation without dequantization or full fine-tuning. The integration automatically wraps quantized linear layers with PEFT adapters, enabling gradient computation only through low-rank adapter matrices while keeping quantized weights frozen, reducing fine-tuning memory by 10-20x compared to full fine-tuning.

Implements architecture-specific fused kernels that combine multiple operations (attention computation, MLP forward pass) into single GPU kernels, reducing memory bandwidth and kernel launch overhead during quantized inference. Fused operations are automatically applied when available for the target architecture and GPU, transparently replacing standard PyTorch operations with optimized implementations that operate directly on quantized weights.

Provides seamless integration with HuggingFace Hub for uploading, downloading, and sharing quantized models with automatic metadata preservation. Quantized models can be saved to Hub with quantization config, enabling one-line loading via AutoGPTQForCausalLM.from_quantized(model_id), with full reproducibility through saved quantization parameters and calibration metadata. The integration handles model versioning, access control, and model card generation automatically.

Provides built-in evaluation tasks (perplexity, benchmark datasets like MMLU, HellaSwag) to measure quantization impact on model performance, enabling quantitative comparison between full-precision and quantized models. The framework supports standard evaluation protocols and datasets, with automatic metric computation and result logging, enabling data-driven decisions about quantization tradeoffs.

Supports distributed quantization across multiple GPUs using data parallelism, enabling quantization of models too large to fit in single GPU memory. The framework automatically partitions model layers across GPUs, synchronizes calibration data, and coordinates quantization process, with transparent communication via PyTorch distributed backend (NCCL, Gloo). Distributed quantization maintains accuracy of single-GPU quantization while reducing wall-clock time through parallelization.

Provides utilities to prepare models for quantization-aware training (QAT), where quantization is simulated during training to learn optimal weights under quantization constraints. The framework includes fake quantization layers that simulate int4 quantization during forward pass while maintaining gradient flow, enabling fine-tuning that adapts weights to quantization before actual quantization. QAT typically preserves 1-2% more accuracy than post-training quantization (PTQ) at the cost of longer training time.