Quantization Aware Adapter Training With Frozen Base Weights

via “quantization with multiple precision formats and calibration strategies”

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Unique: Implements a modular quantization system (src/transformers/quantization_config.py) that abstracts away backend-specific quantization details (bitsandbytes, GPTQ, AWQ) behind a unified QuantizationConfig interface, enabling seamless switching between quantization strategies

vs others: More accessible than standalone quantization libraries because it integrates quantization into model loading via config parameters, automatically handling weight conversion and calibration without requiring separate quantization pipelines

via “quantization with fp8 and low-precision inference”

High-throughput LLM serving engine — PagedAttention, continuous batching, OpenAI-compatible API.

Unique: Implements fused quantization kernels that perform dequantization and matrix multiplication in a single GPU operation, reducing memory bandwidth overhead vs separate dequant+compute steps

vs others: Achieves 4-8x memory reduction with 1-3% accuracy loss vs no quantization, outperforming naive INT8 quantization by using per-token scaling and mixed-precision strategies

via “activation-aware 4-bit weight quantization with minimal accuracy loss”

4-bit weight quantization for LLMs on consumer GPUs.

Unique: Uses activation-aware scaling that analyzes per-channel activation magnitudes from calibration data to selectively protect high-impact weight channels, rather than uniform quantization across all weights. This channel-wise approach with activation-guided clipping preserves model quality better than post-training quantization methods that don't account for activation patterns.

vs others: Outperforms GPTQ and naive post-training quantization by 2-3% accuracy on benchmarks because it preserves activation-salient weights; faster quantization than QLoRA because it doesn't require training, enabling same-day deployment of new models.

via “quantization-aware adapter training (qlora integration)”

Parameter-efficient fine-tuning — LoRA, QLoRA, adapter methods for LLMs on consumer GPUs.

Unique: Implements a gradient routing pattern where the quantized base model is frozen and only adapter parameters receive gradient updates, avoiding the computational cost of dequantization during backpropagation. Integrates with bitsandbytes' quantization kernels to maintain quantized state throughout training while preserving numerical stability in adapter gradients.

vs others: Achieves 4-8x memory reduction compared to standard LoRA on full-precision models while maintaining comparable accuracy, making it the only practical approach for fine-tuning 70B+ models on consumer hardware.

via “quantization-aware fine-tuning with gradient computation on quantized weights”

Optimized quantized LLM inference for consumer GPUs — EXL2/GPTQ, flash attention, memory-efficient.

Unique: Implements quantization-aware fine-tuning by computing gradients through quantized weights using straight-through estimators, keeping weights quantized throughout training. This avoids dequantizing weights and enables efficient fine-tuning on consumer GPUs.

vs others: More memory-efficient than dequantizing weights for fine-tuning because it keeps weights quantized throughout training, whereas naive approaches dequantize weights for gradient computation which doubles memory usage.

via “one-shot post-training quantization with calibration-free execution”

Toolkit for LLM quantization, pruning, and distillation.

Unique: Uses a modifier-based architecture where quantization logic is injected as PyTorch hooks into the model graph, enabling algorithm-agnostic calibration and composition of multiple compression techniques (quantization + pruning + distillation) in a single pipeline without model rewriting

vs others: Faster than AutoGPTQ or GPTQ-for-LLaMA because it abstracts algorithm selection and calibration into reusable modifiers, allowing parallel experimentation; more flexible than ONNX Runtime quantization because it preserves PyTorch semantics and integrates directly with vLLM

via “quantization with multiple precision formats and framework support”

Hugging Face's model library — thousands of pretrained transformers for NLP, vision, audio.

Unique: Integrates multiple quantization backends (bitsandbytes, GPTQ, AWQ) under a unified API where quantization method is specified via config object, enabling transparent switching between quantization schemes. Quantization is applied during model loading via load_in_8bit/load_in_4bit flags, avoiding explicit conversion code.

vs others: More convenient than manual quantization with bitsandbytes because quantization is applied automatically during model loading. More flexible than ONNX quantization because it supports multiple quantization methods and frameworks.

via “quantization and dequantization operations with configurable bit-widths”

8-bit and 4-bit quantization enabling QLoRA fine-tuning.

Unique: Implements both vector-wise (per-column) and block-wise (per-block) quantization with absmax-based scaling, supporting multiple data types (int8, int4, NF4, FP4) through a unified functional API. Uses CUDA kernels for efficient quantization/dequantization without materializing intermediate full-precision tensors.

vs others: Provides more flexible quantization strategies than fixed-scheme quantizers, and achieves better accuracy-efficiency tradeoffs by supporting data-type-specific quantization (NF4 for weights, FP4 for gradients).

via “model quantization for memory and latency reduction”

text-generation model by undefined. 1,60,37,172 downloads.

Unique: Supports both post-training quantization (no retraining) via bitsandbytes and quantization-aware training (better accuracy) via torch.quantization, with automatic calibration dataset selection for minimal accuracy loss

vs others: Faster and simpler than knowledge distillation (which requires training a smaller model), but less accurate than distillation for extreme compression — best for 2-4x size reduction, not 10x+

via “model quantization and compression for edge deployment”

fill-mask model by undefined. 5,92,18,905 downloads.

Unique: Post-training quantization via ONNX Runtime or PyTorch quantization APIs requires no retraining while achieving 4x model size reduction; supports multiple quantization schemes (symmetric, asymmetric, per-channel) for fine-grained accuracy-efficiency control

vs others: Simpler than quantization-aware training (no retraining required) and more portable than framework-specific quantization due to ONNX support

via “block-wise weight-only quantization with optional 4-bit/8-bit compression”

AirLLM 70B inference with single 4GB GPU

Unique: Quantizes weights only while preserving activation precision, differing from standard quantization (QAT/PTQ) that quantizes both weights and activations — maintains better accuracy by avoiding activation quantization noise while still reducing I/O overhead

vs others: Achieves 3x speed improvement with minimal accuracy loss, whereas GPTQ/AWQ require more complex calibration; simpler than mixed-precision quantization but less flexible than per-layer bit-width selection

via “quantization-aware-inference-optimization”

fill-mask model by undefined. 10,73,316 downloads.

Unique: Distilled model size (82M parameters, ~270MB fp32) quantizes to ~70MB (int8) with minimal accuracy loss, enabling deployment on devices with <100MB available memory, whereas RoBERTa-base (125M parameters, ~500MB) quantizes to ~130MB

vs others: Post-training quantization is simpler than quantization-aware training but less accurate; quantized distilled models offer better accuracy-efficiency tradeoff than training smaller models from scratch

via “model quantization for edge deployment”

image-segmentation model by undefined. 1,55,904 downloads.

Unique: Supports standard PyTorch post-training quantization without model-specific modifications, enabling straightforward int8 deployment — though deformable attention operations may not quantize cleanly

vs others: Reduces model size 4x (500MB to 125MB) with minimal accuracy loss vs float32, enabling edge deployment, though 1-2% accuracy degradation and limited hardware support add deployment complexity

via “quantized-model-inference-with-8-bit-precision”

image-segmentation model by undefined. 5,08,692 downloads.

Unique: Post-training quantization applied to pre-trained SegFormer B0 without retraining — uses per-channel scale factors for weights and per-tensor scale factors for activations, optimized for ONNX Runtime's quantization-aware execution

vs others: Simpler than quantization-aware training (no retraining required), smaller than float32 baseline while maintaining comparable accuracy to knowledge distillation approaches, and directly compatible with ONNX Runtime without custom kernels

via “model quantization and compression compatibility”

question-answering model by undefined. 1,45,572 downloads.

Unique: Distributed in safetensors format (safer than pickle, faster to load) with explicit compatibility declarations for ONNX and TensorRT, enabling zero-copy quantization without intermediate format conversions

vs others: Smaller base model (84M vs 110M for BERT-base) quantizes more aggressively with better accuracy retention, and safetensors format eliminates pickle deserialization vulnerabilities present in older model distributions

via “quantization with fp8 and low-precision inference”

A high-throughput and memory-efficient inference and serving engine for LLMs

Unique: Implements FP8 quantization with hardware-accelerated matrix operations on NVIDIA H100/L40S GPUs, using native FP8 Tensor Cores to eliminate quantization overhead. Supports per-token dynamic quantization where activation scales are computed per-token rather than per-batch, improving accuracy.

vs others: Achieves 4-8x model compression with <2% accuracy loss on FP8 (vs. 5-10% loss for INT8 on same models); FP8 inference on H100 is only 5-10% slower than FP16 due to native hardware support, vs. 20-30% slowdown for INT8 on older GPUs.

via “quantization-aware training with 2/4/8-bit precision and bitsandbytes integration”

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Unique: Integrates bitsandbytes quantization kernels with LoRA adapter system to enable 4-bit training with NF4 format, supporting nested quantization (double_quant) for additional memory savings. Automatically handles quantization/dequantization in forward/backward passes without user intervention.

vs others: Native 4-bit quantization with NF4 format vs. alternatives like GPTQ which requires post-training quantization, enabling QLoRA training on consumer GPUs without pre-quantized models.

via “quantization-aware adapter training with frozen base weights”

Parameter-Efficient Fine-Tuning (PEFT)

Unique: Integrates seamlessly with bitsandbytes quantization through the PeftModel wrapper, automatically detecting quantized layer types and routing adapter computations appropriately. The implementation preserves gradient flow through quantized weights without dequantization, achieved via careful handling of backward passes in the adapter injection layer.

vs others: More memory-efficient than QLoRA alternatives because PEFT's unified adapter interface works with any quantization backend, while QLoRA implementations are often tightly coupled to specific quantization libraries. Supports both 4-bit and 8-bit quantization with identical API.

via “mixed-precision training with automatic loss scaling”

* ⭐ 02/2023: [Adding Conditional Control to Text-to-Image Diffusion Models (ControlNet)](https://arxiv.org/abs/2302.05543)

Unique: Implements dynamic loss scaling that monitors gradient statistics and adjusts scale factors per training step, preventing both underflow and overflow without manual intervention. Uses gradient skipping when overflow is detected, maintaining training stability across variable batch sizes and learning rates.

vs others: Achieves 40-50% memory reduction and 1.5-2x speedup vs float32 training with <0.5% accuracy loss, compared to quantization-aware training (which requires post-training calibration) or knowledge distillation (which requires a teacher model). Requires minimal code changes compared to alternatives.

via “lora adapter fine-tuning with frozen quantized base model”

* ⭐ 05/2023: [Voyager: An Open-Ended Embodied Agent with Large Language Models (Voyager)](https://arxiv.org/abs/2305.16291)

Unique: Combines LoRA with 4-bit quantization in a unified framework where adapters are trained in full precision while base weights remain frozen and quantized, enabling end-to-end fine-tuning without dequantization — prior LoRA work assumed full-precision base models or required dequantization during training

vs others: Achieves 10x lower memory consumption than standard LoRA on full-precision models by freezing quantized weights, and enables fine-tuning of 70B models on single GPUs where full-precision LoRA would require multi-GPU setups or gradient checkpointing