Block Wise Weight Only Quantization With Optional 4 Bit 8 Bit Compression

via “quantization with multiple precision formats and calibration strategies”

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Unique: Implements a modular quantization system (src/transformers/quantization_config.py) that abstracts away backend-specific quantization details (bitsandbytes, GPTQ, AWQ) behind a unified QuantizationConfig interface, enabling seamless switching between quantization strategies

vs others: More accessible than standalone quantization libraries because it integrates quantization into model loading via config parameters, automatically handling weight conversion and calibration without requiring separate quantization pipelines

via “quantization with bitsandbytes 4-bit and 8-bit support”

Lightning AI's LLM library — pretrain, fine-tune, deploy with clean PyTorch Lightning code.

Unique: Provides explicit 4-bit and 8-bit quantization configuration with mixed precision support (e.g., selective layer quantization), integrated into model loading pipeline, vs HuggingFace which wraps BitsAndBytes with less control over quantization granularity

vs others: Tighter integration with LitGPT's model loading allows fine-grained control over which layers are quantized, whereas HuggingFace PEFT applies quantization uniformly across the model

via “4-bit and 8-bit quantization for memory-efficient deployment”

Bilingual Chinese-English language model.

Unique: Provides both pre-quantized model variants on Hugging Face Model Hub (eliminating quantization overhead at startup) and on-the-fly quantization support via bitsandbytes integration. Memory footprint reduction is dramatic: 7B model shrinks from 15.3GB (fp16) to 5.1GB (4-bit), enabling deployment scenarios impossible with full precision.

vs others: Pre-quantized models eliminate quantization latency at startup (vs dynamic quantization), while supporting both 4-bit and 8-bit options for fine-grained accuracy-efficiency tradeoffs. Outperforms naive integer quantization by using learned quantization scales.

via “activation-aware 4-bit weight quantization with minimal accuracy loss”

4-bit weight quantization for LLMs on consumer GPUs.

Unique: Uses activation-aware scaling that analyzes per-channel activation magnitudes from calibration data to selectively protect high-impact weight channels, rather than uniform quantization across all weights. This channel-wise approach with activation-guided clipping preserves model quality better than post-training quantization methods that don't account for activation patterns.

vs others: Outperforms GPTQ and naive post-training quantization by 2-3% accuracy on benchmarks because it preserves activation-salient weights; faster quantization than QLoRA because it doesn't require training, enabling same-day deployment of new models.

via “quantization with fp8, fp4, int8, and modelopt support”

Fast LLM/VLM serving — RadixAttention, prefix caching, structured output, automatic parallelism.

Unique: Provides a quantization registry that maps quantization types to optimized kernel implementations, with automatic fallback to slower kernels on unsupported hardware. Supports per-layer and per-channel quantization strategies with integrated calibration.

vs others: Supports more quantization schemes (FP8, FP4, INT8, MXFP4) than vLLM's INT8-only support, with optimized kernels for each scheme and automatic hardware-aware fallbacks.

via “quantization with fp8 and low-precision inference”

High-throughput LLM serving engine — PagedAttention, continuous batching, OpenAI-compatible API.

Unique: Implements fused quantization kernels that perform dequantization and matrix multiplication in a single GPU operation, reducing memory bandwidth overhead vs separate dequant+compute steps

vs others: Achieves 4-8x memory reduction with 1-3% accuracy loss vs no quantization, outperforming naive INT8 quantization by using per-token scaling and mixed-precision strategies

via “efficient quantization support (8-bit and 4-bit) for memory-constrained deployment”

Google's open-weight model family from 1B to 27B parameters.

Unique: Officially validated quantization support across multiple frameworks (bitsandbytes, GPTQ, AWQ) with published quality benchmarks, enabling developers to choose quantization strategy based on deployment constraints without custom optimization work

vs others: Achieves better quality/speed tradeoffs with 4-bit quantization than Llama 2 due to training-aware quantization considerations, and simpler to deploy than custom quantization schemes or model distillation approaches

via “quantization support for memory-efficient deployment”

DeepSeek's 236B MoE model specialized for code.

Unique: Supports multiple quantization formats (FP8, INT8, INT4) through GPTQ/AWQ, reducing 236B model from 40GB to 8-16GB VRAM while maintaining 85-95% of original performance through post-training quantization

vs others: Enables deployment on consumer GPUs through quantization support, whereas many code models require enterprise-grade hardware; trade-off is 5-15% quality loss vs full precision

via “gptq weight quantization with hessian-based optimization”

Toolkit for LLM quantization, pruning, and distillation.

Unique: Implements Hessian-aware quantization where weight importance is determined by second-order Fisher information from calibration data, enabling per-channel and per-group quantization with automatic sensitivity-based bit-width selection

vs others: More accurate than simple magnitude-based quantization because it accounts for weight interactions; faster than full retraining because Hessian computation is one-shot; more flexible than fixed-bit-width schemes because it supports mixed precision

via “nf4 (normal float 4-bit) quantization with information-theoretic optimality”

8-bit and 4-bit quantization enabling QLoRA fine-tuning.

Unique: Uses information-theoretically optimal quantization levels derived from inverse normal CDF, allocating more precision to high-probability regions of weight distributions. Achieves better accuracy than uniform FP4 quantization on transformer weights without requiring per-layer calibration.

vs others: Outperforms FP4 quantization on transformer models by 1-2% accuracy while maintaining same memory footprint, and requires no calibration unlike post-training quantization methods.

via “quantization and model compression for edge deployment”

text-generation model by undefined. 79,12,032 downloads.

Unique: OPT's small size (125M) makes quantization less critical than for larger models, but the permissive license enables unrestricted quantization and redistribution, unlike proprietary models; community has published multiple quantized variants (GGML, GPTQ)

vs others: Easier to quantize than larger models due to smaller size, but quantized quality still lower than larger quantized models (LLaMA-7B INT4); better for extreme edge constraints than quality-critical edge applications

via “block-wise weight-only quantization with optional 4-bit/8-bit compression”

AirLLM 70B inference with single 4GB GPU

Unique: Quantizes weights only while preserving activation precision, differing from standard quantization (QAT/PTQ) that quantizes both weights and activations — maintains better accuracy by avoiding activation quantization noise while still reducing I/O overhead

vs others: Achieves 3x speed improvement with minimal accuracy loss, whereas GPTQ/AWQ require more complex calibration; simpler than mixed-precision quantization but less flexible than per-layer bit-width selection

via “inference optimization through quantization and model compression”

summarization model by undefined. 2,39,806 downloads.

Unique: Supports multiple quantization backends (bitsandbytes, ONNX Runtime, AutoGPTQ) through transformers library, avoiding lock-in to single quantization framework. INT4 quantization via bitsandbytes enables 4x model compression with <2% quality loss, suitable for edge deployment.

vs others: More flexible than framework-specific quantization (TensorFlow Lite, PyTorch mobile) by supporting multiple backends; achieves better compression than distillation-based approaches while maintaining original model architecture.

via “model quantization and compression compatibility”

question-answering model by undefined. 1,45,572 downloads.

Unique: Distributed in safetensors format (safer than pickle, faster to load) with explicit compatibility declarations for ONNX and TensorRT, enabling zero-copy quantization without intermediate format conversions

vs others: Smaller base model (84M vs 110M for BERT-base) quantizes more aggressively with better accuracy retention, and safetensors format eliminates pickle deserialization vulnerabilities present in older model distributions

via “quantization with fp8 and low-precision inference”

A high-throughput and memory-efficient inference and serving engine for LLMs

Unique: Implements FP8 quantization with hardware-accelerated matrix operations on NVIDIA H100/L40S GPUs, using native FP8 Tensor Cores to eliminate quantization overhead. Supports per-token dynamic quantization where activation scales are computed per-token rather than per-batch, improving accuracy.

vs others: Achieves 4-8x model compression with <2% accuracy loss on FP8 (vs. 5-10% loss for INT8 on same models); FP8 inference on H100 is only 5-10% slower than FP16 due to native hardware support, vs. 20-30% slowdown for INT8 on older GPUs.

via “quantization-aware training with 2/4/8-bit precision and bitsandbytes integration”

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Unique: Integrates bitsandbytes quantization kernels with LoRA adapter system to enable 4-bit training with NF4 format, supporting nested quantization (double_quant) for additional memory savings. Automatically handles quantization/dequantization in forward/backward passes without user intervention.

vs others: Native 4-bit quantization with NF4 format vs. alternatives like GPTQ which requires post-training quantization, enabling QLoRA training on consumer GPUs without pre-quantized models.

via “1-bit ternary weight quantization with lookup table matrix operations”

Official inference framework for 1-bit LLMs, by Microsoft. [#opensource](https://github.com/microsoft/BitNet)

Unique: Uses LUT-based matrix operations (not traditional arithmetic) for ternary weight quantization, achieving 16x memory bandwidth reduction; extends llama.cpp's mature inference infrastructure with specialized 1-bit kernels rather than building from scratch

vs others: Faster than standard quantization methods (2.37-6.17x speedup on x86) because LUT operations eliminate floating-point arithmetic entirely; more energy-efficient than GPTQ/AWQ because ternary representation requires minimal computation

via “4-bit quantization with nf4 data type for llm weight compression”

* ⭐ 05/2023: [Voyager: An Open-Ended Embodied Agent with Large Language Models (Voyager)](https://arxiv.org/abs/2305.16291)

Unique: Introduces NF4 (Normal Float 4) data type specifically designed for normally-distributed LLM weights, combined with block-wise absmax scaling and double quantization of quantization constants, achieving 4x compression with minimal accuracy loss — prior work used uniform or symmetric quantization schemes that were less suited to weight distributions

vs others: Outperforms standard 8-bit quantization (e.g., QAT, post-training quantization) by enabling 4-bit precision without significant accuracy degradation, and surpasses naive 4-bit approaches by using NF4 data type optimized for neural network weight distributions rather than generic floating-point formats

via “quantization-aware inference (8-bit and 4-bit)”

* ⭐ 04/2022: [PaLM: Scaling Language Modeling with Pathways (PaLM)](https://arxiv.org/abs/2204.02311)

Unique: Uses symmetric per-layer quantization with learned scale factors optimized for transformer architectures, achieving 95%+ quality retention at 8-bit while maintaining compatibility with standard inference frameworks without custom kernels

vs others: More practical than dynamic quantization (which adds per-batch overhead) and simpler than quantization-aware training (which requires retraining), enabling immediate deployment on consumer hardware with minimal quality loss