1 Bit Ternary Weight Quantization With Lookup Table Matrix Operations

via “matrix multiplication with quantized operands (gemm operations)”

8-bit and 4-bit quantization enabling QLoRA fine-tuning.

Unique: Implements on-the-fly dequantization within CUDA kernels during GEMM, avoiding materialization of full-precision intermediates and reducing memory bandwidth by 50-75%. Supports mixed-precision output and integrates with PyTorch autograd for gradient computation.

vs others: Achieves better memory efficiency than naive dequantize-then-multiply approaches, and provides faster inference than full-precision GEMM while maintaining numerical stability through careful scaling factor management.

via “gptq weight quantization with hessian-based optimization”

Toolkit for LLM quantization, pruning, and distillation.

Unique: Implements Hessian-aware quantization where weight importance is determined by second-order Fisher information from calibration data, enabling per-channel and per-group quantization with automatic sensitivity-based bit-width selection

vs others: More accurate than simple magnitude-based quantization because it accounts for weight interactions; faster than full retraining because Hessian computation is one-shot; more flexible than fixed-bit-width schemes because it supports mixed precision

via “block-wise weight-only quantization with optional 4-bit/8-bit compression”

AirLLM 70B inference with single 4GB GPU

Unique: Quantizes weights only while preserving activation precision, differing from standard quantization (QAT/PTQ) that quantizes both weights and activations — maintains better accuracy by avoiding activation quantization noise while still reducing I/O overhead

vs others: Achieves 3x speed improvement with minimal accuracy loss, whereas GPTQ/AWQ require more complex calibration; simpler than mixed-precision quantization but less flexible than per-layer bit-width selection

via “1-bit ternary weight quantization with lookup table matrix operations”

Official inference framework for 1-bit LLMs, by Microsoft. [#opensource](https://github.com/microsoft/BitNet)

Unique: Uses LUT-based matrix operations (not traditional arithmetic) for ternary weight quantization, achieving 16x memory bandwidth reduction; extends llama.cpp's mature inference infrastructure with specialized 1-bit kernels rather than building from scratch

vs others: Faster than standard quantization methods (2.37-6.17x speedup on x86) because LUT operations eliminate floating-point arithmetic entirely; more energy-efficient than GPTQ/AWQ because ternary representation requires minimal computation