1 Bit Ternary Weight Quantization With Lookup Table Matrix Operations

via “matrix multiplication with quantized operands (gemm operations)”

8-bit and 4-bit quantization enabling QLoRA fine-tuning.

Unique: Implements on-the-fly dequantization within CUDA kernels during GEMM, avoiding materialization of full-precision intermediates and reducing memory bandwidth by 50-75%. Supports mixed-precision output and integrates with PyTorch autograd for gradient computation.

vs others: Achieves better memory efficiency than naive dequantize-then-multiply approaches, and provides faster inference than full-precision GEMM while maintaining numerical stability through careful scaling factor management.