Mixed Precision Training With Precision Bottleneck Relaxation Pb Relax

via “mixed-precision training with precision bottleneck relaxation (pb-relax)”

Text-to-Image generation. The repo for NeurIPS 2021 paper "CogView: Mastering Text-to-Image Generation via Transformers".

Unique: Implements precision bottleneck relaxation (PB-relax) as a custom CUDA kernel that dynamically adjusts attention logit precision during mixed-precision training, preventing overflow without sacrificing gradient flow. This is a novel technique introduced in the CogView paper and is baked into the training pipeline via arguments.py configuration.

vs others: More stable than standard mixed-precision training (PyTorch AMP) for large transformers, but requires custom CUDA code and hardware-specific tuning; simpler than gradient checkpointing but less memory-efficient than DeepSpeed ZeRO.