Contrastive Loss Optimization For Response Quality Differentiation

via “contrastive decoding for improved generation quality”

* ⏫ 07/2023: [Meta-Transformer: A Unified Framework for Multimodal Learning (Meta-Transformer)](https://arxiv.org/abs/2307.10802)

Unique: Implements contrastive decoding as a self-contained inference-time method within the single decoder rather than requiring separate quality models or ensemble approaches, enabling quality improvements without architectural overhead

vs others: Lighter-weight than ensemble-based quality improvement (e.g., DALL-E 3's approach) because it reuses the same model for candidate generation and selection; more practical than training separate discriminators or quality models

* ⏫ 06/2023: [Faster sorting algorithms discovered using deep reinforcement learning (AlphaDev)](https://www.nature.com/articles/s41586-023-06004-9)

Unique: Uses a sigmoid-based contrastive loss that directly operates on log-probability ratios rather than converting preferences to reward labels, enabling end-to-end differentiable optimization without intermediate reward model predictions

vs others: More computationally efficient than PPO-based RLHF because it avoids on-policy sampling and reward model inference; more stable than margin-based losses because sigmoid provides smooth gradients across the entire probability space