trl

Implements supervised fine-tuning (SFT) for causal language models using a standard next-token prediction loss across instruction-response pairs. The trainer wraps Hugging Face Transformers' Trainer class, automatically handling data formatting, tokenization, and gradient accumulation across distributed setups. It supports both full-model and parameter-efficient fine-tuning (LoRA/QLoRA) through integration with the peft library, enabling memory-efficient training on consumer hardware.

Implements the RLHF pipeline (reward modeling + policy optimization) using a two-stage approach: first trains a reward model on human preference pairs (chosen vs rejected responses), then uses PPO (Proximal Policy Optimization) to optimize the language model policy against the learned reward signal. The implementation includes KL divergence penalties to prevent policy drift from the base model and supports both online (generate-then-score) and offline (pre-computed scores) training modes.

Provides utilities to format and preprocess datasets for different training objectives (SFT, RLHF, DPO, etc.). Includes data collators that handle variable-length sequences, automatic padding/truncation, and format conversion (e.g., instruction-response to prompt-completion). Supports streaming datasets for memory-efficient processing of large corpora and automatic train/validation splitting.

Provides utilities to merge LoRA adapters into base models and compose multiple adapters for multi-task inference. Supports weighted merging (combining multiple adapters with different weights), sequential composition (stacking adapters), and adapter pruning (removing low-importance parameters). Handles numerical stability during merging and supports saving merged models in standard formats.

Integrates with popular logging platforms (Weights & Biases, TensorBoard, Hugging Face Hub) to track training metrics, model checkpoints, and hyperparameters. Automatically logs loss curves, evaluation metrics, learning rate schedules, and gradient statistics. Supports custom metric logging and integration with external monitoring systems via callbacks.

Implements Direct Preference Optimization (DPO), a single-stage alternative to RLHF that directly optimizes the language model on preference pairs without training a separate reward model. DPO uses a contrastive loss that maximizes the likelihood ratio between preferred and dispreferred responses, implicitly learning a reward function. The implementation includes support for IPO (Identity Preference Optimization) and other preference optimization variants, with built-in handling of prompt-level weighting and batch-level preference balancing.

Implements Generative Reward Preference Optimization (GRPO), which combines reward modeling with policy optimization in a single end-to-end differentiable process. GRPO trains a model to generate both responses and reward scores simultaneously, using the generated rewards to optimize the policy via policy gradient methods. This approach reduces the two-stage complexity of RLHF while maintaining explicit reward signals, using a shared or separate reward head on the language model.

Provides utilities to score model outputs using a trained reward model and rank responses by quality without requiring full RLHF training. Supports batch processing of completions through a reward model, with configurable scoring strategies (e.g., per-token vs full-sequence rewards). Includes utilities for converting scores to preference pairs and filtering low-quality examples, enabling offline dataset creation for DPO or other preference-based methods.

Abstracts distributed training across multiple GPUs and nodes using Hugging Face Accelerate library, automatically handling data parallelism, gradient synchronization, and mixed-precision training. Supports both single-machine multi-GPU (DataParallel, DistributedDataParallel) and multi-node setups with automatic device placement and loss scaling. Includes built-in support for gradient accumulation to simulate larger effective batch sizes on memory-constrained hardware.

Integrates peft library to enable Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA) fine-tuning, which trains only small adapter matrices instead of full model weights. LoRA adds trainable rank-r decompositions to weight matrices, reducing parameters by 99%+. QLoRA further quantizes the base model to 4-bit precision, enabling fine-tuning of 70B+ parameter models on consumer GPUs. Automatically handles adapter merging, saving, and loading.

Provides utilities for generating completions from trained models and evaluating them against reference outputs or metrics. Includes batch generation with configurable decoding strategies (greedy, beam search, sampling), automatic tokenization and detokenization, and integration with common evaluation metrics (BLEU, ROUGE, exact match). Supports both offline evaluation on fixed datasets and online evaluation during training with periodic checkpointing.

Implements gradient checkpointing (activation checkpointing) to reduce peak memory usage during training by recomputing activations during backpropagation instead of storing them. Automatically applies checkpointing to transformer blocks, reducing memory by 50-70% at the cost of ~15-20% training time overhead. Supports selective checkpointing (only checkpoint expensive layers) and integration with quantization for extreme memory efficiency.

Provides extensible framework for implementing custom loss functions and training objectives beyond standard SFT/RLHF/DPO. Includes base classes for custom trainers that override loss computation, allowing researchers to implement novel alignment methods (e.g., contrastive learning, multi-task learning, curriculum learning). Supports per-example loss weighting, task-specific loss scaling, and loss combination strategies.