TRL

Trains language models on instruction-response pairs using standard supervised learning with automatic chat template formatting. Extends transformers.Trainer with built-in support for multiple chat formats (ChatML, Alpaca, Llama 2, etc.), handling tokenization, padding, and loss masking for instruction-response boundaries. Supports both single-turn and multi-turn conversations with configurable prompt/response masking to ensure gradients only flow through response tokens.

Implements DPO training that aligns models to human preferences by directly optimizing the log-likelihood ratio between preferred and dispreferred responses, eliminating the need for a separate reward model. Uses a reference model (frozen copy of the base model) to compute KL divergence penalties, with optional weight sharing to reduce memory overhead. Supports multiple loss variants (standard DPO, IPO, KTO) and automatic reference model synchronization across distributed training.

Trains reward models that score intermediate steps in a reasoning process (e.g., math problem-solving steps) rather than final outputs. Supports step-level annotations with automatic aggregation to trajectory-level rewards, and includes utilities for parsing structured reasoning formats (e.g., step-by-step math solutions). Integrates with standard TRL trainers for seamless PRM-based training.

Extends TRL trainers to support vision-language models by handling image inputs alongside text, with automatic image tokenization and alignment with text tokens. Supports multiple vision encoders (CLIP, DINOv2, etc.) and integrates with chat templates for multi-modal conversations. Includes utilities for image dataset loading, augmentation, and format conversion.

Provides a command-line interface for launching training jobs with YAML configuration files, eliminating the need to write Python training scripts. Supports all TRL trainers (SFT, DPO, GRPO, etc.) with automatic argument parsing and validation. Includes utilities for hyperparameter sweeps, distributed training setup, and job submission to cloud platforms.

Implements asynchronous GRPO where generation and training happen on separate GPU processes, decoupling the generation bottleneck from training. Uses a queue-based architecture to pipeline generation and training steps, with automatic load balancing and memory management. Supports both local multi-GPU setups and distributed training across multiple machines.

TRL implements RLOO, a policy gradient method that generates multiple completions per prompt and uses leave-one-out variance reduction to estimate policy gradients. Reduces variance compared to standard REINFORCE while avoiding the need for a separate value function. Integrates with vLLM for efficient generation and supports custom reward functions.

Implements GRPO, an online RL method that generates multiple responses per prompt, scores them with a reward function, and optimizes the policy using group-relative advantages. Integrates with vLLM for high-throughput batch generation (100+ tokens/sec) and supports both server mode (external vLLM process) and colocate mode (in-process generation with memory management). Handles reward function composition, advantage normalization, and policy gradient updates with optional KL clipping.

Trains reward models that score responses on a continuous scale, supporting both regression (MSE) and ranking (pairwise margin) objectives. Handles preference pair formatting, automatic reference model loading, and loss variants including Bradley-Terry and Elo-based scoring. Integrates with TRL's data pipeline for automatic chat template formatting and supports both single-model and dual-model architectures.

Integrates Hugging Face PEFT library to enable parameter-efficient fine-tuning using LoRA (Low-Rank Adaptation), QLoRA (quantized LoRA), and other adapters. Automatically handles adapter configuration, merging, and unloading, with seamless integration across all TRL trainers. Supports 4-bit and 8-bit quantization via bitsandbytes, enabling training of 70B+ models on consumer GPUs.

Leverages Hugging Face Accelerate library to abstract away distributed training complexity, supporting data parallelism, distributed data parallelism (DDP), and model parallelism across multiple GPUs/TPUs. Handles gradient accumulation, mixed precision training (fp16/bf16), and automatic loss scaling. All TRL trainers inherit Accelerate integration, enabling single-line scaling from 1 GPU to 8+ GPUs without code changes.

Provides a unified data pipeline that automatically detects and applies chat templates (ChatML, Alpaca, Llama 2, Mistral, etc.) to raw instruction-response data, handling tokenization, padding, and attention mask generation. Supports multiple input formats (JSON, CSV, Hugging Face datasets) and automatically infers schema from data. Includes utilities for dataset validation, train/test splitting, and format conversion.

Provides extensible callback system for monitoring training progress, computing custom metrics, and triggering actions at key points (epoch end, step end, evaluation). Integrates with Hugging Face Hub for automatic model uploading, Weights & Biases for experiment tracking, and TensorBoard for visualization. Callbacks have access to trainer state, model, and optimizer for advanced monitoring.

Implements Kahneman-Tversky Optimization (KTO) and Odds Ratio Preference Optimization (ORPO) as alternatives to DPO, using different loss formulations for preference learning. KTO uses a reference model and asymmetric loss weighting to handle imbalanced preferences, while ORPO combines preference optimization with language modeling loss to prevent reward hacking. Both methods support the same preference pair format as DPO but with different hyperparameter sensitivity.

Implements RLOO, a variance-reduced policy gradient method that trains models by comparing each response against a baseline computed from other responses in the same batch. Reduces variance compared to standard REINFORCE while avoiding the computational overhead of value function training. Supports both on-policy and off-policy variants with optional importance weighting.