AReaL

Orchestrates large-scale reinforcement learning training across distributed clusters using pluggable training engines (FSDP, Megatron, Archon) that support multiple parallelism strategies including tensor parallelism, pipeline parallelism, sequence parallelism (Ulysses), and MoE expert parallelism. The system abstracts away distributed training complexity through a unified TrainEngine API while managing device meshes, process groups, and weight synchronization protocols across heterogeneous hardware configurations.

Manages high-throughput inference serving through pluggable backends (SGLang, vLLM) with asynchronous rollout execution that decouples inference from training. The InferenceEngine API abstracts backend-specific details while supporting dynamic weight updates via a protocol-based system that allows training engines to push updated weights to inference servers without service interruption. Handles server lifecycle management, async request batching, and multi-turn conversation state tracking.

Implements a comprehensive configuration system using Python dataclasses with CLI argument parsing and validation. The system supports hierarchical configuration with allocation_mode syntax for specifying parallelism strategies, training engine parameters, inference configurations, and algorithm-specific settings. Configuration validation ensures compatibility between different components (e.g., parallelism constraints) before training starts. Supports configuration inheritance and overrides through CLI arguments.

Enables multi-node training across SLURM, Ray, and SkyPilot clusters with automatic validation of shared storage accessibility and performance. The system checks that all nodes can access shared storage before training starts, preventing silent failures due to misconfigured NFS or S3 paths. Supports different storage backends (NFS, S3) with backend-specific validation. Handles checkpoint and data synchronization across nodes through shared storage.

Enables reinforcement learning training for multi-turn agent interactions through an ArealOpenAI client that proxies OpenAI-compatible APIs, capturing tool calls, multi-turn conversations, and intermediate rewards. The system tracks interaction sessions via InteractionCache, assigns rewards with configurable discounting schemes, and exports complete trajectories for RL training. Tool call integration allows agents to use external functions while maintaining full observability of the interaction flow for reward assignment.

Implements multiple reinforcement learning algorithms (PPO, GRPO and variants) with configurable hyperparameters, reference model management, and critic networks. The system supports asynchronous training orchestration where multiple rollout workers feed trajectories into a centralized trainer that computes policy gradients, value function losses, and KL divergence penalties. Reference models and critic networks are managed separately to enable efficient computation of advantage estimates and policy divergence constraints.

Implements efficient data processing through a MicroBatchSpec system that handles sequence packing, padding strategies, and memory-aware batching. The system normalizes and estimates memory requirements for different batch configurations, enabling automatic selection of batch sizes that maximize GPU utilization without OOM errors. Supports variable-length sequences with configurable packing strategies (e.g., pack multiple sequences into single training example) and normalization schemes for fair comparison across different batch configurations.

Provides a RolloutWorkflow API that abstracts the interaction between rollout collection and training, enabling custom implementations for different agent types and task structures. The system supports multi-turn and vision workflows through pluggable workflow implementations that define how agents interact with environments, how rewards are assigned, and how trajectories are exported. Rollout coordination ensures proper synchronization between multiple rollout workers and the training engine.

Manages distributed training job scheduling through pluggable launcher backends (Local, Ray, SLURM, SkyPilot) that abstract away cluster-specific details. The Scheduler API coordinates worker allocation, job lifecycle management, and RPC communication between training and inference engines. Supports automatic shared storage validation to ensure checkpoints and data are accessible across all nodes. Each launcher backend handles cluster-specific job submission, resource allocation, and failure recovery.

Implements distributed checkpoint saving and recovery with automatic metadata tracking for training state, model weights, and optimizer state. The system supports incremental checkpointing where only changed weights are saved, reducing storage overhead. Checkpoint metadata includes training step, algorithm state, and configuration information, enabling resumption from any checkpoint with full reproducibility. Handles checkpoint coordination across distributed training engines to ensure consistency.

Provides integrated performance tracing and session visualization tools for debugging distributed training and inference. The system captures detailed traces of training steps, inference requests, and inter-engine communication, enabling identification of bottlenecks and performance issues. Session tracing tracks multi-turn agent interactions with timing information, allowing analysis of agent behavior and reward assignment. Trace visualization tools help developers understand system behavior and optimize configurations.

Provides seamless integration with HuggingFace model hub through automatic architecture detection and model loading utilities. The system detects model architecture (LLaMA, Qwen, Mistral, etc.) and automatically selects appropriate training engine configurations and parallelism strategies. Supports LoRA fine-tuning as an alternative to full model training, reducing memory requirements and training time. Handles model tokenizer loading and configuration validation.