Ray

Ray Core executes Python functions and classes as distributed tasks across a cluster using an actor model with optional compiled DAG acceleration. Tasks are submitted to Raylets (per-node schedulers) which manage local execution, while the Global Control Store (GCS) coordinates cluster state. Compiled DAGs bypass the task submission overhead by pre-planning execution graphs, enabling near-native performance for complex workflows without serialization delays.

Ray Data provides a distributed DataFrame-like API (Dataset) that executes transformations (map, filter, groupby, aggregate) in streaming fashion across cluster nodes. Unlike batch systems, Ray Data schedules tasks based on available resources and data locality, pulling data through the object store in chunks. Supports multiple data sources (Parquet, CSV, S3, Delta Lake) and sinks, with automatic partitioning and lazy evaluation until .materialize() or action calls trigger execution.

Ray Data enables large-scale batch inference by applying a model to a distributed dataset. Users define a UDF (user-defined function) that loads a model and applies it to batches of data, then use Ray Data's map() to parallelize across partitions. Integrates with Ray Serve for serving the same model as an HTTP endpoint, enabling code reuse between batch and online inference. Supports automatic batching, GPU allocation per task, and result writing to cloud storage.

Ray Jobs API allows submitting Python scripts or functions as isolated jobs to a Ray cluster, with automatic resource allocation and priority-based scheduling. Each job runs in its own namespace with isolated actor/task state, preventing interference between concurrent jobs. Jobs can be submitted via CLI (ray job submit) or Python API, with support for dependency specification (runtime environments) and result retrieval. Integrates with Ray's autoscaler for automatic cluster scaling based on job resource requirements.

Ray's Global Control Store (GCS) is a distributed metadata service (built on Redis) that maintains cluster state: node membership, task/actor metadata, object locations, and job status. All Ray components (head node, Raylets, workers) query GCS for cluster topology and coordinate via GCS. Enables features like task scheduling (Raylets query GCS for available nodes), object location tracking (workers find objects via GCS), and fault recovery (GCS detects node failures and triggers task re-submission).

KubeRay is a Kubernetes operator that manages Ray clusters as Kubernetes custom resources (RayCluster). Enables declarative Ray cluster definition via YAML, automatic node scaling via Kubernetes HPA, and integration with Kubernetes networking and storage. KubeRay handles Ray head node and worker pod lifecycle, including health checks, rolling updates, and resource requests/limits. Supports Ray Jobs API for job submission to KubeRay-managed clusters.

Ray Train (v2) abstracts distributed training across PyTorch, TensorFlow, and HuggingFace Transformers using a controller-worker architecture. The controller coordinates training state and checkpointing, while workers execute training loops with automatic distributed data loading. Supports multi-node distributed training (DDP, DeepSpeed), automatic fault recovery via checkpointing, and integration with Ray Tune for hyperparameter search. Handles dependency installation via runtime environments and GPU/CPU resource allocation.

Ray Tune executes hyperparameter search by spawning multiple training trials (each a Ray actor) and scheduling them based on available resources. Supports multiple search algorithms (grid, random, Bayesian optimization via Optuna, population-based training) and early stopping schedulers (ASHA, median stopping rule). Each trial reports metrics back to Tune's trial manager, which decides whether to continue, pause, or terminate based on scheduler logic. Integrates with Ray Train for distributed training trials and Ray Serve for model evaluation.

Ray Serve deploys models as HTTP endpoints by wrapping them in Deployment classes that handle request routing, batching, and scaling. Each deployment runs as a Ray actor pool, with a router component that batches incoming requests and forwards them to available replicas. Supports dynamic scaling based on queue depth or custom metrics, automatic model reloading without downtime, and composition of multiple deployments (e.g., preprocessing → model → postprocessing). Integrates with Ray Data for batch inference and Ray Train for serving trained models.

Ray's autoscaler monitors cluster resource utilization and automatically launches/terminates nodes based on pending task demand. The autoscaler reads resource requirements from tasks (CPU, GPU, memory, custom resources) and compares against available capacity, launching new nodes if demand exceeds supply. Supports multiple cloud providers (AWS, GCP, Azure) via cloud-specific launch templates, and Kubernetes via KubeRay. Includes node failure detection and automatic recovery by re-submitting failed tasks to healthy nodes.

Ray's runtime environment system automatically installs Python dependencies (pip packages, conda environments, local code) on worker nodes at task submission time. Dependencies are specified per-task or per-job, packaged into a runtime environment, and distributed to nodes via the object store. Supports pip requirements files, conda YAML specs, and local Python packages. Enables reproducible distributed execution without pre-baking dependencies into container images.

Ray provides observability through three channels: (1) web dashboard showing cluster topology, task/actor status, and resource utilization in real-time; (2) metrics API exposing Prometheus-compatible metrics (task latency, throughput, object store usage) for external monitoring; (3) state API querying cluster state (tasks, actors, nodes, jobs) programmatically. All three integrate with Ray's internal state management (GCS) to provide consistent views of cluster health.

Ray's object store is a distributed in-memory cache (one per node) that stores serialized objects using Apache Arrow format for zero-copy access. When a task returns a value, it's serialized to Arrow format and stored locally; other tasks access it via ObjectRef (a pointer to the object's location and ID). Arrow's columnar format enables zero-copy deserialization for NumPy arrays and Pandas DataFrames, reducing memory overhead. Object store is managed by the Raylet (per-node scheduler) and coordinates with GCS for object location tracking.

Ray RLlib provides distributed RL training via a modular architecture supporting multiple algorithms (PPO, DQN, A3C, SAC, etc.) and environments (OpenAI Gym, custom). Training uses Ray's distributed execution to parallelize environment rollouts (data collection) and model updates across workers. Supports off-policy and on-policy algorithms, multi-agent RL, and curriculum learning. Integrates with Ray Tune for hyperparameter search and Ray Serve for policy serving.