ray

Ray executes Python functions and methods as distributed tasks across a cluster using a centralized scheduler (Raylet) that assigns work to worker processes based on resource availability and data locality. Tasks are serialized, transmitted to remote workers, executed in isolated processes, and results are stored in a distributed object store (Apache Arrow-based) for efficient retrieval. The scheduler uses a two-level hierarchy: global GCS (Global Control Store) for cluster-wide state and per-node Raylets for local task scheduling and resource management.

Ray Actors are long-lived, stateful objects that run on remote workers and expose methods callable from the driver or other actors. Each actor maintains mutable state across method calls, uses a message queue for serialized method invocations, and executes methods sequentially (by default) or with concurrency control. Actors are created with @ray.remote decorator, instantiated on a specific worker, and method calls return ObjectRefs that can be chained or awaited. This pattern enables building distributed services like parameter servers, model replicas, or stateful microservices without manual socket/RPC management.

Ray provides comprehensive observability through a web-based dashboard, Prometheus-compatible metrics, and a State API for querying cluster state. The dashboard displays real-time cluster status (nodes, workers, tasks), task execution timelines, actor state, and resource utilization. Metrics are exported in Prometheus format for integration with monitoring systems. The State API allows programmatic queries of cluster state (tasks, actors, nodes, jobs) via REST or Python SDK, enabling custom monitoring and debugging. Logs are aggregated from all workers and accessible via the dashboard or API.

Ray's object store is a distributed in-memory storage system (based on Apache Arrow) that stores task results and intermediate data across worker nodes. Objects are stored in a shared memory region on each node, enabling zero-copy access for tasks on the same node and efficient serialization for remote access. The object store uses a least-recently-used (LRU) eviction policy to manage memory, spilling to disk when necessary. Object references (ObjectRefs) are lightweight pointers that can be passed between tasks without copying the underlying data, enabling efficient data sharing in distributed pipelines.

Ray Jobs API allows submitting, monitoring, and managing long-running jobs on a Ray cluster. Jobs are submitted via ray job submit command or Python API, executed with isolated namespaces and resource allocation, and tracked via job IDs. The Jobs API handles job scheduling (respecting resource requirements), execution monitoring (logs, status), and cleanup (automatic termination on completion or timeout). Jobs support dependencies (pip packages, local files) and can be submitted to specific node groups or with specific resource constraints. Job status is queryable via API or dashboard.

Ray's Compiled DAG feature allows developers to define a static directed acyclic graph (DAG) of tasks and actors, compile it into an optimized execution plan, and execute it with minimal scheduling overhead. The compilation step analyzes data dependencies, removes redundant serialization, and generates a C++ execution engine that bypasses the Python scheduler for each step. This is particularly effective for inference pipelines or iterative algorithms where the computation graph is fixed but executed many times. DAGs are defined using ray.dag API and compiled with dag.experimental_compile().

Ray Data provides a distributed DataFrame-like API for processing large datasets across a cluster using lazy evaluation and streaming execution. Datasets are partitioned across workers, transformations (map, filter, groupby, join) are defined lazily and executed only when materialized (via .take(), .write(), or .iter_batches()), and execution uses a streaming model where partitions flow through the pipeline without materializing intermediate results. Ray Data integrates with popular formats (Parquet, CSV, JSON, images) and frameworks (Pandas, NumPy, PyTorch, TensorFlow) for seamless data loading and transformation.

Ray Tune is a distributed hyperparameter optimization framework that supports multiple search algorithms (grid search, random search, Bayesian optimization via Optuna, population-based training, CMA-ES) and scheduling strategies (FIFO, ASHA, PBT, HyperBand). Tune manages trial execution across workers, tracks metrics in real-time, implements early stopping based on performance, and supports multi-objective optimization. Trials are executed as Ray actors or tasks, metrics are reported via callbacks, and the framework automatically scales trials based on available resources. Integration with popular ML frameworks (PyTorch Lightning, TensorFlow, Hugging Face) is built-in.

Ray RLlib is a distributed reinforcement learning library that trains policies (neural networks) using algorithms like PPO, DQN, A3C, and IMPALA. It parallelizes environment simulation across workers (using ray.remote), collects experience trajectories, trains policies on batches of experience, and implements off-policy and on-policy learning. RLlib uses a centralized policy server (Ray actor) that workers query for actions, and a learner process that updates the policy based on collected experience. The framework abstracts away distributed training complexity, handling synchronization, gradient aggregation, and checkpointing automatically.

Ray Train provides a distributed training framework that abstracts away cluster management for PyTorch, TensorFlow, Hugging Face, and other frameworks. It launches distributed training jobs across workers, handles gradient synchronization and communication backends (NCCL, Gloo), manages checkpointing and fault recovery, and provides a simple API for single-machine code to scale to multi-machine training. Ray Train v2 uses a controller-worker architecture where the controller orchestrates training and workers execute training loops, with automatic recovery from worker failures via checkpoint restoration.

Ray Serve is a distributed serving framework that deploys ML models as HTTP endpoints with automatic request batching, dynamic scaling based on load, and support for multi-model pipelines. Models are wrapped in Serve deployments (Ray actors), requests are routed to deployments via a load balancer, batching is applied to improve throughput, and scaling is controlled by metrics (queue depth, latency) or custom policies. Serve supports model composition (chaining deployments) and traffic splitting for A/B testing. Integration with popular frameworks (PyTorch, TensorFlow, Hugging Face, scikit-learn) is built-in.

Ray's autoscaler automatically scales the cluster up or down based on pending tasks and resource demand. It monitors the task queue, detects when tasks cannot be scheduled due to insufficient resources, launches new nodes (via cloud provider APIs like AWS, GCP, Azure), and terminates idle nodes to save costs. The autoscaler uses a resource-aware scheduler that matches task resource requirements (CPU, GPU, memory, custom resources) to available nodes, and supports node labels for task placement constraints. Autoscaling policies are configurable via YAML and support custom scaling logic.

Ray's runtime environment feature allows specifying Python dependencies, environment variables, and working directories that are automatically installed and configured on remote workers before task execution. Dependencies can be specified as pip packages, conda environments, or local Python files, and Ray handles downloading, installing, and activating them on each worker. This enables reproducible execution across heterogeneous clusters and simplifies dependency management without requiring pre-built Docker images. Runtime environments are specified per-job or per-task, and Ray caches installed environments to avoid redundant installation.