Determined AI

Enables multi-GPU and multi-node PyTorch training through a custom trial harness that wraps the training loop and automatically handles distributed data loading, gradient aggregation, and checkpoint synchronization across workers. Uses PyTorch's DistributedDataParallel under the hood with Determined's allocation service managing worker coordination via gRPC, eliminating manual distributed training boilerplate.

Implements distributed hyperparameter optimization using pluggable search algorithms (grid, random, Bayesian, population-based training) that spawn multiple trial instances and intelligently allocate GPU resources based on performance. The master service orchestrates search via the allocation service, which tracks trial metrics and feeds them back to the search algorithm to guide next trial configurations.

Provides a Context object (determined.core.Context) that training code uses to report metrics, save checkpoints, and receive hyperparameter updates. Implements a callback system that hooks into training loops (PyTorch, Keras) to automatically save checkpoints, report metrics, and handle preemption signals. The context is injected into trial code at runtime, allowing training code to remain agnostic of the underlying distributed training setup.

Automatically manages checkpoint storage by implementing configurable garbage collection policies (keep best N checkpoints, keep checkpoints from last M hours, keep all). The master service periodically scans the checkpoint store and deletes old checkpoints based on the policy, freeing storage space. Supports dry-run mode to preview which checkpoints would be deleted before actually deleting them.

Provides tools to compare metrics across multiple experiments and trials, enabling analysis of how hyperparameters affect model performance. The web UI supports filtering, sorting, and exporting experiment results for statistical analysis. The Python SDK provides programmatic access to experiment data for custom analysis notebooks.

Experiments are defined in YAML files that specify training code, hyperparameters, searcher algorithm, resource requirements, and checkpoint storage. Master service validates YAML against a schema (master/internal/config/config.go) before creating experiments. YAML supports templating and variable substitution, allowing reuse across experiments. Configuration is versioned and stored in PostgreSQL for reproducibility.

Manages heterogeneous GPU clusters (single-node, multi-node, Kubernetes, on-prem agents) through a pluggable resource manager architecture that tracks available GPUs, memory, and compute capacity. The allocation service uses a priority queue and bin-packing algorithm to schedule experiment tasks, preempting lower-priority jobs to fit higher-priority ones, with support for resource pools (e.g., reserved GPUs for specific teams).

Tracks experiment state (queued, running, completed, failed) through the master service's core experiment manager, which persists experiment metadata and trial results to Postgres. Automatically saves model checkpoints at configurable intervals and on trial completion, storing them in a pluggable backend (local filesystem, S3, GCS, Azure Blob). Supports resuming experiments from checkpoints, allowing interrupted training to continue without data loss.

Provides a React-based web UI that connects to the master service via REST and gRPC APIs to display real-time training metrics, loss curves, and hyperparameter comparisons. The UI streams metrics from the database and renders interactive charts using a custom metrics decoder that handles various metric formats (scalars, histograms, embeddings). Supports filtering, sorting, and exporting experiment results.

Provides a Keras trial harness that wraps tf.keras.Model training with Determined's distributed training and checkpoint management. Automatically converts eager-mode training to graph mode for performance optimization, handles distributed batch splitting across workers, and integrates Keras callbacks with Determined's metric reporting system.

Exposes the master service through dual REST (HTTP/JSON) and gRPC APIs defined in Protocol Buffers, with auto-generated Python and TypeScript SDKs using protoc and gRPC code generators. The REST API uses gRPC-JSON transcoding for compatibility with web clients, while gRPC provides low-latency streaming for real-time metric updates. All APIs are versioned (v1, v2) to support backward compatibility.

Provides a command-line interface (det CLI) that communicates with the master service via REST API to submit experiments, monitor progress, manage checkpoints, and configure resource pools. Supports YAML experiment configuration files, interactive experiment creation, and shell completion for common commands. Handles authentication via API tokens or username/password.

Provides Helm charts that deploy Determined master and agents as Kubernetes workloads with proper RBAC roles, service accounts, and network policies. The master runs as a Deployment with persistent volume for Postgres, while agents run as DaemonSets or Deployments on GPU nodes. Supports multi-cluster setups with multiple resource managers coordinating via the master service.

Validates experiment YAML configurations against a strict schema before submission, checking for required fields, valid hyperparameter ranges, and compatible resource requests. The master service enforces schema validation on the server side, rejecting invalid configurations with detailed error messages. Supports configuration inheritance and templating for reusable experiment definitions.