ClearML

Intercepts training loops and framework calls (TensorFlow, PyTorch, scikit-learn, XGBoost) via monkey-patching and SDK hooks to automatically log metrics, hyperparameters, model checkpoints, and system resources without explicit logging statements. Uses a Task object that wraps the training context and captures stdout/stderr, git metadata, and environment variables. Stores all artifacts in a local or remote backend (file system, S3, GCS, Azure Blob).

Manages immutable dataset snapshots with content-addressable storage (SHA256-based deduplication) and tracks data lineage across preprocessing, training, and inference pipelines. Datasets are registered as ClearML Dataset objects with metadata (schema, statistics, splits), stored in a backend (local, S3, GCS), and linked to experiments via task dependencies. Supports incremental uploads, data validation rules, and automatic cache invalidation when upstream data changes.

Provides a flexible API for logging custom metrics (scalars, histograms, images, plots) during training via Task.log_scalar(), Task.log_histogram(), Task.log_image(). Metrics are timestamped and stored in the backend with configurable aggregation (e.g., per-epoch vs per-batch). Supports nested metric hierarchies (e.g., 'train/loss', 'val/accuracy') for organized metric browsing. Histograms can track weight distributions or gradient norms for debugging.

Enables creating new experiments by cloning existing Task objects, which copies hyperparameters, code version, and dataset references while allowing selective parameter overrides. Cloned tasks inherit the parent task's configuration but execute as independent experiments. Supports batch cloning for creating multiple variants (e.g., grid search) without manual task creation. Task templates can be stored and reused across teams.

Manages task execution via named queues (e.g., 'gpu_queue', 'cpu_queue') with priority-based scheduling and resource constraints (GPU type, memory requirements, CPU cores). Tasks are enqueued with metadata specifying required resources, and agents poll queues matching their capabilities. Supports dynamic queue assignment and task rescheduling on resource unavailability. Queue state is persisted in ClearML Server.

Enables querying experiments via flexible filtering on tags, hyperparameters, metrics, date range, and custom metadata. Supports full-text search on experiment names and descriptions. Results can be sorted by metric values (e.g., best validation accuracy) and aggregated (e.g., average metric across runs). Filtering is performed server-side for scalability. Saved filters can be bookmarked for repeated use.

Distributes training and inference tasks across heterogeneous compute resources (local machines, cloud VMs, Kubernetes clusters, HPC) via a pull-based agent architecture. The ClearML Agent polls a task queue, pulls code and data from git/artifact storage, sets up isolated Python environments (via venv or Docker), and executes tasks with resource constraints (GPU allocation, memory limits, CPU affinity). Task queues are priority-ordered and support dynamic resource matching (e.g., 'run on GPU with >16GB VRAM').

Defines multi-stage ML workflows as directed acyclic graphs (DAGs) where each node is a ClearML Task with explicit input/output artifact dependencies. Pipelines are defined programmatically via PipelineController API or declaratively via YAML, with support for conditional branching, parallel execution, and dynamic task creation. The controller manages task queuing, monitors execution state, and propagates artifacts between stages (e.g., preprocessed data → training → evaluation).

Automates hyperparameter search via a HyperparameterOptimizer that spawns multiple training tasks with different parameter combinations. Supports grid search, random search, Bayesian optimization (via Optuna integration), and population-based training (PBT). The optimizer monitors task metrics in real-time, prunes unpromising trials early, and allocates compute resources dynamically. Results are aggregated and ranked by a configurable objective metric (e.g., validation accuracy).

Deploys trained models as HTTP REST endpoints via ClearML Serving, which wraps model artifacts (PyTorch, TensorFlow, scikit-learn, ONNX) in a FastAPI application with automatic request/response serialization. Supports model versioning, A/B testing (traffic splitting between model versions), and canary deployments. Models are fetched from artifact storage at startup, and inference requests are logged for monitoring. Deployment targets include Docker containers, Kubernetes, or local processes.

Provides a web dashboard for browsing experiments, comparing metrics across runs, visualizing training curves, and inspecting artifacts (logs, model checkpoints, plots). The UI queries the ClearML Server backend to fetch experiment metadata, metrics time-series, and artifact listings. Supports filtering by tags, date range, and metric thresholds; allows side-by-side metric comparison and custom metric aggregation (e.g., best validation accuracy across runs).

Automatically captures git metadata (commit hash, branch, remote URL, uncommitted changes) when a Task is created, enabling reproducible experiment execution by checking out the exact code version used. Supports both public and SSH-authenticated git repositories. When a task is cloned or rerun, the agent checks out the original commit and executes the code, ensuring bit-for-bit reproducibility. Uncommitted changes are detected and logged as warnings.

Converts trained models between frameworks (PyTorch ↔ ONNX, TensorFlow ↔ ONNX) and exports to standardized formats for cross-platform inference. Uses framework-native export APIs (torch.onnx.export, tf2onnx) with automatic input shape inference and optimization. Exported models are stored as artifacts and can be served via ClearML Serving or external inference engines. Supports quantization and pruning for model compression.

Continuously monitors system resources (CPU, GPU, memory, disk I/O, network) during task execution and logs them as time-series metrics. Uses psutil for CPU/memory and nvidia-ml-py for GPU metrics. Metrics are sampled at configurable intervals (default 30s) and stored alongside experiment metrics. Enables detection of resource bottlenecks (e.g., GPU underutilization, memory leaks) and cost optimization analysis.