Valohai

Valohai stores pipeline definitions (YAML/configuration format) alongside application code in Git repositories, enabling version-controlled ML workflows where pipeline structure, parameters, and code evolve together. The platform syncs with Git to track pipeline changes, trigger runs on commits, and maintain complete lineage between code versions and experiment runs. This approach eliminates separate pipeline storage systems and leverages existing Git workflows for reproducibility.

Valohai automatically captures experiment metadata (hyperparameters, metrics, artifacts, environment) during pipeline runs without explicit logging code, then provides dashboards for comparing metrics across runs and tracing complete lineage (code version → data version → model output). The platform uses a metadata collection layer that intercepts training outputs and correlates them with Git commits, dataset versions, and infrastructure configuration.

Valohai provides real-time visibility into compute costs across multi-cloud infrastructure, tracking spending per job, pipeline, and project. The platform generates alerts when infrastructure is underutilized (e.g., GPUs idle, compute allocated but unused), enabling teams to optimize resource allocation and reduce costs. Cost tracking integrates with the per-user licensing model, separating infrastructure costs from platform licensing.

Valohai provides native integrations with popular data sources (Snowflake, BigQuery, Redshift), labeling platforms (Labelbox, V7 Labs), and ML frameworks (Hugging Face, Super Gradients) to simplify data loading and model integration. These integrations abstract authentication, data transfer, and API interactions, reducing boilerplate code. However, Valohai's architecture supports running arbitrary code, so teams are not limited to pre-built integrations.

Valohai maintains comprehensive audit logs tracking all platform actions (experiment runs, model deployments, data access, user actions) with timestamps and user attribution. These logs enable compliance with regulatory requirements (HIPAA, SOC2, GDPR) and provide accountability for ML model decisions. Audit logs are stored in Valohai and can be exported for compliance audits. Specific log retention policies and encryption are not documented.

Valohai abstracts compute infrastructure across AWS, GCP, Azure, on-premises, and private cloud environments through a unified job submission interface. Users define resource requirements (CPU, GPU, memory) in pipeline configurations, and Valohai's scheduler routes jobs to available infrastructure, auto-scaling compute up/down based on queue depth and workload. The platform supports Kubernetes, Slurm, and Docker-based execution, enabling teams to run the same pipeline across heterogeneous infrastructure without code changes.

Valohai tracks dataset versions and their relationships to experiments through a versioning system that claims to avoid data duplication (mechanism unspecified). The platform maintains lineage between datasets, pipeline runs, and models, enabling users to understand which data version produced which model and to reproduce experiments with exact dataset snapshots. Integration with data sources (Snowflake, BigQuery, Redshift) and labeling platforms (Labelbox, V7 Labs) enables tracking of unstructured data lineage.

Valohai supports deploying trained models for both batch inference (processing large datasets asynchronously) and real-time inference (serving predictions on-demand). The platform abstracts deployment infrastructure, allowing models to be deployed to the same multi-cloud environments used for training. Deployment configuration is defined in pipeline YAML, enabling version-controlled model serving. Real-time inference mechanism (API endpoints, containerization, scaling) is not detailed in documentation.

Valohai supports distributed training by orchestrating multi-node jobs across its infrastructure abstraction layer, enabling teams to scale training across multiple GPUs or CPUs without manual distributed training setup. The platform handles job coordination, resource allocation, and communication between nodes. Specific distributed training frameworks supported (Horovod, PyTorch DDP, TensorFlow distributed) are not documented.

Valohai supports hyperparameter optimization by enabling teams to define parameter search spaces in pipeline configurations and automatically running multiple experiments with different hyperparameter combinations. The platform orchestrates parallel hyperparameter tuning jobs across available infrastructure and tracks results for comparison. Specific optimization algorithms (grid search, random search, Bayesian optimization) are not documented.

Valohai supports human-in-the-loop workflows by enabling pipelines to pause for human review or decision-making before proceeding to next steps. This allows teams to implement approval gates (e.g., model validation before deployment), manual data labeling, or human feedback loops within automated pipelines. Specific implementation (UI for approvals, API for feedback) is not detailed.

Valohai exposes REST APIs and webhooks enabling external systems (CI/CD, data platforms, monitoring tools) to trigger pipeline runs, query experiment results, and integrate with existing workflows. Pipelines can be triggered via API calls, scheduled on intervals, or triggered by Git events. Webhooks enable Valohai to notify external systems of pipeline completion or status changes. Specific API endpoints, authentication mechanisms, and webhook payload formats are not documented.

Valohai's Model Hub provides centralized storage and versioning for trained model artifacts, enabling teams to track model versions, metadata, and relationships to training runs. Models can be tagged, compared across versions, and deployed directly from the Hub. The Hub integrates with experiment tracking to link models to specific training runs and hyperparameters. Specific artifact formats supported (SavedModel, ONNX, HDF5, etc.) and storage backend are not detailed.