Argo Workflows

Defines workflows as Kubernetes Custom Resource Definitions (Workflow, WorkflowTemplate, ClusterWorkflowTemplate) using YAML manifests, supporting both directed acyclic graph (DAG) and sequential step execution models. Each workflow step executes in an isolated container, with the workflow-controller reconciling the desired state against actual pod execution. Templates can be reused across workflows and namespaces via WorkflowTemplate and ClusterWorkflowTemplate CRDs.

Executes multiple workflow steps concurrently within configurable parallelism bounds, using Kubernetes scheduler to place pods on available nodes. Supports step-level parallelism limits, global workflow parallelism caps, and pod resource requests/limits (CPU, memory, GPU) for heterogeneous workloads. The workflow-controller submits pods to Kubernetes API and monitors their completion via pod status watches.

Abstracts workflow step execution through pluggable executor implementations (Docker, Kubelet, K3s, PNS - Process Namespace Sharing). The workflow-controller can be configured to use different executors based on cluster capabilities and security requirements. Each executor handles artifact staging, environment variable injection, and step lifecycle management differently. The argoexec sidecar is injected into step pods regardless of executor type.

Integrates with Kubernetes RBAC to control workflow submission, execution, and monitoring permissions. Workflows are namespace-scoped resources; users can only access workflows in namespaces where they have RBAC permissions. ClusterWorkflowTemplate resources enable cluster-wide template sharing with namespace-level access control. The argo-server enforces RBAC checks on all API requests.

Tracks workflow execution state through Workflow CRD status subresource, recording step-level execution metrics (start time, end time, duration, exit code, retry count). The workflow-controller continuously updates workflow status as pods complete, enabling real-time progress monitoring. Status includes DAG node status, artifact references, and error messages. Historical workflow data can be queried via REST API or archived to external database.

Enables workflow steps to mount Kubernetes volumes (PersistentVolumeClaim, ConfigMap, Secret, emptyDir, hostPath) for data sharing and configuration injection. Volumes are defined in workflow spec and mounted into step containers at specified paths. Supports both read-only and read-write mounts. The workflow-controller injects volume definitions into pod specs before submission.

Manages workflow artifacts (files, datasets, model checkpoints) across S3, GCS, Azure Blob Storage, Git, and HTTP sources using a pluggable artifact driver architecture. The argoexec sidecar container automatically stages artifacts into/out of step containers, handling compression, deduplication, and retry logic. Artifacts are referenced by name within workflows and automatically passed between steps via shared storage or direct pod-to-pod transfer.

Executes workflow steps conditionally using when expressions that evaluate against previous step outputs, parameters, and workflow variables. Supports boolean logic (AND, OR, NOT), string comparisons, and numeric comparisons. Expressions are evaluated by the workflow-controller before pod submission, enabling dynamic workflow branching without step execution overhead. Failed step conditions skip step execution and propagate to downstream steps.

Implements configurable retry policies at the step level with exponential backoff, jitter, and maximum retry limits. Timeouts are enforced per-step and per-workflow using Kubernetes pod deadline mechanism. Failed steps can be retried automatically without manual intervention, with backoff intervals increasing exponentially (e.g., 1s, 2s, 4s, 8s). Timeout violations trigger pod termination and step failure.

Enables data flow between workflow steps using parameters, which are passed as environment variables, command-line arguments, or template substitutions. Parameters can be defined at workflow submission time, generated by step outputs, or sourced from workflow variables. The workflow-controller performs variable substitution using {{}} syntax before pod creation, supporting nested parameter references and default values.

Schedules workflows to execute on a recurring basis using CronWorkflow CRD, which wraps a Workflow template and executes it at specified intervals. Supports standard cron expressions (minute, hour, day, month, day-of-week) with timezone awareness. The workflow-controller watches CronWorkflow resources and creates new Workflow instances at scheduled times, with configurable history limits and concurrency policies.

Provides argo-server component exposing REST and gRPC APIs for workflow submission, status queries, log retrieval, and artifact download. The web UI (React-based) consumes these APIs to display workflow DAG visualization, step-level logs, artifact browser, and execution metrics. APIs support filtering, pagination, and real-time updates via gRPC streaming. Authentication is integrated with Kubernetes RBAC and optional SSO providers.

Provides client libraries (Python, Go, JavaScript/TypeScript) for programmatically submitting workflows, querying status, and monitoring execution. SDKs wrap REST/gRPC APIs and provide type-safe interfaces for workflow definition and submission. Python SDK includes Argo Workflows Python DSL for defining workflows in Python code instead of YAML, with automatic CRD generation.

Manages workflow lifecycle from creation through completion, including automatic cleanup of completed workflows and archiving to persistent storage. Configurable garbage collection policies delete old workflows based on age or count limits. Workflow archiving stores completed workflow metadata in external database (PostgreSQL, MySQL) for long-term retention and audit trails. The workflow-controller enforces these policies during reconciliation.