Kestra

Enables users to define complex orchestration workflows in YAML with built-in schema validation, type checking, and auto-completion. The system parses YAML into a strongly-typed Flow model that validates task dependencies, input parameters, and output references at definition time before execution. Uses a custom YAML parser with Kestra-specific extensions for templating and variable interpolation.

Implements a controller-worker distributed execution model where the controller schedules tasks to a pool of stateless workers via a message queue. Workers pull tasks from the queue, execute them in isolated containers or processes, and report results back to the controller. The RunContext object carries execution state (variables, outputs, secrets) through the execution chain using Pebble templating for dynamic value resolution.

Implements namespace-based isolation for workflows, executions, and secrets, enabling multi-tenant deployments. Each namespace is a logical boundary with its own workflows, execution history, and secrets. Access control is enforced at the namespace level, allowing fine-grained permission management (read, write, execute). Namespaces support hierarchical organization (e.g., `team.project.environment`) and can be used to segregate environments (dev, staging, prod) or teams.

Integrates an AI copilot that generates workflow YAML from natural language descriptions and provides intelligent code suggestions. The copilot uses LLM APIs (OpenAI, Anthropic) to understand user intent and generate syntactically valid Kestra workflows. It can suggest task chains, recommend plugins for integrations, and auto-complete workflow definitions based on context. The system learns from existing workflows in the namespace to provide contextually relevant suggestions.

Provides a file storage system for managing workflow artifacts, intermediate data, and execution outputs. Files are stored in a configurable backend (local filesystem, S3, GCS, Azure Blob) and organized by namespace and execution. The system supports file upload/download via API and UI, automatic cleanup of old artifacts based on retention policies, and file versioning. Artifacts can be referenced across tasks using file paths, enabling data sharing between workflow steps.

Provides a distributed key-value store for persisting workflow state, caching intermediate results, and sharing data across executions. The KV store is namespace-isolated and supports atomic operations (get, set, delete, increment). Values can be complex objects (JSON) or simple scalars, with optional TTL for automatic expiration. Tasks can read and write to the KV store using dedicated task types, enabling stateful workflows and cross-execution data sharing.

Enables version control of workflows through Git integration, allowing workflows to be stored in Git repositories and synced with Kestra. Each workflow version is tracked with commit history, enabling rollback to previous versions. The system supports multiple deployment strategies (manual sync, automatic CI/CD, polling). Workflows can be deployed from Git branches, enabling environment-specific configurations (dev, staging, prod) without duplicating workflow definitions.

Provides a webhook-based event ingestion system that captures external events (API calls, file uploads, database changes) and triggers workflow executions in real-time. Events are validated against a schema, stored in the event log, and matched against registered triggers using pattern matching. The trigger system supports multiple event sources (HTTP webhooks, Kafka topics, database polling) and can fan-out to multiple workflows based on event attributes.

Provides a scheduler component that evaluates cron expressions to trigger workflows on fixed schedules. The scheduler runs on the controller and checks all registered schedules at configurable intervals, creating execution instances when cron conditions are met. Supports timezone-aware scheduling, backfill for missed executions, and schedule pausing/resuming without workflow redefinition.

Implements a plugin architecture where tasks are implemented as pluggable components with standardized interfaces. Each plugin declares inputs, outputs, and configuration via annotations, which are automatically discovered and documented. Plugins can wrap external tools (Python, Node.js, Bash scripts), APIs (AWS, GCP, Slack), or databases (PostgreSQL, MongoDB). The plugin system generates documentation, validates inputs at runtime, and handles dependency injection for configuration and credentials.

Implements a parameter resolution system that validates workflow inputs against a schema, applies type coercion, and resolves default values. Inputs can be marked as required or optional, with support for complex types (arrays, objects, enums). The system generates input forms in the UI based on parameter definitions, enabling non-technical users to provide inputs without editing YAML. Input values are validated before execution and errors are reported with field-level detail.

Integrates the Pebble templating engine to enable dynamic value resolution throughout workflow execution. Variables can reference task outputs, inputs, system variables (execution date, flow name), and secrets using template syntax. Pebble expressions support filters, conditionals, and loops, allowing complex transformations without custom code. Variables are resolved at task execution time using the current RunContext, enabling data-dependent branching and dynamic task configuration.

Provides real-time monitoring of workflow executions with live task status updates, execution logs, and performance metrics. The execution view displays task DAG with color-coded status (running, success, failed, skipped), execution timeline, and detailed logs per task. Metrics include task duration, memory usage, and resource consumption. The system streams execution updates to the UI via WebSocket, enabling real-time visibility without polling.

Implements conditional execution logic allowing tasks to branch based on previous task outputs or execution context. Tasks can be skipped, executed conditionally, or trigger alternative branches on failure. The system supports if/else branching via task conditions, error handlers that catch failures and trigger recovery tasks, and retry logic with exponential backoff. Conditions are evaluated using Pebble expressions against RunContext, enabling data-dependent branching.

Provides a centralized secret store for managing credentials, API keys, and sensitive configuration. Secrets are encrypted at rest and injected into task execution environments via environment variables or task inputs. The system supports secret rotation, audit logging of secret access, and namespace-level secret isolation. Secrets are referenced in workflows using a special syntax (e.g., `{{ secret.api_key }}`), and access is logged for compliance.