Kestra

Kestra enables workflow definition through declarative YAML syntax that gets parsed and validated against a Flow model schema. The system uses Pebble templating engine (integrated via PebbleExpressionService in core/runners) to enable dynamic variable interpolation, conditional logic, and expression evaluation within workflow definitions. YAML is deserialized into strongly-typed Flow objects with built-in validation, allowing developers to define complex orchestration logic without imperative code while maintaining type safety and IDE support through schema validation.

Kestra implements a modular plugin system where tasks are loaded dynamically from a registry of 500+ pre-built plugins covering databases, cloud platforms, messaging systems, and data tools. Each plugin is a self-contained module with its own build.gradle configuration that implements task interfaces and registers handlers with the core execution engine. The plugin system includes automatic documentation generation and schema validation, allowing developers to extend Kestra with custom tasks by implementing standard interfaces without modifying core code.

Kestra provides script task types that execute arbitrary code in multiple languages (Python, Bash, Node.js, PowerShell, etc.) within containerized environments. The Script Tasks system (core/runners) handles language detection, dependency installation, and execution isolation, allowing developers to embed custom logic directly in workflows without creating separate plugins. Scripts can access the execution context through environment variables and stdin, and return results through stdout or files, enabling flexible integration of custom code with the orchestration platform.

Kestra includes native AI task types that integrate with LLM providers (OpenAI, Anthropic, etc.) to enable AI-powered workflow steps. These tasks accept prompts, context, and configuration parameters, send requests to LLM APIs, and return structured results that can be used in downstream tasks. The AI integration is implemented as standard tasks within the plugin system, allowing workflows to incorporate AI-powered decision-making, content generation, and data analysis without external orchestration.

Kestra enables workflows to be stored in Git repositories and synced with the Kestra server, providing version control, change tracking, and collaborative workflow development. Workflows are defined as YAML files that can be committed to Git, enabling teams to use standard Git workflows (branches, pull requests, code review) for workflow changes. The system supports bidirectional sync between Git and Kestra, allowing workflows to be edited in the UI or in Git and synchronized automatically.

Kestra provides a secrets management system that stores sensitive credentials (API keys, database passwords, etc.) in encrypted form within the persistent data layer. Secrets are scoped to namespaces and can be referenced in workflow definitions using a special syntax (e.g., `{{ secret.api_key }}`), which are resolved at execution time. The system supports multiple secret backends (encrypted database storage, external vaults) and provides audit logging for secret access.

Enables version control of workflows through Git integration, allowing workflows to be stored in Git repositories and synced with Kestra. Each workflow version is tracked with commit history, enabling rollback to previous versions. The system supports multiple deployment strategies (manual sync, automatic CI/CD, polling). Workflows can be deployed from Git branches, enabling environment-specific configurations (dev, staging, prod) without duplicating workflow definitions.

Kestra implements a distributed execution model with a Controller component that manages workflow scheduling and state, and Worker components that execute individual tasks in isolation. The architecture uses a message queue (Kafka or in-memory) for task distribution and state synchronization across workers. Workers pull tasks from the queue, execute them in containerized environments (Docker or native), and report results back to the Controller, enabling horizontal scaling and fault isolation without requiring shared state between workers.

Kestra implements a trigger system that enables workflows to be initiated by external events through HTTP webhooks, message queue events (Kafka, RabbitMQ), or scheduled cron expressions. The Execution API exposes webhook endpoints that accept event payloads, validate them against trigger schemas, and immediately queue workflow executions with the event data as input. The trigger system integrates with the Scheduler component to handle both time-based and event-based activation patterns, supporting complex trigger conditions and event filtering without requiring polling.

Kestra automatically generates input forms and validation rules from workflow YAML schemas using a JSON Schema-based approach. The Input Resolution system validates user-provided inputs against defined input types (string, integer, select, file, etc.) before workflow execution, with support for conditional inputs, default values, and dynamic field visibility. The frontend (ui/src/components/InputForms) renders these schemas as interactive forms, enabling non-technical users to provide workflow inputs through a UI rather than manually constructing YAML or API calls.

Kestra's RunContext provides a unified execution context that tracks workflow state, task outputs, and variables throughout execution. The PebbleExpressionService evaluates Pebble template expressions within this context, enabling dynamic variable interpolation, conditional logic, and access to execution metadata (task outputs, execution ID, timestamps). Variables are resolved at task execution time, allowing downstream tasks to reference upstream task outputs using expressions like `{{ outputs.taskName.result }}`, with support for complex object navigation and filtering.

Kestra maintains a complete execution history in a persistent data layer (supporting PostgreSQL, MySQL, H2) that stores execution records, task results, logs, and audit trails. The Data Persistence Layer uses JDBC drivers (jdbc-postgres, jdbc-mysql, jdbc-h2) to abstract database interactions, enabling queries across execution history for debugging, compliance, and analytics. Execution logs are stored alongside execution metadata, allowing developers to retrieve full execution traces including task outputs, error messages, and timing information for any past execution.

Kestra's Scheduler component manages time-based workflow triggers using cron expressions and interval specifications. The Scheduler runs as a separate service (SchedulerCommand) that evaluates trigger conditions at regular intervals and queues workflow executions when conditions are met. It integrates with the distributed execution system to ensure that scheduled workflows are executed reliably even in clustered deployments, with built-in deduplication to prevent duplicate executions if multiple scheduler instances are running.

Kestra provides real-time execution monitoring through WebSocket connections that stream execution status updates, task progress, and log output to connected clients. The frontend (ui/src/components/ExecutionMonitoring) subscribes to execution updates and displays live progress, task status changes, and log streaming without requiring polling. The backend maintains execution state in memory and broadcasts updates to all connected clients, enabling multiple users to monitor the same execution simultaneously with sub-second latency.

Kestra implements namespace-based isolation where workflows, executions, and secrets are scoped to namespaces, enabling multi-tenant deployments where different teams or customers have isolated workflow spaces. Namespaces are enforced at the API level and in the data persistence layer, ensuring that users can only access workflows and executions within their assigned namespaces. The Storage and KV Store systems also respect namespace boundaries, enabling per-namespace configuration and secrets management without cross-tenant data leakage.