prefect

Enables developers to define workflows as standard Python functions decorated with @flow and @task, converting imperative Python code into orchestrated DAGs without requiring domain-specific languages. The system uses Python's function introspection and async/await support to automatically capture task dependencies, parameter types, and return values, building an execution graph at definition time that can be serialized and deployed independently of the defining code.

Implements a deterministic state machine where each task and flow transitions through defined states (Pending → Running → Completed/Failed/Cancelled) with automatic persistence to a backend database. The execution engine tracks state transitions, captures timestamps and result metadata, and automatically applies retry logic with exponential backoff, timeout handling, and failure recovery based on configurable policies stored in the database as orchestration policies.

Provides a Python client library that enables local workflow execution (without a server) and programmatic interaction with Prefect servers. The client handles flow and task execution, state management, and communication with the Prefect API. It supports both synchronous and asynchronous execution models and can be used in scripts, notebooks, or as a library. The client includes utilities for testing workflows locally before deployment and for querying server state from external applications.

Provides a comprehensive command-line interface for managing workflows, deployments, and server operations. The CLI supports commands for creating/updating deployments, running flows locally, querying execution history, managing blocks, and configuring Prefect settings. Commands are organized hierarchically (e.g., `prefect deployment create`, `prefect flow run`) and support both interactive and non-interactive modes. The CLI uses Typer for command definition and supports shell completion for common commands.

Provides a modern React-based web UI (v2) for monitoring workflow execution, managing deployments, and querying execution history. The dashboard displays real-time flow run status, task execution timelines, logs, and state transitions. It supports filtering and searching across flows, deployments, and runs, and provides interactive controls for pausing/resuming deployments and triggering manual flow runs. The UI communicates with the Prefect API and supports role-based access control.

Provides mechanisms to limit concurrent task execution and enforce rate limits on task runs. Concurrency limits are defined per-tag and are enforced globally across all workers, preventing more than a specified number of tagged tasks from running simultaneously. Rate limiting can be applied per-task or per-flow to control resource consumption. The system uses a distributed lock mechanism to enforce concurrency limits across multiple workers without requiring a centralized coordinator.

Decouples task scheduling from execution by routing tasks to named work queues that are consumed by distributed workers running on heterogeneous infrastructure (local machines, Kubernetes, cloud VMs). Workers poll work queues via the Prefect API, pull task execution requests, execute them in isolated processes or containers, and report results back to the server, enabling horizontal scaling and infrastructure-agnostic task distribution without modifying workflow code.

Provides an event system where external systems (webhooks, cloud services, custom applications) emit events to Prefect, which are stored in a time-series database and matched against user-defined automation rules. Rules specify event filters (event type, source, attributes) and actions (trigger flow run, send notification, update deployment), enabling workflows to react to external state changes without polling or manual intervention. Events are queryable and can be used for debugging and audit purposes.

Packages flow code, dependencies, and configuration into versioned deployment artifacts that map to specific infrastructure (work pools, workers, environment variables). Deployments are created via CLI or API, stored in the Prefect database, and include metadata about code location, parameter defaults, schedule triggers, and execution environment. The deployment system supports multiple code storage backends (local filesystem, S3, GitHub) and can automatically pull code from version control before execution, enabling separation of flow definition from deployment configuration.

Enables workflows to run on fixed schedules defined via cron expressions or interval specifications (e.g., every 5 minutes, daily at 2 AM). Schedules are stored as deployment metadata and evaluated by a background scheduler service that creates flow run records at scheduled times. The scheduler supports timezone-aware scheduling, automatic daylight saving time handling, and can be paused/resumed without modifying deployment configuration. Schedule evaluation is deterministic and idempotent, preventing duplicate runs even if the scheduler restarts.

Enables dynamic task parallelization by mapping a task over a collection of inputs, automatically creating multiple task runs with different parameters. Mapping is specified via the `.map()` method on task calls, which accepts iterables or upstream task results. The execution engine expands mapped tasks into individual runs that execute in parallel (subject to worker concurrency limits) and collects results into a list. Mapping supports nested structures and can be combined with conditional logic via task result filtering.

Captures logs from task execution across distributed workers and aggregates them into a centralized logging system accessible via the Prefect UI and API. Logs are structured with metadata (task name, flow run ID, worker ID, timestamp) and can be filtered by flow run, task, or worker. The logging system supports multiple log levels and can be configured per-task or globally. Logs are stored in the Prefect database and are queryable, enabling debugging of distributed workflows without SSH access to worker machines.

Provides a pluggable block system where reusable configuration objects (credentials, connection strings, API keys) are stored encrypted in the Prefect database and can be referenced by name in task code. Blocks are defined as Python classes with type annotations and can be created via CLI, API, or UI. The block system supports custom block types via plugin registration, enabling integration with external systems (cloud providers, databases, APIs). Blocks are versioned and can be updated without modifying task code.

Exposes Prefect's core functionality via a FastAPI-based REST API with auto-generated OpenAPI/Swagger documentation. The API supports CRUD operations on flows, deployments, flow runs, tasks, and blocks, as well as querying execution history and logs. The API uses Pydantic models for request/response validation and supports both JSON and form-encoded request bodies. Authentication is handled via API keys or OAuth2 tokens, and authorization is enforced via role-based access control. The API is versioned and maintains backward compatibility across minor versions.