Hatchet

Hatchet executes complex multi-step workflows defined as directed acyclic graphs (DAGs) stored in the v1_dag table, with built-in hierarchical concurrency management that enforces resource limits at workflow, step, and action levels. The system uses a state machine approach for task lifecycle management (v1_task table) with automatic persistence, enabling workflows to survive service restarts and coordinate dependencies across distributed workers via gRPC streaming.

Hatchet triggers workflow runs in response to external events using a CEL (Common Expression Language) expression matcher stored in v1_filter and v1_match tables. When an event is published to the system, the dispatcher evaluates CEL expressions against event payloads to determine which workflows should be triggered, enabling complex conditional logic without hardcoding trigger rules. This architecture decouples event producers from workflow definitions.

Hatchet scales horizontally by running multiple dispatcher instances, each managing a subset of worker connections based on worker affinity or hash-based sharding. Workers register with a specific dispatcher instance, and the system routes task assignments to the appropriate dispatcher based on worker availability. The architecture supports adding/removing dispatcher instances without downtime, with workers automatically reconnecting to available dispatchers on failure.

Hatchet includes built-in observability through structured logging (api/v1/server/middleware/telemetry/telemetry.go) and metrics export to OpenTelemetry-compatible backends. The system logs task execution events, worker lifecycle events, and API requests with structured fields (tenant_id, workflow_id, task_id) for easy filtering and correlation. Metrics include task latency, success rates, worker utilization, and dispatcher throughput, exported via OpenTelemetry SDK.

Hatchet supports PostgreSQL PGMQ as a built-in message queue alternative to RabbitMQ, eliminating the need for a separate message broker in simpler deployments. PGMQ uses PostgreSQL tables for queue storage, with the same API as RabbitMQ but without external dependencies. This is suitable for deployments where PostgreSQL is already required and operational complexity should be minimized.

Hatchet stores workflow definitions with versioning, allowing multiple versions of a workflow to coexist. Each workflow run is bound to a specific workflow version, ensuring that historical runs can be replayed or analyzed against the exact workflow definition that executed them. The system maintains immutable run history in the v1_workflow_run table, preventing accidental modification of historical data.

Hatchet's dispatcher service (dispatcher_v1.go) maintains persistent gRPC streaming connections to workers, pushing task assignments in real-time rather than workers polling a queue. The dispatcher monitors worker heartbeats and automatically reassigns tasks from dead workers, implementing a pull-based model where workers declare availability and the dispatcher matches them to queued tasks. This architecture reduces latency and enables fair scheduling across heterogeneous worker pools.

Hatchet persists task state in the v1_task table with built-in retry logic that automatically re-executes failed tasks using exponential backoff (configurable base and max multiplier). Each task has a timeout enforced at the dispatcher level; if a task exceeds its timeout, the dispatcher marks it as failed and triggers the retry mechanism. The system tracks retry count and can enforce a maximum retry limit, with all retry history persisted for debugging.

Hatchet implements complete data isolation per tenant at the database level, with all tables partitioned or filtered by tenant_id. The system supports configurable resource limits per tenant (concurrency limits, rate limits, storage quotas) enforced at the API and dispatcher layers. Tenants cannot access each other's workflows, runs, or events, and resource consumption is tracked separately for billing and enforcement.

Hatchet uses a dual-database schema: v1-core for operational data (tasks, workflows, runs) optimized for transactional consistency and fast writes, and v1-olap for analytical data (event aggregations, metrics) optimized for reporting and analytics. The system asynchronously replicates data from v1-core to v1-olap, enabling complex analytical queries without impacting operational performance. This architecture allows operational tables to be heavily partitioned for scalability while analytical tables maintain denormalized views.

Hatchet stores task input and output payloads in the v1_task_payload table, with automatic offloading to external object storage (S3, GCS, etc.) when payloads exceed a configurable size threshold. The system maintains references to offloaded payloads in the database, transparently fetching them when needed. This architecture prevents database bloat from large payloads while maintaining a single logical view of task data.

Hatchet implements rate limiting at multiple levels: per-workflow, per-step, and per-action, with fairness scheduling that ensures no single workflow starves others when shared resources are constrained. The system uses token bucket algorithms with configurable rates and burst sizes, stored in the hierarchical concurrency control layer. This is specifically optimized for LLM API calls where rate limits are common and fairness is critical for multi-tenant systems.

Hatchet provides Python and TypeScript SDKs that are automatically generated from the OpenAPI specification (api-contracts/openapi/openapi.yaml), ensuring consistency between API and SDK. The SDKs include high-level abstractions for defining workflows, registering actions, and triggering runs, with type safety through generated data models. The generation pipeline (pkg/client/rest/gen.go) is part of the build process, ensuring SDKs stay in sync with API changes.

Hatchet provides a web-based dashboard (frontend/app) built with React that displays workflow definitions, execution history, and real-time task status. The dashboard queries the v1-olap analytical schema for historical data and the API for real-time status, with WebSocket support for live updates. Users can trigger workflow runs, inspect task inputs/outputs, view retry history, and debug failed tasks through the UI.