Hatchet

Hatchet executes multi-step workflows defined as directed acyclic graphs (DAGs) stored in the v1_dag table, with hierarchical concurrency management that enforces limits at workflow, step, and action levels. The system uses a state machine approach for task lifecycle management (v1_task table) with automatic persistence, enabling workflows to survive process failures and resume from checkpoints. Concurrency constraints are evaluated at dispatch time via the dispatcher service, preventing resource exhaustion while maintaining fairness across concurrent workflow runs.

Hatchet triggers workflow runs in response to external events matched against CEL (Common Expression Language) filters stored in v1_filter and v1_match tables. The event matching system evaluates incoming events against registered workflow triggers, supporting complex conditional logic (e.g., 'event.type == "payment" && event.amount > 100') without requiring code changes. Events are persisted in the OLAP analytics schema (v1-olap) for audit trails and analytics, enabling both real-time triggering and historical event analysis.

Hatchet stores task payloads (inputs and outputs) in the v1_task_payload table as JSONB by default, but automatically offloads large payloads (>threshold, typically 1MB) to external storage (S3, GCS, Azure Blob Storage). The system stores a reference (URL or object key) in the database and fetches the payload on-demand when needed. This prevents PostgreSQL bloat and enables handling of very large payloads (e.g., multi-MB LLM responses, large file contents). Payload offloading is transparent to the application — the SDK handles fetching and caching automatically.

Hatchet abstracts the message queue layer to support both RabbitMQ (for high-throughput deployments) and PostgreSQL-based PGMQ (for simpler deployments without external dependencies). The message queue is used for task distribution, event publishing, and inter-service communication. The abstraction layer (pkg/config/shared/shared.go) allows switching between queue implementations via configuration without code changes. PGMQ is particularly useful for development and small deployments because it requires only PostgreSQL; RabbitMQ is recommended for production deployments with high throughput.

Hatchet includes a web-based dashboard (frontend/app/src/lib/api/generated/Api.ts) for monitoring workflow execution, viewing run history, and managing workflows. The dashboard displays real-time workflow status, step-by-step execution details, task logs, and failure reasons. Users can trigger workflow runs manually, view analytics (execution time trends, failure rates), and configure workflow settings. The dashboard is built with TypeScript/React and communicates with the API server via REST endpoints. Authentication is integrated with the API layer, supporting API keys and JWT tokens.

Hatchet workers register with the dispatcher service via gRPC streaming (internal/services/dispatcher/dispatcher_v1.go), establishing persistent bidirectional connections for real-time task assignment. Workers send heartbeats and availability signals; the dispatcher maintains worker state (ACTIVE, INACTIVE, DRAINING) and assigns tasks based on worker capacity and concurrency constraints. Task assignment is pull-based (workers request work) rather than push-based, reducing dispatcher load and enabling workers to control their own throughput. The dispatcher uses a state machine to track action assignment lifecycle (PENDING_ASSIGNMENT → ASSIGNED → STARTED → COMPLETED).

Hatchet enforces complete data isolation per tenant at the database schema level (all tables include tenant_id foreign key) and API layer (authentication middleware validates tenant context). Each tenant can configure resource limits (max concurrent workflows, max workers, rate limits) stored in configuration tables. The system uses PostgreSQL row-level security (RLS) policies to prevent cross-tenant data leakage, and the API server validates tenant context on every request via middleware (api/v1/server/middleware/telemetry/telemetry.go). Tenant-scoped metrics and analytics are isolated in the OLAP schema.

Hatchet persists task state in the v1_task table with configurable retry policies (max retries, backoff multiplier, max backoff duration) and timeout constraints. When a task fails or times out, the system automatically reschedules it with exponential backoff (e.g., 1s, 2s, 4s, 8s) up to a maximum retry count. Timeouts are enforced by the dispatcher (soft timeout) and workers (hard timeout via context cancellation). Failed tasks are marked with failure reason and stack trace for debugging. The retry logic is deterministic and idempotent — retrying a task with the same input produces the same result.

Hatchet implements rate limiting at multiple levels: per-workflow-run (max concurrent steps), per-step (max concurrent actions), and per-action (via dispatcher fairness scheduling). The dispatcher uses a fairness algorithm to distribute available capacity across competing workflow runs, preventing starvation when multiple workflows request the same action. Rate limits are stored in v1_workflow_concurrency_limit and v1_step_concurrency_limit tables and evaluated at dispatch time. The system supports both hard limits (reject excess requests) and soft limits (queue and backoff). This is particularly useful for LLM API calls where rate limits are strict and overages are expensive.

Hatchet uses two separate PostgreSQL schemas: v1-core for operational data (tasks, workflows, runs with high write frequency) and v1-olap for analytics (events, metrics, aggregates optimized for read-heavy queries). The v1-core schema uses row-level partitioning by tenant and time to manage table size and enable efficient pruning. The v1-olap schema stores denormalized event data and pre-aggregated metrics for reporting without impacting operational query performance. Data flows from v1-core to v1-olap via asynchronous ETL (event processing pipeline), enabling independent scaling and optimization of each schema.

Hatchet provides Python (pkg/client/rest/gen.go) and TypeScript (frontend/app/src/lib/api/generated/Api.ts) SDKs auto-generated from an OpenAPI specification (api-contracts/openapi/openapi.yaml). The SDKs expose high-level APIs for workflow definition, task submission, and result retrieval, abstracting away gRPC and REST details. Code generation ensures SDK consistency with server API changes — when the OpenAPI spec is updated, SDKs are regenerated automatically. The SDKs include type-safe request/response models (data-contracts.ts) and handle authentication, serialization, and error handling transparently.

Hatchet persists the complete state of each workflow run in PostgreSQL (v1_workflow_run table with status: PENDING, RUNNING, COMPLETED, FAILED) along with step execution state (v1_step_run table). When a dispatcher or worker crashes, the system can resume workflow execution from the last completed step without re-executing already-finished work. Step outputs are persisted in v1_task_payload table, enabling downstream steps to access results from previous steps. The system uses optimistic locking (version columns) to prevent concurrent state updates and ensure consistency.

Hatchet integrates structured logging (via middleware in api/v1/server/middleware/telemetry/telemetry.go) and metrics export for monitoring workflow execution. The system logs all significant events (task assignment, step completion, failures) with structured fields (tenant_id, workflow_id, step_id, duration) enabling easy filtering and correlation. Metrics are exported in Prometheus format (task count, execution duration, failure rate) and can be scraped by monitoring systems. Telemetry middleware captures request/response details and injects trace IDs for distributed tracing across services.