haystack

Haystack uses a decorator-based component system (@component) where any Python class can be registered as a reusable building block with typed inputs/outputs. Components connect via a directed acyclic graph (DAG) pipeline that validates type compatibility at graph construction time, enabling explicit control over data routing between retrieval, ranking, and generation stages. The Pipeline class manages execution order, handles variadic type conversion, and supports both sync and async execution paths with automatic serialization of component state.

Haystack provides end-to-end RAG by combining document retrieval (via vector databases or BM25), optional reranking stages (using cross-encoders or LLM-based rankers), and generation. The architecture separates retrieval from ranking from generation as distinct pipeline stages, allowing developers to swap retrievers (Elasticsearch, Weaviate, Pinecone) and rankers (Cohere, ColBERT, LLM-based) independently. Document preprocessing (splitting, embedding, metadata extraction) is handled by pluggable converters and embedders that support batch processing and streaming.

Haystack supports both synchronous and asynchronous pipeline execution through AsyncPipeline, enabling non-blocking I/O for external API calls, database queries, and file operations. Components can be marked as async, and the pipeline automatically handles concurrent execution where possible. This is critical for production systems where blocking on I/O would waste resources.

Haystack abstracts document storage through a DocumentStore interface that supports multiple backends (Weaviate, Pinecone, Qdrant, Chroma, Elasticsearch, In-Memory). Developers write document indexing and retrieval code once and can swap backends by changing configuration. The framework handles backend-specific details (API calls, query syntax, authentication) internally, enabling easy migration between databases.

Haystack supports serializing entire pipelines to YAML or JSON, enabling reproducible execution and version control of pipeline definitions. Developers can save a pipeline configuration, commit it to git, and recreate the exact same pipeline later. Component state (model weights, configuration) is also serializable, enabling checkpoint-and-restore workflows.

Haystack's agent system enables autonomous agents that iteratively reason over tool outputs using a loop pattern: agent receives query → selects tool → invokes tool → observes result → repeats until task complete. Tools are registered as components with type-safe schemas, and the agent uses an LLM to decide which tool to invoke based on the current state. The framework supports both simple tool-calling (via OpenAI/Anthropic function-calling APIs) and complex multi-step reasoning with memory of previous tool invocations.

Haystack abstracts LLM provider differences through a unified ChatMessage interface and pluggable generator components. Developers write once against the Haystack API and can swap between OpenAI, Anthropic, Cohere, Hugging Face, Azure, AWS Bedrock, and local models without changing pipeline code. The framework handles provider-specific details (API authentication, request formatting, response parsing) internally, and supports streaming responses, function calling, and vision capabilities where available.

Haystack provides a modular document processing pipeline that converts raw files (PDF, DOCX, HTML, Markdown) into structured Document objects, splits them into chunks, extracts metadata, and generates embeddings. Converters handle file format parsing, splitters implement various chunking strategies (fixed-size, semantic, recursive), and embedders integrate with external APIs (OpenAI, Hugging Face) or local models. The entire pipeline is composable — developers can chain converters, splitters, and embedders in custom sequences and apply them at scale.

Haystack integrates with multiple vector databases (Weaviate, Pinecone, Qdrant, Chroma, Elasticsearch) through pluggable DocumentStore implementations. The framework handles embedding generation, vector indexing, and similarity search with configurable distance metrics (cosine, dot product, Euclidean). Developers define retrieval strategies (top-k, threshold-based, hybrid BM25+vector) and the pipeline automatically handles batching, filtering by metadata, and result ranking.

Haystack provides a PromptBuilder component that constructs prompts from templates with variable substitution and chat message formatting. Templates support Jinja2 syntax for conditional logic and loops, and the builder automatically formats messages according to the target LLM's requirements (OpenAI's message format, Anthropic's format, etc.). Developers can define reusable prompt templates and compose them in pipelines, with support for few-shot examples and dynamic prompt engineering.

Haystack includes built-in evaluation components for assessing retrieval quality (precision, recall, MRR, NDCG) and generation quality (BLEU, ROUGE, semantic similarity). Developers can define evaluation pipelines that run queries against a gold standard dataset, compare retrieved documents to expected results, and score generated answers. The framework supports custom metrics and integrates with external evaluation libraries (e.g., RAGAS for RAG evaluation).

Haystack supports human-in-the-loop (HITL) patterns where agents or pipelines pause for human review and approval before proceeding. Developers can insert approval components that collect human feedback, validate decisions, or request clarification. The framework handles state persistence across human interactions and supports both synchronous (blocking) and asynchronous (non-blocking) approval patterns.

Haystack provides structured logging and tracing capabilities that capture component execution, LLM API calls, and pipeline state at each step. The framework integrates with OpenTelemetry for distributed tracing and supports custom instrumentation. Developers can trace execution flows, measure latency at each pipeline stage, and debug failures by inspecting intermediate results and error logs.