Haystack

Haystack provides a decorator-based component system (@component) where any Python class becomes a composable unit with typed inputs/outputs. Components are connected into directed acyclic graphs (DAGs) via a Pipeline class that validates socket connections, enforces type safety, and manages data flow between components. The pipeline system supports both sync (Pipeline) and async (AsyncPipeline) execution with automatic variadic type conversion, enabling developers to wire together retrievers, rankers, generators, and custom logic without boilerplate orchestration code.

Haystack abstracts document persistence and retrieval through a DocumentStore interface supporting multiple backends (Elasticsearch, Pinecone, Weaviate, In-Memory, etc.). Each backend implements hybrid search combining dense vector similarity with sparse keyword matching, supporting filtering by metadata, custom scoring, and batch operations. The abstraction layer handles connection pooling, index creation, and query translation, allowing pipelines to swap backends without code changes.

Haystack pipelines can be serialized to YAML/JSON format for version control and deployment. The serialization captures component configurations, connections, and metadata, enabling pipelines to be deployed without code changes. Deserialization reconstructs the pipeline from serialized format, supporting dynamic component loading and configuration injection from environment variables or config files.

Haystack provides a PromptBuilder component that constructs prompts from templates with variable placeholders, supporting Jinja2-style templating with Python type hints. Templates can include system messages, few-shot examples, and dynamic content, and the builder validates that all required variables are provided before rendering. The rendered prompts are converted to ChatMessage objects for LLM consumption, enabling reusable prompt templates across different models.

Haystack enables developers to create custom components by decorating Python classes with @component, defining typed inputs and outputs via method signatures. The framework validates component contracts at pipeline construction time, ensuring type compatibility with connected components. Custom components can be stateful (holding model instances), async, and integrated seamlessly into pipelines without special handling.

Haystack abstracts LLM providers (OpenAI, Anthropic, Cohere, Hugging Face, Azure, AWS Bedrock, local models) through a unified Generator component accepting ChatMessage objects. The system handles provider-specific API differences, token counting, streaming, and response parsing transparently. Developers define prompts as ChatMessage templates with variable interpolation, and the same prompt code works across providers by swapping the generator component.

Haystack includes DocumentConverter components that extract text from multiple formats (PDF, HTML, DOCX, Markdown, etc.) and convert them to Document objects. The preprocessing pipeline chains converters with splitters (recursive character splitting, semantic splitting) and cleaners (whitespace normalization, HTML tag removal) to prepare raw documents for embedding. Each converter handles format-specific parsing (PDF layout analysis, HTML structure extraction) and outputs normalized Document objects with preserved metadata.

Haystack provides Embedder components (supporting OpenAI, Hugging Face, local models) and Ranker components (cross-encoders, diversity rankers, custom scorers) that can be composed in pipelines to optimize retrieval quality. Embedders convert text to dense vectors with configurable batch sizes and pooling strategies. Rankers re-score retrieved documents using cross-encoder models or custom scoring functions, enabling multi-stage ranking (BM25 → dense retrieval → cross-encoder reranking) without code duplication.

Haystack's Agent system enables autonomous workflows where an LLM iteratively reasons about tasks, invokes tools (function calls), and processes results until reaching a goal. Agents use a schema-based tool registry where Python functions are decorated with @tool and automatically converted to OpenAI/Anthropic function-calling schemas. The agent loop handles tool selection, execution, error handling, and result integration back into the LLM context, supporting both synchronous and asynchronous tool execution.

Haystack includes evaluation components for measuring RAG pipeline quality through metrics like BLEU, ROUGE, MRR, NDCG, and semantic similarity. Evaluators compare generated outputs against ground truth or reference answers, and retrieval evaluators measure whether relevant documents are ranked highly. The evaluation system integrates with pipelines, allowing developers to run evaluations on datasets and track metrics across model/retriever changes.

Haystack provides components for integrating human feedback into RAG pipelines, enabling workflows where users validate or correct agent actions, retrieved documents, or generated answers. The system captures feedback (relevance judgments, corrections, ratings) and can use it to improve future pipeline runs through reranking, fine-tuning signals, or online learning. Feedback is stored alongside pipeline execution traces for analysis and model improvement.

Haystack provides built-in observability through execution tracing that captures component inputs/outputs, execution time, and errors. Traces are structured as trees matching the pipeline DAG, enabling developers to inspect exactly what data flowed through each component and identify performance bottlenecks. Integration with external tracing systems (e.g., OpenTelemetry) allows exporting traces to monitoring platforms for production debugging.

Haystack provides AsyncPipeline that executes components concurrently using Python's asyncio, enabling high-throughput processing without thread management complexity. Components can be marked as async-compatible, and the pipeline automatically schedules concurrent execution where dependencies allow. Async execution is particularly valuable for I/O-bound operations (API calls, database queries) where traditional synchronous pipelines would block.