haystack-ai

Haystack uses a directed acyclic graph (DAG) pipeline architecture where components (retrievers, generators, readers, etc.) are connected as nodes with typed inputs/outputs. Pipelines serialize to YAML/JSON for reproducibility and support both linear chains and complex branching logic. This enables developers to define multi-step LLM workflows declaratively without writing orchestration boilerplate, with automatic type validation between component connections.

Haystack's Retriever components embed documents into vector space using transformer models (BERT, DPR, etc.) and query against pluggable vector database backends (Weaviate, Pinecone, Qdrant, Elasticsearch, in-memory). The framework abstracts the vector store interface so developers can swap backends without changing retrieval logic. Supports hybrid search (dense + sparse/BM25) and metadata filtering across multiple vector store implementations.

Haystack provides a @component decorator and base class pattern enabling developers to create custom components with type-safe input/output contracts. Components declare inputs and outputs as type-hinted function parameters, and the framework validates connections at pipeline construction time. Custom components integrate seamlessly with the registry, serialization, and dependency injection systems. Supports both sync and async implementations.

Haystack's document converters support multi-modal content extraction including images, tables, and structured data from PDFs and web pages. PDFToDocument can extract images as separate Document objects with metadata linking to source pages. Table extraction preserves structure as markdown or HTML. Enables RAG systems to reason over visual content and structured data alongside text.

Haystack includes utilities for managing LLM context windows by tracking token counts, truncating documents to fit within limits, and prioritizing relevant content. The framework can estimate token usage before API calls and automatically truncate retrieved documents or conversation history to stay within model limits. Supports different tokenization strategies (OpenAI, HuggingFace, etc.) and can optimize context by removing low-relevance content.

Haystack includes Reader components that perform extractive question-answering by identifying answer spans within retrieved documents. Readers use transformer models (BERT, RoBERTa, ALBERT) fine-tuned on SQuAD-like datasets to extract exact answers from text. The framework supports both local reader models and API-based readers. Readers can be combined with retrievers in a two-stage pipeline (retrieve relevant documents, then extract answers).

Haystack provides format-specific document converters (PDFToDocument, MarkdownToDocument, HTMLToDocument, etc.) that extract text and metadata from various file types, followed by configurable chunking strategies (sliding window, recursive, semantic). Converters use specialized libraries (PyPDF2, python-docx, BeautifulSoup) and preserve document structure/metadata during conversion. Chunking strategies support overlap and can be tuned for different content types.

Haystack's Generator component abstracts LLM APIs (OpenAI, Anthropic, HuggingFace, Ollama, Azure, local models) behind a unified interface with consistent prompt templating, token counting, and response parsing. Supports both chat and completion endpoints with configurable parameters (temperature, max_tokens, top_p). Handles API key management, retries, and fallback logic. Enables swapping LLM providers without changing application code.

Haystack's Agent component uses an agentic loop (think, act, observe) where an LLM decides which tools to call based on a query, executes tools (retrievers, APIs, calculators), and iterates until reaching a final answer. Tools are registered via a schema-based interface with automatic function calling support for OpenAI/Anthropic models. Agents maintain conversation history and can handle multi-step reasoning tasks. Supports both ReAct-style prompting and function-calling APIs.

Haystack's PromptBuilder component uses Jinja2-style templating to construct dynamic prompts with variable interpolation, conditional logic, and few-shot example injection. Prompts can reference pipeline variables (query, retrieved documents, metadata) and support multi-turn conversation formatting. Templates are composable and can be versioned in YAML. Supports prompt engineering patterns like chain-of-thought, role-based prompting, and structured output formatting.

Haystack includes evaluation components (Evaluator, EvaluationRunResult) that measure RAG system quality across multiple dimensions: retrieval metrics (NDCG, MRR, precision@k), generation metrics (BLEU, ROUGE, semantic similarity), and end-to-end QA metrics (exact match, F1). Evaluators can run against ground-truth datasets and produce aggregated reports. Supports custom metric implementations via pluggable evaluator interface.

Haystack uses a component registry pattern where all pipeline components (retrievers, generators, evaluators) are registered with metadata (inputs, outputs, parameters) and can be instantiated from configuration (YAML/JSON). The framework provides dependency injection to wire components together based on type signatures. Components are serializable and can be saved/loaded with their configuration, enabling reproducible pipelines and model checkpointing.

Haystack abstracts document storage behind a DocumentStore interface supporting multiple backends (Elasticsearch, Weaviate, Pinecone, in-memory, SQL databases). Documents are stored with metadata and can be queried by ID, metadata filters, or semantic similarity. The abstraction enables switching storage backends without changing retrieval code. Supports batch operations (write, delete, filter) for efficient data management.

Haystack pipelines support async/await execution patterns enabling non-blocking I/O for API calls, database queries, and LLM requests. Components can be marked as async and the framework handles coroutine scheduling. Streaming responses are supported for generators, allowing token-by-token output without waiting for full completion. Enables building responsive applications with reduced latency for I/O-bound operations.