Danswer (Onyx)

Danswer implements a modular connector architecture that ingests documents from heterogeneous sources (Slack, Google Drive, Confluence, GitHub, web crawlers) into a unified vector store. Each connector handles source-specific authentication, pagination, and metadata extraction, then chunks documents and generates embeddings via configurable embedding models. The framework supports incremental indexing with change detection to avoid re-processing unchanged documents.

Danswer implements a hybrid search pipeline that combines dense vector similarity (via embeddings) with sparse lexical matching (BM25) to retrieve relevant documents. The system ranks results using a learned combination of both signals, improving recall for keyword-heavy queries while maintaining semantic understanding. Search results include source attribution, relevance scores, and direct links back to original documents.

Danswer provides a web-based admin dashboard for managing connectors, configuring indexing parameters, monitoring sync status, and viewing system health. The dashboard displays indexing progress, error logs, and document statistics. Admins can trigger manual re-indexing, configure LLM and embedding providers, and manage user access. The dashboard is role-based, restricting sensitive operations to administrators.

Danswer implements incremental sync for connectors, detecting changes in source systems and only re-indexing modified documents. The system tracks document versions, timestamps, and checksums to identify changes. Incremental sync reduces indexing time and API calls to source systems. Supports both full re-index and incremental update modes. Change detection is source-specific — some connectors support efficient change detection while others require full re-indexing.

Danswer allows customization of system prompts and response templates used during RAG-powered chat. Admins can define custom instructions for the LLM (e.g., 'always cite sources', 'be concise'), control response tone and format, and add domain-specific guidance. Prompts are versioned and can be A/B tested. The system supports prompt variables for dynamic content (e.g., user name, current date).

Danswer implements a conversational AI layer that retrieves relevant documents for each user query, passes them as context to an LLM (OpenAI, Anthropic, Ollama), and generates grounded responses with citations. The system maintains conversation history, allowing follow-up questions to reference previous context. Citations include direct links to source documents, enabling users to verify answers and explore related content.

Danswer preserves and enforces document-level access controls during indexing and retrieval. When documents are ingested from sources like Slack, Confluence, or Google Drive, their permission metadata (who can read) is captured. During search and chat, results are filtered to only include documents the current user has access to, preventing unauthorized information disclosure. This is implemented via user identity mapping and permission checks at query time.

Danswer provides a native Slack bot that allows users to search and chat with indexed documents directly within Slack. The bot handles Slack message parsing, thread context, and user identity mapping. Users can mention the bot in channels or DMs, ask questions, and receive responses with citations. The integration supports slash commands for advanced queries and configuration. Slack user identities are mapped to document access controls, ensuring permission enforcement within Slack.

Danswer includes a web crawler that discovers and indexes public web pages (e.g., company documentation sites, public wikis). The crawler follows links up to a configurable depth, respects robots.txt, and extracts text content from HTML. Crawled pages are chunked, embedded, and stored alongside other indexed documents. The crawler supports scheduling for periodic re-indexing to keep content fresh.

Danswer supports multiple embedding model providers (OpenAI, Ollama, HuggingFace, Cohere) and allows switching between them without re-indexing. The system abstracts embedding generation behind a provider interface, enabling users to choose based on cost, latency, or privacy requirements. Embedding dimensions are automatically detected and validated. The framework supports both cloud-hosted and self-hosted embedding models.

Danswer implements multiple document chunking strategies (fixed-size, semantic, recursive) to split large documents into embeddings-friendly chunks. Users can configure chunk size, overlap, and strategy per document type. The system preserves chunk metadata (source, page number, section) to enable accurate source attribution. Chunking is applied during indexing and can be re-applied without re-downloading documents.

Danswer abstracts LLM interactions behind a provider interface supporting OpenAI, Anthropic, Ollama, and other compatible APIs. Users can switch LLM providers via configuration without code changes. The system handles provider-specific API differences (token limits, function calling, streaming) transparently. Supports both cloud-hosted and self-hosted models. Enables cost optimization by routing queries to different models based on complexity.

Danswer extracts and preserves document metadata during indexing (author, creation date, modification date, file type, source system, permissions). Metadata is stored alongside embeddings and used for filtering, sorting, and source attribution. The system supports custom metadata fields per connector. Metadata is included in search results and citations, enabling users to assess document freshness and credibility.