cognita

Provides a structured framework that organizes RAG components (data sources, indexing, retrieval, LLM integration) into discrete, independently deployable modules with FastAPI-based REST endpoints. Uses a layered architecture where each component (Model Gateway, Vector DB, Metadata Store, Query Controllers) is loosely coupled and can be extended or replaced without affecting others, enabling teams to move from experimental prototypes to production systems without architectural rewrites.

Implements a stateful indexing pipeline that compares the current state of data sources against the Vector Database to identify newly added, updated, and deleted documents, then selectively re-indexes only changed files. The system maintains metadata about each indexing run (status, timestamps, file hashes) in a Metadata Store, enabling efficient incremental updates without full re-indexing. Supports multiple data source types (local directories, URLs, GitHub repos, TrueFoundry artifacts) through an extensible loader interface.

Provides Docker Compose configuration and cloud deployment templates (TrueFoundry YAML) for deploying Cognita to production environments. Includes containerized backend (FastAPI), frontend (React), and supporting services (Vector DB, Metadata Store). Deployment configuration is externalized through environment variables and YAML files, enabling environment-specific customization (dev, staging, production) without code changes. Supports scaling through container orchestration platforms.

Enables developers to extend Cognita by implementing custom classes that inherit from base abstractions: custom Parsers for new document formats, custom DataSources for new data origins, custom QueryControllers for different retrieval strategies, custom Model providers for new LLM/embedding services. The modular architecture allows these custom components to be registered and used without modifying core Cognita code. Documentation and examples guide developers through the extension process.

Provides a single abstraction layer that unifies access to embedding models, LLMs, rerankers, and audio processors across multiple providers (OpenAI, Anthropic, Ollama, Infinity Server, custom providers). The Model Gateway exposes a consistent Python API regardless of underlying provider, allowing applications to switch providers by changing configuration without code changes. Internally routes requests to provider-specific APIs and handles response normalization, error handling, and fallback logic.

Provides a pluggable parser system that handles multiple document formats (PDF, TXT, DOCX, MD, HTML, JSON, etc.) with format-specific extraction logic. Each parser inherits from a base Parser class and implements format-specific chunking, metadata extraction, and content normalization. The system stores parsing configuration per data source in the Metadata Store, allowing different sources to use different parsers and chunk sizes. Supports custom parsers for domain-specific formats through inheritance and registration.

Abstracts vector database operations behind a unified interface that supports multiple backends (Qdrant, MongoDB, Milvus, Weaviate) for storing and querying embedded document chunks. The system handles vector storage, similarity search, metadata filtering, and collection management through provider-agnostic methods. Queries are executed by converting user questions to embeddings via the Model Gateway, then performing semantic similarity search in the Vector DB, with optional reranking to improve result quality.

Organizes documents into named collections, each with associated data sources, embedding configuration, and vector DB collection mappings. The Metadata Store maintains collection metadata (name, description, vector DB collection name, embedding model, parsing configuration) and tracks associations between collections and data sources. Collections enable multi-tenant or multi-project document organization within a single Cognita instance, with independent indexing and querying per collection.

Provides a React-based web interface (DocsQA) for non-technical users to upload documents, manage collections, configure data sources, and query the RAG system without writing code. The UI communicates with the FastAPI backend through REST endpoints, handling file uploads, collection creation, data source registration, and query submission. Supports drag-and-drop document upload, collection browsing, and interactive query results with source attribution.

Orchestrates the RAG query pipeline by retrieving relevant documents from the Vector DB via semantic search, optionally reranking results, and passing them to an LLM for answer generation. Query Controllers implement different retrieval strategies (simple similarity search, multi-query expansion, hypothetical document embeddings) and integrate with the Model Gateway for both embedding generation and LLM inference. Supports streaming responses and configurable result ranking/filtering.

Provides an extensible interface for connecting to various data sources (local directories, web URLs, GitHub repositories, TrueFoundry artifacts, custom sources) through pluggable loader classes. Each data source loader implements methods to list available documents, download content, and track changes. The system stores data source configuration in the Metadata Store and associates sources with collections. Custom loaders can be implemented by inheriting from a base DataSource class and registering with the system.

Provides a persistent store (SQLite, PostgreSQL, or compatible) that maintains all system configuration and state: collection definitions, data source associations, parsing configurations, embedding model settings, indexing run history, and document metadata. The Metadata Store enables reproducible indexing, audit trails, and state recovery after failures. Queries against the Metadata Store inform indexing decisions (e.g., which documents have changed since last run) and collection management.