privateGPT

Converts documents into vector embeddings using local embedding models (no cloud calls) and stores them in a local vector database for semantic search. Uses a pluggable embedding provider architecture that supports multiple embedding models (e.g., sentence-transformers, Ollama embeddings) and vector stores (Chroma, Weaviate, Milvus), enabling fully offline document indexing without external API dependencies.

Executes LLM inference locally using pluggable LLM providers (Ollama, LlamaCPP, local Hugging Face models) or connects to local/self-hosted endpoints without internet connectivity. Implements a provider abstraction layer that normalizes different LLM APIs (streaming, token counting, model parameters) into a unified interface, allowing seamless switching between models and inference engines.

Processes multiple documents in batch mode, parsing, chunking, embedding, and indexing them into the vector database with progress tracking and error handling. Implements parallel processing where possible (embedding generation, parsing) to reduce total ingestion time, with resumable indexing for interrupted batches.

Splits documents into semantically-aware chunks using configurable strategies (fixed-size, recursive, semantic boundaries) and manages context windows for LLM consumption. Implements chunk overlap and metadata preservation to maintain document structure and enable accurate source attribution, with support for different chunking strategies per document type.

Orchestrates a retrieval-augmented generation (RAG) pipeline that retrieves relevant document chunks via semantic search, constructs a context-aware prompt, and generates answers using local LLMs. Implements ranking and filtering of retrieved chunks to manage context window constraints, with support for follow-up questions that maintain conversation history.

Extracts text and metadata from multiple document formats (PDF, DOCX, TXT, Markdown, CSV) using format-specific parsers and preserves structural information (headings, tables, page numbers). Implements a pluggable parser architecture that allows adding custom parsers for additional formats without modifying core logic.

Maintains multi-turn conversation state by storing and retrieving message history, with automatic context pruning strategies to prevent exceeding LLM context windows. Implements sliding window, summarization, and selective retention approaches to manage conversation length while preserving semantic continuity.

Provides a web-based interface (built with modern frontend framework) for uploading documents, asking questions, and viewing answers with source citations. Implements real-time streaming responses, document management UI, and conversation history display without requiring backend API knowledge.

Exposes configuration options (via YAML, environment variables, or code) to select and customize LLM providers, embedding models, vector databases, and other components at runtime. Implements a dependency injection pattern that allows swapping implementations without code changes, supporting multiple configuration sources with precedence rules.

Streams LLM responses token-by-token to the client in real-time rather than waiting for complete generation, reducing perceived latency and enabling progressive display of answers. Implements streaming protocol support for both local LLM providers (Ollama, LlamaCPP) and API-based providers, with proper handling of stream interruption and error states.

Tracks which document chunks contributed to each LLM answer and provides source citations with document names, page numbers, and chunk references. Implements metadata propagation through the RAG pipeline to maintain source information from retrieval through generation, enabling users to verify answer provenance.