Text Generation WebUI

Dynamically loads language models from multiple backends (llama.cpp, ExLlamaV2/V3, Transformers, TensorRT-LLM) through a hub-and-spoke architecture where models.py acts as a loader dispatcher that populates shared.model and shared.tokenizer global state. The system detects model format (GGUF, GPTQ, safetensors) and routes to the appropriate backend loader, abstracting backend-specific initialization complexity behind a single load_model() interface.

Implements a text generation pipeline (text_generation.py) that streams tokens in real-time using backend-specific generate() methods while applying configurable sampling strategies (temperature, top-p, top-k, repetition penalty, etc.). The pipeline supports both greedy decoding and stochastic sampling, with per-model preset configurations stored in models_settings.py that override global defaults, enabling fine-grained control over generation behavior without code changes.

Integrates HuggingFace Hub integration for discovering, downloading, and caching models directly from the web UI. The system manages model downloads with progress tracking, supports resumable downloads, and caches models in a configurable directory to avoid re-downloading. Users can search for models by name or filter by size/quantization format, with automatic detection of model format (GGUF, safetensors, etc.) and routing to the appropriate backend loader.

Builds the entire web UI using Gradio 3.40+, which provides responsive HTML/CSS/JavaScript frontend with real-time streaming support via WebSockets. The interface is organized into tabs (Chat, Notebook, Training, Model Menu, Extensions) with Gradio components (Textbox, Slider, Dropdown, etc.) that automatically handle state management and event binding. Streaming responses are rendered in real-time as tokens arrive, with automatic UI updates without page refresh.

Implements intelligent context window management that counts tokens in the conversation history using the actual model's tokenizer and automatically truncates old messages when approaching the model's context limit. The system maintains a configurable buffer (e.g., 200 tokens) to ensure generation space. Truncation strategy is configurable (remove oldest messages, summarize, or sliding window). The context window size is auto-detected from model metadata or can be manually specified per model.

Abstracts backend-specific implementation details (llama.cpp, ExLlama, Transformers) behind a unified Python interface in models.py. Each backend is loaded lazily (only when needed) to minimize startup time. The abstraction layer handles backend-specific initialization (e.g., ExLlama's context manager, llama.cpp's server startup) and exposes a common generate() method. Backend selection is automatic based on model format or can be explicitly specified via command-line flag.

Exposes 15+ sampling methods (temperature, top-p, top-k, min-p, DRY, mirostat, etc.) via a configuration system that allows users to create and save custom sampling presets. Presets are stored in user_data/presets.yaml and can be selected via UI dropdown or API parameter. The sampling pipeline (text_generation.py) applies samplers in a configurable order, allowing composition of multiple sampling strategies. Advanced users can implement custom samplers as Python functions and register them with the sampling registry.

Provides a Gradio-based chat UI (ui.py, ui_chat.py) that maintains conversation history as a list of {role, content} dicts, automatically formats messages according to model-specific chat templates (Alpaca, ChatML, Llama2, etc.), and renders streaming responses in real-time. The system detects the appropriate template from model metadata and applies it during generation, handling edge cases like system prompts and multi-turn conversations without manual formatting.

Integrates LoRA (Low-Rank Adaptation) fine-tuning through a dedicated training tab that manages training datasets, hyperparameters (learning rate, rank, alpha), and model checkpoints. The system loads base models, applies LoRA adapters on top, and trains using HuggingFace transformers Trainer API with support for multi-GPU training and gradient accumulation. Trained LoRA weights are saved separately and can be merged with the base model or applied dynamically during inference.

Implements a plugin architecture where extensions are Python modules loaded dynamically from the extensions/ directory, with hooks into the UI (custom tabs), generation pipeline (pre/post-processing), and API layer. Built-in extensions include an OpenAI-compatible REST API (compatible with ChatGPT client libraries) that exposes the local model as /v1/chat/completions and /v1/completions endpoints, allowing drop-in replacement of OpenAI API calls with local inference.

Provides a Jupyter-like notebook interface where users can write markdown and code cells, execute them sequentially with persistent state, and interact with the loaded model through a Python API. The notebook mode maintains a shared execution context across cells, allowing users to call the model, process outputs, and build complex workflows without leaving the web UI. Supports both synchronous and asynchronous execution with streaming output.

Implements a configuration system where model-specific settings (sampling parameters, chat template, system prompt, LoRA adapters) are stored in YAML files in models/ directory and automatically loaded when a model is selected. The system merges model-specific settings with global defaults, allowing per-model customization without UI changes. Configuration includes generation presets, quantization settings, and backend-specific optimizations that are applied transparently during model loading.

Integrates image generation capabilities through extensions that wrap Stable Diffusion models, allowing users to generate images from text prompts within the same web UI. The system manages separate image model loading, prompt processing, and output rendering alongside text generation. Supports multiple Stable Diffusion variants (SD 1.5, SDXL) with configurable sampling steps, guidance scale, and seed control for reproducible image generation.

Implements VRAM-aware model loading that automatically selects quantization formats (GGUF, GPTQ, AWQ) based on available GPU memory, supports layer offloading to CPU when VRAM is insufficient, and provides memory profiling to estimate model size before loading. The system tracks allocated VRAM across models and can unload models to free memory for new ones. Backend-specific optimizations (ExLlama's VRAM pooling, llama.cpp's memory mapping) are applied transparently based on available resources.

Implements a comprehensive argument parsing system using Python's argparse that handles 50+ command-line flags for model selection, backend configuration, UI settings, and API options. Arguments are merged with YAML-based persistent settings from user_data/settings.yaml, with command-line arguments taking precedence. The system supports environment variable overrides and generates a settings file on first run with sensible defaults, enabling both CLI-driven and UI-driven configuration workflows.