Text Generation WebUI

Implements a hub-and-spoke architecture (shared.py as central state hub) that abstracts over 5+ model backends (llama.cpp, ExLlamaV2/V3, Transformers, TensorRT-LLM, ctransformers) through a unified loader interface in modules/loaders.py. The system maintains a single shared.model and shared.tokenizer instance, with backend selection delegated to loaders.py which dynamically imports and instantiates the appropriate backend class based on model format detection and command-line arguments. Model switching is handled by unloading the current model from VRAM before loading the next, managed through models.py.

Orchestrates the text generation pipeline through text_generation.py which wraps backend-specific generate() calls with a unified streaming interface. Implements parameter presets system (stored in user_data/presets.yaml) allowing users to save/load generation configurations (temperature, top_p, top_k, repetition_penalty, etc.). The pipeline supports both synchronous and streaming output modes, with streaming implemented via Python generators that yield tokens as they're produced by the backend, enabling real-time UI updates through Gradio's streaming components.

Provides two distinct conversation modes: 'Instruct' mode treats each input as an independent instruction with no history, while 'Chat' mode maintains conversation history and formats messages according to model-specific chat templates. Chat templates (stored in model metadata) define how to format user/assistant/system messages for the specific model architecture. The system automatically applies the correct template based on the loaded model, handling variations like ChatML, Alpaca, Llama2-Chat, etc. without requiring user intervention.

Integrates llama.cpp (C++ inference engine) through the llama-cpp-python binding, enabling CPU-only inference and support for GGUF quantized models. The integration is handled through modules/llama_cpp_server.py which spawns a separate llama.cpp server process and communicates via HTTP. This allows running models on CPU-only systems or offloading to CPU when VRAM is limited. GGUF quantization provides extreme compression (1-2 bits per weight) enabling 70B models to run on 8GB RAM.

Integrates ExLlama (optimized inference engine for Llama models) through modules/exllamav2.py and modules/exllamav3.py, providing fast inference with dynamic quantization support. ExLlama uses a custom CUDA kernel implementation optimized for Llama architecture, achieving 2-3x speedup over transformers backend on the same hardware. The backend supports EXL2 quantization format which allows dynamic per-token quantization, balancing speed and quality better than static quantization.

Integrates Hugging Face transformers library as a backend, providing the most flexible model support including vision models, multimodal models, and models with custom architectures. The transformers backend loads models directly from HuggingFace Hub or local files, applies quantization through bitsandbytes library, and handles image preprocessing for vision models. This backend is the most feature-complete but also the slowest due to lack of optimization.

Implements centralized state management through shared.py which acts as a hub providing access to shared.model, shared.tokenizer, shared.args, and shared.settings. All components (UI, generation pipeline, extensions) read from and write to shared state rather than passing state explicitly through function parameters. This pattern simplifies component communication but creates tight coupling and makes testing difficult. The shared module also handles command-line argument parsing and settings loading from YAML files.

Exposes the local model through an OpenAI-compatible API endpoint (implemented as a built-in extension) that mirrors the /v1/chat/completions and /v1/completions endpoints. Supports function calling via JSON schema definitions, allowing external applications to invoke the model as a drop-in replacement for OpenAI's API. The API layer translates between OpenAI request/response formats and the internal text_generation.py pipeline, enabling existing OpenAI client libraries (Python, JavaScript, etc.) to work without modification.

Provides an integrated training interface for parameter-efficient fine-tuning via LoRA (Low-Rank Adaptation). The system loads training data from JSON/JSONL files, manages training state through the LoRA system module, and supports training on top of loaded base models without modifying original weights. Trained LoRA adapters are saved as separate files and can be merged back into the base model or loaded dynamically at inference time. Training parameters (learning rate, epochs, batch size) are configurable through the UI.

Implements a Gradio-based chat UI supporting both text and image inputs, with backend support for vision-capable models (via transformers backend) and image generation through integrated extensions. The chat interface maintains conversation history in memory, formats messages according to model-specific chat templates (stored in model metadata), and supports role-based message formatting (user/assistant/system). Image inputs are preprocessed and embedded alongside text tokens, while image generation is handled through separate extension hooks.

Provides a plugin architecture allowing developers to extend functionality through Python modules in the extensions/ directory. Extensions can hook into the Gradio UI creation process (create_interface() in server.py), register custom API endpoints, modify text generation behavior, and add new tabs/components. The extension system loads all .py files from extensions/ at startup, with each extension implementing optional callbacks (ui(), api(), etc.). Extensions have access to shared state (shared.model, shared.tokenizer, shared.settings) enabling deep integration with core functionality.

Manages model-specific settings through a two-tier system: user_data/models_settings.yaml for global model configurations and per-model YAML files in the model directory. The models_settings.py module loads these configurations and merges them with default settings, storing metadata like chat templates, context length, quantization info, and backend-specific parameters. This allows different models to have different generation defaults, chat formatting rules, and optimization settings without requiring code changes.

Provides a Jupyter-like notebook interface where users can write and execute code blocks that maintain state across executions. Unlike chat mode which treats each message independently, notebook mode preserves variables and imports between cells, allowing iterative development and experimentation. The interface supports both code input and model generation within the same notebook, enabling workflows like 'generate code → execute → analyze results → generate next code block'.

Implements memory optimization through multiple strategies: support for quantized model formats (GPTQ, AWQ, EXL2) that reduce model size by 4-8x, CPU offloading for layers that don't fit in VRAM, and dynamic model unloading when switching models. The system tracks VRAM usage through backend-specific APIs and provides command-line flags (--gpu-memory, --cpu-memory) to configure memory allocation. Quantization is handled transparently by the backend loaders — users select a quantized model and the loader automatically applies the appropriate quantization scheme.

Integrates with HuggingFace Hub to automatically download models on first use, with caching to avoid re-downloading. Users specify models by HuggingFace repo ID (e.g., 'meta-llama/Llama-2-7b-hf') and the system handles downloading model files, tokenizers, and configuration. The downloader respects HuggingFace's authentication (for gated models) and supports resuming interrupted downloads. Downloaded models are cached in a configurable directory (default: models/) and reused on subsequent loads.