Ollama

Executes large language models on consumer hardware by automatically detecting and routing inference to available accelerators (NVIDIA CUDA, AMD ROCm, Apple Metal, Vulkan) via a unified GGML backend abstraction layer. The system manages KV cache allocation, GPU memory, and multi-backend fallback chains to maximize throughput while respecting hardware constraints. Inference runs through a request scheduler that queues and batches operations across multiple runner instances.

Manages models as composable layers stored in a content-addressed blob store, enabling efficient model sharing, versioning, and customization via Modelfile syntax. Models are pulled from the Ollama registry (or custom registries) and stored locally with manifest-based deduplication; custom models are created by layering base models with system prompts, parameters, and tools. The system uses blob transfer with authentication to handle large model downloads with resume capability.

Provides an interactive command-line interface (REPL) for chatting with models, with features like multi-line input, command history, syntax highlighting, and model switching. The CLI uses the Ollama API client to send requests and streams responses in real-time. Users can switch models, adjust parameters, and view conversation history without restarting the CLI.

Provides Docker images and Compose configurations for deploying Ollama as a containerized service, with support for GPU passthrough (NVIDIA Container Runtime, AMD GPU support), volume mounting for model persistence, and environment-based configuration. Docker deployment enables reproducible, isolated Ollama instances suitable for production and cloud environments.

Exposes model inference parameters (temperature, top_p, top_k, repeat_penalty, num_predict) via API and CLI, enabling fine-grained control over model behavior without retraining. Parameters are passed per-request and override model defaults defined in Modelfiles. The system validates parameters and applies them during token generation, affecting output diversity, length, and quality.

Integrates web search capabilities into models, enabling them to query the internet and retrieve current information for answering time-sensitive questions. The system uses a search backend (e.g., Brave Search API) to fetch results and passes them to the model as context. This enables agentic workflows where models can research topics and synthesize information from multiple sources.

Provides drop-in compatibility with OpenAI and Anthropic API schemas, allowing existing client libraries (openai-python, @anthropic-sdk/sdk) to route requests to local Ollama models without code changes. The compatibility layer translates incoming API requests to Ollama's native /api/generate and /api/chat endpoints, maps response formats, and handles streaming. Authentication uses API keys stored in Ollama's key management system.

Enables models to request execution of external tools via a schema-based function registry, where tool definitions are provided as JSON schemas and model outputs are parsed to extract function calls. The system supports native tool calling for models that understand function schemas (e.g., Mistral, Hermes) and fallback prompt-based tool calling for models without native support. Tool execution is orchestrated by the client; Ollama returns structured function call requests.

Processes images alongside text by encoding images into embeddings and passing them to vision-capable models (e.g., LLaVA, Qwen-VL) via a unified chat API. Images are provided as base64-encoded data or file paths; the system handles image preprocessing (resizing, normalization) and concatenates image embeddings with text embeddings for joint reasoning. Vision models output text descriptions, answers to visual questions, or structured analysis of image content.

Generates fixed-dimension vector embeddings for text using embedding models (e.g., nomic-embed-text, mxbai-embed-large) via the /api/embed endpoint. Embeddings are computed locally without cloud API calls, enabling private semantic search, similarity matching, and RAG applications. The system batches embedding requests and returns vectors in standard format (float32 arrays) compatible with vector databases.

Manages concurrent inference requests by queuing them and distributing across multiple runner instances (one per model or GPU), with automatic load balancing and memory-aware scheduling. The scheduler prevents GPU memory overload by tracking KV cache allocation per request and unloading models when necessary. Requests are processed in FIFO order with optional priority levels; streaming responses are multiplexed to clients via HTTP chunked encoding.

Provides a templating system (Handlebars-based) for defining reusable prompt structures within Modelfiles, enabling dynamic prompt construction with variable substitution, conditionals, and formatting. Templates are applied at model creation time and can reference user input, system context, and model parameters. This enables prompt engineering without modifying application code.

Converts full-precision models (FP32, FP16) to quantized formats (GGUF with INT4, INT5, INT8 quantization) to reduce model size and memory requirements while maintaining inference quality. Quantization is performed during model import or via the Ollama conversion pipeline; quantized models are stored as GGUF blobs in the layer system. The system supports multiple quantization levels, enabling trade-offs between model size, memory usage, and accuracy.

Automatically detects available GPU hardware (NVIDIA CUDA, AMD ROCm, Apple Metal, Intel Arc, Vulkan) at startup and routes inference to the optimal backend without user configuration. The system queries GPU capabilities (VRAM, compute capability), loads the appropriate GGML backend library, and falls back to CPU inference if no GPU is detected. Backend selection is transparent to the user; the same model runs on any supported hardware.