ollama

Executes large language models locally on consumer hardware by automatically detecting and routing inference through optimized backends (CUDA for NVIDIA, ROCm for AMD, Metal for Apple Silicon, Vulkan for cross-platform GPU support). Uses GGML backend with ML context management and KV cache system to minimize memory footprint while maintaining inference speed. The LlamaServer runner implementation handles request scheduling and memory allocation across detected hardware, enabling models to run without cloud dependencies.

Manages models as composable layers stored in a content-addressed blob store, enabling efficient model distribution and customization through Modelfile syntax. Models are pulled from the Ollama library registry, decomposed into quantized weights, adapters, and system prompts as separate blobs, then reassembled on-device. The manifest system tracks layer dependencies and enables incremental updates — only changed layers are re-downloaded. Custom models can be created by layering base models with LoRA adapters, custom prompts, and parameters via Modelfile declarations.

Streams inference results token-by-token to clients via HTTP streaming (chunked transfer encoding), allowing real-time display of model output without waiting for full completion. Each token is sent as a separate JSON object in the response stream, with metadata (timestamp, token ID, logits if requested). The streaming implementation uses Go's http.Flusher to send tokens immediately after generation, not buffering. Clients receive tokens as they're generated, enabling responsive UIs and early stopping based on partial results.

Provides a command-line interface (CLI) for model management (pull, push, list, delete) and an interactive REPL for conversational inference. The interactive mode supports multi-line input, command history, and model switching without restarting. The REPL implements a stateful conversation context, maintaining chat history across turns and managing token limits. The CLI also exposes server control (start, stop, logs) and debugging tools (show model details, inspect layers).

Supports models with extended reasoning capabilities (e.g., OpenAI o1-style thinking models) that generate internal reasoning tokens before producing final output. The inference pipeline handles thinking tokens separately from output tokens, allowing models to 'think' through problems before responding. Thinking tokens are typically hidden from users but can be exposed for debugging. The KV cache system manages thinking token overhead, which can be 10-100x larger than output tokens for complex reasoning tasks.

Provides Docker images for containerized Ollama deployment, with built-in GPU support (NVIDIA CUDA, AMD ROCm) and multi-platform builds (Linux x86_64, ARM64). Docker images include the Ollama server, CLI, and all dependencies, enabling one-command deployment. GPU support is handled via docker run --gpus flag, automatically mounting GPU devices into the container. The Docker setup supports volume mounts for model persistence across container restarts.

Provides drop-in compatibility with OpenAI and Anthropic API schemas, allowing existing client libraries and applications to redirect requests to local Ollama inference without code changes. The compatibility layer translates incoming OpenAI-format requests (e.g., /v1/chat/completions) to Ollama's native /api/chat endpoint, maps request parameters (temperature, max_tokens, stop sequences), and reformats responses to match expected OpenAI/Anthropic schemas. Streaming responses are converted to server-sent events (SSE) format matching OpenAI's stream protocol.

Enables models to declare and invoke external tools through a schema-based function registry. Models receive tool definitions as JSON schemas in their context, generate structured tool calls (name + arguments) in response, and Ollama routes those calls to registered handlers. The template system embeds tool schemas into the prompt, and the runner validates generated tool calls against declared schemas before execution. Supports both synchronous tool execution (blocking until result) and asynchronous patterns where tool results are fed back into the model for further reasoning.

Supports vision-language models that accept both text and image inputs, processing images through the model's vision encoder before feeding to the language decoder. Images are embedded as base64 or file paths in requests, automatically converted to the model's expected format (e.g., image tokens for LLaVA), and processed alongside text in a single inference pass. The template system handles image encoding and prompt formatting for different vision architectures (LLaVA, Qwen-VL, etc.), abstracting away model-specific image handling.

Generates dense vector embeddings for text inputs using embedding-specific models (e.g., nomic-embed-text, mxbai-embed-large), producing fixed-dimensional vectors suitable for semantic search, clustering, or similarity comparison. The /api/embed endpoint accepts text strings and returns normalized embedding vectors. Embeddings can be stored in external vector databases (Pinecone, Weaviate, Milvus) or used directly for in-memory similarity search. The embedding models are optimized for low latency and small VRAM footprint compared to generative models.

Manages concurrent inference requests through a request scheduler that queues incoming requests and routes them to available runner instances. The scheduler implements fairness policies (FIFO, priority-based) and manages GPU memory allocation across concurrent requests. When multiple requests arrive, the scheduler decides whether to batch them together (if models support batching) or queue them sequentially. The KV cache system is shared across requests when possible, reducing memory overhead. The runner implementation (LlamaServer) handles context switching and memory cleanup between requests.

Loads quantized models (GGUF format with INT4, INT8, FP16 quantization levels) and executes inference without dequantizing to full precision, maintaining quantization throughout the inference pipeline. The GGML backend handles quantized matrix multiplications natively, reducing memory footprint and improving inference speed. Models are stored in quantized format on disk, and the loader automatically selects the appropriate quantization kernel based on hardware capabilities. Quantization is transparent to users — the same API works for quantized and full-precision models.

Provides a declarative template system that abstracts model-specific prompt formatting, system prompts, and parameter handling. Templates define how user messages, system prompts, and tool schemas are formatted into the exact token sequence each model expects. Different models have different prompt formats (Llama uses [INST], Mistral uses [TOOL_CALLS], etc.), and the template system handles these differences transparently. Templates are defined in Modelfiles and applied automatically during inference, eliminating manual prompt engineering per-model.

Imports models from external sources (Hugging Face, local GGUF files, SafeTensors, PyTorch checkpoints) and converts them to Ollama's internal format (GGUF with manifest). The import pipeline handles format detection, quantization (if needed), and layer decomposition into the blob store. Users can import models via CLI (ollama import) or by providing a Modelfile with a FROM statement pointing to an external model source. The conversion process is transparent — users don't need to manually run quantization tools.