Llamafile

Packages LLMs as self-contained executable files by combining llama.cpp inference engine with Cosmopolitan Libc, embedding model weights directly into the binary. Uses a polyglot shell script + binary structure that detects the host OS/architecture (AMD64, ARM64) at runtime and executes the appropriate compiled binary, eliminating the need for installation, dependency management, or external model downloads.

Leverages the GGML tensor library for efficient matrix operations underlying LLM inference, supporting multiple quantization formats (Q4, Q5, Q8, etc.) that reduce model size and memory footprint while maintaining inference quality. The system uses GGML's memory allocator (ggml-alloc.c) to manage KV cache and intermediate tensors, with support for both CPU and GPU acceleration paths that are selected at runtime based on hardware availability.

Supports conversion of models from various formats (PyTorch, Hugging Face, ONNX) into GGUF (GGML Universal Format), a standardized quantized format optimized for inference. The quantization process reduces model size by 4-8x (Q4 vs FP32) while maintaining inference quality. GGUF is a self-describing format that embeds model metadata (architecture, tokenizer, quantization info) in the file, enabling automatic model detection and configuration without external metadata files.

Manages the Key-Value (KV) cache that stores attention keys and values for all previous tokens, enabling efficient incremental inference without recomputing attention for past context. The system allocates KV cache based on configured context size (--ctx-size), reuses cache across multiple inference steps within a single request, and supports context sliding (dropping oldest tokens when context exceeds max length) to maintain bounded memory usage. KV cache is allocated in GPU memory when GPU acceleration is enabled, minimizing CPU-GPU transfers.

Uses Cosmopolitan Libc, a portable C standard library, to compile a single binary that runs natively on Windows, macOS, and Linux without modification. The binary is structured as a polyglot file (shell script + binary) that detects the host OS and architecture at runtime and executes the appropriate compiled code path. This eliminates the need for separate builds, installers, or platform-specific distributions while maintaining native performance.

Implements a complete text generation pipeline via llama_tokenize() for input encoding, llama_decode() for forward passes through the model, and llama_sampling_sample() for probabilistic token selection. Supports multiple sampling strategies (temperature, top-k, top-p, min-p, typical sampling) that control output diversity and coherence, with configurable stopping conditions (max tokens, EOS token, custom stop sequences) that terminate generation when criteria are met.

Extends text-only inference to support multimodal models like LLaVA by using a CLIP image encoder to convert images into embeddings, then projecting those embeddings into the LLM's token embedding space via a learned multimodal projector (stored as separate .gguf weights). Image embeddings are interleaved with text tokens in the input sequence, allowing the model to jointly process visual and textual information for tasks like visual question answering and image captioning.

Exposes the inference engine via a built-in HTTP server (llama.cpp/server/server.cpp) that implements OpenAI-compatible endpoints (/v1/chat/completions, /v1/completions, /v1/embeddings) for drop-in compatibility with existing LLM client libraries and applications. The server manages concurrent requests via a slot-based system that queues inference tasks, handles streaming responses via Server-Sent Events (SSE), and provides metrics/monitoring endpoints for observability.

Manages multiple simultaneous inference requests via a slot-based scheduling system where each slot represents a parallel inference context with its own KV cache. Requests are queued and assigned to available slots; when a slot completes, the next queued request is scheduled. This design allows configurable parallelism (default 1 slot = sequential, but can be increased for multi-GPU or high-memory systems) without requiring separate model instances.

Provides hardware-accelerated inference via CUDA (NVIDIA GPUs) and ROCm (AMD GPUs) by offloading tensor operations to GPU memory and compute. The system detects available GPU at runtime and automatically routes matrix multiplications and attention operations to GPU via GGML's GPU backend, while keeping model weights and KV cache in GPU memory for minimal CPU-GPU transfer overhead. Fallback to CPU inference occurs if GPU is unavailable or incompatible.

Provides a comprehensive CLI (llama.cpp/main/main.cpp) for running inference with extensive parameter control: model selection (--model), context size (--ctx-size), sampling parameters (--temp, --top-k, --top-p), hardware acceleration (--gpu-layers), multimodal inputs (--image, --mmproj), and output formatting (--json, --stream). Supports both interactive mode (REPL-style chat) and batch processing (reading prompts from files or stdin), making it suitable for scripting and automation.

Includes a built-in web UI (served by the HTTP server) that provides a browser-based chat interface for interacting with the model. The UI allows real-time parameter adjustment (temperature, top-k, top-p, context size) without restarting the server, displays token counts and generation metrics, and supports streaming responses with visual feedback. Model selection dropdown allows switching between multiple llamafiles or models without server restart.

Integrates OpenAI's Whisper speech-to-text model (available in GGUF format) to transcribe audio files into text, which can then be fed into the LLM for processing. The system handles audio decoding, Whisper inference, and text generation in a unified pipeline, supporting common audio formats (WAV, MP3, FLAC, OGG). Whisper model is quantized and runs locally without cloud API calls.