llama-cpp-python

Loads and executes quantized language models (GGUF format) directly on CPU using llama.cpp's optimized C++ backend, with Python bindings that expose low-level inference parameters. Supports multiple quantization formats (Q4, Q5, Q8) and CPU-specific optimizations like BLAS acceleration, enabling inference on consumer hardware without GPU requirements. The binding layer marshals tensor operations between Python and the native C++ runtime, handling memory management and model state across the FFI boundary.

Generates text tokens incrementally with callback functions invoked per-token, enabling real-time streaming output to clients without buffering the entire response. The implementation uses a generator pattern where the C++ backend yields tokens one at a time, and Python callbacks (user-provided functions) process each token immediately for display, logging, or downstream processing. This pattern decouples token generation from output handling, allowing flexible integration with web frameworks, CLI tools, or message queues.

Provides direct Python bindings to llama.cpp's C++ API through ctypes/CFFI, exposing low-level inference functions while maintaining memory safety through reference counting and automatic cleanup. The binding layer handles marshaling between Python objects and C++ data structures, managing tensor allocation/deallocation, and ensuring proper cleanup of model state. This approach provides zero-overhead access to the C++ backend while preventing memory leaks or dangling pointers.

Exposes fine-grained control over text generation sampling via parameters like temperature, top-k, top-p (nucleus sampling), and repetition penalty, allowing users to tune the randomness and diversity of generated text. The implementation maps Python parameters directly to llama.cpp's sampling pipeline, which applies these filters sequentially to the logit distribution before token selection. Supports multiple sampling strategies (greedy, temperature-based, top-k, top-p) and their combinations, enabling experimentation with different generation behaviors without modifying model weights.

Supports hardware acceleration through multiple backends (CUDA, Metal, OpenCL, BLAS) selected at load time, allowing the same Python code to run on different hardware without modification. The binding layer detects available accelerators and routes tensor operations to the appropriate backend (e.g., CUDA kernels on NVIDIA GPUs, Metal on Apple Silicon, OpenBLAS on CPU). Backend selection is configured via environment variables or constructor parameters, enabling deployment flexibility across heterogeneous infrastructure.

Manages the model's context window (maximum sequence length) with support for sliding window attention, which limits the attention computation to recent tokens rather than the full history. This reduces memory usage and computation time for long sequences by only attending to the last N tokens. The implementation exposes context size configuration at model load time and supports KV cache management, allowing users to trade off context length against memory consumption and inference speed.

Generates fixed-size embedding vectors from text using the model's internal representations, enabling semantic search and similarity comparisons without generating text. The implementation extracts the model's final hidden state or pooled representation and returns it as a float vector, which can be indexed in vector databases or used for similarity calculations. This capability reuses the same quantized model for both generation and embedding tasks, avoiding the need for separate embedding models.

Processes multiple prompts sequentially with fine-grained control over token generation per prompt, including the ability to set different sampling parameters, context windows, or stopping conditions for each batch item. The implementation maintains separate inference state for each prompt and allows users to configure per-prompt generation parameters, enabling heterogeneous batch processing without code duplication. Batch processing is sequential (not parallel) but allows efficient reuse of model state across prompts.

Exposes token-level probabilities and raw logits from the model's output distribution, enabling inspection of model confidence and alternative token predictions. The implementation returns the full probability distribution over the vocabulary for each generated token, allowing users to analyze model uncertainty, debug generation behavior, or implement custom decoding strategies. This capability is useful for understanding model behavior and implementing advanced sampling techniques.

Constrains text generation to follow user-defined EBNF grammar rules, ensuring outputs conform to specific formats (JSON, SQL, code, etc.) without post-processing. The implementation integrates llama.cpp's grammar engine which filters the token selection at each step to only allow tokens that could lead to valid grammar completions. This approach guarantees syntactic correctness while maintaining semantic quality from the model, enabling reliable structured output generation.

Supports multiple GGUF quantization formats (Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, Q8_0, etc.) with automatic format detection from model files, enabling users to load different quantization levels without code changes. The implementation reads GGUF metadata to determine quantization parameters and configures the inference engine accordingly. This flexibility allows users to experiment with different quality/speed trade-offs by simply swapping model files.