llm (Simon Willison)

Implements a dual sync/async base class architecture (Model, AsyncModel, KeyModel, AsyncKeyModel) defined in llm/models.py that abstracts away provider-specific implementation details. All models inherit from these base classes and implement a common prompt()/execute() interface, allowing identical code to work across OpenAI, Anthropic, Google, and local models without conditional logic. The plugin system auto-discovers and registers models via entry points, enabling runtime model swapping without code changes.

Automatically logs all model interactions to a SQLite database (logs.db) with full conversation state preservation. The Conversation class maintains multi-turn dialogue state, and the logging system records prompts, responses, model metadata, tokens used, and timestamps. Conversations can be resumed, queried, and exported. The database schema supports efficient retrieval of conversation history and enables analytics on model usage patterns across sessions.

Implements streaming responses using Python iterators, allowing models to return output incrementally as tokens are generated. The Response and AsyncResponse classes provide both streaming (via __iter__) and buffered (via text()) interfaces, enabling developers to choose between real-time output and complete responses. Streaming is transparent to the caller—the same code works with streaming and non-streaming models. The CLI uses streaming by default for responsive user experience.

Automatically tracks token usage (input/output tokens) and estimated costs for each model interaction. The Response class includes a usage() method that returns token counts and cost estimates based on model pricing. Usage data is logged to the SQLite database alongside conversation history, enabling analytics on cost per conversation, cost per model, and token efficiency. The system supports custom pricing definitions for models, allowing accurate cost tracking for non-standard pricing models.

Provides full async/await support through AsyncModel and AsyncKeyModel base classes, enabling non-blocking LLM interactions in async applications. All core operations (prompt execution, tool calling, embedding generation) have async equivalents that return coroutines. The system supports both sync and async models in the same application, with automatic detection of execution context. Async responses use AsyncResponse with async iterators for streaming, enabling efficient concurrent LLM calls.

Enables models to call Python functions via a Tool abstraction and Toolbox collection system. Developers decorate Python functions with @llm.tool() to register them, and the system serializes function signatures into schemas that models understand (OpenAI function calling, Anthropic tool_use, etc.). When a model requests tool execution, the framework automatically invokes the Python function, captures the result, and feeds it back to the model in a loop until completion. Tools can be organized into named Toolbox collections for reuse across conversations.

Provides a Schema system that allows developers to define expected output structure (via JSON Schema or Pydantic models) and pass it to models. The framework serializes the schema and sends it to the model provider (e.g., OpenAI's JSON mode, Anthropic's structured output). Model responses are automatically validated against the schema and parsed into structured objects. This enables reliable extraction of specific fields (e.g., name, email, sentiment) from model outputs without regex parsing or post-hoc validation.

Provides an EmbeddingModel abstraction for generating vector embeddings from text. The system supports both single embed() and batch embed_batch() operations, with embeddings stored in a separate SQLite database (embeddings.db). Embeddings can be used for semantic search, similarity comparisons, and clustering. The framework handles provider-specific embedding APIs (OpenAI, Anthropic, local models) through the same interface, and embeddings are cached to avoid redundant API calls.

Provides a template system that allows developers to define reusable prompt templates with variable placeholders. Templates are stored as files or registered via the plugin system, and variables are interpolated at runtime using Jinja2-style syntax. This enables prompt engineering best practices like prompt versioning, A/B testing, and separation of prompt logic from application code. Templates can include system prompts, examples, and tool definitions, making complex prompts composable and maintainable.

Supports attaching images, audio, files, and other media to prompts via the Prompt class. Attachments are represented as Fragment objects that encapsulate file paths, MIME types, and metadata. The system handles encoding attachments into formats that models understand (base64 for images, file references for documents). Multi-modal models (e.g., GPT-4 Vision, Claude 3 Vision) automatically receive attachments in their native format without requiring manual encoding.

Implements a plugin architecture using Python entry points for dynamic discovery and registration of models, tools, and templates. Plugins are Python packages that define entry points in their setup.py/pyproject.toml, and the llm package auto-discovers them at runtime without requiring code changes. Plugins can add new models (e.g., local Ollama models, custom fine-tuned models), tools, templates, and commands. The plugin system is extensible—developers can create plugins without modifying the core llm codebase.

Provides a command-line interface (llm command) that supports both one-shot prompts and interactive multi-turn chat. The CLI streams model responses in real-time using iterators, enabling responsive user experience without waiting for full response completion. Interactive mode maintains conversation state across turns, with readline support for command history and editing. The CLI integrates with all core features (tools, templates, schemas, embeddings) and supports piping input/output for shell integration.

Manages API keys, model aliases, and user preferences through configuration files (typically ~/.llm/config.yaml or environment variables). The system supports multiple API key sources (environment variables, config files, keychain) with a priority order. Model aliases allow users to define shortcuts (e.g., 'gpt4' -> 'gpt-4-turbo') and set default models. Configuration is loaded at startup and can be modified via CLI commands (llm keys set, llm aliases set) without manual file editing.