LLM

Abstracts away provider-specific API differences (OpenAI, Anthropic, Ollama, local models) behind a single `llm prompt` command, routing requests to configured model providers and normalizing response handling. Uses a plugin-based provider registry pattern where each provider implements a standard interface for authentication, request formatting, and response parsing, enabling seamless switching between remote APIs and local model servers without changing invocation syntax.

Maintains conversation state across multiple CLI invocations using a local SQLite database, storing messages, model metadata, and conversation metadata. Each conversation is identified by a unique key, and the CLI automatically appends new user messages and retrieves prior context before sending to the LLM provider, enabling natural multi-turn interactions from the command line without manual context juggling.

Manages API keys and credentials for multiple LLM providers using secure local storage (encrypted files or OS credential stores like macOS Keychain, Windows Credential Manager). Supports both environment variables and interactive prompts for credential entry, with automatic credential rotation and expiration tracking.

Exposes the CLI functionality as a Python library with a high-level API for invoking LLMs, managing conversations, and accessing plugins. The library wraps the CLI's provider abstraction and conversation management, enabling developers to build Python applications that leverage the same multi-provider support and configuration system as the CLI.

Allows users to define named aliases for model configurations (e.g., `gpt4-vision` → `gpt-4-turbo` with specific system prompts and parameters), stored in a YAML or JSON config file. The CLI resolves aliases at invocation time, enabling users to swap model implementations globally without changing scripts, and supports per-alias configuration of temperature, max tokens, system prompts, and provider-specific parameters.

Supports Jinja2-style templating in prompts, allowing users to define variables (e.g., `{{filename}}`, `{{user_input}}`) that are substituted at invocation time from command-line arguments, environment variables, or stdin. Templates can include conditional logic and loops, enabling dynamic prompt generation without writing custom code.

Provides native integration with Ollama, a local LLM runtime, allowing users to run open-source models (Llama 2, Mistral, etc.) on their machine without cloud API calls. The CLI auto-detects Ollama instances running on localhost:11434, manages model downloads and caching, and routes requests to the appropriate local model with full streaming support.

Processes multiple prompts in sequence or parallel, reading from a file or stdin (one prompt per line or JSON array), and aggregates results into a structured output format (JSON, CSV, or plain text). Supports batching across different models and configurations, with built-in progress reporting and error handling for individual prompt failures.

Parses LLM responses and formats them into structured outputs (JSON, YAML, CSV, markdown tables) using pattern matching and optional JSON schema validation. Supports extracting specific fields from free-form responses via regex or JSON path queries, enabling downstream tools to consume LLM outputs without manual parsing.

Allows users to extend the CLI with custom LLM providers, output formatters, and commands by writing Python plugins that implement standard interfaces. Plugins are discovered from a plugins directory and registered at runtime, enabling third-party integrations without modifying core code. Supports both built-in plugins (OpenAI, Anthropic, Ollama) and user-defined plugins with full access to CLI context.

Streams LLM responses token-by-token to stdout as they arrive, rather than buffering the entire response before display. Supports both raw streaming (tokens printed as-is) and formatted streaming (with progress indicators, timing information, and token counts), enabling real-time feedback for long-running inferences.

Calculates token counts for prompts and responses using provider-specific tokenizers (e.g., tiktoken for OpenAI, claude-tokenizer for Anthropic), and estimates API costs based on current pricing. Supports both pre-execution estimation (for prompt planning) and post-execution reporting (for cost tracking and auditing).