magentic

Converts Python functions into LLM-powered equivalents using a @prompt decorator that intercepts function calls and routes them to language models. The decorator preserves function signatures, type hints, and docstrings while transparently replacing execution with LLM inference, enabling developers to define LLM behavior through standard Python function definitions rather than prompt templates or API calls.

Provides a unified interface to multiple LLM providers (OpenAI, Anthropic, Ollama, local models) through a pluggable backend system that abstracts provider-specific API differences. Developers specify the LLM provider once (via environment variable or explicit parameter) and the same decorated function works across all supported backends without code changes, handling differences in API formats, token counting, and response parsing internally.

Automatically parses LLM text responses into Python objects matching the function's return type annotation using a combination of prompt engineering (instructing the LLM to output structured formats like JSON) and post-processing validation. Supports dataclasses, TypedDict, Pydantic models, and primitive types, with intelligent fallback strategies when LLM output doesn't match the expected schema (retry with clarified prompt, partial parsing, or error propagation).

Enables streaming LLM responses token-by-token through Python iterators, allowing applications to display partial results in real-time without waiting for full completion. Internally manages provider-specific streaming protocols (Server-Sent Events for OpenAI, streaming for Anthropic) and yields tokens as they arrive, with optional buffering for structured output types that require complete responses for parsing.

Automatically incorporates function parameters into the LLM prompt by introspecting function arguments at call time and embedding them as context. The decorator extracts parameter names, types, and values, then constructs a prompt that includes both the function's docstring (task description) and the actual parameter values, enabling the LLM to make decisions based on dynamic input without requiring manual string formatting or f-string construction.

Provides async/await support for LLM function calls through async-decorated variants, enabling non-blocking execution in async Python applications. Internally uses asyncio to manage concurrent requests to LLM providers, allowing multiple LLM calls to execute in parallel without blocking the event loop, with proper error propagation and cancellation support through Python's asyncio.Task interface.

Allows developers to customize how prompts are constructed by parsing function docstrings and extracting task descriptions, parameter documentation, and output format instructions. The decorator interprets docstring conventions (Google-style, NumPy-style, or plain text) to build context-aware prompts that include parameter descriptions and expected output formats, with optional hooks for custom prompt builders that override default behavior.

Provides built-in error handling for LLM API failures, rate limits, and malformed responses through configurable retry strategies with exponential backoff. When an LLM call fails (network error, rate limit, invalid response), the decorator automatically retries with increasing delays, with customizable retry counts, backoff multipliers, and jitter to prevent thundering herd problems in concurrent scenarios.

Estimates the number of tokens consumed by LLM calls before execution and calculates approximate costs based on provider pricing models. Uses provider-specific tokenizers (OpenAI's tiktoken, Anthropic's token counter) to count input and output tokens, then multiplies by per-token rates to estimate costs, enabling developers to monitor and optimize LLM spending without waiting for actual API billing.

Manages LLM context windows by tracking token consumption and automatically truncating or summarizing input when approaching provider limits. Monitors cumulative tokens from function parameters, system prompts, and conversation history, then applies truncation strategies (sliding window, summarization, priority-based filtering) to keep total tokens within the model's maximum context window without exceeding limits.