DSPy

DSPy enables users to define LM tasks through Python type-annotated signatures (input/output fields with descriptions) rather than hand-crafted prompt strings. The framework parses these signatures at runtime to generate task-specific prompts dynamically, supporting field-level documentation, type constraints, and optional few-shot examples. This decouples task logic from prompt implementation, allowing the same signature to work across different LM providers and optimization strategies without code changes.

DSPy's optimizer system (teleprompters) automatically tunes prompts and few-shot examples by running a program against a training dataset, measuring performance with a user-defined metric function, and iteratively refining prompts to maximize that metric. Optimizers include few-shot example selection (BootstrapFewShot), instruction optimization (MIPROv2), and reflective strategies (GEPA, SIMBA). The compilation process generates optimized prompts that are then frozen for inference, replacing manual trial-and-error prompt engineering.

DSPy integrates with vector databases and retrieval systems to enable retrieval-augmented generation (RAG) patterns. The framework provides dspy.Retrieve module that queries a vector store (Weaviate, Pinecone, FAISS, etc.) to fetch relevant context, which is then passed to LM modules. DSPy also includes caching mechanisms to avoid redundant LM calls and vector store queries, reducing latency and API costs. The retrieval and caching layers are transparent to the program logic, allowing RAG to be added or modified without changing module code.

DSPy programs can be serialized to JSON or Python code, enabling deployment to production environments without requiring the DSPy framework at runtime. The serialization captures optimized prompts, few-shot examples, and module structure, which can then be executed using lightweight inference code. This allows teams to optimize programs in a development environment (with full DSPy tooling) and deploy optimized artifacts to production (with minimal dependencies). Serialization also enables version control and reproducibility of optimized programs.

DSPy supports parallel and asynchronous execution of modules to improve throughput and reduce latency. Programs can use Python's asyncio to run multiple LM calls concurrently, and the framework provides utilities for batch processing and parallel module execution. This enables efficient processing of large datasets and concurrent requests without blocking. Async execution is particularly useful for I/O-bound operations like API calls, where multiple requests can be in-flight simultaneously.

DSPy provides a built-in evaluation framework that runs programs on test datasets and computes user-defined metrics. The framework supports standard metrics (exact match, F1, BLEU, ROUGE) and custom metric functions that can evaluate semantic correctness, task-specific properties, or business metrics. Evaluation results are aggregated and reported with detailed breakdowns, enabling teams to assess program quality and compare different optimization strategies. The evaluation framework integrates with optimizers to guide prompt tuning based on metrics.

DSPy provides built-in support for multi-turn conversations through history management modules that track dialogue context across turns. The framework automatically manages conversation state, including previous messages, user inputs, and LM responses. Modules can access conversation history to provide context-aware responses, and the history is automatically threaded through the program. This enables building chatbots and dialogue systems without manual context management, and supports optimization of dialogue strategies through the standard optimizer framework.

DSPy integrates with vector databases (Weaviate, Pinecone, Chroma) to enable semantic retrieval of documents or examples. The framework can automatically embed inputs, query the vector database, and inject retrieved results into LM prompts. This enables building retrieval-augmented generation (RAG) systems where the LM has access to relevant context.

DSPy supports the Model Context Protocol (MCP), enabling dynamic discovery and invocation of tools from MCP servers. This allows LM programs to access tools defined in external MCP servers without hardcoding tool definitions. The framework handles MCP communication, schema discovery, and tool invocation transparently.

DSPy provides comprehensive execution tracing that captures all LM calls, module invocations, and intermediate results. The framework generates execution traces that can be inspected for debugging, logged for monitoring, or exported for analysis. Traces include timing information, LM settings, and output values, enabling detailed program analysis.

DSPy programs are built by composing reusable modules (Predict, ChainOfThought, ReAct, etc.) that automatically thread context and outputs through the computation graph. Each module inherits from dspy.Module and implements a forward() method that calls other modules or LM predictions. The framework handles prompt generation, LM invocation, and output parsing transparently, allowing developers to write imperative Python code that reads like standard control flow while maintaining declarative task definitions underneath.

DSPy abstracts over multiple LM providers (OpenAI, Anthropic, Ollama, HuggingFace, Azure, etc.) through a unified dspy.ChainOfThought or dspy.Predict interface that works identically regardless of backend. The framework uses LiteLLM under the hood to normalize API differences, handle retries, and manage rate limiting. Users configure a provider once via dspy.settings.configure() and all modules automatically use that provider without code changes, enabling easy model switching and A/B testing across providers.

DSPy includes a dspy.Assertion system that validates LM outputs against user-defined predicates during program execution. Assertions can check output format, value ranges, semantic properties, or custom logic. When an assertion fails, DSPy can automatically trigger recovery strategies: backtracking to retry with different prompts, calling alternative modules, or raising an exception. This enables robust error handling in LM pipelines without manual try-catch boilerplate, and integrates with optimizers to learn prompts that satisfy assertions.

DSPy's BootstrapFewShot optimizer automatically selects or synthesizes few-shot examples from a training dataset to improve LM performance on a task. The optimizer runs the program on training examples, identifies failures (using the metric function), and selects diverse, representative examples that demonstrate correct behavior. These examples are then added to the prompt as in-context demonstrations. Advanced optimizers like MIPROv2 jointly optimize example selection with instruction tuning, while GEPA uses reflective reasoning to generate synthetic examples that target specific failure modes.

MIPROv2 (Multi-prompt Instruction Parameter Refinement Optimizer v2) jointly optimizes both the task instructions and few-shot examples by treating them as learnable parameters. The optimizer uses a combination of gradient-free search (Bayesian optimization, genetic algorithms) and LM-based proposal generation to explore the instruction space. It generates candidate instructions, evaluates them on the training set, and iteratively refines the best instructions. This approach discovers more effective task descriptions than hand-written prompts, often improving performance by 5-20% on complex tasks.

GEPA (Guided Example Proposal Agent) uses reflective reasoning to improve LM performance by having the model analyze its own failures and propose corrective examples. The optimizer runs the program, identifies failures, and prompts the LM to generate synthetic examples that address those failures. These synthetic examples are then added to the prompt, creating a feedback loop where the model learns from its mistakes. This approach is particularly effective for tasks where the model can articulate why it failed and generate corrective demonstrations.

SIMBA (Stochastic Iterative Model-Based Adaptation) optimizes prompts using stochastic search and model-based adaptation, treating prompt optimization as a black-box optimization problem. The optimizer samples candidate prompts, evaluates them on the training set, and uses the results to guide future samples. Unlike MIPROv2's deterministic search, SIMBA uses randomization to explore the prompt space more broadly, making it effective for tasks where the optimal prompt is far from the initial guess. SIMBA can also adapt to different model families (e.g., switching from GPT-4 to Claude).

DSPy's adapter system enables LM modules to call external tools and functions through a unified interface. Adapters normalize function calling across different LM providers (OpenAI's function calling, Anthropic's tool_use, etc.) and map LM outputs to function calls. The framework supports defining tools as Python functions with type annotations, automatically generating tool schemas, and handling tool execution and result parsing. This enables LM agents to interact with APIs, databases, and custom code without manual prompt engineering for tool invocation.