guidance

Generates text from language models while enforcing constraints defined as an Abstract Syntax Tree (AST) of GrammarNode subclasses (LiteralNode, RegexNode, SelectNode, JsonNode). Uses TokenParser and ByteParser engines that work at the text level rather than token level, implementing token healing to correctly process text boundaries. The execution engine accumulates generated text into stateful lm objects that maintain both output and captured variables across generation steps.

Provides a unified interface for executing guidance programs across heterogeneous language model backends including local models (llama-cpp, Hugging Face Transformers) and remote APIs (OpenAI, Anthropic, Azure OpenAI, Google VertexAI). Each backend implements a common model interface that handles tokenization, state management, and generation, allowing the same guidance program to run on different models without code changes. The abstraction layer handles backend-specific details like API authentication, context window management, and token counting.

Supports both stateful and stateless execution modes, with optional caching of generation results. Stateless mode allows guidance programs to be executed without maintaining state between calls, reducing memory overhead. Caching can be enabled to store results of expensive generations (e.g., long prompts with complex constraints) and reuse them for identical inputs. The caching layer integrates with the model backend to avoid redundant API calls or model inference.

Allows guidance programs to interleave Python control flow (if/else, for loops, function calls) with constrained text generation using the @guidance decorator. The decorator transforms Python functions into guidance programs that can mix imperative logic with declarative grammar constraints. This enables complex workflows where generation decisions depend on previous outputs, external data, or application logic.

Integrates with the llguidance library to enforce grammar constraints at the token level during model inference. The grammar AST is compiled into a state machine that tracks which tokens are valid at each generation step, preventing the model from generating invalid tokens. This is implemented through a custom sampling function that filters the model's token logits based on the current grammar state, ensuring only valid tokens are sampled.

Supports RuleNode grammar constraints that define reusable patterns and recursive grammar rules. Rules can be defined once and referenced multiple times, reducing grammar duplication and improving maintainability. Recursive rules enable generation of nested structures (e.g., nested JSON, nested lists) without explicitly defining the nesting depth. Rules are compiled into the grammar AST and can be parameterized with arguments.

Maintains execution state through immutable lm objects that accumulate generated text, captured variables, and model state across multiple generation steps. Variables are captured using named capture groups in regex patterns or JSON schema fields, and can be referenced in subsequent generation steps. The stateful model object preserves the full generation history, enabling introspection, debugging, and chaining of multiple constrained generations in sequence.

Generates valid JSON output that conforms to a provided JSON schema by using JsonNode grammar constraints. The schema is converted into a grammar that enforces field types, required fields, nested objects, and arrays at generation time. The generated JSON is automatically parsed and made available as Python objects in the captured variables, eliminating the need for post-generation validation or repair.

Constrains generation to match regular expressions using RegexNode grammar nodes, enabling precise control over text format and structure. Named capture groups in regex patterns are automatically extracted into variables for downstream use. The regex constraints are compiled into the grammar AST and enforced at the token level during generation, preventing the model from generating text that violates the pattern.

Implements SelectNode grammar constraints that force the model to choose from a predefined set of alternatives, with each alternative being a grammar subtree. The selection is enforced at generation time, preventing the model from generating text outside the allowed options. Supports both simple string selections and complex nested grammar selections, with captured variables from the selected branch available in the output state.

Provides built-in support for multi-turn conversations with role-based message formatting (system, user, assistant). Chat templates are automatically applied based on the model's tokenizer, handling model-specific formatting requirements (e.g., ChatML, Llama2 chat format). The framework maintains conversation history as part of the lm state, enabling stateful multi-turn interactions where each turn can use grammar constraints.

Enables language models to call external functions or tools by generating function calls that conform to a schema-based registry. Functions are registered with their signatures, and the model generates structured function calls (typically as JSON) that are automatically parsed and dispatched to the registered functions. The framework handles schema validation, parameter extraction, and return value integration back into the generation context.

Supports repetition patterns (one_or_more, zero_or_more, repeat) using RepeatNode grammar constraints that allow the model to generate multiple instances of a pattern. Useful for generating lists, arrays, or repeated structures where each repetition must conform to the same grammar. The framework handles variable capture across repetitions, accumulating results into lists that are accessible in the lm state.

Provides Jupyter notebook widgets that visualize guidance program execution, showing the generated text, captured variables, and grammar constraints in real-time. The visualization includes token-by-token generation progress, constraint violations (if any), and interactive exploration of the grammar AST. Enables developers to debug guidance programs directly in notebooks without external tools.