NeMo Guardrails

Defines conversational flows using Colang, a domain-specific language that compiles to a state machine executed by the LLMRails orchestrator. Colang 2.x uses event-driven state transitions with explicit flow lifecycle management, enabling developers to specify dialog paths, user intents, and bot responses as declarative rules rather than imperative code. The runtime processes incoming messages through the state machine, matching patterns and triggering actions based on flow definitions.

Implements a configurable pipeline of input rails, dialog rails, retrieval rails, output rails, and tool rails that intercept and filter messages at different stages of LLM processing. Each rail stage can apply regex patterns, LLM-based classifiers, or custom actions to detect and block harmful content, enforce topic boundaries, or validate tool calls before they reach the LLM or user. The pipeline architecture allows composition of multiple safety checks without modifying core LLM logic.

Provides a pluggable action system where developers can register custom Python functions as actions that can be invoked from Colang flows or rails. Actions are registered with metadata (name, description, parameters) and can be called from flow definitions or as part of rail enforcement. The action system handles parameter binding, error handling, and integration with the LLMRails orchestrator. Actions can be synchronous or asynchronous, and can access the conversation context and state.

Centralizes guardrails configuration in YAML files (config.yml, prompts.yml) that define LLM providers, rails, flows, actions, and generation parameters. The RailsConfig class parses and validates configuration, providing a programmatic interface to access settings. Configuration validation catches errors early (missing required fields, invalid types, unsupported options). The system supports configuration inheritance and composition, allowing modular configuration files.

Provides an HTTP server that exposes guardrails as a REST API, allowing applications to interact with guardrails over HTTP without embedding the framework directly. The server handles request/response serialization, streaming, and error handling. CLI tools allow testing guardrails locally, generating configuration templates, and running evaluation benchmarks. The server supports both request/response and event-based APIs for different integration patterns.

Provides observability through span-based tracing that captures the execution of flows, actions, and LLM calls. Each operation (flow step, action execution, LLM inference) is wrapped in a span with metadata (name, duration, status, parameters). Traces can be exported to external systems (e.g., Datadog, Jaeger) for monitoring and debugging. LLM caching layer caches LLM responses based on prompt hash, reducing API costs and latency for repeated queries.

Integrates LLM-based self-check actions that ask the LLM to evaluate its own outputs for factual accuracy, consistency, and safety before returning responses to users. The system uses prompt engineering and structured reasoning traces to extract the LLM's confidence and reasoning, then applies configurable thresholds to decide whether to accept, regenerate, or reject the response. This approach leverages the LLM's own reasoning capabilities rather than external fact-checking services.

Detects jailbreak attempts using a combination of LLM-based classifiers and regex pattern matching on user inputs. The system applies pre-configured prompts that ask an LLM to identify adversarial patterns, prompt injections, and role-play attempts, then combines these signals with rule-based detection to block suspicious inputs before they reach the main LLM. Detection results are cached and logged for analysis.

Detects and redacts personally identifiable information (PII), API keys, credentials, and other sensitive data in user messages and LLM responses using regex patterns, NER models, or LLM-based classification. The system can mask, hash, or remove sensitive data before it reaches the LLM or is returned to the user, preventing data leakage and compliance violations. Detection rules are configurable and composable.

Enforces topic boundaries by using embeddings-based semantic similarity to detect when user queries or LLM responses drift outside allowed topics. The system maintains a set of allowed topics (as text descriptions or example queries), embeds them using a configurable embeddings model, and compares incoming messages against these embeddings using cosine similarity. Messages below a configurable threshold are blocked or redirected. This enables semantic topic control without explicit keyword lists.

Provides a schema-based function registry that allows LLMs to call external tools and APIs through structured function calling. The system defines tool schemas in YAML or Python, validates LLM-generated function calls against these schemas, and executes approved calls through a pluggable action system. Tool rails can intercept and validate calls before execution, preventing unauthorized or malformed tool invocations. Supports native function calling APIs from OpenAI and Anthropic.

Supports streaming LLM responses while applying rails checks incrementally to streamed tokens. The system buffers streamed output and applies output rails (safety checks, topic control, fact-checking) to chunks of text as they arrive, allowing early termination if a violation is detected mid-stream. This enables low-latency streaming while maintaining safety guarantees. Streaming configuration allows tuning buffer sizes and check frequency.

Abstracts LLM provider APIs (OpenAI, Anthropic, Ollama, etc.) behind a unified interface, allowing applications to switch providers without code changes. The system manages provider configuration, API key handling, and model selection through YAML configuration. Prompt templating allows parameterizing prompts with variables and filters, enabling reuse across different models and providers. The LLM Task Manager handles prompt compilation, parameter injection, and response parsing.

Integrates embeddings-based retrieval to augment LLM context with relevant documents or knowledge base entries. The system embeds documents using configurable embeddings models, stores them in a vector database, and retrieves top-k similar documents for each user query. Retrieved documents are injected into the LLM prompt as context, enabling RAG (Retrieval-Augmented Generation) patterns. Retrieval rails can filter or re-rank retrieved documents before they reach the LLM.