NeMo Guardrails

Defines conversational flows using Colang, a domain-specific language that compiles to state machines for managing dialog turns, branching logic, and context transitions. The Colang 2.x runtime executes these flows as event-driven state machines, processing user messages through defined states and triggering actions based on flow conditions. This enables declarative specification of multi-turn conversations without imperative control flow.

Implements a configurable pipeline of safety and constraint enforcement layers that process requests before LLM invocation (input rails), after LLM generation (output rails), during dialog turns (dialog rails), before retrieval operations (retrieval rails), and around tool calls (tool rails). Each rail stage can apply custom validators, filters, and transformations using Python actions or LLM-based checks, enabling fine-grained control over what enters and exits the LLM.

Integrates with embedding models (OpenAI, Hugging Face, local models) and vector stores (Chroma, Pinecone, FAISS) to support semantic search and retrieval-augmented generation (RAG). Handles embedding generation, vector storage, similarity search, and result ranking. Supports both in-memory and persistent vector stores, enabling guardrails to retrieve relevant context for fact-checking, topic validation, and knowledge-based responses.

Provides built-in observability through span-based tracing that tracks request flow, LLM calls, action execution, and rail decisions. Integrates with OpenTelemetry for distributed tracing, logs detailed execution traces, and supports exporting traces to external systems (Datadog, Jaeger, etc.). Enables debugging of complex guardrail flows and performance monitoring of LLM calls.

Provides seamless integration with LangChain chains and agents, allowing guardrails to wrap LangChain components or be wrapped by them. Supports using LangChain tools within guardrails, integrating guardrails into LangChain agent loops, and sharing context between guardrails and chains. Enables building complex agentic systems with guardrails applied at multiple points in the execution flow.

Provides command-line tools for validating guardrail configurations, running tests, generating documentation, and deploying guardrails. Includes commands for checking YAML syntax, validating Colang flows, running test suites, and generating API documentation. Enables CI/CD integration and local development workflows without requiring Python code.

Uses secondary LLM calls to validate outputs and detect attacks through structured prompting. Implements jailbreak detection by analyzing user inputs against known attack patterns, and hallucination detection by having the LLM verify its own outputs against retrieved facts or user context. These checks run asynchronously or synchronously depending on configuration, using the same LLM provider or a separate safety-focused model.

Uses semantic embeddings (via configurable embedding models) to classify user messages and LLM outputs against allowed topics and content categories. Compares input/output embeddings against a knowledge base of topic examples or safety categories, using cosine similarity thresholds to determine if content is on-topic or violates safety policies. This enables semantic understanding beyond keyword matching, supporting nuanced topic boundaries and content policies.

Detects and redacts sensitive information (PII, credentials, secrets) in user inputs and LLM outputs using a hybrid approach: pattern-based detection (regex for credit cards, SSNs, API keys) combined with LLM-based recognition for context-dependent sensitive data (names, addresses). Detected sensitive data can be redacted, masked, or flagged for manual review, with configurable handling per data type.

Provides a framework for binding Python functions as actions that can be invoked from Colang flows or rails. Actions are registered in a central registry, support async/sync execution, accept typed parameters, and can access the current conversation context. The action system handles parameter marshaling, error handling, and result propagation back to flows, enabling seamless integration of custom business logic into guardrails.

Abstracts LLM provider differences (OpenAI, Anthropic, Ollama, Azure, etc.) behind a unified interface, handling provider-specific API formats, authentication, and parameter mapping. Supports streaming responses with configurable chunk handling, token counting, and caching. The provider system allows swapping LLM backends without changing guardrail logic, and supports using different providers for different tasks (main generation vs. safety checks).

Defines guardrail configurations in YAML format with a strict schema that validates structure, required fields, and parameter types at load time. The RailsConfig parser converts YAML into Python objects, performs validation, and raises clear errors for misconfigurations. Supports configuration inheritance, variable substitution, and environment-based overrides, enabling version-controlled, auditable guardrail definitions.

Exposes guardrails as an HTTP server with two interaction modes: request-response (single turn) and event-based (streaming, multi-turn). The server handles request parsing, context management, streaming response chunking, and error serialization. Supports both REST endpoints and WebSocket connections for real-time streaming, enabling integration with web frontends, mobile apps, and other HTTP clients.

Manages prompts as templates with variable substitution, filters for formatting/transformation, and automatic context injection. Prompts are defined in YAML or Python, support Jinja2-style templating, and can reference conversation history, user context, and retrieved documents. The prompt system handles prompt composition (system + user messages), token counting, and parameter passing to LLMs.