Fiddler AI

Instruments autonomous AI agents and multi-step workflows to capture execution traces in real-time, recording each agent action, decision point, tool invocation, and state transition with sub-100ms latency overhead. Traces include full execution context (prompts, model outputs, tool responses, intermediate states) enabling post-hoc analysis of agent behavior and decision paths without requiring code modifications to the agent itself.

Provides a framework for evaluating LLM outputs using other LLMs as judges, supporting both built-in evaluation templates and custom evaluator functions. Implements a 'bring your own judge' pattern allowing teams to define domain-specific evaluation criteria (factuality, tone, safety, business logic compliance) and deploy them as reusable evaluators across experiments and production monitoring. Evaluators can be chained and composed for multi-dimensional assessment.

Provides a framework for defining, versioning, and managing LLM prompts as first-class artifacts. Enables teams to store prompt templates with variables, version them, and track changes over time. Supports prompt composition (combining multiple prompts) and prompt chaining (sequential prompts). Integrates with experiments to enable A/B testing of prompt variants and with monitoring to track prompt performance in production.

Generates comprehensive audit trails of AI system decisions, including execution traces, evaluation results, policy enforcement actions, and fairness analysis. Produces compliance reports documenting model behavior, fairness metrics, and decision explanations for regulatory review. Supports data retention policies and export capabilities for compliance documentation. Designed for regulated industries requiring transparent, auditable AI systems.

Provides observability capabilities across multiple deployment models: SaaS (all tiers), VPC (Enterprise only), and on-premise (Enterprise only). Enables organizations to choose deployment based on data residency, compliance, and security requirements. Instrumentation and monitoring logic remain consistent across deployment options, allowing teams to migrate between deployments without code changes. Enterprise deployments support custom integrations and infrastructure requirements.

Implements a consumption-based pricing model where customers pay per trace (Developer tier: $0.002 per trace) with free tier for real-time guardrails only. Trace definition and granularity not publicly documented, making cost estimation difficult without contacting sales. Enterprise tier offers custom pricing. Pricing model incentivizes efficient trace collection and filtering to minimize costs.

Analyzes model predictions across demographic groups and protected attributes to detect disparate impact, bias, and fairness violations. Computes fairness metrics (documented in 'Fairness Metrics Reference' but specifics not provided) across slices of data defined by protected attributes (e.g., gender, race, age) and identifies systematic differences in model behavior that may indicate discriminatory outcomes. Supports both pre-deployment analysis and continuous monitoring of fairness in production.

Monitors input feature distributions and model performance metrics over time to detect drift (changes in data distribution) and performance degradation. Uses statistical tests and comparison against baseline distributions to identify when model inputs or outputs have shifted, signaling potential model retraining needs. Supports both univariate drift detection (per-feature) and multivariate drift detection (joint distribution changes). Integrates with alerting to notify teams of detected drift.

Monitors the health and quality of Retrieval-Augmented Generation (RAG) systems by analyzing retrieval quality, chunk relevance, and answer grounding. Detects when retrieved documents are irrelevant to queries, when answers are not grounded in retrieved context, and when retrieval quality has degraded. Provides metrics on retrieval precision, recall, and relevance to help teams optimize RAG pipelines and identify when knowledge bases need updating or retrieval logic needs refinement.

Deploys real-time guardrails that intercept and validate LLM outputs or agent actions against defined policies before they reach users. Guardrails execute with <100ms latency and can enforce policies such as content filtering, PII redaction, toxicity detection, jailbreak prevention, and custom business logic constraints. Supports both Fiddler-provided guardrails and custom guardrails defined by teams. Free tier includes real-time guardrails; enterprise tier adds 'Fiddler Trust Models' for advanced policy enforcement.

Provides a framework for running controlled experiments on LLM prompts, model selections, and agent configurations. Enables teams to define experiment variants (different prompts, models, parameters), run them against test datasets, evaluate results using custom evaluators or LLM-as-a-Judge, and compare performance across variants. Integrates with Fiddler's evaluation and monitoring capabilities to provide statistical significance testing and automated winner selection.

Allows users to query ML metrics, model performance data, and observability events using natural language instead of SQL or custom query languages. Translates natural language questions (e.g., 'What is the average latency for predictions on mobile devices?') into queries against Fiddler's metrics database, returning results with visualizations. Leverages LLMs to understand intent and map natural language to metric definitions.

Monitors LLM API usage across multiple providers (OpenAI, Anthropic, and others — specific providers not documented) and tracks costs, token usage, and performance metrics. Aggregates metrics across providers to give unified visibility into LLM spending and usage patterns. Supports cost attribution by application, user, or other dimensions for chargeback and optimization.

Analyzes which input features contributed most to individual model predictions, providing local explainability (per-prediction) and global explainability (across all predictions). Uses techniques such as SHAP values, feature importance, or other attribution methods (specific methods not documented) to quantify feature contributions. Enables users to understand model decisions and debug unexpected predictions by identifying which features drove the outcome.