Fiddler AI

Deploys sub-100ms inference-time protection against hallucinations, toxicity, PII/PHI exposure, prompt injection, and jailbreak attempts using proprietary Fiddler Trust Models that are task-specific and context-aware. Operates as a synchronous policy enforcement layer that intercepts AI system outputs before they reach users, with configurable thresholds and remediation actions (block, flag, redact) per threat type.

Captures and visualizes the complete execution trace of autonomous agents and multi-agent systems, including tool calls, state transitions, decision points, and reasoning steps. Builds a directed acyclic graph (DAG) of agent actions with full context (prompts, model outputs, tool inputs/outputs, timestamps), enabling root cause analysis and debugging of agent failures without requiring code instrumentation beyond SDK integration.

Enables teams to define, version, and test prompts as first-class artifacts in the platform. Supports prompt templates with variable placeholders, version control with change tracking, and A/B testing infrastructure to compare prompt variations against evaluation metrics. Integrates with evaluation framework to automatically run tests on new prompt versions and track performance over time.

Allows teams to integrate custom or proprietary models (not just OpenAI/Anthropic LLMs) into Fiddler for monitoring and evaluation. Supports model-agnostic integration via API or SDK, enabling observability of in-house models, fine-tuned models, or models from non-standard providers. Provides the same monitoring, evaluation, and guardrails capabilities regardless of model source.

Provides a framework for defining and executing custom evaluation rules that use LLMs or deterministic logic to assess AI system outputs against user-defined criteria (correctness, relevance, safety, style). Supports both rule-based evaluation (regex, schema validation) and LLM-based judgment (prompt-based scoring), with built-in comparison of outputs across multiple models or prompts and configurable scoring rubrics.

Monitors retrieval-augmented generation (RAG) systems by tracking retrieval quality, context relevance, and answer grounding. Analyzes whether retrieved documents are relevant to queries, whether the LLM is grounding answers in retrieved context, and identifies failure modes (hallucinations despite relevant context, irrelevant retrievals). Provides metrics and dashboards for RAG pipeline health without requiring code changes to retrieval or generation logic.

Tracks traditional ML model performance metrics (accuracy, precision, recall, AUC) in production and detects data drift (input distribution shifts), model drift (prediction distribution shifts), and fairness issues (performance disparities across demographic groups). Uses statistical tests (Kolmogorov-Smirnov, chi-square) to identify drift and compares performance metrics across subgroups to flag fairness violations, with configurable thresholds and alerting.

Allows users to query model performance metrics, observability data, and evaluation results using natural language questions (e.g., 'What was the average latency for model X last week?' or 'Which demographic group had the lowest accuracy?') rather than writing SQL or using dashboard filters. Translates natural language to structured queries against Fiddler's metrics database and returns results in natural language or visualizations.

Generates explanations for model predictions by computing feature importance (which inputs most influenced the prediction) and decision explanations (why the model made a specific prediction). Uses techniques like SHAP, LIME, or attention-based explanations depending on model type, and visualizes explanations in dashboards for debugging and model understanding.

Provides role-based access control (RBAC) and SSO integration to manage who can view observability data, modify guardrails, run evaluations, and access audit logs. Supports custom roles with granular permissions (e.g., 'can view fairness metrics but not raw predictions'), audit logging of all platform actions, and compliance-ready access controls for regulated industries.

Runs evaluations and tests on batches of data (historical logs, test datasets) to assess model or agent performance at scale. Supports experiment tracking (comparing results across multiple runs with different configurations), version control for prompts and evaluators, and integration with CI/CD pipelines for automated quality gates. Stores experiment results with full lineage (data, evaluators, models, parameters) for reproducibility.

Provides a single observability platform that monitors AI systems deployed across multiple environments (SaaS, VPC, on-premise) and integrates data from different deployment targets into unified dashboards. Allows cross-deployment comparisons (e.g., 'How does model performance differ between SaaS and on-premise deployments?') and centralized alerting across all deployments.