Galileo Observe

Detects when LLM outputs contain factually incorrect or unsupported claims by comparing generated text against provided context/retrieval sources. Uses proprietary Luna distilled models (97% cheaper than LLM-as-judge) that run inference on trace data to classify hallucinations with >70% F1 accuracy, enabling automated flagging of unreliable outputs in RAG pipelines without expensive API calls to external LLMs.

Measures how closely LLM-generated responses adhere to and are grounded in provided retrieval context by scoring semantic alignment between output and source documents. Implemented as a Luna distilled evaluator that runs on ingested traces to produce adherence scores, enabling teams to identify when models ignore or contradict retrieved information and track adherence trends across production traffic.

Enables A/B testing and comparative evaluation of different LLM models, prompts, retrieval strategies, and configurations by running the same evaluation metrics across variants and comparing results. Traces are tagged with variant identifiers, and the platform computes comparative metrics (e.g., hallucination rate for Model A vs. Model B) to help teams identify which configuration performs best.

Routes real-time alerts from production guardrails and monitoring rules to Slack channels, email, or custom webhooks, enabling teams to be notified immediately when quality thresholds are breached. Alerts can be configured with custom thresholds, severity levels, and routing rules to ensure the right team members are notified of relevant failures.

Offers Enterprise tier deployment options beyond Galileo-hosted infrastructure, including VPC (customer-managed) and on-premises deployment for teams with data residency, compliance, or security requirements. Luna models and evaluation infrastructure can be deployed to customer infrastructure, enabling evaluation to run within customer networks without data leaving the organization.

Provides evaluation metrics grounded in research (founder background in BERT, speech recognition, and AI systems) with automatic tuning to customer datasets. Rather than using generic LLM-as-judge prompts that achieve <70% F1 accuracy, Galileo auto-tunes Luna models per customer to achieve >70% F1 accuracy on domain-specific evaluation tasks, adapting metrics to customer data distributions and quality criteria.

Evaluates the quality of documents retrieved by RAG systems through built-in metrics that assess relevance, ranking order, and retrieval completeness. Ingests trace data containing queries, retrieved documents, and ground-truth relevance labels to compute metrics (specific metrics like precision, recall, NDCG not explicitly documented) and identify retrieval failures, enabling teams to diagnose whether poor LLM outputs stem from bad retrieval or bad generation.

Ingests structured trace data from production LLM and RAG systems in real-time, capturing signals across models, prompts, functions, context/retrieval, datasets, and traces. Traces are stored and indexed to enable millions of signals to be tracked simultaneously, with the platform analyzing patterns across traces to surface failure modes, hidden patterns, and performance trends without requiring batch reprocessing.

Analyzes ingested production traces to automatically identify failure patterns, classify failure modes (e.g., 'hallucination caused incorrect tool input'), and surface hidden patterns across millions of signals. The insights engine correlates failures across prompts, models, functions, and context to prescribe root causes and remediation steps without requiring manual log analysis.

Allows teams to define custom evaluation logic beyond built-in metrics by creating custom evaluators that can be applied to traces. Custom evaluators are distilled into Luna models for production deployment, enabling teams to encode domain-specific quality criteria (e.g., 'response must cite sources') and run them at scale without external API calls. Evaluators can be versioned and deployed as production guardrails.

Deploys optimized evaluators (Luna models) as production guardrails that monitor 100% of traffic in real-time, triggering alerts when quality thresholds are breached. Guardrails can be deployed to Galileo-hosted, VPC, or on-premises infrastructure (Enterprise tier) and are configured with alert rules that notify teams via Slack, email, or webhooks when failures occur, enabling rapid response to production quality degradation.

Evaluates multi-turn agent behavior and workflow execution by analyzing sequences of LLM calls, tool invocations, and state transitions across conversation turns. Built-in evaluators assess tool selection correctness, workflow completion, and multi-turn coherence by ingesting traces that capture the full agent execution graph, enabling teams to identify where agents fail in complex reasoning tasks.

Manages evaluation datasets (synthetic, development, production-sourced) and versions evaluation metrics and custom evaluators as 'Luna models' that can be tracked, compared, and deployed. Datasets can be created from production traces, labeled with ground truth, and used to train and validate custom evaluators, enabling teams to maintain reproducible evaluation pipelines and compare evaluator performance across versions.

Integrates with Model Context Protocol (MCP) servers to evaluate external tools, functions, and data sources used by LLM applications. Traces can include MCP server interactions, and evaluators can assess whether tools are being called correctly, returning expected data, and being used appropriately by the LLM, enabling end-to-end evaluation of tool-augmented LLM systems.