Galileo

Ingests execution traces from external LLM applications (models, prompts, functions, context, datasets) and reconstructs multi-turn agent workflows to surface failure modes, tool selection success rates, and cost breakdowns per interaction. Uses a proprietary trace schema to correlate model outputs with downstream function calls and context usage, enabling post-hoc debugging without code instrumentation.

Provides 20+ out-of-the-box evaluators optimized for RAG, agents, safety, and security use cases. Each metric is implemented as a distilled Luna model (proprietary LLM-as-judge variant) that runs at 97% lower cost than full GPT-4o evaluation while maintaining comparable accuracy. Metrics are applied to evaluation datasets in batch mode and scored against ground truth or reference outputs.

Integrates with Model Context Protocol (MCP) servers to ingest context and tool definitions from external systems. Enables Galileo to evaluate LLM applications that use MCP-compatible tools and context sources, allowing evaluation of agent behavior with real-world tool integrations.

Integrates with NVIDIA NeMo Guardrails via 'Galileo Protect' to enforce guardrails in production. Galileo evaluations (hallucination detection, safety checks) feed into NeMo Guardrails to block or flag unsafe outputs. Enables production deployment of evaluation-driven safety policies without custom guardrail logic.

Tracks evaluation metrics over time and automatically detects regressions (quality drops) in model outputs. Compares current metric values against historical baselines and alerts when metrics fall below configured thresholds. Supports trend visualization and statistical significance testing to distinguish real regressions from noise.

Allows users to define custom evaluation metrics via a framework (implementation details unknown) and automatically tunes metric thresholds based on live production feedback. The platform ingests production traces, correlates metric scores with actual user outcomes or business KPIs, and adjusts metric parameters to improve precision/recall without manual retraining.

Detects when LLM outputs contain factually incorrect or unsupported claims using Luna-based evaluators that analyze output against provided context or ground truth. Integrates with NVIDIA NeMo Guardrails via 'Galileo Protect' to enforce guardrails in production, blocking or flagging hallucinated outputs before they reach users.

Enables creation and management of evaluation datasets from multiple sources: synthetic data (generated by LLMs), development data (from internal testing), and production data (from live traces). Datasets are versioned and can be used to create ground truth for custom evaluators or to benchmark model versions. Synthetic data generation approach is undocumented but implied to use LLM-based generation.

Enables custom metrics to be integrated into CI/CD pipelines as automated evaluation gates that block deployments if metric thresholds are not met. Evaluation results are reported back to CI/CD systems (webhook or API integration assumed but undocumented) to gate code promotion. Supports offline evaluation of model changes before production deployment.

Analyzes ingested execution traces to identify recurring failure patterns, surface hidden failure modes, and prescribe fixes. Uses an 'insights engine' (implementation unknown) to correlate failures with input characteristics, model outputs, tool selections, and context to identify root causes. Provides actionable recommendations for prompt tuning, tool selection logic, or data augmentation.

Tracks LLM API costs at the granularity of individual trace steps (model calls, tool invocations, context retrievals) and aggregates costs per conversation turn, session, or user. Provides cost breakdowns and identifies high-cost interactions for optimization. Integrates with Luna model cost savings (97% reduction claimed) to show cost impact of using distilled evaluators vs full LLM-as-judge.

Deploys distilled Luna models as production guardrails that run evaluations in real-time on LLM outputs before they reach users. Luna models are optimized for low-latency inference (specific latency SLA unknown) and run at 97% lower cost than LLM-as-judge evaluators. Supports multiple deployment options: Galileo-hosted, customer VPC, or on-premises (Enterprise tier only).

Supports multiple LLM providers as evaluation judges (GPT-4o explicitly mentioned; others unknown) and allows users to select which judge to use for each evaluation. Evaluation results can be compared across different judges to assess judge agreement and identify ambiguous cases. Integrates with Luna models as a cost-optimized alternative to full LLM-as-judge evaluation.