Athina AI

Executes 50+ pre-built evaluation metrics (Ragas-based and custom) against LLM outputs without requiring metric implementation. Metrics include RagasAnswerCorrectness, RagasContextPrecision, RagasContextRelevancy, RagasContextRecall, RagasFaithfulness, ResponseFaithfulness, Groundedness, ContextSufficiency, DoesResponseAnswerQuery, ContextContainsEnoughInformation, and Faithfulness. Integrates with external LLM providers (OpenAI confirmed) to compute metric scores in parallel batches with configurable concurrency (max_parallel_evals parameter).

Allows teams to define custom evaluation metrics beyond the 50+ presets by implementing metric logic that integrates with the EvalRunner orchestration system. Custom metrics are stored in Athina's platform and versioned alongside datasets and prompts. Implementation approach unknown but likely supports Python function definitions or declarative metric schemas that hook into the parallel evaluation pipeline.

Integrates with external LLM providers (OpenAI confirmed, others unknown) to execute evaluations and run AI workflows. Manages API keys securely via AthinaApiKey.set_key() and OpenAiApiKey.set_key() methods. Abstracts provider-specific API differences, allowing teams to swap models without changing evaluation code. Handles API rate limiting, retries, and error handling transparently.

Provides loaders (athina.loaders.Loader) to import evaluation datasets from various sources (CSV, JSON, API, pre-built datasets like yc_query_mini) and transform them into Athina's internal format. Loaders handle schema mapping, data validation, and format conversion. Pre-built datasets are available for quick prototyping. Supports programmatic dataset construction via Python tuples or objects.

Maintains a complete history of evaluation runs, including metadata (timestamp, user, configuration), input datasets, metrics, and results. Each run is linked to specific prompt versions, model selections, and retriever configurations, creating an audit trail. Teams can retrieve past runs, compare results, and reproduce evaluations. Likely uses a database to store run metadata and results with queryable indexes.

Aggregates metric scores across evaluation samples and computes statistical summaries (mean, standard deviation, percentiles, min/max). Supports filtering and grouping by dimensions (e.g., by sample type, query length, retriever). Likely uses NumPy or similar for efficient computation. Enables teams to understand metric distributions and identify outliers.

Manages evaluation datasets with versioning, annotation, and SQL-based querying capabilities. Datasets are stored in Athina's platform with version history, enabling teams to track changes and regenerate datasets by modifying model, prompt, or retriever configurations. Includes pre-built datasets (e.g., yc_query_mini) and loaders for importing external data. Supports side-by-side dataset comparison with SQL query interface for data scientists.

Orchestrates batch evaluation runs across multiple metrics and dataset samples using parallel execution with configurable concurrency (max_parallel_evals parameter). EvalRunner.run_suite() method accepts a list of evaluation metrics, a dataset, and concurrency settings, then distributes evaluation work across worker threads/processes. Results are aggregated and returned as structured evaluation reports. Handles API rate limiting and error handling for external LLM provider calls.

Monitors LLM-powered applications in production to detect quality degradation, hallucinations, and context relevance issues in real-time. Integrates with running applications to capture LLM inputs/outputs and compute evaluation metrics continuously. Detects anomalies such as response quality drops, increased hallucination rates, or context mismatches. Implementation details unknown but likely uses streaming evaluation and statistical anomaly detection.

Manages and versions prompts across multiple LLM providers (OpenAI confirmed, others unknown) with side-by-side comparison and evaluation capabilities. Teams can test the same prompt against different models (e.g., GPT-4 vs GPT-3.5) and compare results. Prompts are versioned in Athina's platform and linked to evaluation runs, enabling teams to track which prompt version produced which results. Supports prompt templates with variable substitution.

Allows teams to configure and evaluate different retrieval strategies (e.g., different vector databases, chunking strategies, embedding models) and measure their impact on RAG pipeline quality. Datasets can be regenerated by changing retriever configuration, enabling A/B testing of retrieval approaches. Evaluation metrics like RagasContextPrecision and RagasContextRelevancy measure retrieval quality. Implementation details unknown but likely supports pluggable retriever interfaces.

Enables non-technical users (product managers, business analysts) to prototype multi-step AI workflows without code. Provides a visual interface for chaining prompts, models, and retrievers together. Workflows can be tested against datasets and evaluated using preset metrics. Implementation details unknown but likely uses a DAG-based flow editor with drag-and-drop components.

Supports human annotation of evaluation datasets alongside automated metrics, enabling teams to create ground truth labels for model evaluation. Annotators can review LLM outputs and provide feedback (e.g., correctness, relevance, hallucination presence). Annotations are stored in Athina and can be used to validate automated metric accuracy. Implementation details unknown but likely includes annotation UI, reviewer management, and inter-rater agreement tracking.

Generates side-by-side comparison reports of evaluation runs, enabling teams to understand how changes (prompt, model, retriever) impact metric scores. Reports show metric deltas, statistical significance (if applicable), and sample-level breakdowns. Supports filtering and sorting by metric, sample, or other dimensions. Likely uses statistical aggregation and visualization to surface insights.