comet-ml

Provides an Experiment object that acts as a container for a single training run, allowing developers to imperatively log hyperparameters, metrics, and artifacts via method calls (e.g., log_parameters(), log_metrics()). The system persists all logged data to Comet's cloud or self-hosted backend, enabling later retrieval and comparison across runs. Uses a stateful session model where a single Experiment instance maintains context throughout a training loop.

Enables side-by-side comparison of metrics, parameters, and artifacts across multiple training runs using a web-based dashboard. Developers can filter, sort, and group experiments by tags or metadata, and create custom visualization templates to display metrics in domain-specific ways (e.g., ROC curves, confusion matrices). The comparison engine indexes all logged data and supports search queries across experiment metadata.

Comet enables versioning of training datasets, allowing developers to create snapshots of datasets at specific points in time and link them to experiments. Each dataset version is immutable and can be retrieved later to reproduce past results. The system tracks which dataset version was used for each experiment, creating an audit trail for reproducibility. Dataset versions can be tagged and organized by project.

Comet provides pre-built integrations with popular ML frameworks (specific frameworks not detailed in documentation) that automatically instrument training loops to log metrics, parameters, and artifacts without requiring manual API calls. Integrations are available for LlamaIndex (RAG systems), Kubeflow (orchestration), and Predibase (LLM fine-tuning). Each integration provides framework-specific adapters that hook into the framework's callback or event system to capture training data automatically.

Comet exposes a REST API that allows developers to programmatically query experiments, retrieve metrics and artifacts, and create custom integrations. The API supports filtering, sorting, and exporting experiment data in structured formats (JSON, CSV). Developers can build custom dashboards, analysis tools, or integrations with external systems using the REST API. Authentication is via API key.

Comet provides SDKs in multiple programming languages (Python, JavaScript, Java, R) enabling developers to integrate experiment tracking into projects regardless of primary language. Each SDK exposes the same core API (Experiment, logging methods, artifact management) with language-specific idioms. SDKs are maintained by Comet and released in sync with the core platform.

Comet is available as a cloud-hosted SaaS platform (Comet Cloud) and as a self-hosted open-source version (Opik). Enterprise customers can deploy Comet on-premises or in a private VPC with custom configurations. The deployment model affects data residency, compliance, and integration options. Cloud deployment is managed by Comet; self-hosted deployment requires infrastructure management by the customer.

Provides a versioned artifact storage system where developers can log binary files (model checkpoints, datasets, plots) alongside experiments. Each artifact is assigned a version number and stored in Comet's backend with metadata linking it to the experiment that produced it. The system supports querying artifacts by experiment, version, or tag, and provides APIs to retrieve specific artifact versions for reproducibility. Artifacts are immutable once logged and can be accessed via REST API or SDK.

The Opik component provides a @track decorator that automatically captures execution flow through LLM application functions, logging inputs, outputs, and intermediate steps as a structured trace. When a decorated function is called, Opik records the function name, arguments, return value, and execution time, then sends the trace to the Comet backend for visualization and analysis. Traces are hierarchical — nested function calls create parent-child relationships in the trace tree. The system supports tracing across multiple LLM providers and custom functions without code modification beyond adding the decorator.

Opik provides a test suite system where developers write assertions in plain English (e.g., 'the response should be helpful and relevant') that are evaluated against LLM traces using an LLM-as-judge approach. When a test suite is run, Opik sends the trace data and assertions to an LLM (provider configurable) which evaluates whether the trace output satisfies the assertions. Results are returned as pass/fail with reasoning, enabling automated evaluation of LLM application quality without hand-crafted metrics.

Opik provides a test suite system where developers can create datasets of test cases (input-output pairs) and associate assertions with each case. The system stores test datasets in Comet's backend and enables running evaluations against traces produced by LLM applications. Test suites support versioning, tagging, and filtering, allowing teams to organize evaluation datasets by use case or model version. Evaluation results are linked back to the test suite and traces for analysis.

Opik provides an Agent Playground feature that allows developers to execute LLM agents in a sandboxed environment before deploying to production. The sandbox captures all agent actions, tool calls, and decision-making steps, enabling inspection and debugging without affecting production systems. Developers can modify agent code, inputs, or configuration and re-run in the sandbox to test changes. The sandbox execution is fully traced and logged to Comet for analysis.

Opik includes Ollie, a built-in coding agent that can automatically suggest or apply fixes to code based on test failures or evaluation results. When a test suite fails or an assertion is violated, Ollie analyzes the failure and generates code changes to address the issue. Developers can review and approve suggested fixes before applying them. Ollie integrates with the sandbox environment to test fixes before deployment.

Opik provides production monitoring capabilities that track LLM application behavior in live environments, logging traces, costs, and compliance metrics. The system captures all LLM API calls, token usage, and costs, enabling cost attribution and budget tracking. Governance features include audit logs, access controls, and compliance reporting. Monitoring data is streamed to Comet's backend and visualized in dashboards for real-time visibility.

Comet provides a model registry where developers can register trained models, assign version numbers, and track metadata (training parameters, evaluation metrics, artifacts). The registry integrates with CI/CD systems to enable automated model deployment workflows. Models can be tagged with metadata (e.g., 'production-ready', 'experimental') and queried by version or tag. The registry supports model lineage tracking — linking models to the experiments and datasets that produced them.