Comet ML

Captures and logs ML experiment runs by instrumenting training code with SDK calls to record parameters, metrics, hyperparameters, and automatic code snapshots. The platform stores run metadata in a centralized database, enabling side-by-side comparison of experiments across multiple dimensions (accuracy, loss, training time, hardware utilization). Code snapshots are captured at experiment start, preserving the exact training script state for reproducibility and debugging.

Provides a centralized registry for storing model versions with associated metadata (training parameters, performance metrics, dataset references, custom tags). Models are registered from experiment runs or uploaded directly; the registry maintains a version history with rollback capability. Metadata is queryable and can be linked to CI/CD pipelines for automated model promotion workflows, though specific CI/CD integration mechanisms are not detailed in documentation.

Enables deployment of Comet (specifically Opik, the open-source LLM observability component) on user-managed infrastructure (Kubernetes, Docker, VMs) or on-premises data centers. Users can self-host the full Opik platform, maintaining data within their own network and avoiding cloud vendor lock-in. Self-hosted instances can be configured with custom storage backends (PostgreSQL, etc.) and integrated with existing infrastructure (VPCs, firewalls, etc.). Enterprise support is available for custom deployments.

Enables searching and exporting experiment data (metrics, parameters, code, artifacts) in bulk. Users can filter experiments by tags, metrics, parameters, or date range, then export results as CSV or JSON for external analysis. Search is performed via the web UI or REST API, allowing programmatic access for automation. Exported data includes all logged metadata, enabling integration with external analytics tools (Pandas, SQL, etc.).

Provides pre-built integrations with popular LLM frameworks and libraries (LlamaIndex, LangChain, etc.) to simplify instrumentation. Integrations typically provide decorators or middleware that automatically capture function inputs/outputs and LLM API calls without requiring manual SDK calls. Framework-specific adapters handle the details of extracting relevant metadata (prompts, completions, model names, token counts) from framework objects.

Provides an admin dashboard for managing Comet workspaces, teams, and users. Admins can view workspace usage statistics (number of experiments, storage consumption, API calls), manage team memberships, configure SSO and audit logging, and set workspace-level policies. The dashboard displays real-time metrics and historical trends, enabling capacity planning and cost optimization.

Via the Opik component, captures execution traces from LLM applications and AI agents by instrumenting code with @track decorators or SDK calls. Traces record function inputs, outputs, latency, token counts, and LLM API calls (prompts, completions, model used). The platform visualizes traces as interactive trees showing the full execution path, enabling debugging of multi-step LLM workflows. Traces are indexed and searchable, with filtering by latency, cost, model, or custom attributes.

Enables creation of test suites for LLM applications using plain-English assertions evaluated by an LLM-as-judge. Users define test cases with inputs and expected outputs, then run them against LLM application traces. The platform uses an LLM (configurable, likely GPT-4 by default) to evaluate whether outputs meet criteria (e.g., 'response is factually accurate', 'response is concise'). Results are aggregated and visualized, showing pass/fail rates and failure reasons.

A built-in AI agent (Ollie) that analyzes LLM application traces and test failures, identifies root causes, and generates code fixes. Ollie reads trace data and test results, reasons about what went wrong, writes code patches, and commits them to the user's codebase with version control integration. The agent includes regression testing to verify fixes don't break existing functionality. Execution happens in a sandboxed Agent Playground before deployment.

Monitors deployed LLM applications in production by collecting traces, aggregating metrics (latency, error rate, token usage), and calculating costs based on LLM API pricing. The platform provides dashboards showing real-time performance, cost per request, and cost trends over time. Governance features (mentioned but not detailed) likely include access controls and audit logs for compliance. Alerts can be configured for cost spikes or performance degradation.

Provides version control for datasets and training artifacts (model checkpoints, preprocessed data, feature sets) by storing them in a versioned artifact store. Users can log artifacts from experiments, tag them with metadata, and retrieve specific versions for reproducibility. The platform tracks lineage (which experiment produced which artifact) and enables comparison across versions. Artifacts can be stored locally or remotely (S3, GCS, etc.).

Provides interactive dashboards for comparing multiple experiments side-by-side across metrics, hyperparameters, and other dimensions. Users can select experiments and view parallel coordinates plots, scatter plots, and tables showing how parameter changes correlate with performance. The platform includes a library of pre-built visualization templates and a custom visualization builder for domain-specific charts. Comparisons can be filtered, sorted, and exported.

Integrates with hyperparameter optimization frameworks (Optuna, Ray Tune, Hyperopt, etc.) to log optimization runs and visualize the search space. Users define parameter ranges and optimization objectives, run the optimizer, and Comet logs each trial as an experiment. The platform visualizes the optimization landscape (parameter values vs. objective metric) and can suggest next trials based on past results. Integration is framework-specific; each optimizer requires custom integration code.

Provides enterprise-grade access control via Single Sign-On (SSO) integration with identity providers (Okta, Azure AD, etc.) and detailed audit logging of all platform actions (experiment creation, model registration, trace access, etc.). Audit logs record who performed what action, when, and from which IP address, enabling compliance with regulatory requirements (SOC 2, HIPAA, etc.). Fine-grained permissions can be assigned to service accounts for programmatic access.