Comet ML

Captures and stores experiment metadata including hyperparameters, metrics, and code snapshots during ML training runs. Works by instrumenting training scripts via the comet_ml SDK, which intercepts log calls (exp.log_parameters, exp.log_metrics) and sends them to Comet's backend for centralized storage and versioning. Code snapshots are automatically captured at experiment start, enabling reproducibility by preserving the exact code state that generated results.

Provides a unified dashboard for comparing metrics, parameters, and artifacts across multiple experiments using a table-based interface with filtering, sorting, and custom visualization options. The platform stores experiment data in a queryable backend that supports cross-experiment aggregation, allowing users to identify patterns, outliers, and optimal configurations through interactive charts and parallel coordinates plots.

Offers SDKs in multiple programming languages (Python, JavaScript, Java, R) enabling experiment tracking and integration from diverse ML ecosystems. The Python SDK (comet_ml) is the primary and most feature-complete, while other SDKs provide core functionality with varying levels of feature parity. SDKs handle authentication, metric/parameter logging, artifact upload, and integration with language-specific ML frameworks.

Opik, the LLM observability component, is available as open-source software (19,000+ GitHub stars) enabling self-hosted deployment on-premises or in private cloud environments. Self-hosted Opik provides the same trace capture and visualization capabilities as the cloud version but with data stored in the user's infrastructure. Deployment is via Docker containers or Kubernetes, with configuration for custom databases and storage backends.

Provides native integrations with popular ML frameworks and libraries (PyTorch, TensorFlow, scikit-learn, XGBoost, etc.) enabling automatic logging of training metrics, model architecture, and hyperparameters without explicit instrumentation. Integrations are implemented as callbacks or hooks that intercept framework events (epoch end, batch end, etc.) and log relevant data to Comet. Framework-specific integrations reduce boilerplate code and ensure consistent metric logging.

Maintains a centralized registry of model versions with metadata including training parameters, performance metrics, and deployment status. Models are stored as references (not the actual model files) with links to external storage, and the registry integrates with CI/CD pipelines to enable automated promotion from staging to production. Version history is preserved with rollback capabilities, allowing teams to track which model version is deployed where.

Captures detailed execution traces from LLM applications and agents via the Opik SDK, recording each step in a chain including LLM calls, tool invocations, context retrievals, and user feedback. Traces are structured hierarchically (parent-child relationships between steps) and visualized in a timeline view with full context, enabling developers to debug LLM application behavior and identify bottlenecks. Traces appear in the platform 'almost instantly' even at high volumes, using asynchronous logging to avoid blocking application execution.

Enables creation of test suites for LLM applications using plain-English assertions evaluated by an LLM-as-a-judge approach. Tests are defined declaratively (e.g., 'output should be factually accurate', 'response should be under 100 words') and executed against a dataset of inputs, with results aggregated to provide pass/fail metrics. The platform uses LLM evaluation rather than traditional metrics, allowing subjective quality assessment without requiring labeled ground truth data.

Monitors deployed models in production for performance degradation, data drift, and prediction anomalies. The platform collects predictions and ground truth labels (when available) from production endpoints, compares current performance against baseline metrics established during training, and triggers alerts when performance drops below configured thresholds. Monitoring data is visualized in dashboards with drill-down capabilities to identify which data segments are affected.

Manages versioning of training artifacts (datasets, preprocessed data, model checkpoints, evaluation results) with content-addressed storage and deduplication. Artifacts are stored in Comet's artifact storage or referenced from external storage (S3, GCS), with version history preserved and queryable by experiment. The platform tracks artifact lineage, showing which experiments produced which artifacts and which artifacts were used in downstream experiments.

Integrates with hyperparameter optimization frameworks (Optuna, Ray Tune, Hyperopt, etc.) to automatically log and track optimization runs. The platform captures the optimization algorithm's suggestions, experiment results, and convergence progress, enabling visualization of the optimization landscape and identification of optimal hyperparameter regions. Integration is achieved via SDK hooks that intercept optimization callbacks.

Enables team-based access control to experiments, models, and artifacts with role-based permissions (viewer, editor, admin). Teams can be organized hierarchically with project-level and organization-level permissions. Access control is enforced at the API and UI level, with audit logs recording all access and modifications. SSO integration (mentioned as enterprise feature) allows centralized identity management.

Provides a REST API for programmatic access to experiments, models, metrics, and artifacts, enabling integration with external tools and custom workflows. The API supports CRUD operations on experiments, querying metrics and parameters, downloading artifacts, and managing model registry entries. Authentication is via API keys, and responses are JSON-formatted with pagination support for large result sets.