Metric Computation And Monitoring During Training

Multi-backend deep learning API for JAX, TF, and PyTorch.

Unique: Keras 3's metrics use a stateful accumulation pattern where each `keras.metrics.Metric` object maintains internal state (e.g., running sum and count for averaging) across batches, enabling memory-efficient metric computation without storing all predictions, and supporting distributed training via state synchronization.

vs others: More memory-efficient than PyTorch's approach of storing all predictions and computing metrics post-hoc, and more flexible than TensorFlow's built-in metrics because custom metrics can override any part of the computation pipeline.

via “metric computation and evaluation with task-specific measures”

PyTorch toolkit for all speech processing tasks.

Unique: Integrates task-specific metric computation (WER, EER, MCD) directly into the training loop via the `compute_metrics()` method, enabling automatic evaluation without separate evaluation scripts. Unlike manual metric computation, this approach ensures consistent evaluation across training and test sets.

vs others: More convenient than computing metrics separately, more consistent than manual evaluation, and enables easy comparison of models using standard metrics.

via “training metrics tracking and visualization”

A Python library for fine-tuning LLMs [#opensource](https://github.com/unslothai/unsloth).

Unique: Integrated metrics tracking that automatically computes common metrics (loss, perplexity, gradient norms) without requiring manual implementation, with optional logging to multiple backends through a unified interface

vs others: Simpler setup than manual TensorBoard/W&B integration with automatic metric computation, and more flexible than HuggingFace Trainer's fixed metrics while maintaining compatibility with standard logging backends

via “metric computation and tracking during training”

Multi-backend Keras

Unique: Implements metrics as stateful objects in keras/src/metrics/ that accumulate values across batches and compute aggregate statistics. Metrics are compiled into models and automatically computed during training/evaluation, with support for both eager and graph execution modes across all backends.

vs others: Unlike PyTorch (requires manual metric computation) or TensorFlow (metrics are TensorFlow-specific), Keras provides a unified metric system across all backends with built-in metrics for common use cases and automatic computation during training.

via “training-stability-monitoring”