pyannote-audio

Performs speaker diarization by combining neural segmentation models (trained on Pyannote's proprietary datasets) with speaker embedding extraction and clustering. The pipeline uses a two-stage approach: first, a temporal convolutional network (TCN) or transformer-based segmentation model identifies speaker boundaries and speech/non-speech regions frame-by-frame; second, speaker embeddings are extracted and clustered using agglomerative hierarchical clustering with dynamic threshold tuning. The system supports both batch processing and streaming inference modes.

Extracts fixed-dimensional speaker embeddings (typically 192-512 dims) from audio segments using pretrained speaker verification models (e.g., ECAPA-TDNN, ResNet-based architectures). The embeddings capture speaker-specific acoustic characteristics and are designed to be speaker-discriminative while speaker-invariant to content. Embeddings can be extracted at segment or utterance level and are compatible with standard distance metrics (cosine, Euclidean) for downstream clustering or similarity matching.

Provides utilities for visualizing diarization results, including speaker timeline plots, embedding space visualizations (t-SNE, UMAP), and spectrogram overlays with speaker labels. Includes debugging tools for analyzing segmentation errors, embedding quality, and clustering decisions. Supports interactive HTML visualizations and static plots for reports. Can overlay ground truth annotations for error analysis.

Provides utilities for processing large collections of audio files in batches with automatic job scheduling, error handling, and result aggregation. Supports parallel processing across multiple CPU cores or GPUs, with configurable batch sizes and queue management. Includes checkpointing to resume interrupted jobs and logging for monitoring progress. Can be integrated with workflow orchestration tools (e.g., Airflow, Prefect) for production pipelines.

Performs frame-level speaker activity detection and speaker change detection using neural segmentation models (TCN or transformer-based) that process audio spectrograms and output per-frame probabilities for speech/non-speech and speaker boundaries. The model operates on fixed-size windows (typically 10-20ms frames) and uses temporal convolutions or attention mechanisms to capture context across frames. Outputs are post-processed (smoothing, peak detection) to produce clean segment boundaries.

Provides a unified interface for discovering, downloading, and loading pretrained diarization and speaker embedding models from Hugging Face Model Hub. Models are versioned, cached locally, and can be instantiated with a single function call. The system handles model card parsing, dependency resolution, and automatic fallback to CPU if GPU is unavailable. Users can also upload custom models to Hugging Face Hub for sharing and reproducibility.

Clusters speaker embeddings using agglomerative hierarchical clustering (bottom-up merging) with dynamic threshold selection based on embedding statistics. The algorithm computes pairwise distances between embeddings (cosine or Euclidean), builds a dendrogram, and cuts at a threshold that maximizes cluster separation. Threshold tuning can be automatic (based on silhouette score, gap statistic) or manual. Supports custom linkage criteria (complete, average, ward) and distance metrics.

Performs speaker diarization on streaming audio by processing frames incrementally and updating speaker clusters in real-time. The system maintains a running set of speaker embeddings and updates cluster assignments as new frames arrive. Segmentation is performed frame-by-frame, and new speakers are detected by comparing incoming embeddings against existing speaker clusters using a dynamic threshold. Supports both online (single-pass) and semi-online (buffered) modes for latency/accuracy tradeoffs.

Provides utilities for converting raw audio waveforms into acoustic features (mel-spectrograms, MFCCs, chromagrams) required by neural models. Handles audio resampling, normalization, windowing, and feature computation using librosa or torchaudio backends. Supports both offline (batch) and online (streaming) feature extraction with configurable window sizes, hop lengths, and frequency ranges. Features are cached and can be reused across multiple model runs.

Reads, writes, and manipulates speaker diarization annotations in RTTM (Rich Transcription Time Marked) format, a standard format for speaker diarization ground truth and predictions. Provides utilities for parsing RTTM files into Python objects, filtering/merging segments, computing metrics (DER, JER, purity, coverage), and exporting results back to RTTM. Supports validation of RTTM files and conversion between RTTM and other formats (JSON, CSV).

Enables distributed inference across multiple GPUs or machines using PyTorch's distributed data parallel (DDP) and model parallel patterns. The system automatically partitions audio files across GPUs, processes segments in parallel, and aggregates results. Supports both data parallelism (same model on multiple GPUs) and model parallelism (large models split across GPUs). Handles synchronization, gradient aggregation, and result merging transparently.

Provides a training framework for fine-tuning pretrained diarization models on custom datasets or training models from scratch. Includes data loaders for RTTM-annotated audio, loss functions (e.g., focal loss for imbalanced data), optimization strategies (Adam, SGD with learning rate scheduling), and validation/evaluation loops. Supports mixed-precision training for memory efficiency and gradient accumulation for large batch sizes. Integrates with Weights & Biases for experiment tracking.