speechbrain

Provides end-to-end neural ASR pipelines using PyTorch with pretrained checkpoints for multiple languages and acoustic conditions. Implements CTC (Connectionist Temporal Classification) and attention-based sequence-to-sequence architectures that map raw audio spectrograms to text tokens, with built-in support for language model rescoring and beam search decoding. Models are loaded via a unified checkpoint system that handles feature extraction, acoustic modeling, and text decoding in a single inference pass.

Extracts fixed-dimensional speaker embeddings (typically 192-512 dims) from variable-length audio using neural speaker encoders trained on large-scale speaker datasets. Implements x-vector and ECAPA-TDNN architectures that learn speaker-discriminative features through metric learning (e.g., AAM-Softmax, Prototypical Networks). Embeddings can be compared via cosine similarity for speaker verification (1:1 matching) or used as features for speaker clustering and identification tasks.

Provides end-to-end training infrastructure for speech models with support for distributed training across multiple GPUs/TPUs, automatic mixed precision (AMP) for memory efficiency, and gradient accumulation for large batch sizes. Implements PyTorch DistributedDataParallel (DDP) for multi-GPU training with automatic synchronization, combined with gradient scaling for stable training. Includes logging, checkpointing, and early stopping for efficient model development.

Provides recipe-based experiment templates that bundle model architecture, training hyperparameters, data preprocessing, and evaluation metrics in a single configuration file (YAML/JSON). Recipes are self-contained and reproducible, enabling one-command training and evaluation with automatic logging of all hyperparameters and results. Supports recipe composition and inheritance for systematic experimentation and ablation studies.

Provides standard evaluation metrics for speech tasks including WER (Word Error Rate) for ASR, speaker verification EER (Equal Error Rate) and minDCF, diarization DER (Diarization Error Rate), and emotion recognition accuracy/F1-score. Implements efficient metric computation with support for batch processing and distributed evaluation across multiple GPUs. Includes benchmark datasets and baseline comparisons for standardized evaluation.

Reduces background noise and enhances speech quality using neural beamforming techniques that leverage multi-channel audio (if available) or single-channel neural enhancement. Implements learnable beamformers (e.g., MVDR-like networks) that estimate speech and noise subspaces from spectrograms, combined with masking-based enhancement (ideal ratio mask, phase-aware mask) to suppress noise while preserving speech intelligibility. Can operate on raw waveforms or spectrograms with configurable feature representations (MFCC, Fbank, raw spectrograms).

Segments audio into speaker turns and clusters segments by speaker identity using a pipeline of speaker change detection, speaker embedding extraction, and hierarchical clustering. Implements end-to-end diarization via neural segmentation (predicting speaker change points) combined with speaker embedding-based clustering (e.g., spectral clustering, agglomerative clustering with cosine distance). Outputs speaker labels with timestamps, enabling downstream analysis of who spoke when.

Detects speech presence in audio by classifying short frames (typically 20-40ms) as speech or non-speech using neural networks trained on large-scale labeled datasets. Implements CNN or RNN-based classifiers that operate on spectrograms (MFCC, Fbank) or raw waveforms, outputting frame-level probabilities that can be aggregated into segment-level decisions via smoothing or post-processing. Enables efficient audio processing by skipping non-speech regions.

Classifies emotional states (e.g., happy, sad, angry, neutral) from speech audio using neural classifiers that extract emotion-relevant features from spectrograms or embeddings. Implements CNN or RNN architectures trained on emotion-labeled speech datasets (e.g., IEMOCAP, RAVDESS), learning prosodic and spectral patterns associated with different emotions. Outputs class probabilities for each emotion category, enabling both hard classification and confidence-based ranking.

Separates individual speaker sources from mixed multi-speaker audio using neural source separation models that learn to decompose spectrograms into speaker-specific components. Implements Conv-TasNet, Conformer, or attention-based architectures that estimate speaker-specific masks or directly generate speaker waveforms. Can operate in supervised mode (known number of speakers) or unsupervised mode (unknown speaker count) with optional speaker embedding conditioning for speaker-specific extraction.

Classifies the language spoken in audio using neural classifiers trained on multilingual speech datasets. Implements CNN or RNN architectures that learn language-specific acoustic patterns from spectrograms, outputting probabilities for each supported language. Enables automatic language detection for multilingual ASR pipelines or language-specific processing workflows.

Extracts diverse audio representations (MFCC, Fbank, spectrogram, mel-spectrogram, raw waveform) from audio files using PyTorch-based feature computation. Implements efficient batch processing of variable-length audio with configurable frame sizes, hop lengths, and frequency bins. Features are normalized and can be augmented (time-stretching, pitch-shifting, SpecAugment) for data augmentation in training pipelines.

Provides a unified checkpoint system for loading, saving, and fine-tuning pretrained speech models with automatic handling of model architecture, weights, and hyperparameters. Implements checkpoint serialization that bundles model definition, weights, and training metadata, enabling reproducible model loading and transfer learning. Supports fine-tuning workflows with configurable learning rates, layer freezing, and gradient accumulation for efficient adaptation to new tasks or domains.