Audio Codec Compression With Discrete Token Representation

via “neural audio compression with encodec”

Meta's library for music and audio generation.

Unique: Uses residual vector quantization across multiple codebooks (typically 4) to represent audio at different frequency bands and temporal resolutions, enabling variable bitrate compression while maintaining perceptual quality. Trained end-to-end with adversarial loss for realistic reconstruction.

vs others: Achieves better perceptual quality than traditional codecs (MP3, AAC) at equivalent bitrates and enables discrete token representation required for language model-based generation; more efficient than raw waveform processing.

via “coarse audio structure generation via semantic-to-codebook mapping”

Open-source text-to-audio — speech, music, sound effects, 13+ languages, runs locally.

Unique: Implements a two-stage hierarchical audio codec approach where coarse tokens establish acoustic structure before fine-grained details are added, enabling efficient progressive refinement and potential latency optimization

vs others: Faster than single-pass models for coarse-only use cases; enables streaming or progressive audio output unlike end-to-end TTS systems

via “discrete audio token generation with speaker embedding control”

A generative speech model for daily dialogue.

Unique: Uses discrete audio tokens (learned via DVAE quantization) rather than continuous spectrograms, enabling stable, controllable audio generation with explicit speaker embeddings that condition the token sequence. This discrete approach is inspired by VQ-VAE and allows the model to learn a compact, interpretable audio representation that separates content (text) from speaker identity (embedding).

vs others: More speaker-controllable than end-to-end TTS models (e.g., Tacotron 2) because speaker embeddings are explicitly separated from text encoding, enabling voice cloning without fine-tuning. More stable than continuous spectrogram generation because discrete tokens have well-defined boundaries and are less prone to artifacts at token boundaries.

via “quantized-codebook-learning-for-discrete-speech-units”

automatic-speech-recognition model by undefined. 12,10,723 downloads.

Unique: Uses product quantization with straight-through estimators to learn discrete speech units without requiring phonetic labels — the quantizer acts as a learned bottleneck that forces the model to discover meaningful acoustic patterns, unlike supervised phoneme-based approaches that require manual annotation

vs others: Discovers more linguistically-relevant discrete units than k-means clustering on MFCC features because the quantizer is jointly optimized with the feature extractor, resulting in units that better preserve phonetic information (phoneme error rate 15% lower on downstream tasks)

A single-stop code base for generative audio needs, by Meta. Includes MusicGen for music and AudioGen for sounds. #opensource

Unique: Combines convolutional autoencoders with vector quantization to create a learned codec that produces discrete tokens suitable for language model training, rather than using traditional codecs (MP3, AAC) or continuous latent representations that don't integrate naturally with transformer architectures

vs others: More efficient than raw waveform generation because it reduces sequence length by 50-100x, and more flexible than traditional audio codecs because the discrete representation is learned end-to-end for the downstream task rather than optimized for human perception alone

via “hybrid-tokenization audio encoding with dual-stream representation”

* ⭐ 09/2022: [AudioGen: Textually Guided Audio Generation (AudioGen)](https://arxiv.org/abs/2209.15352)

Unique: Uses a hybrid dual-stream tokenization combining masked LM activations with neural codec codes, rather than relying on a single tokenization source. This architectural choice explicitly addresses the trade-off between structural coherence (from LM tokens) and acoustic quality (from codec tokens) that single-stream approaches face.

vs others: Outperforms single-codec tokenization approaches (like Jukebox's VQ-VAE) by preserving long-term semantic structure through LM tokens, while maintaining acoustic quality through codec tokens—a design choice not present in prior audio generation systems.

via “real-time streaming audio encoding with quantized latent representation”

* ⭐ 12/2022: [Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)](https://arxiv.org/abs/2212.04356)

Unique: Uses a single multiscale spectrogram adversary instead of traditional multi-discriminator approaches, combined with a novel loss balancer mechanism that decouples loss weight from loss scale, enabling more stable training of the quantized latent space. Streaming architecture supports real-time encoding/decoding without buffering entire audio segments.

vs others: Outperforms baseline codecs across speech, noisy speech, and music domains according to MUSHRA subjective evaluation, while maintaining real-time performance on standard hardware — a capability gap for traditional neural codecs that typically require offline processing or significant computational overhead.

via “encodec-based audio tokenization and reconstruction”

A transformer-based text-to-audio model. #opensource

via “neural codec-based discrete speech representation learning”

* ⭐ 01/2023: [MusicLM: Generating Music From Text (MusicLM)](https://arxiv.org/abs/2301.11325)

Unique: Uses residual vector quantization (RVQ) with hierarchical token streams instead of single-level VQ, capturing both coarse acoustic structure and fine prosodic details in separate token sequences, enabling the language model to learn different prediction patterns at different granularities

vs others: More efficient than waveform-based language models (smaller token vocabulary, shorter sequences) and more expressive than single-level VQ because hierarchical tokens preserve multi-scale acoustic information needed for natural speech synthesis