Quantized Codebook Learning For Discrete Speech Units

via “quantized-codebook-learning-for-discrete-speech-units”

automatic-speech-recognition model by undefined. 12,10,723 downloads.

Unique: Uses product quantization with straight-through estimators to learn discrete speech units without requiring phonetic labels — the quantizer acts as a learned bottleneck that forces the model to discover meaningful acoustic patterns, unlike supervised phoneme-based approaches that require manual annotation

vs others: Discovers more linguistically-relevant discrete units than k-means clustering on MFCC features because the quantizer is jointly optimized with the feature extractor, resulting in units that better preserve phonetic information (phoneme error rate 15% lower on downstream tasks)

via “discrete visual tokenization with learned codebook”

* ⭐ 09/2022: [PaLI: A Jointly-Scaled Multilingual Language-Image Model (PaLI)](https://arxiv.org/abs/2209.06794)

Unique: Uses learned discrete codebooks to tokenize images, creating a bridge between continuous vision features and discrete language tokens. This enables applying BERT-style masked language modeling directly to images without pixel-level reconstruction.

vs others: Provides better semantic alignment with language models than continuous feature representations because discrete tokens create a shared vocabulary between modalities, improving joint vision-language learning compared to approaches using separate continuous representations.

via “neural codec-based discrete speech representation learning”

* ⭐ 01/2023: [MusicLM: Generating Music From Text (MusicLM)](https://arxiv.org/abs/2301.11325)

Unique: Uses residual vector quantization (RVQ) with hierarchical token streams instead of single-level VQ, capturing both coarse acoustic structure and fine prosodic details in separate token sequences, enabling the language model to learn different prediction patterns at different granularities

vs others: More efficient than waveform-based language models (smaller token vocabulary, shorter sequences) and more expressive than single-level VQ because hierarchical tokens preserve multi-scale acoustic information needed for natural speech synthesis