Speaker Conditioned Autoregressive Speech Generation

via “acoustic decoder with speaker-conditioned speech generation”

text-to-speech model by undefined. 1,71,519 downloads.

Unique: Speaker conditioning via natural language descriptions rather than speaker embeddings or ID-based selection, allowing zero-shot voice control without speaker enrollment. Decoder architecture uses cross-attention between text and acoustic sequences, enabling fine-grained alignment and prosody control.

vs others: Offers semantic speaker control (text descriptions) instead of speaker ID or embedding-based approaches, making it more accessible for developers who lack speaker enrollment data while maintaining competitive audio quality through transformer-based acoustic modeling.

via “three-stage autoregressive-to-diffusion speech synthesis”

A high quality multi-voice text-to-speech library

Unique: Combines autoregressive content generation with diffusion-based acoustic refinement rather than end-to-end autoregressive generation, enabling independent control over semantic content and acoustic quality. The diffusion decoder stage specifically addresses prosody naturalness through iterative refinement rather than single-pass generation.

vs others: Produces more natural prosody and intonation than single-stage autoregressive TTS systems (like Glow-TTS) because diffusion refinement captures fine-grained acoustic details; slower than FastPitch but higher quality for complex linguistic phenomena.

via “autoregressive audio continuation generation from prompt conditioning”

* ⭐ 09/2022: [AudioGen: Textually Guided Audio Generation (AudioGen)](https://arxiv.org/abs/2209.15352)

Unique: Applies language modeling directly to raw audio tokens rather than requiring intermediate representations (text, phonemes, MIDI, or symbolic notation). The model learns audio structure end-to-end from raw waveforms, enabling it to capture prosodic and acoustic patterns that symbolic approaches miss.

vs others: Generates more natural prosody and speaker consistency than text-to-speech baselines because it conditions directly on audio rather than text, and maintains longer-term coherence than codec-only models because it uses LM tokens that capture semantic structure.

via “prompt-based speech generation with acoustic conditioning”

A cross-lingual neural codec language model for cross-lingual speech synthesis.

via “speaker-conditioned autoregressive speech generation”

* ⭐ 01/2023: [MusicLM: Generating Music From Text (MusicLM)](https://arxiv.org/abs/2301.11325)

Unique: Conditions the language model on speaker embeddings extracted from reference audio rather than requiring explicit speaker labels or IDs, enabling zero-shot adaptation to new speakers without retraining and allowing speaker characteristics to be learned implicitly from the reference audio

vs others: More flexible than speaker-ID-based conditioning (works for any speaker, not just those in training set) and more natural than concatenative synthesis because the language model learns to generate coherent acoustic sequences rather than selecting pre-recorded units