tortoise-tts

Generates speech by chaining three neural models: an autoregressive GPT-like model (UnifiedVoice) that produces mel spectrogram codes from tokenized text conditioned on voice embeddings, a diffusion decoder (DiffusionTts) that refines codes into high-quality mel spectrograms through iterative denoising, and a HiFiGAN vocoder that converts spectrograms to waveforms. This multi-stage approach decouples content generation from acoustic refinement, enabling both prosody control and high-fidelity output.

Extracts speaker embeddings from reference audio samples (5-30 seconds) using a speaker encoder, then conditions the autoregressive and diffusion models on these embeddings to synthesize speech in the cloned voice. The voice conditioning system integrates embeddings at multiple points in the generation pipeline, enabling voice characteristics to influence both content generation timing and acoustic refinement without requiring fine-tuning.

Provides two CLI tools: do_tts.py for single-phrase synthesis and read.py for long-form text reading. These tools expose core API functionality through command-line arguments, enabling non-programmatic users to generate speech without writing code. The CLI handles file I/O, argument parsing, and progress reporting. This enables integration into shell scripts and batch processing workflows.

Manages downloading, caching, and loading of pre-trained model weights (autoregressive, diffusion, vocoder, speaker encoder) from remote repositories. Models are downloaded on-demand and cached locally to avoid repeated downloads. The TextToSpeech API handles lazy loading, where models are loaded into GPU memory only when needed, reducing startup time and memory footprint for inference-only workflows.

Processes multiple text inputs in configurable batch sizes through the autoregressive model, with automatic batch size selection based on available GPU memory. Implements KV-cache optimization to reduce redundant computation during autoregressive decoding and supports half-precision (FP16) computation to reduce memory footprint. The TextToSpeech API orchestrates batch processing across all three pipeline stages while managing device placement and memory allocation.

Processes long documents by splitting text into sentences, synthesizing each sentence independently, and concatenating audio outputs with optional silence padding. The read.py and read_fast.py modules implement streaming generation where sentences are synthesized sequentially and can be output to audio files or streamed in real-time. This approach avoids loading entire documents into memory and enables progressive audio generation without waiting for full synthesis.

The DiffusionTts decoder refines mel spectrogram codes from the autoregressive model through iterative denoising, where each step removes noise and improves acoustic quality. The number of diffusion steps is configurable (typically 5-50 steps), trading off quality for inference speed. This stage operates on mel spectrogram space rather than waveform space, making it computationally efficient while capturing fine-grained acoustic details like formant structure and spectral smoothness.

Converts mel spectrograms to audio waveforms using a pre-trained HiFiGAN generative adversarial network, which uses multi-scale discriminators and periodic/aperiodic decomposition to generate high-fidelity audio. The vocoder operates on 24kHz mel spectrograms (80-128 mel bins) and produces 24kHz waveforms with minimal artifacts. This stage is the final step in the synthesis pipeline and is computationally efficient compared to autoregressive or diffusion stages.

Preprocesses input text by tokenizing into subword units, extracting linguistic features (phonemes, stress, intonation markers), and converting to numerical representations suitable for the autoregressive model. The text processing pipeline handles multiple languages, special characters, and punctuation normalization. Tokenization uses a learned vocabulary (similar to GPT) rather than character-level encoding, enabling the model to capture linguistic structure efficiently.

Converts audio waveforms to mel-scale spectrograms (80-128 mel bins, 24kHz sample rate) for use as voice conditioning input and intermediate representations. The audio processing pipeline applies windowing, FFT, mel-scale filtering, and optional normalization. This representation is used both for extracting speaker embeddings from reference audio and as the target representation for the diffusion decoder.

Integrates with DeepSpeed library to enable model parallelism across multiple GPUs, distributing the autoregressive and diffusion models across devices. This allows inference on larger models or with larger batch sizes than single-GPU memory permits. DeepSpeed handles gradient checkpointing, activation partitioning, and communication optimization to minimize overhead.

Provides multiple optimization modes (standard, fast, ultra-fast) that trade off audio quality for inference speed by adjusting autoregressive batch size, diffusion steps, and model precision. The API exposes parameters like autoregressive_batch_size, diffusion_steps, and use_half_precision, enabling users to tune synthesis for their specific latency/quality requirements. This is implemented through separate API classes (TextToSpeech for standard, TextToSpeechFast for optimized).