indic-parler-tts

Generates natural-sounding speech from text input across 16 Indic languages and English using a transformer-based architecture adapted from Parler TTS. The model leverages a dual-encoder design with a text encoder that processes linguistic features and a speaker/prosody encoder that captures voice characteristics, then decodes to mel-spectrograms which are converted to waveforms via a neural vocoder. This architecture enables fine-grained control over speaker identity, pitch, and speaking rate while maintaining language-specific phonetic and prosodic patterns.

Enables precise voice selection and speaker characteristics through learned speaker embedding vectors that are injected into the decoder during synthesis. The model uses a speaker encoder that maps voice characteristics (pitch range, timbre, speaking style) into a fixed-dimensional embedding space, allowing users to select from pre-defined speakers or interpolate between speaker embeddings to create novel voice variations. This design decouples speaker identity from linguistic content, enabling the same speaker to pronounce text in different languages.

Generates mel-spectrograms with language-aware prosodic features (pitch contours, duration patterns, energy envelopes) that reflect linguistic and paralinguistic characteristics of Indic languages. The decoder produces frame-level mel-spectrogram features conditioned on both text embeddings and speaker embeddings, with implicit modeling of prosodic variation through the transformer attention mechanism. Prosodic patterns are learned from training data rather than explicitly specified, enabling natural-sounding synthesis that respects language-specific intonation patterns.

Generates mel-spectrograms that are compatible with multiple neural vocoder backends (HiFi-GAN, Glow-TTS vocoder, WaveGlow) for conversion to raw audio waveforms. The model outputs mel-spectrograms in a standard format (80-128 frequency bins, 12.5ms frame shift) that can be fed into any vocoder without model-specific preprocessing. This design decouples speech generation from waveform synthesis, allowing users to choose vocoder implementations based on latency, quality, or computational constraints.

Processes multiple text inputs in batch mode with automatic language detection and routing to language-specific tokenizers and phoneme inventories. The model accepts batched text inputs, detects the language of each input (or accepts explicit language tags), and applies language-specific preprocessing before encoding. Batch processing is implemented at the transformer encoder level, enabling efficient GPU utilization for multiple texts simultaneously while maintaining language-specific linguistic constraints.

Extracts rich linguistic representations from input text using a transformer encoder that processes character or subword tokens and produces contextual embeddings. The encoder uses multi-head self-attention to capture long-range linguistic dependencies (e.g., subject-verb agreement, pronoun resolution) and produces frame-level embeddings that are aligned with mel-spectrogram frames via attention mechanisms. This design enables the decoder to condition speech generation on deep linguistic context rather than surface-level text features.

Maps input text to language-specific phoneme inventories and applies language-aware tokenization that respects phonological rules of each Indic language. The model maintains separate phoneme sets for each language (e.g., Hindi has different phoneme inventory than Bengali) and applies language-specific grapheme-to-phoneme conversion rules. Tokenization is implemented as a preprocessing step that converts text to phoneme sequences before encoder input, enabling the model to work with consistent phonological units across languages.

Enables a single speaker to synthesize speech in multiple Indic languages by mapping language-specific phonemes to a shared acoustic space where speaker characteristics are language-independent. The model learns a shared speaker embedding space that captures voice characteristics (pitch range, timbre, speaking style) independent of language, allowing speaker embeddings extracted from one language to be applied to synthesis in other languages. This is implemented through a speaker encoder that processes speaker reference audio and produces language-agnostic embeddings, which are then injected into the decoder for any target language.

Supports streaming inference where mel-spectrograms are generated incrementally as text is processed, enabling real-time speech synthesis with minimal latency. The model uses a streaming-compatible decoder architecture that generates mel-spectrogram frames in a left-to-right manner without requiring the full text to be processed before output generation begins. Streaming is implemented through attention masking that prevents future context from influencing current frame generation, and frame-level output buffering that allows vocoder to process frames as they become available.

Provides mechanisms for fine-tuning the model on custom voice data or adapting to new languages through parameter-efficient techniques (LoRA, adapter modules) or full fine-tuning. The model exposes trainable components (speaker encoder, language-specific tokenizers, decoder) that can be adapted with relatively small amounts of custom data. Fine-tuning is implemented through standard PyTorch training loops with support for distributed training, gradient accumulation, and mixed-precision training to reduce memory requirements.