mSLAM: Massively multilingual joint pre-training for speech and text (mSLAM)

Performs unified pre-training across 143+ languages on both speech and text modalities simultaneously using a shared encoder architecture. The model learns cross-modal and cross-lingual representations through contrastive learning objectives that align speech and text embeddings in a common latent space, enabling zero-shot transfer across language pairs and modalities without task-specific fine-tuning.

Leverages the shared multilingual embedding space to perform speech recognition in a target language without any labeled speech data in that language. The model uses representations learned from high-resource languages and text data in the target language to enable ASR through alignment in the common embedding space, effectively transferring knowledge from data-rich to data-poor languages.

Enables bidirectional retrieval between speech and text using the shared embedding space: given a speech query, retrieve matching text documents, or given text, retrieve matching speech. The model computes similarity scores between speech and text embeddings using cosine distance or other metrics in the common latent space, supporting both exact matching and semantic similarity-based retrieval across languages.

Learns language-agnostic speech representations by training on contrastive objectives (e.g., InfoNCE or similar) that push speech embeddings from the same utterance closer together while pushing embeddings from different utterances apart, across all 143+ languages simultaneously. This approach learns universal phonetic and linguistic features that generalize across languages without explicit language labels during training.

Learns language-agnostic text representations using a shared tokenizer and embedding space across 143+ languages, enabling the model to understand text in any language without language-specific vocabularies. The approach uses masked language modeling or similar objectives on multilingual text corpora, learning to predict masked tokens in context while sharing parameters across all languages.

Aligns speech audio with corresponding text transcriptions across 143+ languages by learning to match speech embeddings with text embeddings in the shared space. The model uses the contrastive objectives to enforce that speech and text from the same utterance have similar embeddings, enabling automatic alignment without explicit alignment annotations or forced alignment tools.

Implicitly performs language identification by analyzing the learned embeddings, which encode language-specific phonetic and linguistic patterns despite being trained as language-agnostic. The model can identify the language of a speech utterance or text by analyzing the embedding distribution or using a lightweight classifier on top of the embeddings, without explicit language labels during pre-training.

Enables efficient fine-tuning of the pre-trained multilingual embeddings for downstream tasks (speech recognition, machine translation, sentiment analysis, etc.) by freezing or partially unfreezing the pre-trained encoder and training a task-specific head on top. The shared multilingual representations provide a strong initialization that requires minimal labeled data for fine-tuning compared to training from scratch.