Multilingual Language Routing Via Mbart Tokenizer

via “multilingual tokenization with mbert's shared vocabulary”

token-classification model by undefined. 2,49,148 downloads.

Unique: Uses mBERT's 119K shared vocabulary across 104 languages, enabling unified tokenization without language detection; WordPiece subword segmentation preserves morphological information across language families (e.g., Germanic, Romance, Slavic)

vs others: Simpler than language-specific tokenizer pipelines while maintaining reasonable compression; more consistent across languages than separate tokenizers, reducing entity boundary misalignment

via “multilingual-language-routing-via-mbart-tokenizer”

summarization model by undefined. 40,872 downloads.

Unique: Inherits mBART's language-agnostic encoder-decoder design where language tokens are embedded in the tokenizer vocabulary, enabling zero-shot language routing without separate language classifiers or routing logic

vs others: Single model handles 25 languages vs maintaining 25 separate models, reducing deployment complexity and memory footprint, but with performance trade-offs compared to language-specific models like Italian-BERT