tokenizers

Implements Byte Pair Encoding (BPE) algorithm in Rust with FFI bindings to Python and Node.js, achieving 10-100x faster tokenization than pure Python implementations. The Rust core uses efficient data structures and memory management to process text into token IDs and offsets, with the tokenization pipeline flowing through normalizers, pre-tokenizers, and post-processors as composable stages.

Implements WordPiece algorithm (used by BERT, DistilBERT) that greedily matches the longest subword tokens from a vocabulary, prefixing continuation tokens with '##' to indicate non-initial positions. The algorithm processes pre-tokenized words character-by-character, falling back to [UNK] tokens for out-of-vocabulary subwords, enabling efficient representation of rare words and morphological variants.

Provides language-specific bindings that expose the Rust core to Python and Node.js via PyO3 and napi-rs FFI technologies. PyO3 bindings use Arc<RwLock> for thread-safe shared state and integrate with tokio for async support; napi-rs bindings compile to native addons for multiple platforms (Linux gnu/musl, Windows, macOS, Android). Both bindings maintain API parity with the Rust core while providing idiomatic interfaces for each language.

Supports efficient batch tokenization of multiple texts simultaneously, with optional parallelization across CPU cores. The batch API accepts lists of strings and returns lists of Encoding objects, with internal parallelization via Rayon (Rust) or thread pools. Batch processing reduces per-text overhead and enables better CPU cache utilization compared to sequential tokenization.

Returns Encoding objects that encapsulate complete tokenization results: token IDs, token strings, character offsets, attention masks, token type IDs (for sequence pairs), and special token positions. The Encoding structure provides convenient accessors for common operations (e.g., getting tokens for a span, padding to length) and supports serialization to/from dictionaries for integration with ML frameworks.

Implements decoders that reconstruct original text from token sequences, reversing the tokenization process. Different decoders handle different tokenization schemes: BPE decoder removes ## markers and merges subword tokens, WordPiece decoder handles ## continuation markers, Unigram decoder reconstructs from byte-level tokens. Decoders support optional space insertion and special character handling.

Implements Unigram tokenization (used by SentencePiece) that models tokenization as a probabilistic process where each token has an associated loss value. During encoding, the algorithm finds the most likely tokenization sequence that minimizes loss, and during training, iteratively removes low-loss tokens from the vocabulary. This approach naturally handles variable-length tokens and rare characters without explicit [UNK] fallback.

Implements the simplest tokenization strategy: direct vocabulary lookup where each whitespace-separated word maps to a token ID, with [UNK] for out-of-vocabulary words. This approach requires explicit pre-tokenization and is primarily used for legacy models or as a baseline, but provides maximum interpretability and minimal computational overhead.

Provides a modular pipeline where text flows through configurable stages: Normalizer (Unicode normalization, lowercasing, accent removal), PreTokenizer (whitespace/punctuation splitting, language-specific segmentation), Model (BPE/WordPiece/Unigram/WordLevel), PostProcessor (adding special tokens like [CLS]/[SEP], handling sequence pairs), and Decoder (reconstructing text from tokens). Each stage is independently composable, allowing users to build custom tokenizers by chaining components.

Implements BPE training algorithm that iteratively merges the most frequent byte/character pairs in a corpus to build a vocabulary. The algorithm starts with character-level tokens, counts pair frequencies, merges the top-frequency pair, and repeats until reaching the target vocabulary size. Training supports byte-level BPE (for any Unicode text) and character-level BPE, with configurable minimum frequency thresholds and special token handling.

Implements Unigram language model training using Expectation-Maximization (EM) to optimize token loss values. The algorithm initializes vocabulary with frequent substrings, computes token loss via forward-backward algorithm, and iteratively removes low-loss tokens until reaching target vocabulary size. This approach naturally balances vocabulary coverage and compression efficiency.

Implements training for WordPiece and WordLevel tokenizers by computing subword statistics from a pre-tokenized corpus. For WordPiece, the algorithm identifies frequent subword pairs and builds a vocabulary with ## continuation markers; for WordLevel, it simply counts word frequencies and selects the top-K words. Both approaches support minimum frequency thresholds and special token handling.

Implements save/load functionality for tokenizers via JSON configuration files that capture the complete pipeline state: normalizer settings, pre-tokenizer rules, model parameters (vocabulary, merge rules, loss values), post-processor configuration, and decoder settings. Serialization enables reproducible tokenization across environments and version control of tokenizer configurations.

Tracks character-level offsets (start/end positions in original text) for each token, enabling reverse mapping from token positions back to original text spans. The Encoding object stores offset tuples for each token, allowing users to extract original text for specific tokens or identify which tokens correspond to a given character range. This is essential for entity extraction, question answering, and other span-based NLP tasks.