span-marker-mbert-base-multinerd

Performs token-level classification using a span-marker architecture built on mBERT (multilingual BERT), enabling detection and classification of named entities across 10+ languages simultaneously. The model uses a two-stage span-based approach: first identifying entity boundaries via token classification, then assigning entity type labels to detected spans. This differs from traditional sequence labeling by operating on variable-length spans rather than individual tokens, reducing cascading errors from boundary misalignment.

Leverages mBERT's multilingual embedding space to classify entity types consistently across languages without language-specific fine-tuning. The model encodes text through mBERT's 12 transformer layers, projecting tokens into a shared 768-dimensional space where entity semantics align across languages. This enables zero-shot or few-shot entity classification for languages not explicitly seen during training, as long as they're covered by mBERT's 104-language pretraining.

Classifies detected entities into 10+ distinct entity types (person, organization, location, product, event, etc.) as defined by the MultiNERD dataset, enabling fine-grained information extraction beyond simple binary entity/non-entity classification. The model learns type-specific patterns through supervised training on MultiNERD's annotated corpus, using mBERT's contextual representations to disambiguate entities with identical surface forms but different types (e.g., 'Apple' as company vs. fruit).

Processes multiple documents or long documents through efficient span enumeration, where the model identifies all possible entity spans (up to a configurable maximum length, typically 8-10 tokens) and classifies each span's entity type. This approach avoids redundant token-level computations by leveraging mBERT's contextual representations across the entire document, then scoring spans post-hoc. Batch processing is optimized through padding and masking to handle variable-length inputs efficiently.

Exposes mBERT's intermediate layer representations (768-dimensional contextual embeddings) for each detected entity span, enabling downstream tasks like entity linking, coreference resolution, or entity similarity matching. The model outputs not just entity type labels but also the pooled contextual representation of each entity span, computed by averaging mBERT's hidden states across the span's tokens. These representations capture semantic and syntactic context, enabling vector-based entity operations.

Uses safetensors format for model weights instead of traditional PyTorch pickle format, enabling faster model loading, reduced memory overhead, and protection against arbitrary code execution during deserialization. Safetensors is a binary format that stores tensor data with explicit type and shape information, allowing zero-copy memory mapping on compatible systems. The model is distributed as a single safetensors file, eliminating the need for separate config and weight files.

Leverages mBERT's 119K shared vocabulary across 104 languages, enabling consistent tokenization of multilingual text without language-specific tokenizers. The WordPiece tokenizer handles subword segmentation for out-of-vocabulary words, preserving morphological information across languages. This unified tokenization approach ensures that entities in different languages are represented in a shared token space, enabling the span-marker model to apply consistent entity classification rules across languages.