spacy

Breaks raw text into tokens using a Cython-compiled tokenizer (spacy/tokenizer.pyx) that applies language-specific exception rules and morphological boundaries. The tokenizer maintains a rule registry per language and uses finite-state matching to handle contractions, punctuation, and special cases (e.g., 'don't' → ['do', "n't"]). Tokens are stored as lightweight views into a Doc's underlying TokenC struct array, enabling zero-copy access to token attributes.

Implements a transition-based dependency parser (spacy/pipeline/parser.pyx) that uses a neural network to predict syntactic head-dependent relationships. The parser maintains a shift-reduce state machine, processing tokens left-to-right and predicting transitions (shift, left-arc, right-arc) via a feed-forward or transformer-based neural model. Parsed dependencies are stored in the Doc's head and dep attributes, enabling downstream tasks like relation extraction and semantic role labeling.

Maintains language-specific data (tokenization rules, morphological features, stop words, lemmatization rules) in JSON files (website/meta/languages.json) that are loaded at runtime. Each language has a Language subclass (e.g., English, German, French) that defines language-specific tokenization exceptions and morphological rules. Users can add custom languages by creating a new Language subclass and registering it with @Language.factory. The system supports 70+ languages with unified API despite diverse linguistic properties.

Serializes trained models to disk in a binary format that preserves all components, configuration, and weights. Models are saved as directories containing component files (e.g., model.pkl for neural weights), config.cfg, and metadata.json. Deserialization loads the model back into memory with all components ready for inference. The system supports incremental model updates (e.g., adding new entities to NER without retraining) via component-level serialization.

Allows users to attach custom attributes to Token, Doc, and Span objects via the extension system (Token.set_extension, Doc.set_extension, Span.set_extension). Extensions can be properties (computed on-the-fly), attributes (stored in memory), or methods. Extensions are registered globally and available on all instances of the target class. This enables adding domain-specific metadata (e.g., sentiment scores, custom NER labels) without modifying spaCy's core classes.

Provides batch processing via the nlp.pipe() method that processes multiple documents efficiently by batching them through the pipeline. Internally, spaCy uses DocBin format to store multiple Doc objects in a single binary file, enabling efficient serialization and deserialization. The system supports streaming processing where documents are yielded as they're processed, enabling memory-efficient handling of large corpora.

Combines two NER approaches: (1) neural sequence labeling via a BiLSTM or transformer model that predicts BIO tags (Begin, Inside, Outside) for each token, and (2) rule-based matching using PhraseMatcher and Matcher for pattern-based entity extraction. Neural predictions are stored in the Doc's ents attribute; rule-based matches can be added via EntityRuler pipeline component. Both approaches integrate into a unified Doc.ents interface, allowing hybrid NER systems.

Assigns part-of-speech (POS) tags and morphological features (tense, mood, case, gender, number) to each token using a statistical tagger trained on annotated corpora. The tagger uses a feed-forward neural network or transformer to predict tags based on word embeddings and context. Morphological features are stored in the Token.morph attribute as a MorphAnalysis object, enabling fine-grained linguistic analysis. The system supports 70+ languages with language-specific tagsets (e.g., Universal Dependencies).

Links named entities to entries in a knowledge base (KB) using a learned entity linker that scores candidate entities based on context and entity similarity. The linker uses a neural model to compute entity embeddings and context embeddings, then ranks KB candidates by similarity. Linked entities are stored in the Doc.ents with kb_id attributes. The KB is a custom data structure (KnowledgeBase class) that stores entity vectors and aliases, enabling fast candidate retrieval.

Allows users to compose custom NLP pipelines by registering and chaining pipeline components (tokenizer, tagger, parser, NER, etc.) via a factory pattern. Each component is a callable that takes a Doc and returns a modified Doc. The Language class maintains a pipeline list and executes components sequentially, with each component's output feeding into the next. Components can be enabled/disabled, and the pipeline can be serialized/deserialized with configuration files (config.cfg in INI format).

Provides two pattern-matching engines: (1) Matcher for token-level patterns using attribute-based rules (e.g., 'find tokens where POS=VERB followed by NOUN'), and (2) PhraseMatcher for fast phrase matching using a trie-based algorithm. Both engines return Span objects with start/end positions and match IDs. Patterns are defined as lists of token dictionaries or phrase strings, compiled into finite-state automata for efficient matching. Matches can be used for entity extraction, relation extraction, or custom annotations.

Provides a TextCategorizer pipeline component that uses a neural model (feed-forward or transformer-based) to classify documents or sentences into predefined categories. The model learns from annotated training data via backpropagation. Classification scores are stored in the Doc.cats attribute as a dictionary mapping category names to confidence scores (0-1). Supports multi-class and multi-label classification. Users can train custom classifiers on domain-specific data using the training API.

Stores pre-trained word vectors (embeddings) in a Vectors object that maps words to dense vectors (typically 300-dim). Vectors are loaded from external sources (e.g., Word2Vec, GloVe, fastText) and integrated into the Vocab. Token similarity is computed via cosine distance between vectors. The Doc and Span classes provide similarity() methods that average token vectors. Vectors are memory-mapped for efficient loading of large models.

Provides a training API and CLI (spacy train) that uses configuration files (config.cfg) to define training hyperparameters, model architecture, and data paths. Training uses a callback-based system where components are updated via backpropagation. The system supports multi-task learning (training multiple components jointly, e.g., NER + POS tagging). Training progress is logged and models are evaluated on validation data. Trained models are serialized to disk with all components and configuration.