spaCy

Constructs NLP processing pipelines by declaratively composing named components (tagger, parser, NER, textcat, etc.) in a TOML-based `.cfg` configuration file with no hidden defaults. Each component processes Doc objects sequentially, enabling reproducible, version-controlled NLP workflows. Configuration specifies component order, hyperparameters, batch sizes, and GPU allocation, making training runs fully transparent and auditable.

Provides 84 pre-trained statistical and transformer-based pipelines across 25 languages, enabling immediate tokenization, POS tagging, dependency parsing, lemmatization, and NER without training. Pipelines are language-specific (e.g., `en_core_web_sm`, `de_core_news_md`) and optimized for speed via Cython-based tokenization and efficient memory management. Supports both CPU-based statistical models and GPU-accelerated transformer models (BERT, etc.) for higher accuracy.

Categorizes arbitrary text spans (not just named entities) into user-defined categories via a trainable span categorization component. Unlike NER which identifies entity boundaries, span categorization assumes span boundaries are known (e.g., from NER or manual annotation) and assigns categories to spans. Supports overlapping spans and multiple categories per span. Enables tasks like aspect-based sentiment analysis, attribute extraction, or fine-grained entity typing.

Segments text into sentences by detecting sentence boundaries (periods, question marks, exclamation marks, newlines). Uses rule-based heuristics and optional neural models for ambiguous cases (e.g., abbreviations like 'Dr.' or 'U.S.'). Sentence boundaries are marked in Doc objects, enabling downstream components to process sentences independently. Supports custom sentence segmentation rules via component configuration.

Provides pre-built project templates for common NLP tasks (NER, text classification, relation extraction, etc.) that can be cloned and customized. Templates include directory structure, configuration files, training scripts, and evaluation code, enabling developers to start with a working end-to-end workflow rather than building from scratch. Templates are version-controlled and can be extended with custom components or data.

Provides built-in visualizers for displaying linguistic annotations (dependency trees, named entities, text classifications) in interactive HTML or Jupyter notebooks. Visualizers render Doc objects with color-coded entities, dependency arcs, and annotations, enabling debugging and explanation of model predictions. Supports custom styling and filtering of visualizations.

Packages trained spaCy pipelines as distributable Python packages (wheels, tarballs) that can be installed via pip. Enables versioning, dependency management, and easy deployment to production environments. Packaged models include all trained components, configuration, and metadata; can be installed as `pip install spacy-model-name` and loaded via `spacy.load()`. Supports model versioning and compatibility checking.

Integrates large language models (via spacy-llm package) for few-shot and zero-shot NLP tasks without requiring training data. LLMs are used as components in the pipeline, enabling tasks like entity extraction, text classification, and relation extraction using natural language prompts instead of labeled training data.

Provides pretrained models and language-specific components for 75+ languages, enabling NLP pipelines to process text in diverse languages with language-specific tokenization, POS tagging, parsing, and NER. Language selection is automatic based on model choice or explicit in pipeline configuration.

Implements a trainable NER component that learns to identify and classify custom entity types from annotated text. Uses a neural network architecture (Thinc-based) trained via the configuration system with configurable batch sizes, learning rates, and dropout. Supports both statistical models and transformer-based models; enables users to define arbitrary entity types beyond pre-trained categories (e.g., custom 'PRODUCT', 'COMPETITOR' types). Training requires annotated data in spaCy's JSON format or via the Prodigy annotation tool.

Performs dependency parsing to extract grammatical relationships (subject-verb-object, modifiers, etc.) from sentences, producing a directed acyclic graph of syntactic dependencies. Uses a transition-based neural parser trained via the configuration system; outputs include head tokens, dependency labels (nsubj, dobj, etc.), and subtree information. Enables syntactic tree visualization and programmatic access to sentence structure for downstream NLP tasks like relation extraction or semantic analysis.

Provides a trainable text classification component supporting both multi-class (one label per document) and multi-label (multiple labels per document) scenarios. Uses a neural network architecture trained via the configuration system with configurable thresholds, class weights, and loss functions. Enables classification at document or span level; integrates with the pipeline to classify entire documents or specific text spans. Supports both statistical and transformer-based models.

Integrates Large Language Models into spaCy pipelines via the `spacy-llm` package, enabling LLM-based task execution (NER, classification, relation extraction) without training data. Uses a modular prompting system to convert unstructured LLM responses into robust spaCy-compatible outputs (Doc objects with entities, classifications, etc.). Supports multiple LLM providers (specific providers UNKNOWN from documentation) and enables few-shot prompting for task adaptation. Eliminates need for annotated training data by leveraging LLM zero-shot or few-shot capabilities.

Enables developers to define custom NLP components that integrate into the spaCy pipeline via a component interface. Custom components receive Doc objects, perform arbitrary processing, and return modified Doc objects; can add custom attributes, annotations, or external API calls. Components are registered by name and configured via the `.cfg` file, enabling non-developers to enable/disable or configure custom components without code changes. Supports integration with external ML frameworks (PyTorch, TensorFlow) and APIs.

Processes multiple documents in batches via the `nlp.pipe()` method, enabling efficient processing of large document collections. Batch size is configurable in the `.cfg` file (e.g., `batch_size = 1000`); larger batches improve throughput but increase memory usage. Supports GPU acceleration for transformer-based models; automatically distributes computation across available GPUs. Enables streaming processing of large datasets without loading entire corpus into memory.

Links named entities (extracted by NER) to entries in external knowledge bases (e.g., Wikipedia, Wikidata, custom databases) via entity disambiguation. Uses a neural entity linker trained on entity mention-to-KB-entry pairs; performs candidate generation (retrieve potential KB entries for an entity mention) and ranking (score candidates to select best match). Enables enriching extracted entities with structured information (Wikipedia URLs, entity IDs, properties) from knowledge bases.

Performs morphological analysis to extract morphological features (part-of-speech, case, tense, number, etc.) and lemmatization to reduce words to their base forms. Uses a trainable lemmatizer component (rule-based or neural) configured via `.cfg` files. Morphological features are language-specific and extracted from pre-trained models or custom training. Enables downstream tasks like information extraction or text normalization that benefit from lemmatized forms.