gensim

Decomposes document-term matrices using Singular Value Decomposition to discover latent semantic relationships between documents and terms. Gensim implements sparse SVD via ARPACK, reducing dimensionality while preserving semantic structure, enabling semantic search and document similarity without explicit keyword matching. The implementation handles large sparse matrices efficiently through iterative algorithms rather than dense matrix operations.

Probabilistic generative model that discovers latent topics in document collections using variational inference or Gibbs sampling. Gensim implements online LDA with mini-batch updates, allowing incremental model training on streaming data without reprocessing the entire corpus. The model learns per-document topic distributions and per-topic word distributions through iterative Bayesian inference, enabling dynamic topic discovery as new documents arrive.

Provides serialization and deserialization of trained models (embeddings, topic models, transformations) to disk for reproducibility and production deployment. Gensim implements model saving through pickle and custom binary formats, enabling models to be trained once and reused across multiple applications without retraining. The serialization preserves all learned parameters and statistics, enabling deterministic inference on new data.

Computes and tracks corpus-level statistics including document frequencies, term frequencies, vocabulary size, and term co-occurrence patterns. Gensim's Dictionary class maintains these statistics during corpus iteration, enabling analysis of vocabulary properties without materializing the full corpus. Statistics are used by downstream models (TF-IDF, LDA) to learn appropriate weighting and prior parameters.

Provides native support for reading and writing corpus data in Gensim-optimized formats (Matrix Market, SVMLight) that enable efficient storage and retrieval of sparse document-term matrices. These formats store only non-zero entries, reducing disk space and I/O overhead compared to dense formats. Gensim's corpus readers integrate with the corpus abstraction, enabling seamless iteration over files in these formats.

Provides optional similarity indexing through sparse matrix structures and integration with approximate nearest neighbor libraries (Annoy, FAISS) for efficient similarity queries on large corpora. Gensim's SparseMatrixSimilarity class enables fast similarity computation through sparse matrix multiplication, while optional indexing backends enable sublinear-time nearest neighbor search. This enables semantic search and recommendation systems to scale to millions of documents.

Non-parametric Bayesian topic model that automatically infers the optimal number of topics without manual specification, using a hierarchical Dirichlet process prior. Gensim implements HDP via variational inference, discovering topic hierarchies and sharing statistical strength across topics through the DP structure. Unlike LDA, HDP can grow the topic space dynamically as evidence warrants, making it suitable for exploratory analysis where topic count is unknown.

Learns dense vector representations of words by predicting context words (Skip-gram) or predicting target words from context (CBOW) using shallow neural networks. Gensim implements both architectures with negative sampling and hierarchical softmax for efficient training on large vocabularies. The model captures semantic and syntactic relationships in continuous vector space, enabling word analogy tasks and semantic similarity computation without explicit feature engineering.

Extends Word2Vec by representing words as bags of character n-grams, enabling embeddings for out-of-vocabulary (OOV) words and capturing morphological information. Gensim wraps the fastText algorithm, allowing words to be decomposed into subword units (e.g., 'running' = 'run' + 'nin' + 'ing' + special tokens), so unseen words get representations based on their character composition. This approach handles rare words, misspellings, and morphologically rich languages better than standard Word2Vec.

Learns fixed-size vector representations for entire documents by extending Word2Vec with a document ID token that acts as a memory of document context. Gensim implements both Distributed Memory (DM) and Distributed Bag-of-Words (DBOW) variants, training document vectors alongside word vectors through the same neural network objective. This enables semantic similarity between documents and document classification without explicit feature engineering.

Computes TF-IDF (Term Frequency-Inverse Document Frequency) weights for documents using corpus-wide statistics to identify important terms. Gensim implements TF-IDF as a transformation that learns IDF weights from a training corpus and applies them to new documents, supporting both standard TF-IDF and sublinear TF scaling. The implementation integrates with Gensim's corpus abstraction, enabling memory-efficient processing of large document collections.

Provides abstract corpus and dictionary interfaces that enable memory-efficient processing of document collections larger than RAM through lazy iteration and streaming. The Dictionary maps tokens to integer IDs and tracks corpus statistics, while the corpus abstraction allows documents to be processed one-at-a-time without loading the entire collection into memory. This architecture enables all Gensim models to work with arbitrarily large corpora by iterating through documents on-demand.

Computes semantic similarity between documents, words, or queries using learned representations (embeddings, topic distributions, or TF-IDF vectors). Gensim provides similarity interfaces that support multiple distance metrics (cosine, Euclidean, Jaccard) and enable efficient similarity queries through sparse matrix operations and optional indexing. The abstraction works with any vector representation, enabling similarity computation across different model types.

Enables chaining multiple transformations (TF-IDF, LSI, LDA, normalization) into sequential pipelines that process documents through multiple stages. Gensim implements transformations as objects that learn statistics from training data and apply transformations to new documents, supporting composition through the corpus iteration interface. This enables building complex NLP pipelines (e.g., tokenize → TF-IDF → LSI → similarity) without materializing intermediate representations.