lancedb

Executes approximate nearest neighbor search using state-of-the-art indexing strategies (IVF-PQ for large-scale partitioning and HNSW for hierarchical navigation). The Rust core implements Lance columnar format storage with zero-copy Arrow integration, enabling sub-millisecond queries over millions of vectors. Query execution pipeline applies vector distance metrics (L2, cosine, dot product) with optional scalar filtering and projection pushdown to minimize data materialization.

Provides BM25-based full-text search over text columns using inverted index construction and term frequency/inverse document frequency ranking. The implementation integrates with the Lance storage layer to co-locate FTS indexes alongside vector indexes, enabling hybrid queries that combine semantic and lexical relevance. Query execution applies tokenization, stemming, and relevance scoring without requiring external search engines like Elasticsearch.

Supports streaming inserts and updates via append-only operations that are automatically batched and indexed. New data is immediately queryable without explicit index rebuilds; incremental indexing updates existing indexes in the background. Streaming API accepts Arrow RecordBatch, Pandas DataFrames, or JSON-like dictionaries. Atomic transactions ensure consistency across vector and metadata columns.

Enforces Arrow schema validation on all data operations, automatically coercing compatible types (e.g., Python int to Arrow int64) and rejecting incompatible data. Schema is defined at table creation time and enforced on all inserts/updates. Type mismatches are reported with detailed error messages indicating the problematic column and expected type. Optional columns allow NULL values; required columns reject NULLs.

Integrates embedding models (OpenAI, Hugging Face, local models) directly into the database, enabling automatic vectorization of text during insert/update operations. Embedding functions are registered per-column and applied transparently; raw text is stored alongside embeddings for retrieval. Supports both synchronous and asynchronous embedding generation. Caching prevents duplicate embeddings for identical text.

Provides a fluent, chainable query builder API that constructs query execution plans without immediately executing them. Queries are lazily evaluated; execution is deferred until results are explicitly requested (e.g., .to_list(), .to_arrow()). The query builder supports method chaining for vector search, filtering, projection, limit, and offset operations. Query plans are optimized by the DataFusion query planner before execution.

Combines vector similarity scores and full-text search (BM25) scores using configurable fusion strategies (weighted sum, reciprocal rank fusion, or custom scoring functions). The query builder API accepts both vector and text queries, executes them in parallel against their respective indexes, and merges results using normalized scoring. Filtering and projection pushdown apply to the fused result set, reducing post-processing overhead.

Stores vectors, embeddings, raw multimodal data (images, videos, point clouds), and structured metadata in a single Lance table using Apache Arrow columnar format. Zero-copy semantics allow queries to access vectors and metadata without deserialization overhead. MVCC (multi-version concurrency control) versioning enables time-travel queries and atomic updates across vector and metadata columns, maintaining consistency without locks.

Applies SQL WHERE clauses and column projections directly to vector search queries, pushing filters and projections down to the storage layer for early elimination of non-matching rows. The query builder constructs a filter expression tree that is evaluated during index traversal (for indexed scalar columns) or during result materialization (for non-indexed columns), reducing the number of vectors that must be scored and returned.

Implements multi-version concurrency control (MVCC) at the storage layer, automatically creating immutable snapshots of table state on each write operation. Time-travel queries can retrieve data as it existed at a specific point in time by referencing version tags or timestamps. Version management is transparent to the application; no explicit snapshot creation is required. Compaction and garbage collection clean up old versions to reclaim disk space.

Creates and maintains B-tree and hash indexes on scalar (non-vector) columns to accelerate metadata filtering in vector queries. Index creation is asynchronous and non-blocking; queries can execute while indexes are being built. The query planner automatically selects indexed columns for filter pushdown, reducing the number of rows that must be scanned. Index statistics are maintained for cost-based query optimization.

Provides Python, Node.js, and Java SDKs that wrap a single high-performance Rust core via Foreign Function Interface (FFI) bindings. Each language SDK exposes idiomatic APIs (e.g., async/await in Node.js, context managers in Python) while delegating all compute-intensive operations (indexing, search, filtering) to the shared Rust implementation. FFI overhead is minimal (~1-2% per operation) due to batch processing and zero-copy Arrow data transfer.

Operates in 100% embedded mode (no server required) similar to SQLite, storing all data in a local directory with a single-file or multi-file Lance format. The Rust core runs in-process within the application, eliminating network latency and external dependencies. Suitable for development, testing, and edge deployments. Seamlessly upgrades to remote mode by pointing to a LanceDB Cloud instance without code changes.

Connects to remote LanceDB Cloud or self-hosted Lance server instances using connection strings. Namespaces provide logical table grouping and isolation within a single database instance, enabling multi-tenant deployments or organizational separation without separate database instances. Connection pooling and retry logic handle transient failures automatically. Authentication is supported via API keys.