dlt

Automatically infers table schemas from semi-structured JSON data by analyzing record samples and building a type hierarchy that captures nested objects and arrays as separate normalized tables. Uses a recursive type inference engine that maps JSON structures to SQL-compatible column types, handling deeply nested payloads without manual schema definition. The schema architecture evolves as new data patterns are encountered, automatically adding columns and creating child tables for nested arrays.

Tracks extraction state (cursors, timestamps, IDs) across pipeline runs to load only new or modified records since the last execution. Implements a state sync mechanism that persists cursor positions in the destination and restores them on pipeline restart, enabling efficient incremental loads from APIs and databases without full refreshes. The state context is managed per pipeline and supports both timestamp-based and ID-based incremental strategies through the Incremental class.

Manages sensitive credentials (API keys, database passwords, cloud credentials) through a hierarchical configuration system that resolves secrets from environment variables, .dlt/secrets.toml files, or cloud secret managers. The configuration system uses @with_config decorators to inject resolved credentials into pipeline functions without exposing them in code. Secrets are never logged or persisted in pipeline state, ensuring security compliance.

Provides built-in tracing and telemetry that captures pipeline execution metrics (duration, records processed, errors) and logs them to stdout, files, or external observability platforms. The tracing system instruments extract, normalize, and load stages with timing information and error context, enabling debugging and performance optimization. Telemetry can be configured to send metrics to Datadog, New Relic, or other APM platforms.

Provides decorators and utilities to convert dlt pipelines into Airflow DAGs with automatic task generation for extract, normalize, and load stages. The Airflow integration handles credential injection, state management, and error recovery within Airflow's execution model. Developers can use @dlt.resource decorators to define sources and dlt.run() to execute pipelines as Airflow tasks, with Airflow managing scheduling, retries, and monitoring.

Loads extracted and normalized data into 30+ destinations (Snowflake, BigQuery, Databricks, DuckDB, Postgres, Athena, ClickHouse, vector DBs, filesystems) with configurable write strategies: replace (full refresh), append (insert-only), or merge (upsert with deduplication). The load stage architecture uses job clients that translate normalized data into destination-specific formats and SQL dialects, with write disposition logic determining how records are written or updated. Each destination has a specialized client (e.g., BigQuery client, Snowflake client) that handles authentication, batching, and error recovery.

Provides a declarative REST API source interface that handles pagination, authentication (OAuth, API keys, basic auth), rate limiting, and request retries automatically. The REST API integration uses a schema-based approach where endpoint definitions specify pagination strategy (offset, cursor, keyset), authentication method, and response structure. Internally, the pipe system iterates through paginated responses, yielding records to the extraction pipeline while managing connection state and error recovery.

Extracts data from SQL databases (Postgres, MySQL, Snowflake, etc.) with automatic table discovery, schema reflection, and change data capture (CDC) support. The SQL database source uses database introspection to discover tables and columns, then generates extraction queries that can be incremental (using timestamps or LSN-based CDC) or full refresh. The pipe system manages connection pooling and query execution, yielding rows as normalized records to the extraction pipeline.

Transforms raw extracted records into normalized relational tables by flattening nested objects and arrays into separate child tables with foreign key relationships. The normalization stage (dlt/normalize/normalize.py) processes extracted data through a configurable normalizer that detects nested structures, creates child tables, and maintains referential integrity through synthetic keys. This enables storing complex JSON in SQL-compatible schemas without losing data relationships.

Provides a Pipeline class that orchestrates extract, normalize, and load stages in sequence, with configuration resolution from files, environment variables, and code. The pipeline factory functions (pipeline(), attach(), run()) create or retrieve pipeline instances that manage runtime context, state, and execution flow. Configuration is declarative via @with_config decorators and TOML/YAML files, allowing pipeline behavior to be changed without code changes. The pipeline execution model supports both synchronous runs and async execution via Airflow integration.

Loads normalized data into vector databases (Weaviate, Pinecone, Qdrant, LanceDB) with automatic embedding generation and semantic search indexing. The vector database destination client handles embedding computation (via OpenAI, Hugging Face, or local models), chunking of text fields, and insertion into vector indices with metadata. This enables building RAG (Retrieval-Augmented Generation) systems where extracted data is automatically indexed for semantic search.

Loads data into cloud storage (S3, GCS, Azure Blob) or local filesystems as Parquet, JSON, or CSV files with configurable partitioning by date or column values. The filesystem destination client handles file format conversion, partitioning logic, and cloud storage authentication. Data is organized into directory structures (e.g., s3://bucket/dataset/table/year=2024/month=01/) enabling efficient querying via Athena, BigQuery external tables, or Spark.

Implements a composable pipe system (dlt/extract/pipe.py) that chains extraction, filtering, and transformation operations with optional parallelization. Pipes are generator-based iterables that yield records through a chain of transformers, with support for concurrent execution via thread pools or process pools. The pipe iterator manages backpressure and batching, allowing efficient processing of large datasets without loading everything into memory.