dlt (data load tool)

dlt provides a Pipeline class that acts as a central orchestrator managing the complete ETL lifecycle through three sequential stages: extract (data ingestion), normalize (schema inference and transformation), and load (destination writing). The Pipeline class holds runtime context, manages state persistence, and sequences stage execution with built-in retry logic and error handling. Configuration resolution uses a decorator-based system (@with_config) that binds pipeline parameters to config files and environment variables, enabling environment-agnostic pipeline definitions.

dlt automatically infers schemas from source data during extraction using a built-in type system that maps Python types to destination-specific SQL types. The schema architecture supports evolution — new columns are detected and added automatically, and type changes are tracked. Schema inference happens during the normalize stage, which parses extracted data and generates table definitions without requiring manual schema specification. The type inference system handles nested structures, nullable fields, and precision constraints, with destination-specific type mapping (e.g., BigQuery TIMESTAMP vs Snowflake TIMESTAMP_NTZ).

dlt provides a command-line interface for initializing pipelines, managing pipeline state, and deploying to cloud platforms. The CLI supports commands for creating new pipelines (dlt init), running pipelines (dlt run), inspecting state (dlt state), and deploying to Airflow or cloud functions. The init command scaffolds pipeline code with source templates, reducing boilerplate. The CLI integrates with the configuration system, allowing environment-specific deployments without code changes. Deployment commands generate Airflow DAGs or cloud function definitions from pipeline code, enabling serverless execution.

dlt provides a library of verified sources (pre-built connectors) for popular SaaS platforms (Stripe, Salesforce, HubSpot, GitHub, etc.) and databases. These sources encapsulate API integration logic, pagination handling, authentication, and schema definitions, reducing development time for common data sources. Verified sources are maintained by the dlt community and tested against source APIs, ensuring reliability. Developers can use verified sources directly or customize them for specific needs. The sources are published in a central registry and can be discovered via the CLI or documentation.

dlt provides built-in tracing and telemetry that captures pipeline execution metrics, logs, and errors. The system tracks execution time, data volumes, schema changes, and load statistics, providing visibility into pipeline performance and health. Telemetry is sent to dlt's cloud platform for centralized monitoring and alerting (optional). The tracing system integrates with Python's logging module, allowing custom log handlers and log level configuration. Execution metadata is stored in the pipeline's state, enabling historical analysis of pipeline runs.

dlt supports loading data to vector databases (Weaviate, Qdrant, Pinecone, LanceDB) with automatic embedding generation and storage. The system can generate embeddings from text fields using OpenAI, Hugging Face, or other embedding models, and store them alongside original data in vector databases. Vector database destinations handle schema mapping, embedding storage, and similarity search configuration. This enables building RAG (retrieval-augmented generation) systems and semantic search applications directly from dlt pipelines.

dlt provides an Incremental class that tracks state across pipeline runs to load only new or modified data from sources. The system stores state (e.g., last_updated timestamp, max_id) in the pipeline's state store and uses it to filter source data on subsequent runs. State is persisted after each successful load and can be restored from the destination if local state is lost. The incremental loading mechanism integrates with the pipe system, allowing transformers to access state and apply filtering logic. This enables efficient loading of large datasets by avoiding full re-extraction on each run.

dlt provides a REST API source that handles common API patterns including pagination (offset, cursor, page-based), authentication (API keys, OAuth, basic auth), and retry logic with exponential backoff. The REST API integration uses a declarative configuration approach where developers specify endpoint URLs, pagination parameters, and authentication details, and dlt automatically handles pagination state, rate limiting, and transient failures. The system supports nested resource extraction (e.g., fetching related records from multiple endpoints) through the pipe system, enabling complex multi-endpoint data collection in a single pipeline.

dlt provides a SQL database source that connects to relational databases (PostgreSQL, MySQL, SQL Server, etc.) and automatically discovers tables, columns, and relationships. The system supports table filtering (include/exclude patterns), column selection, and incremental loading based on modification timestamps or primary keys. The SQL source integrates with the pipe system to enable transformations on extracted data before loading. Database connections are managed through SQLAlchemy, supporting a wide range of database engines with consistent configuration and credential management.

dlt's pipe system provides a composable data transformation framework where transformers are generator functions that receive data from upstream sources or pipes and yield transformed records. Transformers integrate with the extraction stage, enabling in-flight transformations before data reaches the normalize stage. The pipe system supports chaining multiple transformers, accessing pipeline state and context, and implementing custom business logic (filtering, enrichment, aggregation). Transformers are executed within the extraction stage using a pool runner that can parallelize transformer execution across multiple workers.

dlt supports loading data to multiple destinations (PostgreSQL, BigQuery, Snowflake, Databricks, DuckDB, Athena, ClickHouse, vector databases) with configurable write dispositions that control how data is written: replace (truncate and reload), append (insert new records), or merge (upsert based on primary keys). The load stage uses destination-specific job clients that generate and execute DDL/DML statements optimized for each destination. Write dispositions are applied at the table level, enabling different strategies for different tables in the same pipeline. The system handles schema creation, data type mapping, and destination-specific optimizations (e.g., BigQuery clustering, Snowflake clustering).

dlt's normalize stage transforms extracted data (often nested JSON) into flat, relational tables with automatic handling of nested objects and arrays. The normalization process infers schemas from data, creates parent-child relationships for nested structures, and generates normalized table definitions. The system handles deeply nested structures by creating separate tables for nested arrays and linking them via foreign keys. Normalization happens automatically after extraction and before loading, eliminating manual data flattening logic. The normalize stage is configurable, allowing control over table naming, column naming, and nesting depth.

dlt provides a configuration system that resolves pipeline parameters from multiple sources (config files, environment variables, Python code) with a clear precedence order. Secrets are managed separately from configuration, supporting secure storage in environment variables, .dlt/secrets.toml files, or external secret managers. The configuration system uses a decorator-based approach (@with_config) that binds function parameters to configuration specs, enabling environment-agnostic code. Configuration is organized into sections (PIPELINES, SOURCES, DESTINATIONS) and supports nested configuration for complex settings. The system validates configuration at runtime and provides clear error messages for missing or invalid settings.

dlt automatically persists pipeline state after each successful load, storing metadata like last_updated timestamps, max_ids, and execution history. State can be restored from the local filesystem or from the destination database if local state is lost, enabling recovery from failures without manual intervention. The state system integrates with incremental loading, allowing pipelines to resume from the last successful checkpoint. State is stored in a .dlt directory and can be synced to the destination for distributed execution. The system provides state inspection and manipulation commands for debugging and recovery.