Databricks

Databricks implements a lakehouse architecture that combines data warehouse and data lake capabilities using Delta Lake as the underlying format. This approach uses ACID transactions, schema enforcement, and time-travel capabilities on cloud object storage (S3, ADLS, GCS), eliminating the need for separate data warehouse and data lake systems. The architecture supports both batch and streaming workloads through a single unified metadata layer, enabling consistent data governance and query semantics across analytics and ML workloads.

Databricks provides distributed query execution across SQL, Python, Scala, and R through a unified Catalyst optimizer and Tungsten execution engine (inherited from Apache Spark). Queries are compiled to optimized physical plans that execute in parallel across a cluster, with automatic partitioning and shuffle optimization. The platform supports both interactive queries via notebooks and batch jobs, with query results cached in memory for interactive exploration and persisted to Delta Lake for reproducibility.

Databricks Mosaic AI provides a suite of tools for building enterprise generative AI applications, including model fine-tuning, RAG (retrieval-augmented generation) pipelines, and evaluation frameworks. The system enables organizations to fine-tune open-source LLMs (Llama, Mistral) on company data, build RAG systems that ground LLM responses in lakehouse data, and evaluate model quality with custom metrics. Mosaic AI integrates with Model Serving for deploying fine-tuned models and with Agent Bricks for building agents.

Databricks Lakebase provides a serverless PostgreSQL-compatible database integrated with the lakehouse, enabling transactional workloads (OLTP) alongside analytical workloads (OLAP) on the same data platform. Lakebase uses a shared storage architecture with Delta Lake, eliminating data duplication and enabling transactions on lakehouse data. The system automatically scales compute based on workload, with per-second billing and no cluster management required.

Databricks uses per-second billing for all compute resources (clusters, jobs, model serving), enabling organizations to pay only for resources actually used without upfront costs or minimum commitments. The platform offers Committed Use Contracts (CUCs) for volume discounts, with flexibility to apply commitments across multiple clouds (AWS, Azure, GCP) and products (compute, model serving, feature store). Billing is transparent with per-SKU pricing published for each cloud provider.

Web-based notebooks (similar to Jupyter) with real-time collaborative editing, allowing multiple users to edit the same notebook simultaneously. Includes built-in version control with commit history, branching, and rollback capabilities. Notebooks are stored in Git-compatible format, enabling integration with GitHub/GitLab for CI/CD. Supports multiple languages (Python, SQL, R, Scala) in the same notebook with automatic language detection.

Organizes users and resources into isolated workspaces with separate compute clusters, data, and configurations. Implements role-based access control (RBAC) with predefined roles (Admin, Analyst, Engineer) and custom roles. Enables fine-grained permissions at the workspace, cluster, job, and notebook levels. Supports SSO integration with external identity providers (Azure AD, Okta, SAML) for centralized user management.

Databricks integrates MLflow as a native model training and experiment tracking system, enabling data scientists to log hyperparameters, metrics, artifacts, and model versions during training runs. MLflow Tracking stores experiment metadata and model artifacts in the lakehouse, while MLflow Model Registry provides centralized model versioning, staging (dev/staging/production), and lineage tracking. The system automatically captures training context (code, environment, data versions) for reproducibility and enables comparison across experiment runs through a web UI.

Databricks Model Serving provides serverless inference endpoints for registered MLflow models, automatically scaling compute based on request volume without requiring manual cluster management. The service exposes models via REST API endpoints with built-in support for A/B testing (traffic splitting between model versions), request/response logging for monitoring, and integration with Unity Catalog for access control. Inference requests are routed to GPU or CPU compute depending on model type, with per-token billing for LLMs and per-request billing for other models.

Databricks Lakeflow provides a declarative workflow orchestration system for scheduling and executing batch ETL jobs and streaming pipelines. Jobs are defined as DAGs (directed acyclic graphs) with dependencies, retry logic, and notifications, executed on Databricks clusters with automatic cluster provisioning and teardown. The system supports both SQL and Python tasks, with built-in integration with Delta Lake for data versioning and Unity Catalog for governance, enabling end-to-end lineage tracking from source data to final output tables.

Databricks Unity Catalog provides a centralized metadata layer for managing data assets across the lakehouse, enabling role-based access control (RBAC), data classification, and lineage tracking. The system uses a three-level namespace (catalog.schema.table) to organize data, with fine-grained permissions at table and column levels. Unity Catalog integrates with cloud identity providers (Azure AD, Okta) for authentication and supports data masking, row-level security, and audit logging for compliance requirements.

Databricks Feature Store provides a centralized repository for managing ML features (computed attributes used in model training and inference), enabling feature reuse across multiple models and teams. Features are defined as SQL transformations on Delta Lake tables, with automatic computation and storage in the lakehouse. The system tracks feature lineage, versions, and metadata, enabling data scientists to discover and reuse features without duplicating computation logic. Feature Store integrates with MLflow to automatically capture feature versions used in training, enabling reproducible model training.

Databricks Genie provides a conversational AI interface that translates natural language questions into SQL queries executed against the lakehouse. The system uses LLMs (likely Claude or GPT-4 via API) to understand user intent, generate SQL, and explain results in natural language. Genie maintains conversation context across multiple turns, enabling follow-up questions and refinements without re-specifying the full query. The system integrates with Unity Catalog for access control, ensuring users only see results they have permission to access.

Databricks Agent Bricks provides a framework for building AI agents that can access data, tools, and models within the Databricks platform. Agents use LLMs (Claude, GPT-4) as the reasoning engine, with built-in integration for tool calling (function definitions), memory management (conversation history), and grounding in lakehouse data via RAG (retrieval-augmented generation). The framework handles agent orchestration, error handling, and logging, enabling developers to focus on defining agent capabilities rather than infrastructure.

Databricks AutoML automatically trains multiple ML models on a dataset, performs hyperparameter tuning, and recommends the best model based on performance metrics. The system supports classification, regression, and forecasting tasks, automatically handling feature engineering, model selection (linear models, tree-based models, neural networks), and hyperparameter optimization. AutoML generates a notebook with the best model's training code, enabling users to understand and modify the approach. Results are logged to MLflow for tracking and comparison.