Databricks

Combines data warehouse and data lake architectures using Delta Lake as the underlying open format, enabling ACID transactions, schema enforcement, and time-travel queries on unstructured and structured data in cloud object storage. Implements a metadata layer that tracks data lineage and versioning, allowing rollback to previous states and concurrent read/write operations without data corruption.

Executes SQL queries across distributed Spark clusters using a vectorized query engine (Photon) that processes data in columnar batches rather than row-by-row, leveraging SIMD CPU instructions and GPU acceleration for 5-10x faster analytics queries. Automatically optimizes query plans based on data statistics and partitioning, with support for complex joins, aggregations, and window functions across petabyte-scale datasets.

Managed Postgres database that integrates with Databricks lakehouse, allowing transactional OLTP workloads to coexist with analytical OLAP workloads in the same system. Lakebase stores data in Delta Lake format, enabling direct querying from Spark while maintaining Postgres compatibility for applications. Automatically syncs data between Postgres and Delta Lake tables, eliminating manual ETL between transactional and analytical systems.

Provides API access to pre-trained large language models (LLMs) hosted on Databricks infrastructure, including open-source models (Llama 2, Mistral) and proprietary models. Models are served via REST endpoints with support for streaming responses, token counting, and batch inference. Pricing is per-token (input and output), with volume discounts for high-volume usage. Models are deployed in Databricks data centers, ensuring data privacy (no data sent to external LLM providers).

Suite of tools for building, evaluating, and deploying generative AI applications. Includes prompt engineering tools (prompt versioning, A/B testing), evaluation frameworks (automated metrics for quality, safety, cost), and deployment orchestration. Integrates with Foundation Models API and external LLM providers (OpenAI, Anthropic). Provides pre-built evaluation metrics (BLEU, ROUGE, semantic similarity) and custom evaluation support via Python functions.

Web-based notebooks (similar to Jupyter) with real-time collaborative editing, allowing multiple users to edit the same notebook simultaneously. Includes built-in version control with commit history, branching, and rollback capabilities. Notebooks are stored in Git-compatible format, enabling integration with GitHub/GitLab for CI/CD. Supports multiple languages (Python, SQL, R, Scala) in the same notebook with automatic language detection.

Organizes users and resources into isolated workspaces with separate compute clusters, data, and configurations. Implements role-based access control (RBAC) with predefined roles (Admin, Analyst, Engineer) and custom roles. Enables fine-grained permissions at the workspace, cluster, job, and notebook levels. Supports SSO integration with external identity providers (Azure AD, Okta, SAML) for centralized user management.

Provides integrated experiment tracking, model versioning, and model registry built on MLflow, allowing data scientists to log hyperparameters, metrics, and artifacts during training runs, compare experiments side-by-side, and promote models through development/staging/production stages. Automatically captures code snapshots, dependencies, and environment configurations, enabling reproducible model training and easy rollback to previous model versions.

Deploys trained models as REST endpoints that automatically scale based on request volume, with built-in support for A/B testing, canary deployments, and traffic routing policies. Handles model loading, batching, and GPU allocation transparently, eliminating manual infrastructure management. Supports multiple model formats (MLflow, ONNX, custom Python) and frameworks (scikit-learn, TensorFlow, PyTorch, LLMs) with automatic dependency resolution.

Centralized repository for ML features with automatic versioning, point-in-time joins (preventing data leakage), and feature lineage tracking. Stores computed features in Delta Lake tables with metadata about feature definitions, transformations, and dependencies. Enables feature reuse across models and teams, with automatic feature freshness management and backfill capabilities for historical feature values.

Automatically generates candidate models by testing multiple algorithms (gradient boosting, neural networks, linear models) with different hyperparameter configurations, feature engineering strategies, and data preprocessing pipelines. Evaluates each candidate on holdout validation sets and ranks them by performance metrics, returning the best model as an MLflow-registered artifact. Requires minimal configuration beyond specifying target column and problem type (classification/regression).

Converts natural language questions into SQL queries and visualizations without requiring SQL knowledge, using an LLM-powered semantic layer that understands table schemas, column semantics, and business context. Users ask questions in English (e.g., 'What was revenue by region last quarter?'), and Genie generates SQL, executes it, and returns charts/tables. Learns from user feedback to improve query generation accuracy over time.

Framework for building autonomous AI agents that can decompose tasks, call external tools/APIs, and iterate based on feedback. Agents are defined as Python classes with tool registries, memory management, and evaluation metrics. Includes built-in support for tool calling via function schemas, multi-step reasoning with chain-of-thought, and continuous improvement through A/B testing and user feedback collection. Agents are deployed as REST endpoints via Model Serving.

Orchestrates data pipelines (ETL/ELT) for both batch and streaming data sources, with automatic scheduling, error handling, and data quality checks. Pipelines are defined as DAGs (directed acyclic graphs) with tasks for data ingestion, transformation, and loading. Supports multiple data sources (databases, APIs, cloud storage, Kafka) and transformations (SQL, Python, Spark). Automatically tracks data lineage and enables rollback to previous pipeline states.

Centralized metadata and access control layer that governs data tables, ML models, dashboards, and AI agents across the entire Databricks workspace. Implements fine-grained access control (row-level, column-level, object-level), data classification, and audit logging. Enables cross-workspace sharing of governed assets and enforces compliance policies (data retention, PII masking, encryption). Integrates with external identity providers (ADFS, Okta) for SSO.