Hopsworks

Hopsworks implements a dual-layer feature store architecture that separates online (low-latency serving) and offline (batch training) storage, with a unified query interface that supports point-in-time lookups via temporal versioning. Features are computed via Apache Spark or Flink pipelines and automatically materialized to both layers, enabling consistent feature access across training and inference while maintaining historical snapshots for reproducible model training datasets.

Hopsworks provides a declarative feature group abstraction that encapsulates feature definitions, schemas, and validation rules as first-class entities in the platform. Feature groups are defined via Python SDK with optional Great Expectations integration for data quality checks, and the platform automatically enforces schema evolution, detects breaking changes, and maintains lineage metadata linking features to source data and downstream models.

Hopsworks integrates with Great Expectations to define, execute, and monitor data quality checks on feature groups, with automatic validation on every insert and periodic monitoring of data quality metrics. Validation results are stored in the metadata database and can trigger alerts or block inserts if data violates defined expectations, with detailed reports showing which records failed validation and why.

Hopsworks maintains a comprehensive metadata repository that tracks lineage from raw data sources through feature groups to training datasets and deployed models, with automatic dependency graph construction showing which features are used by which models and which data sources feed which features. Lineage is queryable via API and visualizable in the UI, enabling impact analysis (e.g., 'which models will be affected if I deprecate this feature?') and debugging (e.g., 'why did this model's performance degrade?').

Hopsworks provides a feature pipeline orchestration layer that coordinates batch and streaming feature computation jobs, with automatic error handling (retries, dead-letter queues), monitoring (job status, latency, data quality), and alerting. Pipelines are defined via Python SDK or YAML configuration and can be triggered on schedule (cron), on-demand, or event-driven (e.g., when new data arrives in S3), with automatic dependency management and job ordering.

Hopsworks implements project-based multi-tenancy where each project is an isolated workspace with its own feature groups, models, and datasets, with fine-grained role-based access control (RBAC) and explicit sharing policies that allow controlled cross-project feature access. The platform uses a centralized authentication system (supporting LDAP, OAuth2, SAML) and maintains audit logs of all data access and model deployments for compliance and governance.

Hopsworks provides a centralized model registry that stores model artifacts (serialized models, weights, code), metadata (hyperparameters, training metrics, feature versions used), and deployment history with automatic lineage tracking to training datasets and features. The registry supports multiple model formats (scikit-learn, TensorFlow, PyTorch, XGBoost) and integrates with the feature store to enforce that deployed models use only features from approved feature groups, preventing training-serving skew.

Hopsworks provides a model serving layer that deploys registered models as REST/gRPC endpoints with automatic feature lookup from the online feature store, request batching for throughput optimization, and optional inference result caching to reduce latency and feature store load. The serving infrastructure supports multiple deployment targets (Kubernetes, serverless platforms) and automatically validates input features against the model's training schema before inference.

Hopsworks provides a job execution framework that submits Spark and Flink jobs to a YARN cluster or Kubernetes for distributed feature computation, with built-in scheduling (cron-based or event-triggered), dependency management, and automatic retry logic. Jobs are defined via Python SDK or uploaded as JAR/Python files, and the platform tracks job execution history, logs, and metrics in the metadata database for debugging and auditing.

Hopsworks provides a training dataset abstraction that combines features from multiple feature groups with labels at a specific point in time, automatically handling temporal joins to prevent data leakage and ensuring that features and labels are aligned to the same event timestamp. Training datasets are versioned and can be exported to multiple formats (Parquet, CSV, TFRecord) for consumption by training frameworks, with automatic schema validation and feature statistics tracking.

Hopsworks provides a managed Jupyter notebook environment integrated with the platform, where notebooks have automatic access to the feature store, model registry, and job execution APIs via pre-configured Python libraries. The platform manages Python dependencies (via conda environments) and provides notebook-to-job conversion, allowing users to develop features and models in notebooks and automatically convert them to scheduled jobs without code changes.

Hopsworks provides a storage connector abstraction that enables feature pipelines to read from and write to external data sources (S3, GCS, Azure Blob Storage, HDFS, databases) via a unified interface, with automatic credential management, connection pooling, and format conversion (Parquet, CSV, JSON, Delta Lake). Connectors are defined once and reused across feature groups and jobs, with support for both batch and streaming data sources.

Hopsworks provides a SQL query interface that allows users to query feature groups and training datasets using standard SQL, with automatic query optimization (predicate pushdown, join reordering) and transparent execution on the underlying storage backend (Spark, Hive, or database). The query interface supports both batch queries (for training dataset creation) and point-in-time queries (for inference feature lookup), with automatic schema inference and type casting.