Hopsworks

Hopsworks orchestrates feature computation pipelines using Apache Spark and Flink as distributed execution engines, with job scheduling via YARN and integrated monitoring. The platform abstracts distributed computing complexity through a unified Python/Scala API that compiles feature transformations into optimized Spark SQL or Flink DataStream jobs, enabling both batch and streaming feature materialization at scale without requiring users to write native Spark/Flink code.

Hopsworks implements temporal versioning of feature groups using Delta Lake or Iceberg table formats, enabling queries to reconstruct feature values as they existed at any historical timestamp. The query system tracks feature group versions, applies time-based filtering, and joins features from multiple versions to ensure training datasets reflect the exact feature state at prediction time, preventing data leakage and enabling reproducible model training.

Hopsworks enables defining feature groups declaratively through Python classes or YAML, specifying schema, primary keys, event timestamps, and materialization strategy. The platform tracks schema changes across versions, supports backward-compatible schema evolution (adding nullable columns, renaming with aliases), and prevents breaking changes. Feature group versions are immutable; schema modifications create new versions with automatic migration of existing data where possible.

Hopsworks provides a job execution framework that schedules and monitors Spark/Flink jobs with configurable retry policies, dependency chains, and failure notifications. Jobs are defined declaratively with input/output specifications, resource requirements (CPU, memory), and scheduling rules (cron, event-triggered). The platform tracks job execution history, logs, and metrics, enabling debugging and performance optimization. Failed jobs can be automatically retried with exponential backoff or escalated to alerts.

Hopsworks maintains a comprehensive metadata catalog of all features, feature groups, training datasets, and models with searchable descriptions, tags, and ownership information. The catalog enables discovery through full-text search, tag-based filtering, and lineage visualization. Metadata includes feature statistics (cardinality, missing values, distribution), data quality metrics, and usage statistics (how many models use each feature). The catalog integrates with external data governance tools via REST API.

Hopsworks enforces schema contracts on feature groups through a declarative validation framework that checks data types, nullability, and custom constraints before features are materialized. The platform integrates Great Expectations for statistical profiling and anomaly detection, tracking data quality metrics over time and alerting on schema violations or statistical drift, enabling early detection of data pipeline failures.

Hopsworks provides a centralized model registry that stores model artifacts, hyperparameters, training metrics, and data lineage through a REST API and Python SDK. The registry tracks which features, training datasets, and code versions produced each model, enabling reproducibility and impact analysis. Integration with MLflow-compatible APIs allows seamless logging from training scripts, while the platform maintains immutable audit trails of model versions and their associated metadata.

Hopsworks provides a model serving layer that deploys registered models as REST endpoints with automatic feature enrichment from the feature store. The serving infrastructure supports both batch prediction (for offline scoring) and real-time inference (sub-100ms latency) by caching frequently-accessed features in-memory and fetching on-demand features from the feature store. The platform handles feature transformation, schema validation, and request routing through a Kubernetes-native deployment model.

Hopsworks implements project-scoped isolation where each project contains its own feature groups, training datasets, models, and jobs with independent access control lists. The platform uses role-based access control (RBAC) with predefined roles (Data Scientist, Engineer, Manager) and fine-grained permissions at the feature group and model level. Authentication integrates with LDAP, OAuth2, and API keys, while audit logs track all data access and modifications for compliance.

Hopsworks provides a Python SDK that integrates with Jupyter notebooks, enabling interactive feature engineering with auto-completion, inline documentation, and direct access to feature store data. The SDK abstracts Spark/Flink complexity through a pandas-like API for small datasets and automatic Spark SQL compilation for large-scale operations. Notebook integration includes kernel management, dependency isolation via conda environments, and seamless switching between local and cluster execution.

Hopsworks abstracts data source connectivity through a pluggable storage connector framework supporting S3, Azure Blob Storage, GCS, HDFS, and JDBC databases. Connectors handle authentication (IAM roles, connection strings, API keys), data format conversion (Parquet, CSV, Delta, Iceberg), and schema inference. The platform manages connector credentials securely in a vault and enables feature groups to read from or write to external sources without exposing credentials in user code.

Hopsworks generates training datasets by joining multiple feature groups with configurable time-series windows, handling feature alignment across different update frequencies. The platform supports event-time joins (using transaction timestamps) and processing-time joins, with automatic handling of late-arriving features and missing values. Generated datasets are versioned, cached in Parquet/Delta format, and linked to the features and models that consume them for lineage tracking.

Hopsworks exposes a comprehensive REST API built on Java EE with OpenAPI documentation, enabling programmatic access to feature groups, training datasets, models, and jobs. The API supports CRUD operations on features, batch and real-time feature retrieval, model predictions, and job management. gRPC endpoints provide low-latency feature serving for high-throughput applications, with request/response streaming for batch operations. All endpoints enforce authentication via API keys or OAuth2 tokens and audit all requests.