Kubeflow

Kubeflow Pipelines enables users to define, compile, and execute multi-step ML workflows as directed acyclic graphs (DAGs) using a Python SDK that generates Kubernetes-native YAML manifests. The platform translates high-level pipeline definitions into containerized Kubernetes pods with automatic dependency management, artifact passing between steps, and built-in support for conditional execution and loops. Pipelines are stored as custom resources and executed by a dedicated controller that monitors step completion and manages inter-pod communication.

Kubeflow Training Operators provide Kubernetes controllers that manage distributed training jobs by translating high-level training specifications into coordinated pod groups with automatic parameter server/worker/chief role assignment. Each operator (TensorFlow Operator, PyTorch Operator, MPI Operator) understands framework-specific communication patterns (gRPC for TensorFlow, NCCL for PyTorch) and handles service discovery, environment variable injection, and fault tolerance. Users define training jobs as Kubernetes custom resources (e.g., TFJob, PyTorchJob) specifying replica counts, resource requests, and container images; the controller provisions pods, manages inter-pod networking, and monitors job completion.

The Notebook Controller is a Kubernetes controller that manages the lifecycle of notebook server pods by watching Notebook custom resources and creating/updating/deleting corresponding pod deployments. When a Notebook resource is created, the controller provisions a pod with the specified container image, mounts persistent volumes for the user's home directory, and exposes the notebook via a Kubernetes service. The controller handles pod restarts, volume mounting, and cleanup when notebooks are deleted. Integration with the Profile Controller ensures notebooks are created in user-specific namespaces with appropriate RBAC and resource quotas.

Kubeflow defines custom Kubernetes resources (CRDs) for ML workloads (TFJob, PyTorchJob, Notebook, Pipeline, Experiment, InferenceService) that enable users to declare ML infrastructure using YAML manifests following Kubernetes conventions. Each CRD has a corresponding controller that watches for resource creation/updates and implements the desired behavior (e.g., TFJob controller provisions training pods, Notebook controller provisions notebook servers). This declarative approach enables GitOps workflows where infrastructure is version-controlled and deployed via kubectl or CI/CD pipelines. CRDs integrate with Kubernetes RBAC, audit logging, and resource quotas, providing enterprise-grade governance.

Kubeflow Notebooks provides a controller that provisions and manages Jupyter, RStudio, and VS Code server instances as Kubernetes pods within user-specific namespaces. The Notebook Controller watches custom resources (Notebook CRDs) and creates corresponding pod deployments with persistent volume claims for user home directories. Integration with the Profile Controller enforces multi-tenant isolation by assigning each notebook to a namespace with RBAC policies and resource quotas, preventing users from accessing other users' data or exceeding cluster resource limits. Notebooks are accessed via the Central Dashboard with authentication/authorization enforced at the ingress layer.

Kubeflow Katib provides a hyperparameter optimization (HPO) and neural architecture search (NAS) platform that runs multiple trial jobs in parallel, each with different hyperparameter configurations, and uses pluggable search algorithms (grid search, random search, Bayesian optimization, genetic algorithms) to iteratively improve parameters. Katib defines an Experiment custom resource specifying the search space, objective metric, and algorithm; the Katib controller spawns trial jobs (as Training Operator jobs or generic Kubernetes pods) with different parameter combinations, collects metrics from each trial, and uses the search algorithm to suggest the next set of parameters. Metrics are collected via a metrics collector sidecar that scrapes logs or integrates with monitoring systems (Prometheus).

Kubeflow integrates KServe (a separate project under the Kubeflow ecosystem) to provide a model serving platform that deploys trained models as scalable inference services on Kubernetes. KServe abstracts framework-specific serving logic (TensorFlow Serving, TorchServe, Triton) behind a unified InferenceService custom resource that handles model loading, request routing, and auto-scaling. Users define an InferenceService specifying the model artifact location (S3, GCS, local PVC), framework, and resource requirements; KServe provisions a predictor pod with the appropriate serving runtime, exposes it via a Kubernetes service, and provides traffic management features like canary deployments (gradual traffic shift) and A/B testing.

Kubeflow's Profile Controller implements multi-tenancy by creating isolated Kubernetes namespaces for each user or team, with automatic RBAC role bindings, resource quotas, and network policies. When a user is created in Kubeflow, the Profile Controller provisions a namespace, creates a ServiceAccount for the user, binds RBAC roles (allowing the user to manage resources in their namespace only), and applies resource quotas (CPU, memory, storage) to prevent resource exhaustion. The controller also manages namespace-level access control, ensuring users can only view and modify resources in their assigned namespace. Integration with the Central Dashboard enforces authentication and maps authenticated users to their namespaces.

Kubeflow's Central Dashboard serves as the primary web interface for accessing all Kubeflow components (notebooks, pipelines, training jobs, model serving, Katib experiments). The dashboard is a single-page application (SPA) built with a web framework (likely React or similar) that provides navigation menus, component-specific web applications (embedded iframes or linked applications), and user authentication/authorization. The dashboard integrates with Kubernetes API to query custom resources (Notebooks, TFJobs, Pipelines, etc.) and displays their status, logs, and metrics. Authentication is enforced at the ingress layer (via OIDC or similar), and the dashboard respects Kubernetes RBAC to show only resources the user has access to.

Kubeflow Model Registry (a separate component) provides a centralized repository for storing model metadata, versions, and lineage information. Models are registered with metadata (name, version, framework, metrics, training parameters) and linked to their training artifacts (pipelines, datasets, hyperparameters). The registry tracks model lineage (which training job produced which model version) and enables model discovery and reuse across teams. Integration with KServe enables automatic deployment of registered models to serving endpoints, and integration with pipelines enables automatic model registration upon successful training.

Kubeflow's Admission Webhook intercepts Kubernetes API requests (create, update, delete) for Kubeflow custom resources and enforces policies before resources are persisted. The webhook validates resource specifications (e.g., ensuring training jobs specify valid frameworks, notebooks request reasonable resource limits), mutates resources (e.g., injecting default values, adding labels for tracking), and rejects requests that violate policies (e.g., resource requests exceeding namespace quotas). The webhook is registered with Kubernetes' ValidatingWebhookConfiguration and MutatingWebhookConfiguration, making it part of the standard Kubernetes admission control flow.

Kubeflow integrates the Spark Operator (a separate project) to enable submission and management of Apache Spark jobs on Kubernetes. The Spark Operator provides a SparkApplication custom resource that abstracts Spark cluster provisioning and job submission. Users define a SparkApplication specifying the Spark application JAR/Python script, resource requirements, and executor count; the operator provisions a Spark driver pod and executor pods, manages inter-pod networking for Spark communication, and monitors job completion. This enables data processing workflows (ETL, feature engineering) to run natively on Kubernetes without requiring a separate Spark cluster.