KServe

KServe implements a Kubernetes operator pattern through Custom Resource Definitions (CRDs) that abstract ML model serving complexity into declarative YAML specifications. The control plane (written in Go at pkg/controller/) runs InferenceService controllers that reconcile desired state, automatically provisioning Kubernetes Deployments, Services, and Ingress resources. This enables GitOps-compatible model deployment where users declare model specs (framework, storage location, resource requirements) and KServe handles the orchestration, networking, and lifecycle management without manual pod configuration.

KServe's data plane (Python framework at python/kserve/kserve/) provides a unified model server that abstracts framework-specific serving logic behind standardized REST and gRPC protocols. The framework implements protocol handlers that translate incoming requests to framework-specific inference calls, supporting TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, and custom models. Request routing uses a ModelServer base class that handles protocol negotiation, request validation, and response serialization, allowing a single container image to serve different model types by swapping the underlying predictor implementation.

KServe's data plane emits Prometheus metrics (python/kserve/kserve/metrics.py) tracking request count, latency percentiles, model inference time, and error rates. The model server exposes a /metrics endpoint in Prometheus format, enabling integration with monitoring stacks (Prometheus, Grafana, Datadog). The control plane can optionally configure ServiceMonitor CRDs (Prometheus Operator) for automatic metric scraping, enabling observability without manual Prometheus configuration. This provides visibility into model performance, enabling SLO tracking, alerting, and capacity planning.

KServe provides a Python SDK (python/kserve/kserve/) with base classes (Model, ModelServer) that enable developers to implement custom inference logic for any framework or proprietary model. Developers extend the Model class, implementing load() and predict() methods, and KServe handles protocol translation, request routing, and lifecycle management. This enables serving models not natively supported by KServe (e.g., custom ensemble logic, proprietary formats) while inheriting REST/gRPC protocol support, autoscaling, and monitoring infrastructure.

KServe implements validating and mutating webhooks (pkg/controller/v1beta1/inferenceservice/) that intercept InferenceService CRD creation/updates to enforce schema validation, apply defaults, and mutate specifications before persistence. The webhooks validate that model storage URIs are accessible, framework specifications are valid, and resource requests are reasonable. This enables policy enforcement at the API level, preventing invalid configurations from being deployed and reducing debugging time.

KServe supports deploying InferenceServices across multiple Kubernetes namespaces with namespace-scoped RBAC, enabling multi-tenant model serving where different teams manage models in isolated namespaces. The control plane respects Kubernetes RBAC, allowing fine-grained access control (e.g., team A can only manage models in namespace-a). Service endpoints are namespace-scoped, preventing cross-namespace model access unless explicitly configured. This enables shared Kubernetes clusters to safely host models from multiple teams.

KServe's ingress controller (pkg/controller/v1beta1/inferenceservice/components/) implements traffic splitting logic that routes requests between predictor, transformer, and explainer components based on configurable percentages. The control plane provisions Kubernetes Ingress resources with traffic weight annotations that map to underlying Service selectors, enabling canary rollouts where new model versions receive a percentage of traffic while the stable version handles the remainder. This is implemented through Knative Serving integration (when enabled) or native Kubernetes Ingress with traffic splitting annotations, allowing gradual validation of new models before full cutover.

KServe integrates with Kubernetes Horizontal Pod Autoscaler (HPA) to automatically scale model server replicas based on request metrics. The data plane emits Prometheus metrics (request count, latency, queue depth) that HPA consumes via the metrics API, scaling up when request rate exceeds thresholds and scaling down during low traffic. The control plane configures HPA resources with target metrics (requests-per-second, CPU, memory) derived from InferenceService annotations, enabling serverless-like autoscaling where infrastructure automatically adjusts to demand without manual replica management.

KServe's storage-initializer component (cmd/storage-initializer/) runs as an init container that downloads model artifacts from cloud storage (S3, GCS, Azure Blob) or local PersistentVolumeClaims before the model server starts. The control plane injects this init container into model server Pods based on the InferenceService storage URI (e.g., s3://bucket/model-path), handling authentication via Kubernetes Secrets and mounting artifacts to a shared volume. This decouples model artifact management from model server code, enabling models to be updated without rebuilding container images.

KServe's explainer component (pkg/controller/v1beta1/inferenceservice/components/explainer.go) provides optional model interpretability by routing requests to SHAP or LIME explainer servers that generate feature importance explanations alongside predictions. The control plane provisions explainer Pods as a separate component in the InferenceService, with request routing logic that calls both predictor and explainer, returning combined prediction + explanation responses. This enables users to understand which input features drove model decisions, critical for regulatory compliance (GDPR, FCRA) and debugging model behavior.

KServe's transformer component (pkg/controller/v1beta1/inferenceservice/components/transformer.go) enables optional request/response transformation by routing inference requests through a transformer server before reaching the predictor. The transformer can implement custom Python logic (via KServe's Transformer base class) to perform feature engineering, data validation, format conversion, or response post-processing. The control plane provisions transformer Pods as a separate component with automatic request routing, allowing complex data pipelines without modifying model code or client applications.

KServe's InferenceGraph CRD (referenced in DeepWiki as 'InferenceGraph for Multi-Model Pipelines') enables composition of multiple models into directed acyclic graphs (DAGs) where outputs from one model feed into inputs of another. The control plane provisions InferenceGraph controllers that manage the lifecycle of component models and implement request routing logic to execute the graph, supporting both sequential pipelines (model A → model B → model C) and parallel branches (model A → [model B, model C] → model D). This enables complex inference workflows without requiring client-side orchestration.

KServe provides OpenAI-compatible REST endpoints (python/kserve/kserve/protocol/rest/openai/) for large language models, enabling drop-in replacement of OpenAI API with self-hosted models. The implementation supports OpenAI's chat completion and text completion APIs, including streaming responses via Server-Sent Events (SSE), allowing clients using OpenAI SDKs to switch to KServe-hosted models without code changes. This is implemented through protocol handlers that map OpenAI request/response schemas to underlying model server implementations (vLLM, HuggingFace, custom).

KServe provides LocalModelCache CRD (pkg/apis/serving/v1alpha1/local_model_cache_types.go) for node-level model caching, reducing model loading times by persisting model artifacts on node local storage across Pod restarts. The control plane manages cache lifecycle, handling cache invalidation, size limits, and multi-model sharing on single nodes. Additionally, KServe integrates with Kubernetes GPU scheduling (nvidia.com/gpu resource requests) and provides KV cache offloading for LLMs, enabling efficient memory management by offloading attention cache to CPU/disk for handling longer sequences.