BentoML

Transforms Python classes into production-grade API services using @bentoml.service and @bentoml.api decorators. The framework introspects decorated methods, generates OpenAPI schemas automatically via src/_bentoml_sdk/service/openapi.py, and maps them to HTTP/gRPC endpoints. Service[T] generic class manages lifecycle, dependency injection, and model binding without requiring explicit routing configuration.

Implements request-level batching in src/_bentoml_impl/server/serving.py that accumulates incoming requests up to a configured batch size or timeout window, then processes them together through the model. Uses a task queue system (Task Queue System in DeepWiki) to manage request buffering, with per-endpoint batch configuration via bentoml.api(max_batch_size=N, batch_window_ms=M). Batching is transparent to the service code—the API method receives either single or batched inputs depending on configuration.

Defines request and response schemas using input/output descriptors (Input/Output Descriptors in DeepWiki) that specify expected data types, shapes, and formats. Descriptors support numpy arrays, images, text, JSON, and custom types. BentoML automatically validates incoming requests against descriptors and serializes responses, handling type conversion and format negotiation. Descriptors are used to generate OpenAPI schemas and gRPC protobuf definitions, ensuring consistency between documentation and actual validation.

Provides built-in integration with Hugging Face Hub (Hugging Face Integrations in DeepWiki) that enables loading models directly from the Hub without manual downloading. BentoML caches downloaded models locally and manages versioning, so repeated loads don't re-download. Integration supports transformers, diffusers, and other Hugging Face libraries. Models are referenced by Hub ID (e.g., 'gpt2', 'stabilityai/stable-diffusion-2') and automatically downloaded on first use.

Provides a hierarchical configuration system (Configuration System in DeepWiki) via bentoml_config.yaml that defines service behavior, resource allocation, and deployment settings. Configuration includes service settings (max_concurrency, timeout), build settings (Python version, dependencies), and image settings (base image, environment variables). Environment-specific overrides are supported via environment variables (BENTOML_* prefix) or separate config files, enabling the same Bento to be deployed with different configurations across environments.

Enables services to stream responses back to clients via gRPC server-side streaming (gRPC Server in DeepWiki). Service methods can yield multiple responses, and BentoML automatically converts them to gRPC streaming responses. Streaming is useful for long-running operations (e.g., token-by-token LLM generation) where clients want to receive results incrementally rather than waiting for the full response. HTTP responses are still buffered fully; streaming is only available via gRPC.

Collects metrics at each stage of the request processing pipeline (Monitoring and Observability in DeepWiki) including request count, latency, error rate, and model inference time. Metrics are exposed in Prometheus format at /metrics endpoint for scraping by monitoring systems. Logging is integrated throughout the framework, with request-level logs including request ID, latency, and errors. Custom metrics can be added via bentoml.metrics API. Observability is designed for Kubernetes deployments with Prometheus + Grafana integration.

Runs dual HTTP (ASGI-based via src/_bentoml_impl/server/app.py) and gRPC servers simultaneously from a single service definition. HTTP server handles REST clients and provides health checks (/healthz), metrics endpoints, and OpenAPI UI. gRPC server (gRPC Server in DeepWiki) auto-generates protobuf definitions from service method signatures and supports streaming. Both servers share the same underlying request processing pipeline and batching logic, with protocol-specific serialization (JSON for HTTP, protobuf for gRPC).

Provides a model registry (Model Management in DeepWiki) that stores trained models with semantic versioning (e.g., my_model:v1, my_model:v2) in a local bentoml_models directory or cloud storage. Models are loaded via bentoml.models.get(model_tag) and bound to service instances at initialization. Supports framework-agnostic model serialization (PyTorch, TensorFlow, scikit-learn, ONNX, custom pickle) with automatic dependency tracking. Service can reference specific model versions, enabling A/B testing and gradual rollouts without code changes.

Packages a service definition, models, and dependencies into a self-contained Bento artifact (bentofile.yaml defines the bundle). The Bento includes a Python environment snapshot (via pip lock file or conda), service code, model references, and build configuration. BentoML generates a Dockerfile automatically from the Bento specification, enabling reproducible container builds. Bentos are versioned (my_service:20240101_abc123) and can be pushed to a registry (local, Docker Hub, or BentoCloud). Deployment reads the Bento and spins up the service without additional configuration.

Implements a multi-stage request processing pipeline (Request Processing Pipeline in DeepWiki, src/_bentoml_impl/server/serving.py) that handles incoming HTTP/gRPC requests through deserialization, batching, model inference, and serialization stages. Concurrency is controlled via worker thread pools (configurable via service_config.runners[].max_concurrency) and per-endpoint rate limiting. Health check endpoints (/healthz, /readyz) report service status, model availability, and resource utilization. Pipeline includes error handling, request logging, and metrics collection at each stage.

Supports serialization and deserialization of models from multiple ML frameworks (PyTorch, TensorFlow, scikit-learn, ONNX, XGBoost, LightGBM, Keras, Paddle, custom pickle) via framework-specific runners. Each model type has a corresponding runner (e.g., PyTorchRunner, TensorFlowRunner) that handles framework-specific initialization, GPU memory management, and inference. Runners track framework versions and dependencies, enabling bentoml to validate compatibility when loading models. Custom runners can be implemented for proprietary or unsupported frameworks.

Enables services to depend on other services or models via dependency injection (Service Dependencies in DeepWiki). A service can reference another service instance (e.g., @bentoml.service(depends_on=[other_service])) or load multiple models, then compose predictions by chaining calls. Dependencies are resolved at service initialization, and the framework manages lifecycle (startup, shutdown) for all dependencies. Composition is transparent—service code calls dependencies as regular Python objects, not via HTTP/gRPC.

Provides bentoml serve command (Local Development Serving in DeepWiki) that runs the service locally with hot-reload on code changes. The development server watches the service file and automatically restarts the service when code is modified, enabling rapid iteration. Debugging is supported via standard Python debuggers (pdb, IDE breakpoints) because the service runs in the main process. Development server includes a built-in OpenAPI UI at /docs for testing endpoints interactively.

Integrates with BentoCloud (BentoCloud Deployment in DeepWiki), a managed platform for deploying Bentos with automatic scaling, monitoring, and traffic management. Users push a Bento to BentoCloud via bentoml deploy command, and the platform handles container orchestration, load balancing, and scaling based on CPU/memory/custom metrics. BentoCloud provides a dashboard for monitoring service health, viewing logs, and managing deployments. Scaling policies are configured via bentoml_config.yaml (e.g., min_replicas, max_replicas, target_cpu_utilization).