Pay Per Second Gpu Compute With Automatic Hardware Selection

via “cpu and gpu deployment with automatic device selection”

Bilingual Chinese-English language model.

Unique: Implements automatic device detection with fallback logic that selects GPU if available, otherwise CPU, with explicit support for Apple Silicon MPS backend. The architecture combines device selection with quantization options to enable deployment across a wide range of hardware from edge devices to high-end GPUs.

vs others: Provides more flexible hardware support than GPU-only frameworks (vLLM, TGI) while maintaining competitive inference speed on GPUs, making it suitable for heterogeneous deployments where hardware varies across environments.