whisper-jax

Performs real-time audio transcription using OpenAI's Whisper model compiled and optimized through JAX's XLA compiler for GPU/TPU acceleration. The implementation leverages JAX's functional programming paradigm and JIT compilation to achieve lower latency and higher throughput than standard PyTorch implementations, with support for streaming audio chunks and batch processing. Integrates with HuggingFace Transformers for model loading and preprocessing pipelines.

Automatically detects the language of input audio and applies language-specific acoustic and language models from Whisper's multilingual variant. The system uses a two-stage approach: first detecting language from a short audio sample (typically 30 seconds), then routing to the appropriate language-specific decoder. Supports 99+ languages with unified preprocessing pipeline that handles different phonetic characteristics and acoustic properties per language.

Provides a Gradio-based web UI deployed on HuggingFace Spaces that accepts audio file uploads, streams them to the JAX-optimized Whisper backend, and displays transcription results with live progress updates. The interface handles file validation, audio format conversion, and streaming responses using WebSocket connections for real-time feedback. Built on Gradio's reactive component system with automatic CORS handling and session management for concurrent users.

Processes multiple audio files concurrently using JAX's vmap (vectorized map) primitive to parallelize inference across batch dimensions without explicit loop unrolling. The system automatically handles variable-length audio sequences through padding and masking, distributes computation across available GPU/TPU cores, and aggregates results with minimal memory overhead. Supports both synchronous batch processing and asynchronous job queuing for large-scale transcription pipelines.

Automatically converts input audio to Whisper's required format (16kHz mono PCM) through a composable preprocessing pipeline that handles resampling, channel mixing, normalization, and silence trimming. Uses librosa for audio I/O and signal processing, with JAX-compatible operations for in-memory transformations. Supports streaming preprocessing for large files without loading entire audio into memory, with configurable chunk sizes and overlap for seamless processing.

Generates transcription with precise timing information at the segment level (typically 30-second chunks), including start/end timestamps for each transcribed segment. Whisper's decoder outputs token-level timing through attention weights, which are aggregated to segment boundaries. The implementation preserves timing information through the JAX inference pipeline and formats output as WebVTT, SRT, or JSON with millisecond precision for subtitle generation and media synchronization.

Reduces Whisper model size through JAX-native quantization techniques (int8, float16) and knowledge distillation, enabling deployment on resource-constrained devices (mobile, edge servers) with minimal accuracy loss. The system uses JAX's dtype casting and custom quantization kernels to compress the 1.5GB large model to 400-600MB while maintaining 95%+ accuracy. Supports both static quantization (post-training) and dynamic quantization (per-batch) with automatic precision tuning based on target hardware.

Provides per-segment and per-token confidence scores from Whisper's decoder output, enabling downstream applications to identify low-confidence regions and trigger alternative processing (e.g., manual review, re-transcription with different model). Implements confidence aggregation strategies (mean, min, weighted) and automatic quality thresholds for flagging potentially incorrect transcriptions. Integrates with JAX's error handling to gracefully degrade on corrupted audio or out-of-distribution inputs.