whisper-large-v3

Converts audio waveforms to text across 99 languages using a transformer-based encoder-decoder architecture trained on 680,000 hours of multilingual audio data from the web. The model uses mel-spectrogram feature extraction with a convolutional stem followed by transformer encoder layers, enabling robust handling of accents, background noise, and technical language without language-specific preprocessing. Inference can run via PyTorch, JAX, or ONNX backends with automatic device placement (CPU/GPU/TPU).

Automatically detects the spoken language from audio segments using the model's internal language classification head, which operates on the transformer encoder's hidden states before decoding. The model outputs a language token (e.g., <|zh|>, <|es|>) as the first token in the sequence, enabling zero-shot language identification without separate language detection models. Supports detection across 99 languages with confidence scores derived from the model's token probability distribution.

Supports fine-tuning the Whisper model on domain-specific audio data to improve accuracy for specialized use cases (medical, legal, technical, accented speech). The implementation uses standard PyTorch training loops with the model's encoder-decoder weights unfrozen, enabling adaptation to new domains with relatively small labeled datasets (100-1000 hours). Fine-tuning leverages the model's pretrained representations, requiring less data than training from scratch while achieving significant accuracy improvements (5-15% WER reduction) on target domains.

Integrates with external speaker diarization systems (e.g., pyannote.audio) to produce speaker-labeled transcripts where each segment is attributed to a specific speaker. The implementation uses diarization output (speaker segments with timestamps) to segment the audio, transcribe each segment independently, and reassemble the transcript with speaker labels. While Whisper itself does not perform diarization, this capability enables end-to-end speaker-aware transcription by combining Whisper with complementary diarization models.

Supports model quantization (INT8, INT4) and distillation to reduce model size and inference latency, enabling deployment on resource-constrained devices (mobile, edge, embedded systems). The implementation uses PyTorch quantization APIs or ONNX quantization tools to convert the 1.5B-parameter large-v3 model to 8-bit or 4-bit precision, reducing model size from ~3GB to ~750MB-1.5GB with minimal accuracy loss (<1% WER degradation). Quantized models enable real-time inference on CPUs and mobile devices.

Generates token-level timestamps for transcribed text by leveraging the model's attention weights and the decoder's autoregressive token generation sequence. The implementation uses the alignment between input mel-spectrogram frames (12.5ms per frame) and output tokens to compute precise start/end times for each word or subword unit. Timestamps are extracted from the model's internal state during inference without requiring separate alignment models, enabling efficient end-to-end processing.

Processes audio in real-time or near-real-time using a sliding-window inference approach where the model processes overlapping chunks of audio (typically 30-second windows with 5-second overlap) and stitches transcripts together. The implementation maintains state across chunks to handle word boundaries and context, using the model's encoder-decoder architecture to process each window independently while preserving continuity. Streaming mode trades some accuracy for latency reduction, enabling live transcription with ~2-5 second delay.

Processes multiple audio files in parallel using PyTorch's DataLoader or JAX's vmap for vectorized inference, enabling efficient GPU utilization when transcribing large audio collections. The implementation pads variable-length audio inputs to a common length within each batch, processes them through the model simultaneously, and unpacks results. Batching reduces per-sample inference overhead and amortizes model loading costs, achieving 3-5x throughput improvement over sequential processing on GPU hardware.

Automatically handles audio preprocessing including resampling to 16kHz, mono conversion, normalization, and silence trimming before transcription. The model expects 16kHz mono PCM audio; the implementation uses librosa or torchaudio to convert arbitrary input formats (MP3, FLAC, 48kHz stereo, etc.) to the required specification. Preprocessing is transparent to the user — the model accepts raw audio files and handles format conversion internally, with optional configuration for silence detection and volume normalization.

Restricts the model's output vocabulary to a predefined set of words or phrases, enabling domain-specific transcription where only relevant terms are recognized. The implementation uses a constrained beam search decoder that masks invalid tokens at each decoding step, forcing the model to output only words from the allowed vocabulary. This is useful for transcribing specialized domains (medical, legal, technical) where out-of-vocabulary terms should be suppressed or replaced with domain-specific alternatives.

Provides token-level and segment-level confidence scores derived from the model's softmax probability distribution over the vocabulary. The implementation extracts log-probabilities from the decoder's output distribution at each step, enabling developers to identify low-confidence regions in the transcript. Confidence scores can be aggregated to word or segment level, and used to flag uncertain transcriptions for human review or to trigger fallback mechanisms.

Accepts optional text prompts to guide transcription toward specific terminology or style, improving accuracy for domain-specific or specialized content. The implementation prepends context tokens to the decoder input, biasing the model toward generating text consistent with the prompt. For example, providing a prompt like 'This is a medical conversation about cardiology' or 'Transcribe the following technical specification' influences token selection during decoding without retraining the model.

Transcribes audio in one language and optionally translates the output to another language using the model's multilingual encoder-decoder architecture. The model was trained on parallel multilingual data, enabling it to perform zero-shot translation (translating to languages not explicitly trained on) by leveraging shared semantic representations across languages. The implementation uses language tokens to specify the target language, enabling on-the-fly translation without separate translation models.