Whisper Large v3

Transcribes audio in 98 languages to text in the original language using a Transformer sequence-to-sequence architecture trained on 680,000 hours of internet audio. The model uses task-specific tokens to signal transcription mode, processes mel spectrograms through an AudioEncoder to generate embeddings, then applies autoregressive TextDecoder with optional beam search or greedy decoding strategies. Language-specific performance varies significantly (English at 65% of training data achieves highest accuracy; lower-resource languages have degraded performance).

Translates non-English speech directly to English text using the same Transformer encoder-decoder architecture but with a translation task token prepended to the decoder input. Bypasses intermediate transcription step by directly mapping audio embeddings to English tokens, reducing error propagation compared to cascaded transcription-then-translation pipelines. Supports 98 source languages but outputs only English.

Uses a Transformer sequence-to-sequence architecture with two main components: (1) AudioEncoder processes mel-spectrograms (3000 × 80 frames) through convolutional layers and Transformer encoder blocks, outputting 1500 × 1280-dimensional audio embeddings; (2) TextDecoder is a Transformer decoder with cross-attention over audio embeddings, generating text tokens autoregressively. The encoder uses sinusoidal positional encodings for audio frames; the decoder uses learned positional embeddings for text tokens. Cross-attention allows the decoder to attend to relevant audio regions while generating each text token, enabling alignment between audio and text without explicit alignment supervision.

Detects the spoken language in audio by prepending a language-detection task token to the decoder and generating a language token as the first output. Uses the same AudioEncoder to process mel spectrograms, then the TextDecoder outputs a single language identifier token from a 98-language vocabulary. Language detection happens as a byproduct of the transcription/translation pipeline and can be extracted independently.

Converts raw audio files in any FFmpeg-supported format (MP3, WAV, M4A, FLAC, OGG) to mel-spectrogram features via three-step pipeline: (1) FFmpeg decodes audio to 16kHz mono PCM, (2) whisper.pad_or_trim() normalizes to exactly 30-second segments (padding with silence or truncating), (3) whisper.log_mel_spectrogram() applies mel-scale filterbank and log compression to produce 80-dimensional mel-spectrogram frames. Output is a fixed-shape tensor (3000 frames × 80 mel bins) fed to AudioEncoder.

Generates transcription/translation text token-by-token using autoregressive decoding, where each token prediction conditions on all previously generated tokens. Supports two decoding strategies via DecodingOptions: (1) greedy decoding (fastest, selects highest-probability token at each step), (2) beam search (slower, maintains K hypotheses and prunes low-probability paths). Decoding is constrained by a 50,257-token vocabulary (tiktoken BPE encoding) and supports optional language/task token constraints to enforce output language or task type.

Extracts precise word-level timing information by decoding with timestamp tokens (special tokens representing 20ms audio intervals) and post-processing to align token boundaries with word boundaries. The transcription pipeline outputs segments (typically 30-second chunks) with segment-level timestamps, then optionally decodes again with timestamp tokens enabled to extract word-level timing. Results are formatted as structured JSON with hierarchical organization: segments → words → character offsets, enabling precise audio-text alignment for subtitle generation, audio editing, or speaker attribution.

Handles variable-length audio by automatically segmenting into overlapping 30-second windows, transcribing each window independently, then merging results while avoiding duplication. The high-level model.transcribe() function implements this: (1) splits audio into 30-second chunks with configurable overlap (default 0.5 seconds), (2) processes each chunk through the full pipeline (preprocessing → encoding → decoding), (3) merges segment results by detecting and removing duplicate text at window boundaries. Overlap ensures context continuity across segment boundaries, reducing word-boundary errors.

Provides six model sizes (tiny: 39M, base: 74M, small: 244M, medium: 769M, large: 1550M, turbo: 809M) with documented tradeoffs: tiny/base/small are 10×/7×/4× faster than large but with lower accuracy; medium is 2× faster; turbo is 8× faster but not trained for translation. Each size has English-only variant (e.g., tiny.en) optimized for English-only transcription. Model selection is exposed via whisper.load_model(model_name) with automatic weight download and caching. VRAM requirements range from 1GB (tiny/base) to 10GB (large).

Supports GPU acceleration via PyTorch CUDA backend, automatically detecting and utilizing NVIDIA GPUs when available. Model loading and inference are GPU-aware: model weights are moved to GPU memory (device='cuda'), audio embeddings are computed on GPU, and decoding runs on GPU. VRAM requirements scale with model size (1GB for tiny, 10GB for large). The implementation uses PyTorch's automatic device management; users can override with device parameter in whisper.load_model(). No explicit memory optimization (gradient checkpointing, activation quantization) is implemented — relies on PyTorch's default memory management.

Uses special task tokens prepended to the decoder input to signal which task (transcription, translation, language detection) to perform, allowing a single model to handle multiple tasks without separate model checkpoints. Task tokens are: <|transcribe|> for transcription, <|translate|> for translation, <|detect_language|> for language detection. Language tokens (e.g., <|en|>, <|fr|>) can optionally constrain the output language. The decoder attends to these tokens during generation, modulating its behavior without architectural changes. This is implemented in the decoding logic via DecodingOptions.task and DecodingOptions.language parameters.