AudioCraft

Generates high-fidelity music from natural language text descriptions using MusicGen, a controllable autoregressive language model that operates on discrete audio tokens produced by EnCodec compression. The model uses a streaming transformer architecture with text conditioning to map descriptions to musical sequences, supporting variable-length generation up to 30 seconds with control over tempo, instrumentation, and style through prompt engineering.

Generates diverse sound effects and general audio from text descriptions using AudioGen, a variant of the MusicGen architecture adapted for non-musical audio synthesis. Operates identically to MusicGen in the tokenization-generation-decoding pipeline but trained on sound effect datasets, enabling generation of environmental sounds, foley effects, and acoustic phenomena from natural language prompts.

Provides a modular architecture where audio generation models are composed from interchangeable components (compression models, language models, conditioners) through configuration files. Enables researchers to experiment with different architectures by swapping components (e.g., replacing EnCodec with alternative codecs, using different transformer variants) without modifying core code.

Provides utilities for audio loading, resampling, normalization, and feature extraction (spectrograms, mel-spectrograms, MFCC) to support data preprocessing and analysis. Includes efficient batch processing for large audio datasets and integration with common audio formats (WAV, MP3, FLAC), enabling end-to-end audio pipelines from raw files to model inputs.

Provides high-level Python API for loading pre-trained models and running inference with minimal code. Abstracts away model architecture details, device management, and configuration, enabling users to generate audio with single function calls. Supports automatic model downloading, caching, and version management.

Compresses audio waveforms into discrete token sequences using EnCodec, a learned neural codec that combines convolutional autoencoders with residual vector quantization. Enables lossless or lossy compression at variable bitrates (1.5-24 kbps) while preserving perceptual quality, serving as the tokenization layer for all generation models. Supports streaming inference and multi-band processing for improved reconstruction.

Generates music and sound effects using MAGNeT, a non-autoregressive masked language model that predicts entire token sequences in parallel rather than sequentially. Uses iterative refinement with confidence-based masking to progressively improve token predictions, reducing generation latency to 2-5 seconds for 30-second audio while maintaining quality comparable to autoregressive MusicGen.

Extends MusicGen with multi-modal conditioning to accept both text descriptions and reference audio (melody, style samples) as input. Uses separate audio conditioners that extract style embeddings from reference audio and fuse them with text embeddings through a joint conditioning system, enabling generation of music that matches specified styles while following text descriptions.

Generates music conditioned on explicit musical structure (chord progressions, melody contours, drum patterns) using JASCO, which extends the MusicGen architecture with music-specific conditioners that parse symbolic musical inputs. Accepts MIDI-like representations or audio transcriptions of chords/melody/drums and generates full arrangements that respect the specified musical structure while maintaining coherence.

Improves audio quality by applying diffusion-based decoding to EnCodec-compressed audio, using MultiBand Diffusion to refine reconstructed waveforms. Operates on frequency bands independently to reduce compression artifacts and enhance perceptual quality, particularly effective for recovering high-frequency details lost during neural compression.

Embeds imperceptible watermarks into generated audio using AudioSeal, a learned watermarking system that adds inaudible signals robust to common audio transformations (compression, noise, time-stretching). Enables detection and verification of AI-generated audio, supporting authenticity claims and copyright attribution through watermark extraction and validation.

Enables large-scale model training across multiple GPUs and nodes using Fully Sharded Data Parallel (FSDP) from PyTorch, automatically partitioning model weights and gradients across devices. Implements gradient checkpointing, mixed-precision training, and communication optimization to efficiently train large audio generation models on clusters, reducing training time from weeks to days.

Implements streaming audio generation using causal attention mechanisms that process audio tokens sequentially without future context, enabling real-time or low-latency generation. Uses sliding window attention and KV-cache optimization to generate audio incrementally, producing output tokens as input arrives rather than waiting for complete input sequences.