Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning (CM3Leon)

CM3Leon implements a decoder-only, token-based multimodal architecture that unifies text and image modalities into a single autoregressive sequence. The model uses a retrieval-augmented approach during pretraining where both text and image tokens are processed through the same transformer decoder, enabling bidirectional generation (text→image and image→text) without separate encoder-decoder branches. This is achieved by tokenizing images into discrete tokens and treating them identically to text tokens in the autoregressive sequence, allowing the model to learn cross-modal dependencies through standard language modeling objectives.

CM3Leon's pretraining stage incorporates retrieval augmentation where relevant text-image pairs are retrieved and concatenated into the training sequences. During pretraining, the model learns to predict both text and image tokens in context of retrieved examples, enabling the model to leverage external knowledge without explicit fine-tuning. The retrieval mechanism operates at the sequence level, pulling related examples from a large corpus and interleaving them with the primary sequence, allowing the autoregressive model to learn in-context patterns and improve generalization through exposure to diverse multimodal contexts.

CM3Leon frames semantic segmentation as a token prediction task within the unified decoder, enabling the model to generate segmentation masks by predicting special segmentation tokens conditioned on image input. During multi-task SFT, the model learns to output segmentation tokens that correspond to semantic classes, converting the segmentation task into sequence prediction. This approach integrates segmentation into the multimodal model without separate segmentation heads or decoders.

CM3Leon supports image infilling where partial images with missing regions are completed based on surrounding context and optional text descriptions. The model conditions on the visible image tokens and text instructions, predicting tokens for the masked regions autoregressively. This capability is learned during multi-task SFT and enables tasks like object removal, hole filling, and content-aware completion without requiring explicit mask inputs or separate inpainting models.

CM3Leon's multi-task SFT stage trains the model on diverse downstream tasks (text-to-image, image-to-text, infilling, editing, segmentation) using instruction-tuning approaches where each task is framed as following natural language instructions. This enables the model to learn task-specific behaviors while maintaining a unified architecture, allowing a single model to handle multiple vision and language tasks. The instruction tuning approach enables the model to generalize to new tasks and instructions not seen during training.

After retrieval-augmented pretraining, CM3Leon undergoes multi-task supervised fine-tuning (SFT) on diverse downstream tasks including text-to-image generation, image infilling, language-guided image editing, image-controlled generation, and segmentation. The SFT stage uses task-specific training data where each task is framed as a sequence prediction problem, allowing the unified decoder to learn task-specific behaviors while maintaining the shared multimodal representation. Contrastive decoding methods are applied during this stage to improve generation quality by contrasting high-quality and lower-quality outputs.

CM3Leon implements a self-contained contrastive decoding method that improves generation quality by contrasting predictions from the model with a reference distribution during inference. Rather than requiring a separate quality model or discriminator, the method operates within the single multimodal decoder by sampling multiple candidate sequences and selecting or reranking them based on contrastive objectives. This approach is integrated into the SFT stage and applied during inference to improve both image and text generation without architectural modifications.

CM3Leon achieves zero-shot image generation capability (without task-specific fine-tuning) through its retrieval-augmented pretraining and unified multimodal architecture. The model generates images directly from text prompts by predicting image tokens autoregressively, achieving MS-COCO FID score of 4.88 without any COCO-specific training. This zero-shot capability emerges from the large-scale pretraining on diverse text-image pairs and the model's ability to leverage retrieved examples during inference, enabling competitive performance on standard benchmarks without task-specific adaptation.

CM3Leon achieves 5x reduction in training compute compared to comparable multimodal methods through its unified decoder-only architecture and retrieval-augmented pretraining approach. The efficiency gains come from eliminating separate vision encoders and cross-modal fusion layers, using a single transformer decoder for all modalities, and leveraging retrieval to improve learning efficiency without scaling model size. The paper documents this efficiency metric but does not provide detailed breakdowns of which architectural choices contribute most to the improvement.

CM3Leon converts images into discrete tokens using an image tokenizer, enabling images to be represented as sequences of integers identical to text tokens. This tokenization approach allows the unified decoder to process images and text through the same autoregressive mechanism without separate vision-specific processing. The discrete tokens are learned during pretraining and enable the model to treat image generation as a sequence prediction problem, though the specific tokenizer architecture (VQ-VAE, learned codebook, etc.) is not detailed in the documentation.

CM3Leon can generate descriptive text captions from images by conditioning the autoregressive decoder on image tokens and predicting text tokens. The bidirectional nature of the unified architecture enables the model to learn image-to-text generation during pretraining without separate caption-specific training. The model leverages the same retrieval-augmented pretraining and multi-task fine-tuning as image generation, allowing it to generate contextually relevant descriptions from visual input.

CM3Leon supports language-guided image editing where users provide text instructions to modify existing images. During the multi-task SFT stage, the model learns to condition on both the original image and text editing instructions, predicting modified image tokens that reflect the requested changes. This capability enables tasks like object removal, style transfer, attribute modification, and other edits specified through natural language without requiring separate editing models or mask inputs.

CM3Leon supports image-controlled generation where a reference image provides visual style, composition, or content guidance for generating new images. During multi-task SFT, the model learns to condition on reference images and text prompts, generating new images that follow the reference's visual characteristics while incorporating the text description. This enables style transfer, composition-guided generation, and other reference-based image synthesis tasks within the unified decoder.