Large Scale Visual Instruction Tuning Corpus

via “instruction-tuned multimodal generation with alignment”

Meta's largest open multimodal model at 90B parameters.

Unique: Provides both base and instruction-tuned variants, allowing users to choose between raw model capability and aligned behavior, with torchtune framework enabling custom fine-tuning on proprietary instruction datasets

vs others: Open-weight instruction-tuned variants enable custom alignment without relying on proprietary API providers, though fine-tuning infrastructure requirements are higher than using managed APIs

via “synthetic-instruction-data-generation-and-curation”

Open multimodal model for visual reasoning.

Unique: First large-scale application of language-only GPT-4 to generate multimodal instruction-following data (158K samples) without human annotation; dataset is publicly released and reproducible, enabling community-driven research on synthetic data quality and effectiveness

vs others: Eliminates annotation costs compared to human-labeled datasets like Visual Genome or Conceptual Captions, while achieving competitive model performance (85.1% relative to GPT-4); enables rapid iteration on model architectures without waiting for manual data labeling

via “instruction-tuning dataset formatting with conversational structure”

200K high-quality multi-turn dialogues for instruction tuning.

Unique: Structures conversations as implicit instruction-response pairs within multi-turn context, enabling instruction-tuning while preserving conversational coherence — differs from single-turn instruction datasets (which lack context) and from generic dialogue datasets (which don't optimize for instruction-following)

vs others: Better for instruction-following than generic dialogue datasets because structure is optimized for SFT; better for conversational coherence than single-turn instruction datasets because full context is preserved

via “large-scale visual instruction tuning corpus”

150K visual instruction examples for multimodal model training.

Unique: Achieves 150K-example scale through systematic GPT-4V-based generation rather than manual annotation, making large-scale instruction tuning datasets feasible. The scale enables training of models with sufficient data diversity to learn generalizable visual understanding patterns.

vs others: Larger than most manually-annotated visual instruction datasets (COCO is 330K images but fewer instruction examples); more cost-effective than human annotation at scale; enables training of models competitive with larger proprietary datasets through efficient generation.

via “instruction tuning and supervised fine-tuning research documentation”

总结Prompt&LLM论文，开源数据&模型，AIGC应用

Unique: Connects instruction tuning research to broader LLM training methodology by showing how SFT relates to in-context learning and RLHF, with papers on instruction diversity and dataset construction that explain why instruction-tuned models generalize better to unseen tasks.

vs others: More comprehensive than framework documentation by covering underlying training research; more practical than pure NLP papers by organizing knowledge around LLM-specific instruction following and generalization patterns.

via “vision-language model instruction tuning via image-text pair alignment”

* ⭐ 04/2023: [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (VideoLDM)](https://arxiv.org/abs/2304.08818)

Unique: Introduces a systematic two-stage alignment approach that decouples vision encoding from language understanding, using adapter modules and LoRA-style parameter-efficient fine-tuning to maintain frozen pre-trained weights while achieving strong instruction-following performance. This contrasts with end-to-end training approaches by reducing memory overhead and enabling faster iteration on instruction datasets.

vs others: More parameter-efficient and faster to train than full model fine-tuning (e.g., BLIP-2, LLaVA v1.0 early approaches) while achieving comparable or superior instruction-following accuracy through explicit alignment objectives rather than implicit joint training.