Multi Task And Meta Learning Frameworks

via “multi-task training with unified loss functions and evaluation metrics”

Salesforce's efficient vision-language bridge model.

Unique: Implements unified multi-task training pipeline via LAVIS Runner system that automatically selects task-specific losses and metrics based on configuration, enabling multi-task learning without task-specific training code

vs others: More flexible than single-task fine-tuning because multi-task learning improves zero-shot transfer, and more maintainable than custom multi-task implementations because LAVIS handles loss weighting and metric computation

via “multi-task learning with shared representations and task-specific heads”

PyTorch NLP framework with contextual embeddings.

Unique: Implements multi-task learning through a unified architecture where a shared BiLSTM encoder feeds into task-specific output heads (CRF for tagging, softmax for classification), enabling flexible combinations of different task types; supports dynamic task weighting during training to balance task contributions

vs others: More efficient than training separate models for each task while maintaining task-specific output constraints; enables knowledge transfer between related tasks, improving performance on low-resource tasks; simpler to implement than complex multi-task architectures with task-specific encoders

via “cross-task knowledge transfer through shared representations”

Microsoft's unified model for diverse vision tasks.

Unique: Achieves knowledge transfer across 6+ vision tasks through a single unified seq2seq architecture, where shared visual encoding and decoder parameters enable cross-task learning without task-specific branches or ensemble methods

vs others: Outperforms task-specific models on low-data scenarios through knowledge transfer, though with 5-10% lower peak performance on high-data tasks compared to specialized models

via “multi-task learning and auxiliary objective training”

fill-mask model by undefined. 1,90,34,963 downloads.

Unique: RoBERTa's improved pretraining produces representations with stronger task-agnostic semantic content, enabling more effective multi-task learning with less task interference compared to BERT — auxiliary tasks improve primary task performance by 1-3% absolute on average

vs others: More effective for multi-task learning than single-task fine-tuning due to stronger base representations; requires more careful tuning than task-specific models but provides better generalization and inference efficiency than ensemble approaches

via “multi-task-learning-with-shared-representations”

A very simple framework for state-of-the-art NLP

Unique: Flair's multi-task learning framework uses shared embedding and encoder layers with task-specific output heads, enabling efficient knowledge transfer while maintaining task-specific prediction heads. This architecture allows fine-grained control over task weighting and loss functions, supporting both hard parameter sharing and soft parameter sharing strategies.

vs others: Flair's multi-task learning is more flexible than single-task pipelines (supports arbitrary task combinations) and more interpretable than end-to-end multi-task transformers, with explicit control over task weighting and loss functions.

via “multi-task instruction tuning for diverse downstream capabilities”

* ⏫ 07/2023: [Meta-Transformer: A Unified Framework for Multimodal Learning (Meta-Transformer)](https://arxiv.org/abs/2307.10802)

Unique: Applies instruction tuning to diverse vision and language tasks within a single unified decoder, enabling flexible task specification through natural language while maintaining a consolidated model architecture

vs others: More flexible than task-specific models because instructions enable dynamic task specification; more parameter-efficient than maintaining separate models for each task, though with potential performance trade-offs

via “multimodal-task-specific-fine-tuning”


Unique: Provides systematic framework for selecting fine-tuning strategy (full fine-tuning vs LoRA vs adapter modules) based on dataset size, computational budget, and task similarity to pre-training distribution — with empirical guidance on when each approach maximizes performance-efficiency trade-offs

vs others: Deeper treatment of multimodal-specific fine-tuning challenges (modality-specific layer freezing, handling missing modalities at test time) compared to generic transfer learning courses focused on single-modality models

via “multi-task adapter composition for vision-language understanding”

* ⭐ 04/2022: [Winoground: Probing Vision and Language Models for Visio-Linguistic... (Winoground)](https://arxiv.org/abs/2204.03162)

Unique: Implements task-specific adapter composition for multimodal models with explicit routing logic, enabling independent training of task adapters while maintaining shared backbone — distinct from single-task adapter approaches and multi-task learning methods that require joint training

vs others: More memory-efficient than training separate full models per task and more flexible than single-task adapters, enabling dynamic task switching without model reloading

via “multi-task and meta-learning frameworks”


Unique: Provides practical implementations of multi-task learning with systematic task weighting strategies and meta-learning approaches (MAML, Prototypical Networks) from scratch, combined with empirical analysis of when multi-task learning helps vs hurts generalization. Includes frameworks for identifying task relatedness and designing shared representations.

vs others: More practical and implementation-focused than academic meta-learning papers by providing working code and systematic frameworks for task weighting and architecture design, while more comprehensive than generic transfer learning tutorials by covering few-shot learning and rapid adaptation.

via “multi-task vision model with shared representation”

* ⏫ 12/2023: [VideoPoet: A Large Language Model for Zero-Shot Video Generation (VideoPoet)](https://arxiv.org/abs/2312.14125)

Unique: Uses single encoder-decoder backbone with shared parameters across all vision tasks, trained on 5.4B diverse annotations to learn unified representation handling variable spatial hierarchies and semantic granularities. Contrasts with ensemble or task-specific approaches by consolidating capabilities into one model.

vs others: Reduces deployment complexity and memory footprint compared to maintaining separate detection (YOLO), segmentation (DeepLab), grounding (ALBEF), and captioning (BLIP) models, though individual task performance vs specialized baselines unknown.

via “multi-task and domain-specific fine-tuning strategies”


Unique: Addresses the practical challenge of fine-tuning on multiple objectives simultaneously, with specific techniques for loss weighting, task-specific adapters, and detecting when one task is degrading performance on another

vs others: More sophisticated than single-task fine-tuning while remaining more practical than training separate models for each task; enables efficient multi-purpose models that maintain performance across diverse use cases

via “multi-task agent learning with shared trajectory representation”

### Other Papers <a name="2023op"></a>

Unique: Enables multi-task learning by conditioning the language model policy on task descriptions, allowing a single agent to learn from trajectories across diverse tasks and generalize to new tasks — this is distinct from task-specific agents that require separate training for each task

vs others: More sample-efficient than single-task agents because it leverages cross-task patterns, and more flexible than fixed multi-task architectures because task conditioning is learned end-to-end